Chuck Lever 66a04cd140 svcrdma: Fix trace point use-after-free race ... commit e28b4fc652 upstream. I hit this while testing nfsd-5.7 with kernel memory debugging enabled on my server: Mar 30 13:21:45 klimt kernel: BUG: unable to handle page fault for address: ffff8887e6c279a8 Mar 30 13:21:45 klimt kernel: #PF: supervisor read access in kernel mode Mar 30 13:21:45 klimt kernel: #PF: error_code(0x0000) - not-present page Mar 30 13:21:45 klimt kernel: PGD 3601067 P4D 3601067 PUD 87c519067 PMD 87c3e2067 PTE 800ffff8193d8060 Mar 30 13:21:45 klimt kernel: Oops: 0000 [#1] SMP DEBUG_PAGEALLOC PTI Mar 30 13:21:45 klimt kernel: CPU: 2 PID: 1933 Comm: nfsd Not tainted 5.6.0-rc6-00040-g881e87a3c6f9 #1591 Mar 30 13:21:45 klimt kernel: Hardware name: Supermicro Super Server/X10SRL-F, BIOS 1.0c 09/09/2015 Mar 30 13:21:45 klimt kernel: RIP: 0010:svc_rdma_post_chunk_ctxt+0xab/0x284 [rpcrdma] Mar 30 13:21:45 klimt kernel: Code: c1 83 34 02 00 00 29 d0 85 c0 7e 72 48 8b bb a0 02 00 00 48 8d 54 24 08 4c 89 e6 48 8b 07 48 8b 40 20 e8 5a 5c 2b e1 41 89 c6 <8b> 45 20 89 44 24 04 8b 05 02 e9 01 00 85 c0 7e 33 e9 5e 01 00 00 Mar 30 13:21:45 klimt kernel: RSP: 0018:ffffc90000dfbdd8 EFLAGS: 00010286 Mar 30 13:21:45 klimt kernel: RAX: 0000000000000000 RBX: ffff8887db8db400 RCX: 0000000000000030 Mar 30 13:21:45 klimt kernel: RDX: 0000000000000040 RSI: 0000000000000000 RDI: 0000000000000246 Mar 30 13:21:45 klimt kernel: RBP: ffff8887e6c27988 R08: 0000000000000000 R09: 0000000000000004 Mar 30 13:21:45 klimt kernel: R10: ffffc90000dfbdd8 R11: 00c068ef00000000 R12: ffff8887eb4e4a80 Mar 30 13:21:45 klimt kernel: R13: ffff8887db8db634 R14: 0000000000000000 R15: ffff8887fc931000 Mar 30 13:21:45 klimt kernel: FS: 0000000000000000(0000) GS:ffff88885bd00000(0000) knlGS:0000000000000000 Mar 30 13:21:45 klimt kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Mar 30 13:21:45 klimt kernel: CR2: ffff8887e6c279a8 CR3: 000000081b72e002 CR4: 00000000001606e0 Mar 30 13:21:45 klimt kernel: Call Trace: Mar 30 13:21:45 klimt kernel: ? svc_rdma_vec_to_sg+0x7f/0x7f [rpcrdma] Mar 30 13:21:45 klimt kernel: svc_rdma_send_write_chunk+0x59/0xce [rpcrdma] Mar 30 13:21:45 klimt kernel: svc_rdma_sendto+0xf9/0x3ae [rpcrdma] Mar 30 13:21:45 klimt kernel: ? nfsd_destroy+0x51/0x51 [nfsd] Mar 30 13:21:45 klimt kernel: svc_send+0x105/0x1e3 [sunrpc] Mar 30 13:21:45 klimt kernel: nfsd+0xf2/0x149 [nfsd] Mar 30 13:21:45 klimt kernel: kthread+0xf6/0xfb Mar 30 13:21:45 klimt kernel: ? kthread_queue_delayed_work+0x74/0x74 Mar 30 13:21:45 klimt kernel: ret_from_fork+0x3a/0x50 Mar 30 13:21:45 klimt kernel: Modules linked in: ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue ib_umad ib_ipoib mlx4_ib sb_edac x86_pkg_temp_thermal iTCO_wdt iTCO_vendor_support coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel glue_helper crypto_simd cryptd pcspkr rpcrdma i2c_i801 rdma_ucm lpc_ich mfd_core ib_iser rdma_cm iw_cm ib_cm mei_me raid0 libiscsi mei sg scsi_transport_iscsi ioatdma wmi ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter nfsd nfs_acl lockd auth_rpcgss grace sunrpc ip_tables xfs libcrc32c mlx4_en sd_mod sr_mod cdrom mlx4_core crc32c_intel igb nvme i2c_algo_bit ahci i2c_core libahci nvme_core dca libata t10_pi qedr dm_mirror dm_region_hash dm_log dm_mod dax qede qed crc8 ib_uverbs ib_core Mar 30 13:21:45 klimt kernel: CR2: ffff8887e6c279a8 Mar 30 13:21:45 klimt kernel: ---[ end trace 87971d2ad3429424 ]--- It's absolutely not safe to use resources pointed to by the @send_wr argument of ib_post_send() _after_ that function returns. Those resources are typically freed by the Send completion handler, which can run before ib_post_send() returns. Thus the trace points currently around ib_post_send() in the server's RPC/RDMA transport are a hazard, even when they are disabled. Rearrange them so that they touch the Work Request only _before_ ib_post_send() is invoked. Fixes: bd2abef333 ("svcrdma: Trace key RDMA API events") Fixes: 4201c74647 ("svcrdma: Introduce svc_rdma_send_ctxt") Signed-off-by: Chuck Lever <chuck.lever@oracle.com> Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>		2020-05-02 17:25:51 +02:00
..
9p.h	License cleanup: add SPDX GPL-2.0 license identifier to files with no license	2017-11-02 11:10:55 +01:00
afs.h	afs: Fix some tracing details	2020-04-02 15:28:19 +02:00
alarmtimer.h	License cleanup: add SPDX GPL-2.0 license identifier to files with no license	2017-11-02 11:10:55 +01:00
asoc.h	ASoC: trace: remove snd_soc_codec	2018-04-16 11:53:35 +01:00
bcache.h	License cleanup: add SPDX GPL-2.0 license identifier to files with no license	2017-11-02 11:10:55 +01:00
block.h	License cleanup: add SPDX GPL-2.0 license identifier to files with no license	2017-11-02 11:10:55 +01:00
bridge.h	net: bridge: use rhashtable for fdbs	2017-12-13 15:10:01 -05:00
btrfs.h	btrfs: tracepoints: Fix bad entry members of qgroup events	2019-10-29 09:20:07 +01:00
cachefiles.h	fscache: Add tracepoints	2018-04-04 13:41:27 +01:00
cgroup.h	cgroup/tracing: Move taking of spin lock out of trace event handlers	2018-07-11 10:48:47 -07:00
clk.h	clk: add duty cycle support	2018-06-19 10:06:29 -07:00
cma.h	License cleanup: add SPDX GPL-2.0 license identifier to files with no license	2017-11-02 11:10:55 +01:00
compaction.h	License cleanup: add SPDX GPL-2.0 license identifier to files with no license	2017-11-02 11:10:55 +01:00
context_tracking.h	License cleanup: add SPDX GPL-2.0 license identifier to files with no license	2017-11-02 11:10:55 +01:00
cpuhp.h	License cleanup: add SPDX GPL-2.0 license identifier to files with no license	2017-11-02 11:10:55 +01:00
devlink.h	License cleanup: add SPDX GPL-2.0 license identifier to files with no license	2017-11-02 11:10:55 +01:00
dma_fence.h	Tracing updates for 4.15:	2017-11-17 14:58:01 -08:00
ext4.h	ext4: force inode writes when nfsd calls commit_metadata()	2019-01-09 17:38:43 +01:00
f2fs.h	treewide: remove large struct-pass-by-value from tracepoint arguments	2018-03-28 22:55:18 +02:00
fib.h	net: Change the layout of structure trace_event_raw_fib_table_lookup	2018-08-13 09:21:05 -07:00
fib6.h	net/ipv6: Udate fib6_table_lookup tracepoint	2018-05-24 23:01:15 -04:00
filelock.h	locks: add tracepoint in flock codepath	2018-08-06 13:15:16 -04:00
filemap.h	License cleanup: add SPDX GPL-2.0 license identifier to files with no license	2017-11-02 11:10:55 +01:00
fs_dax.h	libnvdimm for 4.15	2017-11-17 09:51:57 -08:00
fscache.h	fscache: Maintain a catalogue of allocated cookies	2018-04-06 14:05:14 +01:00
fsi_master_ast_cf.h	fsi: master-ast-cf: Add new FSI master using Aspeed ColdFire	2018-07-23 15:22:52 +10:00
fsi_master_gpio.h	fsi: master-gpio: Add more tracepoints	2018-07-12 12:02:31 +10:00
fsi.h	License cleanup: add SPDX GPL-2.0 license identifier to files with no license	2017-11-02 11:10:55 +01:00
gpio.h	License cleanup: add SPDX GPL-2.0 license identifier to files with no license	2017-11-02 11:10:55 +01:00
host1x.h	gpu: host1x: Remove wait check support	2018-05-18 21:50:04 +02:00
hswadsp.h	License cleanup: add SPDX GPL-2.0 license identifier to files with no license	2017-11-02 11:10:55 +01:00
huge_memory.h	License cleanup: add SPDX GPL-2.0 license identifier to files with no license	2017-11-02 11:10:55 +01:00
i2c.h	i2c: break out smbus support into separate file	2017-05-31 21:01:03 +02:00
initcall.h	tracing: initcall: Ordered comparison of function pointers	2018-04-26 15:02:46 -04:00
intel_ish.h	License cleanup: add SPDX GPL-2.0 license identifier to files with no license	2017-11-02 11:10:55 +01:00
intel-sst.h	License cleanup: add SPDX GPL-2.0 license identifier to files with no license	2017-11-02 11:10:55 +01:00
iommu.h	License cleanup: add SPDX GPL-2.0 license identifier to files with no license	2017-11-02 11:10:55 +01:00
ipi.h	License cleanup: add SPDX GPL-2.0 license identifier to files with no license	2017-11-02 11:10:55 +01:00
irq_matrix.h	genirq/matrix: Add tracepoints	2017-09-25 20:38:26 +02:00
irq.h	License cleanup: add SPDX GPL-2.0 license identifier to files with no license	2017-11-02 11:10:55 +01:00
jbd2.h	License cleanup: add SPDX GPL-2.0 license identifier to files with no license	2017-11-02 11:10:55 +01:00
kmem.h	mm: remove cold parameter from free_hot_cold_page*	2017-11-15 18:21:06 -08:00
kvm.h	KVM: Fix stack-out-of-bounds read in write_mmio	2017-12-18 12:57:01 +01:00
libata.h	License cleanup: add SPDX GPL-2.0 license identifier to files with no license	2017-11-02 11:10:55 +01:00
lock.h	License cleanup: add SPDX GPL-2.0 license identifier to files with no license	2017-11-02 11:10:55 +01:00
mce.h	License cleanup: add SPDX GPL-2.0 license identifier to files with no license	2017-11-02 11:10:55 +01:00
mdio.h	License cleanup: add SPDX GPL-2.0 license identifier to files with no license	2017-11-02 11:10:55 +01:00
migrate.h	mm, sched/numa: Remove rate-limiting of automatic NUMA balancing migration	2018-10-02 11:31:14 +02:00
mmc.h	mmc: core: Fix tracepoint print of blk_addr and blksz	2018-03-15 11:15:22 +01:00
mmflags.h	Drop a bunch of metag references	2018-02-23 14:29:59 +00:00
module.h	License cleanup: add SPDX GPL-2.0 license identifier to files with no license	2017-11-02 11:10:55 +01:00
napi.h	License cleanup: add SPDX GPL-2.0 license identifier to files with no license	2017-11-02 11:10:55 +01:00
net_probe_common.h	net: dccp: Add DCCP sendmsg trace event	2018-01-02 14:27:30 -05:00
net.h	net: core: unwrap skb list receive slightly further	2018-07-04 14:06:19 +09:00
nilfs2.h	License cleanup: add SPDX GPL-2.0 license identifier to files with no license	2017-11-02 11:10:55 +01:00
nmi.h	License cleanup: add SPDX GPL-2.0 license identifier to files with no license	2017-11-02 11:10:55 +01:00
oom.h	License cleanup: add SPDX GPL-2.0 license identifier to files with no license	2017-11-02 11:10:55 +01:00
page_isolation.h	License cleanup: add SPDX GPL-2.0 license identifier to files with no license	2017-11-02 11:10:55 +01:00
page_ref.h	License cleanup: add SPDX GPL-2.0 license identifier to files with no license	2017-11-02 11:10:55 +01:00
pagemap.h	License cleanup: add SPDX GPL-2.0 license identifier to files with no license	2017-11-02 11:10:55 +01:00
percpu.h	License cleanup: add SPDX GPL-2.0 license identifier to files with no license	2017-11-02 11:10:55 +01:00
power_cpu_migrate.h	License cleanup: add SPDX GPL-2.0 license identifier to files with no license	2017-11-02 11:10:55 +01:00
power.h	cpufreq: trace frequency limits change	2018-07-26 10:17:47 +02:00
preemptirq.h	tracing: Change offset type to s32 in preempt/irq tracepoints	2020-01-14 20:07:00 +01:00
printk.h	License cleanup: add SPDX GPL-2.0 license identifier to files with no license	2017-11-02 11:10:55 +01:00
qdisc.h	qdisc: add tracepoint qdisc:qdisc_dequeue for dequeued SKBs	2017-08-16 14:10:10 -07:00
random.h	License cleanup: add SPDX GPL-2.0 license identifier to files with no license	2017-11-02 11:10:55 +01:00
rcu.h	rcu: Remove CPU-hotplug failsafe from force-quiescent-state code path	2018-07-12 15:39:07 -07:00
rdma.h	rdma/ib: Add trace point macros to display human-readable values	2018-01-23 09:44:14 -05:00
regulator.h	License cleanup: add SPDX GPL-2.0 license identifier to files with no license	2017-11-02 11:10:55 +01:00
rpcrdma.h	svcrdma: Fix trace point use-after-free race	2020-05-02 17:25:51 +02:00
rpm.h	License cleanup: add SPDX GPL-2.0 license identifier to files with no license	2017-11-02 11:10:55 +01:00
rseq.h	rseq: Introduce restartable sequences system call	2018-06-06 11:58:31 +02:00
rtc.h	rtc: Add tracepoints for RTC system	2018-02-13 21:30:22 +01:00
rxrpc.h	rxrpc: Fix trace-after-put looking at the put connection record	2020-01-27 14:51:16 +01:00
sched.h	sched/debug: Use symbolic names for task state constants	2019-11-20 18:46:14 +01:00
scsi.h	License cleanup: add SPDX GPL-2.0 license identifier to files with no license	2017-11-02 11:10:55 +01:00
sctp.h	net: sctp: Add SCTP ACK tracking trace event	2018-01-02 14:27:29 -05:00
signal.h	License cleanup: add SPDX GPL-2.0 license identifier to files with no license	2017-11-02 11:10:55 +01:00
siox.h	siox: add support for tracing	2017-12-19 10:56:24 +01:00
skb.h	License cleanup: add SPDX GPL-2.0 license identifier to files with no license	2017-11-02 11:10:55 +01:00
smbus.h	i2c: break out smbus support into separate file	2017-05-31 21:01:03 +02:00
sock.h	net: expose sk wmem in sock_exceed_buf_limit tracepoint	2018-07-02 22:40:56 +09:00
spi.h	License cleanup: add SPDX GPL-2.0 license identifier to files with no license	2017-11-02 11:10:55 +01:00
spmi.h	License cleanup: add SPDX GPL-2.0 license identifier to files with no license	2017-11-02 11:10:55 +01:00
sunrpc.h	sunrpc: use-after-free in svc_process_common()	2019-01-16 22:04:37 +01:00
sunvnet.h	License cleanup: add SPDX GPL-2.0 license identifier to files with no license	2017-11-02 11:10:55 +01:00
swiotlb.h	License cleanup: add SPDX GPL-2.0 license identifier to files with no license	2017-11-02 11:10:55 +01:00
syscalls.h	License cleanup: add SPDX GPL-2.0 license identifier to files with no license	2017-11-02 11:10:55 +01:00
target.h	License cleanup: add SPDX GPL-2.0 license identifier to files with no license	2017-11-02 11:10:55 +01:00
task.h	License cleanup: add SPDX GPL-2.0 license identifier to files with no license	2017-11-02 11:10:55 +01:00
tcp.h	tcp: minor optimization around tcp_hdr() usage in receive path	2018-05-31 13:20:47 -04:00
thermal_power_allocator.h	License cleanup: add SPDX GPL-2.0 license identifier to files with no license	2017-11-02 11:10:55 +01:00
thermal.h	cpu_cooling: Drop static-power related stuff	2017-12-07 22:52:01 +01:00
thp.h	License cleanup: add SPDX GPL-2.0 license identifier to files with no license	2017-11-02 11:10:55 +01:00
timer.h	hrtimer: Add clock bases and hrtimer mode for softirq context	2018-01-16 03:00:50 +01:00
tlb.h	License cleanup: add SPDX GPL-2.0 license identifier to files with no license	2017-11-02 11:10:55 +01:00
udp.h	License cleanup: add SPDX GPL-2.0 license identifier to files with no license	2017-11-02 11:10:55 +01:00
ufs.h	scsi: ufs: add trace event for ufs upiu	2018-04-18 23:37:39 -04:00
v4l2.h	License cleanup: add SPDX GPL-2.0 license identifier to files with no license	2017-11-02 11:10:55 +01:00
vb2.h	License cleanup: add SPDX GPL-2.0 license identifier to files with no license	2017-11-02 11:10:55 +01:00
vmscan.h	mm, vmscan, tracing: use pointer to reclaim_stat struct in trace event	2018-04-11 10:28:30 -07:00
vsock_virtio_transport_common.h	License cleanup: add SPDX GPL-2.0 license identifier to files with no license	2017-11-02 11:10:55 +01:00
wbt.h	block: Fix writeback throttling W=1 compiler warnings	2019-12-31 16:34:59 +01:00
workqueue.h	tracing: Add missing forward declaration	2018-04-19 11:05:48 -04:00
writeback.h	Merge branch 'for-4.15/block' of git://git.kernel.dk/linux-block	2017-11-14 15:32:19 -08:00
xdp.h	bpf: fix redirect to map under tail calls	2018-08-17 15:56:23 -07:00
xen.h	tracing: xen: Ordered comparison of function pointers	2020-01-29 16:43:21 +01:00