From c6dc0a288d4e787c578d7ab0464b9c943ad88ecc Mon Sep 17 00:00:00 2001 From: Josef Bacik Date: Sat, 17 Oct 2026 16:37:08 +0000 Subject: [PATCH] bpf: Forbid sleeping in bpf_find_vma() callbacks A sleepable program can call bpf_find_vma() and use a sleepable helper such as bpf_copy_from_user() or bpf_copy_from_user_task() in the callback. The verifier accepts this, but bpf_find_vma() runs the callback with the mmap_lock of the mm held for read. If the user copy faults on that mm, the fault handler has to take mmap_lock again. Lockdep reports recursive locking, and if a writer queued up in between, the fault blocks behind the writer, which waits for the read lock the helper already holds: sleepable prog other thread of the same mm bpf_find_vma(current, ...) mmap_read_trylock(mm) mmap_write_lock(mm) <- waits callback: copy_from_user() page fault mmap_read_lock(mm) <- queued behind the writer This is the same problem as with sleepable task_vma iterator programs. The verifier checks synchronous callbacks in the caller's context, so in_sleepable_context() is true in the callback of a sleepable program. Verify the callback like a hash map for_each callback, as running in an RCU read-side critical section where sleeping is not allowed, and actually hold rcu_read_lock() around it, so that global functions the callback calls, which are then verified for a non-sleepable context, see the RCU protection they are verified with. Non-sleepable programs already run the callback under RCU, nothing changes for them. A sleepable program whose bpf_find_vma() callback calls any helper, kfunc or global function that may sleep is now rejected by the verifier, including ones that would be harmless under mmap_lock as they don't touch user memory. Fixes: 7c7e3d31e785 ("bpf: Introduce helper bpf_find_vma") Signed-off-by: Josef Bacik Signed-off-by: Alexei Starovoitov Link: https://patch.msgid.link/20261017163745.3341787-7-alexei.starovoitov@gmail.com --- kernel/bpf/task_iter.c | 6 ++++++ kernel/bpf/verifier.c | 2 ++ 2 files changed, 8 insertions(+) diff --git a/kernel/bpf/task_iter.c b/kernel/bpf/task_iter.c index cf2bf388e181..2fa6b8dae8c4 100644 --- a/kernel/bpf/task_iter.c +++ b/kernel/bpf/task_iter.c @@ -810,8 +810,14 @@ BPF_CALL_5(bpf_find_vma, struct task_struct *, task, u64, start, vma = find_vma(mm, start); if (vma && vma->vm_start <= start && vma->vm_end > start) { + /* + * The callback must not sleep while mmap_lock is held, and the + * verifier checks it as an RCU read-side critical section. + */ + rcu_read_lock(); callback_fn((u64)(long)task, (u64)(long)vma, (u64)(long)callback_ctx, 0, 0); + rcu_read_unlock(); ret = 0; } bpf_mmap_unlock_mm(work, mm); diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 883cf148235c..430d5309eb5c 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -11629,6 +11629,8 @@ static int set_find_vma_callback_state(struct bpf_verifier_env *env, bpf_mark_reg_not_init(env, &callee->regs[BPF_REG_4]); bpf_mark_reg_not_init(env, &callee->regs[BPF_REG_5]); callee->in_callback_fn = true; + /* bpf_find_vma() runs the callback under mmap_lock and RCU. */ + callee->in_rcu_callback_fn = true; callee->callback_ret_range = retval_range(0, 1); return 0; }