Delete all elements of BPF_F_NO_PREALLOC hash map in one batch. The first
free_bulk() starts RCU tasks trace GP and the rest of the elements are
freed while it's in flight. Wait for call_rcu_ttrace_in_progress to clear
in bpf_mem_cache of every cpu and check that free_by_rcu_ttrace and
waiting_for_gp_ttrace lists are empty.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://lore.kernel.org/bpf/20260930095920.601738-4-alexei.starovoitov@gmail.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Add tests where two packet pointers share an id and tightening one
pointer's umax from its var_off would put it less than their constant
distance from the other's umax: with an index & 0x38 capped at 50, the
base pointer keeps umax 50, so the pointer 8 bytes further on must keep
umax 58, even though its known bits allow at most 56.
These refused a valid program or accepted an out-of-bounds access before
the fix:
- check the advanced copy, load through the base: valid, was refused;
- check the base, load the byte at base + 1 through a copy advanced by
8: was accepted;
- check base + 4, load 4 bytes at base + 2 through base + 8: reads two
bytes past the checked range, was accepted;
- the same as the second with data_meta pointers checked against data:
was accepted.
These pass with and without the fix and cover nearby paths:
- subtract an unknown scalar from a checked pointer and load below it
(the range is kept across a new id);
- reach a load through two paths whose checks cover 8 and 7 bytes after
the loaded pointer; the second path must not be pruned by the first;
- spill a copy of a pointer, check the pointer, fill the copy and load
one byte past the checked range: the load is refused, and the copy
has the range of the check.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://lore.kernel.org/bpf/20261001145255.855630-2-alexei.starovoitov@gmail.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Since commit 022ac07508 ("bpf: use reg->var_off instead of reg->off
for pointers"), find_good_pkt_pointers() sets the range of all packet
pointers sharing an id from the umax of the compared pointer, and
check_packet_access() requires umax + off + size <= range. That assumes
the umax of two such pointers differ by exactly their constant distance.
reg_bounds_sync() breaks it when var_off tightens one umax and not the
other:
r4 &= 0x38
if r4 > 50 goto exit ; umax 50, var_off (0x0; 0x38)
r5 = pkt + r4 ; umax 50
r6 = r5
r6 += 8 ; umax 56, not 58
Comparing r6 with pkt_end sets the range to 56, and the valid 8-byte
load at r5 is rejected (50 + 8 > 56). Comparing r5 sets it to 50, and
the out-of-bounds 1-byte load at r6 - 7, i.e. r5 + 1, is accepted
(56 - 7 + 1 <= 50).
Don't call reg_bounds_sync() on a packet pointer that keeps its id (a
constant was added or subtracted) or its range (an unknown non-negative
value was subtracted), so that var_off cannot tighten its umax. Only
update the 32-bit bounds from var_off: reg_bounds_sanity_check() wants
them constant when the lower half of var_off is, e.g. for pkt + 8.
This relies on nothing else changing the 64-bit bounds of a packet
pointer, which holds today.
var_off of such a pointer is no longer narrowed by its bounds. Adjust
three verifier_align expectations; the low bits, which the alignment
checks use, don't change. veristat on the selftests shows no verdict
changes and +0.8% insns in test_cls_redirect_subprogs.
Fixes: 022ac07508 ("bpf: use reg->var_off instead of reg->off for pointers")
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://lore.kernel.org/bpf/20261001145255.855630-1-alexei.starovoitov@gmail.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Add a test for the use-after-free fixed by the previous commit. A
sleepable prog on bpf_fentry_test1() blocks a task on a userfaultfd
page, like bpf_mod_race does, while the two progs that run after it in
the same trampoline are detached and freed one after the other, which
the test knows from their .bss maps going away. After the first detach
the task is in an image that isn't the trampoline's current one
anymore. The task is then released and must not call into the freed
progs. This is done with fentry progs, with fexit progs, where the task
is already past the original function, and with a task sleeping in a
fentry prog while fexit progs are detached, where the original function
must still be called. bpf_mod_race's userfaultfd helper moves to
testing_helpers.c so that both tests can use it.
Without the fix, on a kernel with KASAN:
BUG: KASAN: vmalloc-out-of-bounds in __bpf_prog_enter_recur+0xed/0x1e0
Read of size 8 at addr ffa0000000144040 by task test_progs/171
CPU: 6 UID: 0 PID: 171 Comm: test_progs Tainted: G OE 7.2.0+ #1 PREEMPT(full)
Call Trace:
<TASK>
__bpf_prog_enter_recur+0xed/0x1e0
bpf_trampoline_6442545468+0x72/0xe3
bpf_fentry_test1+0x9/0x20
bpf_prog_test_run_tracing+0x183/0x3e0
__sys_bpf+0xd3f/0x38f0
...
Signed-off-by: Florent Revest (Anthropic) <florent.revest@linux.dev>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20260926135605.1217928-4-florent.revest@linux.dev
The existing cpu_flag subtests always prime a key with BPF_F_ALL_CPUS
before any BPF_F_CPU write, so the create path is never covered.
Add a subtest that creates the element with BPF_F_CPU on a map with
max_entries 1, so the key can only reuse the element the previous key
released, and check that the CPUs the update did not name read back
zero. Run it for PERCPU_HASH preallocated and BPF_F_NO_PREALLOC,
whose per-cpu areas come from different allocators, and for
LRU_PERCPU_HASH.
Under BPF_F_NO_PREALLOC the reuse is only guaranteed on the cpu that
ran the delete, so pin the thread across the pair, and name a cpu other
than that one in map_flags.
Signed-off-by: Donggeun Yoo <donggeunyoo.kernel@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20260924102321.2120434-3-donggeunyoo.kernel@gmail.com
Add a selftest to confirm the verifier rejects ALU operations
that return arena or non-arena results depending on code path.
Signed-off-by: Emil Tsalapatis <emil@etsalapatis.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20260922172028.6269-9-emil@etsalapatis.com
Add a selftests that ensures that PTR_TO_PACKET arguments can
only be passed to subprogs that will never adjust the underlying
packet memory, and are rejected otherwise.
Signed-off-by: Emil Tsalapatis <emil@etsalapatis.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20260922172028.6269-7-emil@etsalapatis.com
Add tests to ensure the verifier properly tracks the 0 bit state
and width of the rx_queue_mapping field read from struct sock.
Signed-off-by: Emil Tsalapatis <emil@etsalapatis.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20260922172028.6269-5-emil@etsalapatis.com
Add a selftest to ensure dynptr slices cannot include
past the end of the linear area of an skb.
Signed-off-by: Emil Tsalapatis <emil@etsalapatis.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20260922172028.6269-3-emil@etsalapatis.com
Add a verifier regression test that initializes a numeric iterator at fp-8
and attempts to destroy it through fp+0. The verifier must reject the
non-negative offset instead of treating it as the initialized stack slot.
Check the offset diagnostic to ensure rejection happens at the stack
object address check. The numeric iterator destroy operation is a no-op;
this test checks verifier rejection and does not run the program.
Signed-off-by: Xu Yunxiang <xyx2021@mail.ustc.edu.cn>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Reviewed-by: Sun Jian <sun.jian.kdev@gmail.com>
Link: https://lore.kernel.org/bpf/20260920210423.345636-3-xyx2021@mail.ustc.edu.cn
Build raw program BTF records to exercise local object ownership without
constructing a runtime chain deep enough to threaten the kernel stack.
Cover referenced-kptr and percpu-kptr self-cycles, a two-type cycle, and a
cycle mixing a graph root with a referenced kptr. The existing graph-only
check accepts these local-kptr cycles and over-limit kptr chains; the fix
rejects them with -ELOOP.
Pin the eight-record depth boundary with a terminal plain object. Exercise
both parent-first and child-first BTF orders, and a shared suffix reached
first through a shorter path. These cases require cached suffix depths to
be checked against the remaining depth budget on each path.
Check list and rbtree chains of three, four, eight, and nine types. This
covers the old graph-only depth boundary and the new explicit bound. The
existing linked-list BTF tests still reject pure graph cycles and now accept
the longer acyclic layouts previously rejected by the conservative rule.
Although bpf_percpu_obj_new() currently rejects types with special fields,
require the percpu cycle to fail at BTF load so future support cannot bypass
the ownership bound. Keep a positive control for a non-owning kptr, which
remains outside the ownership graph.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20260914132444.2564218-3-memxor@gmail.com
Program-allocated objects can own other local objects through referenced
kptrs. bpf_obj_free_fields() follows those pointers through
__bpf_obj_drop_impl() synchronously, before the object storage is freed
through RCU. A self-referential local kptr type therefore permits arbitrarily
deep object chains, and dropping the head can exhaust the kernel stack.
Long acyclic type chains have the same problem.
btf_check_and_fixup_fields() still assumes referenced kptrs only point to
kernel types and checks ownership through list and rbtree roots only. Its
existing rule is sufficient for graph-only cycles: the target of each graph
edge must contain a node, so every type in a cycle has both a root and a
node. The rule rejects such a type owning another root, breaking every
cycle. It also limits graph-only chains to three types, or two if the first
type contains a node, and conservatively rejects longer acyclic chains.
The missing local-kptr edges, rather than a missed graph-only cycle, are the
bug introduced by support for bpf_kptr_xchg() into local kptrs.
Replace that restriction with one bounded ownership walk covering graph
roots and local referenced kptrs. Run it after all BTF records have been
fixed up, reject cycles and paths deeper than eight record-bearing types,
and cache each type's suffix depth while checking it against the remaining
budget. This also permits the longer acyclic graph-only layouts rejected
by the old rule; update their existing BTF tests accordingly.
Keep the bound independent of MAX_CALL_FRAMES because recursive destruction
can run below a BPF call chain. A plain local pointee without special-field
metadata adds only a final non-recursing drop. Non-owning kptrs and
kernel-BTF kptrs do not recurse through local records and remain outside the
walk. Include local percpu-kptr edges too, although allocation of percpu
objects with special fields is currently forbidden, so that relaxing that
restriction cannot bypass the ownership bound.
btf_check_and_fixup_fields() continues to initialize graph_root.value_rec,
including for separately allocated map records. The ownership relationships
belong to immutable program BTF and only need validation at BTF load time.
Fixes: b0966c7245 ("bpf: Support bpf_kptr_xchg into local kptr")
Reported-by: Nicholas Carlini <npc@anthropic.com>
Suggested-by: Nicholas Carlini <npc@anthropic.com>
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20260914132444.2564218-2-memxor@gmail.com
Add a finite loop whose progress is represented only by changing the
frame number of a stack pointer. The loop first reads zero from the
caller's stack, switches to the same offset in the callee's stack, and
exits after reading one on its next iteration.
Force frequent checkpoints so the test exercises infinite-loop detection,
and check that the program returns one when run.
Without the frameno comparison in regs_exact(), the program is rejected
with an "infinite loop detected" diagnostic instead of loading
successfully.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Tested-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://patch.msgid.link/20260919014213.1840880-3-memxor@gmail.com
Add a verifier test which retains a map value from an outer callback and
then acquires a lock through an inner callback value before attempting to
release the outer callback value. Both values can denote different elements,
so the verifier must reject the mismatched unlock.
Also exercise callbacks reached through two inner-map lookups. The lookup
results share inner_map_meta but may refer to different one-element arrays,
so their callback values must retain distinct lock identities.
Extend the existing spin_lock failure table and reuse its array and
inner-map fixtures to keep these cases alongside the other lock identity
tests. Update the nested callback reference-leak expectation for the extra
callback value ID.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Link: https://patch.msgid.link/20260917233222.2542500-10-memxor@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Add raw CO-RE relocations that fail to resolve their target enum value.
Place each supported and unsupported instruction form in dead code.
Unsupported targets must fail relocation with a diagnostic even when they
are unreachable. Supported ALU immediates, memory accesses, and ldimm64
instructions must still be poisoned and removed as dead code, allowing the
program to load. Check that both halves of ldimm64 are poisoned.
Load every instruction stream without relocations first to ensure that
rejection is caused by the relocation rather than the original program.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://patch.msgid.link/20260917233222.2542500-8-memxor@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Add a raw program load with CO-RE relocation metadata but no func_info or
line_info. Place the relocation in dead code and require the poisoning log,
proving that the kernel processes standalone CO-RE metadata instead of
silently skipping it.
Also give a subprogram a relocatable immediate as its terminal instruction.
Require the relocation's poisoning log before check_subprogs() rejects the
resulting fall-through. With the old ordering, check_subprogs() rejects the
original terminal instruction before CO-RE can emit the substitution log, so
the test continues to distinguish the ordering after relocation target
validation is tightened.
Submit a trailing ldimm64 first slot with CO-RE metadata and require the early
structural diagnostic. This exercises the check that protects relocation
processing instead of the later regular instruction validation.
Load the standalone instruction stream without relocation metadata first to
ensure that CO-RE processing causes its poisoning diagnostic. Encode the fixed
BTF metadata directly with the selftest BTF helpers.
Suggested-by: Eduard Zingerman <eddyz87@gmail.com>
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://patch.msgid.link/20260917233222.2542500-6-memxor@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Add two paths whose packet pointer ranges are individually compatible at
a join but whose members have different relative displacements. The first
path proves an eight-byte access through one member. On the second path,
the same guard only proves that the access starts before data_end.
An affected verifier prunes the second path and accepts the program. With
packet pointer class displacement preserved, it explores that path and
rejects the out-of-bounds access.
Read the unknown offset and branch selector directly from XDP context
fields, and force state checkpoints so the pruning attempt does not
depend on the verifier checkpoint heuristics.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Link: https://patch.msgid.link/20260917233222.2542500-4-memxor@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Exercise global subprogram verification from workqueue callbacks, which
can run in a sleepable context even when the containing program is not
sleepable. An unprotected callback must not let the global subprogram use
implicit RCU protection inherited from the program.
Add a negative case which loads an RCU-protected task kptr in a global
subprogram reached from a workqueue callback. It fails on an unfixed
kernel because the program is incorrectly accepted. Also cover a
workqueue callback protected by an explicit RCU read-side critical
section, where the same global subprogram remains valid.
Call the same harmless global subprogram directly from the main program
and from an unprotected callback. Mark it __weak __noinline so both calls
survive optimization, and check that its instruction statistics account
for both verification contexts. This also verifies that global calls
from callbacks are not rejected wholesale. Workqueue callbacks return
zero explicitly after the global call, as required by their contract.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://patch.msgid.link/20260914131923.2544250-3-memxor@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
The existing exception tests do not reach a callback that materializes
BPF_REG_FP into a register. They either throw from the main program,
where BPF_REG_FP already holds the value the callback needs, or use a
callback whose only stack accesses are frame pointer relative, which the
arm64 JIT rewrites to be stack pointer relative.
Add a test that throws from a subprogram using its own BPF stack, with a
callback that hands the address of a local variable to
bpf_probe_read_kernel(). The helper and the callback have to name the
same slot for the value read back to be the one the helper stored.
Signed-off-by: Donggeun Yoo <donggeunyoo.kernel@gmail.com>
Link: https://lore.kernel.org/r/20260907130624.611942-3-donggeunyoo.kernel@gmail.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Add tests that place gotox at the end of the main program and a
subprogram, with each jump-table target preceding the gotox instruction.
This tests gotox as a valid non-fallthrough terminal instruction.
Signed-off-by: Siddharth Chintamaneni <sidchintamaneni@gmail.com>
Reviewed-by: Anton Protopopov <a.s.protopopov@gmail.com>
Link: https://lore.kernel.org/r/20260902171414.96165-2-sidchintamaneni@gmail.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Add two subtests.
tcp_timewait: the client shuts down first and the server closes after
it, so the client sock ends up in TIME_WAIT. A tcp iterator then finds
the timewait sock by the cookie it inherited from the client sock and
destroys it. Iterate once more to make sure it is gone. Without the
first fix bpf_sock_destroy() reads past the timewait sock and KASAN
complains.
tcp_listen_pending: connect to a listener but never accept, so the child
sits in the accept queue, then destroy the listener. Without the second
fix the cond_resched() in inet_csk_listen_stop() trips the might_sleep
check under rcu_read_lock().
./test_progs -a sock_destroy
#444/1 sock_destroy/tcp_client:OK
#444/2 sock_destroy/tcp_server:OK
#444/3 sock_destroy/tcp_listen_pending:OK
#444/4 sock_destroy/tcp_timewait:OK
#444/5 sock_destroy/udp_client:OK
#444/6 sock_destroy/udp_server:OK
#444/7 sock_destroy/trace_tcp_destroy_sock:OK
#444 sock_destroy:OK
Summary: 1/7 PASSED, 0 SKIPPED, 0/0 FAILED
Signed-off-by: Jiayuan Chen <jiayuan.chen@linux.dev>
Link: https://lore.kernel.org/r/20260910112810.153801-1-jiayuan.chen@linux.dev
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
The program stashes a bpf_obj_new() object whose type ends with a flexible
array of empty structs, then reads it back as an untrusted kptr. Without
the previous patch this divides by zero in btf_struct_walk() instead of
being rejected.
# ./test_progs -t verifier_btf_flex_array
...
#602 verifier_btf_flex_array:OK
Summary: 1/1 PASSED, 0 SKIPPED, 0/0 FAILED
Signed-off-by: Jiayuan Chen <jiayuan.chen@linux.dev>
Link: https://lore.kernel.org/r/20260910122316.186384-2-jiayuan.chen@linux.dev
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Add a test which builds an ICMP error out of a TCP segment. A tcx prog
on the client's egress side trims the first data segment down to the
target size and pushes the ICMP error headers in front of it to then
reflect the packet back to the sender.
# LDLIBS=-static PKG_CONFIG='pkg-config --static' ./vmtest.sh -- ./test_progs -t tc_change_tail
[...]
#509 tc_change_tail:OK
#510 tc_change_tail_pmtu:OK
Summary: 2/0 PASSED, 0 SKIPPED, 0/0 FAILED
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Link: https://lore.kernel.org/r/20260907121025.1923656-2-daniel@iogearbox.net
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
-----BEGIN PGP SIGNATURE-----
iQIzBAABCAAdFiEE+soXsSLHKoYyzcli6rmadz2vbToFAmqdhRMACgkQ6rmadz2v
bTo8vRAAnP+x1z6FBOgHawGmmBpXtazoipRFeok/+/YGQ4SbS+evvFNvmkeCyihZ
7EeNHGuWEbA6xQBAcMf0StMjmr3jkJvUA6lsHSxBiFLpHGqNt0Fz9IXvAL/8QWPg
F7+CBs5A+J61i66LEmINbWzW1ujDf7baU/1VWui9zfPxnmPu40V/74eXzm6DmH/V
oHTsCscCYNe+N+Wix6B6nLrObY537fcJU12N56uqpkaSc0+6H1fBQJmyFGpYroTX
+4g+DNzEHyIYmt6B3+oDpbT0Zh/pP6ROq5TPDNJSSa+5B/uPo5C5pH520HwXF9hJ
04uaD7y2fpE809Nwl3OEK4ozpJ+mEGU1NKZPvFAxckY6GCiBtfbGd1etFno7r+4F
mZhNBKbQVEQX4XJUlcoqn9n7T2OCDnjiuneZAjIi4vUdevnSnakr0rCTFriiHP54
jg4fEkXxvYjPiqs5SQbxoGxmYSfg9fYaSfK4HgFdlfbwp+3Vmv4BW/tcsIJ9dcPs
L3YWkUU3znjbu2BLy0CP9fGt/1ik8p4/tA8vvZwz0yqpDMJkLy1TziGMef51UQ7k
1Pe/Ln5J3C6xnWs+3TRF11y8PuuCF0y47DGPACfwPoQxca1ufJ5fgmLXRRQToZe4
RjDmoeQvvI9w9kxh6gVhTHoiiEeD0wcQqx2CMHKTrU/PbM7d4Co=
=oz4Z
-----END PGP SIGNATURE-----
Merge tag 'bpf-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf
Pull bpf fixes from Alexei Starovoitov:
"This mainly contains verifier fixes that address bugs reported by
Nicholas Carlini.
- Fix incorrect non-NULL inference in pointer comparisons: pointer
types that may be NULL at runtime, pointers with unbounded offsets,
JMP32 comparisons with zero, and imprecise zero registers (Eduard
Zingerman)
- Fix precision tracking for half-dead zero spills, ld_abs/ld_ind
implicit subprog exit, bpf_loop() callbacks, linked scalar ids and
NULL call arguments (Eduard Zingerman)
- Reject BPF_PSEUDO_FUNC reference to the main program, fix zero
extension of arena 32-bit cmpxchg, don't rewrite bpf_fastcall
patterns entered by a jump (Eduard Zingerman)
- Fix percpu map update and BPF_F_CPU validation with sparse CPU IDs
(Hui Su)
- Fix NULL-ptr-derefs in bpf_snprintf_btf() for void and VAR types,
and reject key-less BTF for hash maps (Jiayuan Chen)
- Various fixes (Kumar Kartikeya Dwivedi):
- Fix out-of-bounds access in disassembler on invalid LDSX
instruction
- mark siginfo of signal tracepoints as scalar and
sched_process_wait argument as nullable
- mark faultable stack helpers as sleepable
- reject tail calls and legacy packet loads from callbacks
- enforce rbtree callback lock restrictions for resilient locks
- require MEM_PERCPU for percpu kptr stores
- clear NON_OWN_REF after RCU protection ends
- mark NULL kptr stores precise
- preserve inner map identity in callback frames
- reject non-scalar bpf_loop() iteration counts
- Fix trampoline allocation slowdown on x86 by using
EXECMEM_MODULE_DATA (Mike Rapoport)
- Keep bpf_refcount_acquire() nullable for borrowed RCU kptrs and
reject untrusted allocated-object pointers (Ning Ding)
- Fix special fields handling in recycled rhtab elements (Nuoqi Gui,
Yuan Chen)"
* tag 'bpf-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf: (86 commits)
bpf, riscv: Make arena support depend on ZACAS
selftests/bpf: Test pointer bpf_loop iteration count rejection
bpf: Reject non-scalar bpf_loop iteration counts
bpf: use mark_arg_precision() in check_mem_size_reg()
bpf: propagate mark_chain_precision() errors out of loop_flag_is_zero()
selftests/bpf: precision of a NULL global subprogram BTF_ID argument
bpf: mark a NULL BTF_ID argument of a global subprogram precise
selftests/bpf: precision of a NULL kfunc argument
bpf: mark a NULL kfunc argument precise
selftests/bpf: precision of a NULL global subprogram memory argument
bpf: mark a NULL memory argument of a call precise
selftests/bpf: precision of a NULL helper argument
bpf: mark a NULL call argument precise
selftests/bpf: Test inner map identities in callbacks
bpf: Preserve inner map identity in callback frames
selftests/bpf: Test imprecise scalar kptr stores
bpf: Mark NULL kptr stores precise
selftests/bpf: Test rhtab kptr cancellation semantics
bpf: Cancel special fields when recycling rhtab elements
selftests/bpf: Test timer field on recycled rhtab element
...
Add a verifier test that leaves the raw tracepoint context pointer in R1
when calling bpf_loop(). This is the smallest trigger for the incorrect
precision backtracking: it reuses an existing callback and needs no maps or
userspace setup.
Expect an ordinary scalar-type rejection. Without the verifier fix, the
test instead reaches precision backtracking and reports an internal
"backtracking misuse" error.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://patch.msgid.link/20260905014735.1452988-3-memxor@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Check that mark_chain_precision() is called for a NULL pointer passed
as a nullable pointer argument of a global subprogram.
(Pointer arguments of the global subprograms are nullable by default).
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://lore.kernel.org/r/20260904-register-is-null-precise-fixes-v1-4-0f5a360ff15d@gmail.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
check_func_arg() allows bpf_register_is_null() for nullable arguments
w/o marking the underlying scalar register precise. Hence a checkpoint
created on such a path would prune against arbitrary scalar value.
check_helper_call() enforces second parameter of the
bpf_get_local_storage() to be zero, w/o marking the underlying scalar
register precise. Hence a checkpoint created on such a path would
prune against arbitrary scalar value.
Grouping these two into one patch, as they share the same fixes tag.
Fixes: b5dc0163d8 ("bpf: precise scalar_value tracking")
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://lore.kernel.org/r/20260904-register-is-null-precise-fixes-v1-1-0f5a360ff15d@gmail.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Add load-only timer_mim coverage for inner map identities propagated
through nested timer and bpf_for_each_map_elem() callbacks.
The negative case initializes a timer in the second inner map with the map
saved from the first inner map timer callback. The positive case pairs the
timer value with the map supplied to the same for-each callback.
Without the verifier fix, the mismatched-map program is accepted while the
same-map control is rejected. Preserving map_uid reverses both verdicts.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Link: https://lore.kernel.org/r/20260904104203.345917-9-memxor@gmail.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Add a verifier regression where an imprecise zero scalar reaches a kptr
store first and a nonzero scalar reaches the same instruction on a second
path.
Without the corresponding verifier fix, the second path is pruned and the
program is unexpectedly accepted. With the fix, the scalar range is
compared and the invalid store is rejected.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Link: https://lore.kernel.org/r/20260904104203.345917-7-memxor@gmail.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Resizable hash-map updates and deletions must not perform full special-field
destruction in their caller context. In particular, a referenced kptr must
remain attached to the allocation until the memory allocator destructor can
release it safely.
Add separate coverage for both affected paths. The update test stores a task
kptr, replaces the ordinary value bytes with BPF_EXIST, and verifies that the
kptr survived. The delete test removes an element and exchanges its kptr
through the still-valid map-value pointer before the allocation is reclaimed.
Both cases observe a NULL kptr when rhtab uses bpf_obj_free_fields(). They
recover and release the reference after rhtab switches to cancellation
semantics.
Signed-off-by: Nuoqi Gui <gnq25@mails.tsinghua.edu.cn>
[ kkd: Split update and delete coverage and rewrote the commit log ]
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Link: https://lore.kernel.org/r/20260904104203.345917-5-memxor@gmail.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Exercise the rhtab special-field lifecycle with the sequence from the
original report. A bpf_for_each_map_elem() callback deletes the sole
element, then initializes and arms a timer through the callback value
pointer while it remains valid.
Use a one-element map and pin userspace and BPF execution to one CPU.
Repeated delete-and-replace cycles drain the per-CPU allocator cache, and
periodic RCU synchronization makes the deleted units available for
recycling.
After each replacement, a second BPF program calls bpf_timer_cancel()
on its value. A successful cancellation proves both that a timer-bearing
unit was recycled and that insertion preserved the timer field. Without
the fix, insertion clears that field and cancellation keeps returning
-EINVAL. A long expiration keeps the timer callback out of the test, so
the regression is detected without accessing freed memory.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Link: https://lore.kernel.org/r/20260904104203.345917-3-memxor@gmail.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
The bpf_nf test allocs a ct, sets snat and dnat with random addr and
port via bpf_ct_set_nat_info(), then looks the ct up and checks the
reply tuple against what was set.
The port comes from bpf_get_prandom_u32() and can be 0. For
bpf_ct_set_nat_info(), port 0 means "port not specified", so only the
addr is mapped and the kernel keeps the original port. The check then
compares that port with 0 and fails, which shows up as a flaky
"Test for source natting" failure in CI [1][2].
Keep the random port in 1..65535 so it is always specified.
[1] https://github.com/kernel-patches/bpf/actions/runs/33830002889/job/100893868791
[2] https://github.com/kernel-patches/bpf/actions/runs/33829976794/job/100893220999
Fixes: b06b45e82b ("selftests/bpf: add tests for bpf_ct_set_nat_info kfunc")
Signed-off-by: Jiayuan Chen <jiayuan.chen@linux.dev>
Link: https://lore.kernel.org/r/20260904073745.363314-1-jiayuan.chen@linux.dev
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Add a sleepable verifier test that loads a refcount-only local kptr in an
explicit RCU read-side critical section, ends the section, and passes the
pointer to bpf_refcount_acquire().
The loaded pointer never carries NON_OWN_REF. After RCU unlock it retains
MEM_ALLOC while becoming PTR_UNTRUSTED, which previously made the kfunc
argument check accept it as a live allocated object. Expect verification to
reject the untrusted argument instead.
Signed-off-by: Ning Ding <dingning04@gmail.com>
[ kkd: Rewrote commit log ]
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Link: https://lore.kernel.org/r/20260904084325.52250-9-memxor@gmail.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Add a sleepable verifier test that loads a graph-node local kptr in an
explicit RCU read-side critical section, then passes its node to
bpf_rbtree_remove() after the section ends.
Before the verifier fix, the stale NON_OWN_REF flag makes the node look like
a live borrowed reference and the program is accepted. After the fix, the
pointer is demoted without NON_OWN_REF and the graph kfunc argument is
rejected.
Also exercise a graph kptr loaded while a spin lock provides implicit RCU
protection. The pointer must be invalidated when the lock is released, which
guards the required ordering between non-owning-reference invalidation and
RCU demotion.
Update the existing fault-protected load test state description. The
post-unlock pointer no longer carries NON_OWN_REF, but remains readable
because the load is rewritten to use BPF_PROBE_MEM.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Link: https://lore.kernel.org/r/20260904084325.52250-7-memxor@gmail.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Add verifier coverage for the distinction between owning and borrowed
arguments to bpf_refcount_acquire().
An owning pointer returned by bpf_obj_new() must continue producing a
non-NULL result without an extra check. An RCU-loaded local kptr is only
borrowed, so a checked result must load successfully while passing an
unchecked result to bpf_obj_drop() must be rejected as possibly NULL.
Use a sleepable syscall program for the borrowed cases so the explicit RCU
critical section is what permits the local kptr load. Without the verifier
fix, the unchecked case is incorrectly accepted. With it, the verifier
rejects the possibly NULL argument.
Signed-off-by: Ning Ding <dingning04@gmail.com>
[ kkd: Rewrote commit log ]
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Link: https://lore.kernel.org/r/20260904084325.52250-5-memxor@gmail.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Add verifier coverage for the two ways a non-percpu pointer can be stored
in a __percpu_kptr field: a program-BTF local allocation returned by
bpf_obj_new(), and a referenced kernel-BTF task_struct pointer.
Without the verifier fix, both programs are unexpectedly accepted and the
negative tests fail. Requiring MEM_PERCPU makes both programs fail
verification with the expected invalid-kptr diagnostic.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Link: https://lore.kernel.org/r/20260904084325.52250-3-memxor@gmail.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Check that a register-form NULL check does not lift PTR_MAYBE_NULL on
a path where the compared register is non-zero. W/o the previous patch
the program is accepted.
Reported-by: Nicholas Carlini <npc@anthropic.com>
Suggested-by: Nicholas Carlini <npc@anthropic.com>
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://lore.kernel.org/bpf/20260904083325.2083493-8-eddyz87@gmail.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Add jmp32_ptr_vs_zero_jne: the fall-through of the 32-bit compare, which
the verifier used to skip, contains an out of bounds map value access,
hence w/o the previous patch the program is accepted. See previous patch
for detailed description.
Reported-by: Nicholas Carlini <npc@anthropic.com>
Suggested-by: Nicholas Carlini <npc@anthropic.com>
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://lore.kernel.org/bpf/20260904083325.2083493-6-eddyz87@gmail.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
linked_regs_too_many_regs checks that collect_linked_regs() ties at most
LINKED_REGS_MAX registers for a single jump. Compare r5 instead of r0,
so that the register the jump compares is itself the member that does
not fit, and check that it comes out of the jump unlinked.
W/o the previous patch env->{false,true}_reg{1,2} bring r5's id back and
insn 7 is logged as "R5=scalar(id=1,...)".
Reported-by: Nicholas Carlini <npc@anthropic.com>
Suggested-by: Nicholas Carlini <npc@anthropic.com>
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://lore.kernel.org/bpf/20260904083325.2083493-4-eddyz87@gmail.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Check that a comparison against a pointer whose offset is not bounded
from above does not make the verifier infer that a nullable pointer is
not NULL, and that a bounded offset still does. W/o the previous patch
the first test is accepted.
Reported-by: Nicholas Carlini <npc@anthropic.com>
Suggested-by: Nicholas Carlini <npc@anthropic.com>
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://lore.kernel.org/bpf/20260904083325.2083493-2-eddyz87@gmail.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
The netfilter framework is allergic to ip header changing after
validation done by ip/ipv6 stack.
Assert that bpf netfilter programs do not allow skb write access.
Following additional tests are expected to be rejected by verifier:
1. alter skb->len.
2. alter skb->data.
3. prog calls bpf_dynptr_slice_rdwr.
4. alter location returned by dynptr API.
Add following test case for bpf runtime:
- alter skb data via bpf_dynptr_write()
Test checks via __retval() that bpf_dynptr_write() returned nonzero value.
Signed-off-by: Florian Westphal <fw@strlen.de>
Reviewed-by: Jiayuan Chen <jiayuan.chen@linux.dev>
Link: https://lore.kernel.org/r/20260903065845.22762-1-fw@strlen.de
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Add verifier coverage for the callback restriction on legacy packet
loads. Exercise BPF_LD_ABS directly in a bpf_loop callback and
BPF_LD_IND from a static subprogram called by the callback, ensuring that
callback context follows nested static calls.
Also exercise a callback which reaches BPF_LD_IND through a global
function and its static descendant. A sibling success case calls the same
global chain outside a callback, preserving support for ordinary global
packet loads. Existing success cases continue to cover loads from ordinary
static subprograms.
The failure cases expect the policy-specific rejection instead of reaching
the implicit-return path, triggering a verifier warning, or being accepted
through a function boundary.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Link: https://lore.kernel.org/r/20260903214758.2727663-9-memxor@gmail.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Add verifier coverage for the sleepable bpf_get_stack() and
bpf_get_task_stack() implementations. Call each helper while preemption is
disabled and require the verifier to reject it as sleepable.
Both programs load when the prototypes lack might_sleep, so the
expected-failure tests fail. Keep success controls outside the
non-preemptible region to ensure ordinary calls from sleepable uprobes
remain valid.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Link: https://lore.kernel.org/r/20260903214758.2727663-7-memxor@gmail.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Add an expected failure case which calls
bpf_btf_find_by_name_kind() from a BPF timer callback. Without the
helper prototype being marked sleepable, the verifier accepts the
program and the load unexpectedly succeeds.
Also add a positive control which calls the helper directly from a
syscall program. This verifies that marking the helper sleepable only
rejects it in non-sleepable regions.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Link: https://lore.kernel.org/r/20260903214758.2727663-5-memxor@gmail.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Add a verifier failure case where an rbtree comparator enters two nested
static subprograms and the innermost subprogram unlocks and relocks the
tree. Restoring the lock keeps the surrounding callback state balanced,
so the test specifically exercises whether the callback restriction follows
the nested calls.
Also add a load-only positive control whose comparator calls a harmless
static subprogram. This preserves the intended support for verified static
subprogram calls while holding the tree lock.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Link: https://lore.kernel.org/r/20260903214758.2727663-3-memxor@gmail.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Check bpf_fastcall pattern detection when the pattern is entered at an
instruction other than the first spill:
- a jump to the first spill allows the rewrite;
- conditional/unconditional a jump to the call or to the fill does not
allow the rewrite.
Reported-by: Nicholas Carlini <npc@anthropic.com>
Suggested-by: Nicholas Carlini <npc@anthropic.com>
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://lore.kernel.org/r/20260903205820.1743087-2-eddyz87@gmail.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
