Test case demonstrating a bug in cnum comparison logic fixed by
previous commit. A pruning point is reached with r6 in two states:
1. 32-bit range of [0x7FFFFFF0, U32_MAX] ∪ [0, 0x10]
2. 32-bit range of [0x100, 0x200]
At pruning point the buggy is_state_visited() logic would assume that
would assume range (2) to be a subset of (1) and fail to explore the
path performing division by zero.
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://lore.kernel.org/r/20260425-cnum-range-within-v1-2-2fdca70cb09d@gmail.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
states.c:range_within() must be updated to properly check if
cnum-based range in an old state is a superset of a range in the cur
state. Currently it makes the decision using min/max accessors:
reg_umin(old) <= reg_umin(cur) <= reg_umax(old)
This is wrong for cnums that cross both UT_MAX/0 and ST_MAX/ST_MIN
boundaries. Consider cnum32{base=0x7FFFFFF0, size=0x80000020},
which represents values [0x7FFFFFF0, ..., U32_MAX, 0, ..., 0x10].
Its projections are u32_min/max=0/U32_MAX, s32_min/max=S32_MIN/MAX.
A register with range [0x100, 0x200] (which lies entirely in the gap
of the wrapping range) would pass the min/max check despite having no
overlap with the actual cnum arc.
This commit replaces min/max comparison with cnum{32,64}_is_subset()
operation. The operation implementation is verified using cbmc model
checker in [1].
[1] https://github.com/eddyz87/cnum-verif/
Fixes: bbc6310855 ("bpf: replace min/max fields with struct cnum{32,64}")
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://lore.kernel.org/r/20260425-cnum-range-within-v1-1-2fdca70cb09d@gmail.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Emil Tsalapatis says:
====================
Introduce arena library and runtime
Add a new subdirectory to tools/testing/selftests/bpf called libarena,
along with programs useful for writing arena-based BPF code. This
patchset adds the following:
1) libarena, a subdirectory where arena BPF code that is generally useful
to BPF arena programs can be easily added and tested.
2) An ASAN runtime for BPF arena programs. BPF arenas allow for accessing
memory after it has been freed or if it is out of bounds, making it more
difficult to triage bugs combined to regular BPF. Use LLVM's recently added
support for address-space based sanitization to selectively sanitize just
the arena accesses.
3) A buddy memory allocator that can be reused by BPF programs to handle
memory allocation/deletion. The allocator uses the ASAN runtime to add
address sanitization if requested.
The patch includes testing for the new allocators and ASAN features that
can be built from the top directory using "make libarena_test" and
"make libarena_test_asan". The generated binaries reside in libarena/.
The patch also adds test-progs-based selftests to the codebase for the
libarena code, so the new tests are run by ./test_progs.
The patchset has the following stucture:
1-3: Create basic libarena scaffolding and refactor existing headers.
4-5: Add the ASAN runtime and associated scaffolding.
6-8: Add the new buddy memory allocator along with selftests.
Signed-off-by: Emil Tsalapatis <emil@etsalapatis.com>
HISTORY
=======
v8->v9 (https://lore.kernel.org/bpf/20260421165037.4736-1-emil@etsalapatis.com)
- Added Matt's Acked-by for Patch 1
- Replaced open coded fls with __builtin based calculation (Matt)
- Add comment explaining the reasoning behind the zero variable (Matt)
- Remove the self-contained runner and present selftests as examples (Kumar).
The reasoning is that including selftests with the library is not
effective because the library will be sync'ed outside the tree
infrequently, so it would be ineffective in catching issues before
they land.
- Adjust syscall API for clarity and forward compatibility (Matt)
- Reset asan_validate state after failed check (Sashiko)
- Fix printf format specifier (Sashiko)
- Rename syscall to be generic arena_info (Matt)
- Namespace libarena headers by moving them to their own directory
- Rename selftest_helpers to libarena/userspace.h to reflect it is not
just for selftests.
v7->v8 (https://lore.kernel.org/bpf/20260412174546.18684-1-emil@etsalapatis.com)
- Duplicate READ_ONCE/WRITE_ONCE instead of moving it to
bpf_experimental.h to keep libarena self-contained (Kumar)
- Add libarena_asan test to test_progs and conditionally compile it if
suppported (Kumar)
- Add stderr parsing for buddy tests when run under test_progs (Kumar)
- Move all arena-related headers into libarena and add its include/
subdirectory in the standard include path (Kumar)
- Remove silent-by-default ASAN, add help message on test_libarena
explaining that -v emits the messages (Kumar)
- Add run_prog_args as a libarena helper
- Add explanation on the use of __weak for the spinlock qnodes
v6->v7 (https://lore.kernel.org/bpf/20260412011857.3387-1-emil@etsalapatis.com)
- Modify patch 1 to allow operations between PTR_TO_ARENA src_reg
and dst_reg of any type. Adjust selftests accordingly (Alexei)
- Remove unnecessary include in patch 5 (Song)
- Removed unused definitions/assignments in patches 8/9, update patch
descriptions
v5->v6 (https://lore.kernel.org/bpf/20260410163041.8063-1-emil@etsalapatis.com)
- Fix subreg_def management for SCALAR += PTR_TO_ARENA operations (AI)
- Add more selftests for the SCALAR += PTR_TO_ARENA patch (Sashiko)
- Adjust fls() operation to be in line with the kernel version (Sashiko)
- Address Sashiko selftests and debugging nits
- Add ASAN loadN and storeN _noabort variants and associated BTF anchor
- Remove unnecessary bit freeing of buddies during block splitting
v4->v5 (https://lore.kernel.org/bpf/20260407045730.13359-1-emil@etsalapatis.com)
Omitting various nits and fixups.
- Properly adjust subreg_def for scalar += ptr_to_arena calls (Sashiko)
- Remove extraneous definition from prog_tests/arena_spin_lock.c (Song)
- Trim extraneous comments from ASAN and buddy (Alexei)
- Remove asan_dummy call and replace with function pointer array (Alexei)
- Remove usersapi.h header and merge it into common.h (Alexei)
- Replace ASAN macros with function calls (Alexei)
- Embed buddy lock into the struct and move the buddy allocator to __arena_global
(Alexei)
- Add commenting for buddy allocator constants (Alexei)
- Add default buddy allocator directly in common.bpf.c, so that the user does
not need to define it.
- Expand test harnesses to dynamically find individual selftests. Now the
selftests also reports each test individually (e.g., 5 entries for the
buddy allocator instead of 1). This brings them to par with the rest of
the test_progs.
v3->v4 (https://lore.kernel.org/bpf/20260403042720.18862-1-emil@etsalapatis.com)
- Add Acks by Song to patches 1-4.
- Expand the verifier's handling of scalar/arena operations to
include all 3-operand operations in Patch 1 (Alexei)
- Add additional tests for arena/arena (allowed) and arena/pointer (not allowed)
operations in Patch 2
- Remove ASAN version of the library from default compilation since it requires
LLVM 22 and up (CI)
- Rework buddy allocator locking for clarity and add comments
- Fix From: email to be consistent with SOB
- Address (most) Sashiko comments
v2->v3 (https://lore.kernel.org/bpf/20260127181610.86376-1-emil@etsalapatis.com)
Nonexhaustive due to significant patch rework.
- Do not duplicate WRITE_ONCE macro (Mykyta, Kumar)
- Add SPDX headers (Alexei)
- Remove bump/stack allocators (Alexei)
- Integrate testing with test_progs (Kumar)
- Add short description of ASAN algorithm at the top of the file (Alexei)
v1->v2 (https://lore.kernel.org/bpf/20260122160131.2238331-1-etsal@meta.com/)
- Added missing format string argument (AI)
- Fix outdated selftests prog name check (AI)
- Fixed stack allocation check for segment creation (AI)
- Fix errors in non-ASAN bump allocator selftests (AI)
- Propagate error value from individual selftests in selftest.c
- Removed embedded metadata from bump allocator as it was needlessly
complicating its behavior
====================
Link: https://patch.msgid.link/20260426190338.4615-1-emil@etsalapatis.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Add code to directly test the output of libarena ASAN tests.
The code reuses testing infrastructure originally for BPF streams
to verify that ASAN emits call stacks when the selftests trigger
a memory error.
Since stderr() testing uses logic from test_progs, it is only
available on the test_progs-based selftest runner. The standalone
runner still uses internal ASAN state to verify access errors are
triaged as expected.
Signed-off-by: Emil Tsalapatis <emil@etsalapatis.com>
Link: https://lore.kernel.org/r/20260426190338.4615-9-emil@etsalapatis.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Introduce selftests for the buddy allocator with and without
ASAN. Add the libarena selftests both to the libarena test
runner and to test_progs, so that they are a) available when
libarena is pulled as a standalone library, and b) exercised
along with all other test programs in this directory.
ASAN for libarena requires LLVM 22. Add logic in the top-level
selftests Makefile to only compile the ASAN variant if the
compiler supports it, otherwise skip the test.
Signed-off-by: Emil Tsalapatis <emil@etsalapatis.com>
Link: https://lore.kernel.org/r/20260426190338.4615-8-emil@etsalapatis.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Add a byte-oriented buddy allocator for libarena. The buddy
allocator provides an alloc/free interface for small arena allocations
ranging from 16 bytes to 512 KiB. Lower allocations values are rounded
up to 16 bytes. The buddy allocator does not handle larger allocations
that can instead use the existing bpf_arena_{alloc, free}_pages() kfunc.
Signed-off-by: Emil Tsalapatis <emil@etsalapatis.com>
Link: https://lore.kernel.org/r/20260426190338.4615-7-emil@etsalapatis.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Expand the arena library selftest infrastructure to support
address sanitization. Add the compiler flags necessary to
compile the library under ASAN when supported.
Signed-off-by: Emil Tsalapatis <emil@etsalapatis.com>
Link: https://lore.kernel.org/r/20260426190338.4615-6-emil@etsalapatis.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Add an address sanitizer (ASAN) runtime to the arena library. The
ASAN runtime implements the functions injected into BPF binaries
by LLVM sanitization when ASAN is enabled during compilation.
The runtime also includes functions called explicitly by memory
allocation code to mark memory as poisoned/unpoisoned to ASAN.
This code is a no-op when sanitization is turned off.
Signed-off-by: Emil Tsalapatis <emil@etsalapatis.com>
Link: https://lore.kernel.org/r/20260426190338.4615-5-emil@etsalapatis.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
The BPF selftest headers include functionality that is
specific to arenas and is required by libarena. Keep libarena
self-contained by moving all functionality into its include/
directory. Also add libarena/include to the standard include
paths for the selftests to make the moved headers easy to
access by existing selftests.
Some functionality is required by libarena but not strictly
arena-related. We still move it to the libarena/include path,
which is an upgrade from directly accessing them from the
selftests/bpf directory using relative paths.
A new bpf_may_goto.h file is split off of bpf_experimental.h.
bpf_arena_spin_lock.h and bpf_arena_common.h are moved to
libarena/include. bpf_atomic.h is also moved to libarena
because it is necessary for arena spinlocks.
For bpf_arena_spin_lock.h, mark the spinlock state array as __weak
to define the spinlock state array in the header while also
being compatible with multi-compilation unit programs. While
we're at it, we remove unnecessary definitions from existing
test programs.
Signed-off-by: Emil Tsalapatis <emil@etsalapatis.com>
Link: https://lore.kernel.org/r/20260426190338.4615-4-emil@etsalapatis.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Add initial code and a Makefile for an arena-based BPF library. Modules
can be added just by including the source file in the library's src/
subdirectory. Future commits will introduce the library code itself.
The code includes workarounds that are removed in subsequent patches
that ensure bisectability.
Signed-off-by: Emil Tsalapatis <emil@etsalapatis.com>
Link: https://lore.kernel.org/r/20260426190338.4615-3-emil@etsalapatis.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
The WRITE_ONCE macro is identically defined both in bpf_atomic.h
and in bpf_arena_common.h. However, the bpf_atomic.h definition has no
ifdef guard. If bpf_atomic.h is included after bpf_arena.common.h,
compilation fails because of the duplicate definition.
Guard the definiton in bpf_atomic.h with and ifdef to let programs
include the two headers in any order. Duplicating the definition is
the simplest solution out of all the alternatives:
- Keeping one of the two existing definitions is not possible because
both BPF atomics and arena programs need the macro, and the two features
are independent. Using one should not require the header for the other.
- Factoring out the definition into a new header that only includes it
is more churn than just duplicating it.
- Factoring out the definition into bpf_experimental.h requires all
users of WRITE_ONCE to include the header. However, the arena library
introduced in subsequent commits must be self-contained, while
bpf_experimental.h is in the base selftests/bpf directory.
Both headers are moved to the arena library in a subsequent patch.
Signed-off-by: Emil Tsalapatis <emil@etsalapatis.com>
Reviewed-by: Matt Bobrowski <mattbobrowski@google.com>
Link: https://lore.kernel.org/r/20260426190338.4615-2-emil@etsalapatis.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Gregory Bell says:
====================
selftests/bpf: Use local types for kfunc declarations
The xdp_flowtable and test_tunnel_kern selftests were previously
rewritten to compile with CONFIG_NF_FLOW_TABLE=m and CONFIG_NET_FOU=m.
While the compilation issues were resolved, the tests fail at
runtime. In test_tunnel_kern.c, struct bpf_fou_encap___local is
defined with the correct fields but the kfunc declarations still
reference the forward-declared struct bpf_fou_encap. Similarly,
xdp_flowtable.c uses a forward-declared struct flow_offload_tuple_rhash
as the return type for bpf_xdp_flow_lookup(). Clang emits these forward
declarations as BTF FWD types, which fail to resolve against the
module-defined STRUCT types at run time:
libbpf: extern (func ksym) 'bpf_xdp_flow_lookup': func_proto [51] incompatible with nf_flow_table [128640]
libbpf: extern (func ksym) 'bpf_skb_get_fou_encap': func_proto [79] incompatible with fou [135045]
This patch updates both selftests to use ___local type suffixes
for kernel struct type, replacing the forward declarations.
struct flow_offload_tuple_rhash___local is defined without fields
because the test only uses the returned pointer for a null check.
This avoids needing to locally define its nested types,
struct rhash_head and struct flow_offload_tuple.
I understand that fixing selftests for specific config options is
generally not a priority, but since these tests were already
rewritten to support these configs, they should work correctly
with them.
Fixes: d17f9b370d ("selftests/bpf: Fix compilation failure when CONFIG_NET_FOU!=y")
Fixes: eeb23b54e4 ("selftests/bpf: fix compilation failure when CONFIG_NF_FLOW_TABLE=m")
Signed-off-by: Gregory Bell <grbell@redhat.com>
change log:
[0] https://lore.kernel.org/all/cover.1776280396.git.grbell@redhat.com/
- Add BPF_NO_KFUNC_PROTOTYPES macro to test_tunnel_kern.c so the test compiles
when CONFIG_NET_FOU=y. Without it the function prototypes conflict with vmlinux.h
====================
Link: https://patch.msgid.link/20260417154122.2558890-1-grbell@redhat.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Replace the forward-declared struct bpf_fou_encap with the existing
bpf_fou_encap___local type in the bpf_skb_set_fou_encap and
bpf_skb_get_fou_encap declarations. This removes the need for
the forward declaration and the explicit casts at each call.
Fixes: d17f9b370d ("selftests/bpf: Fix compilation failure when CONFIG_NET_FOU!=y")
Signed-off-by: Gregory Bell <grbell@redhat.com>
Link: https://lore.kernel.org/r/20260417154122.2558890-3-grbell@redhat.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Define flow_offload_tuple_rhash___local and use it in place of the
forward-declared kernel type for the bpf_xdp_flow_lookup kfunc return
type and tuplehash variable. This is consistent with how
bpf_flowtable_opts___local is already handled in the same file and
avoids relying on a forward declaration of the struct.
Fixes: eeb23b54e4 ("selftests/bpf: fix compilation failure when CONFIG_NF_FLOW_TABLE=m")
Signed-off-by: Gregory Bell <grbell@redhat.com>
Link: https://lore.kernel.org/r/20260417154122.2558890-2-grbell@redhat.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Eduard Zingerman says:
====================
bpf: replace min/max fields with struct cnum{32,64}
This RFC replaces s64, u64, s32, u32 scalar range domains tracked by
verifier by a pair of circular numbers (cnums): one for 64-bit domain
and another for 32-bit domain. Each cnum represents a range as a
single arc on the circular number line, from which signed and unsigned
bounds are derived on demand. See also wrapped intervals
representation as in [1].
The use of such representation simplifies arithmetic and conditions
handling in verifier.c and allows to express 32 <-> 64 bit deductions
in a more mathematically rigorous way.
[1] https://jorgenavas.github.io/papers/ACM-TOPLAS-wrapped.pdf
Changelog
=========
v2 -> v1:
- Fixes in source code comments highlighted by bot+bpf-ci.
RFCv1 -> v2:
- Dropped RFC tag.
- Dropped cnum{32,64}_mul(), too much complexity and no veristat
or selftests gains.
- cnum32_from_cnum64() normalizes to CNUM32_EMPTY when input
cnum64.size >= U32_MAX, previously only checked for > U32_MAX
(bot+bpf-ci).
- cnum32_from_cnum64() and cnum64_cnum32_intersect() now check for
empty inputs (sashiko).
- In regs_refine_cond_op() case BPF_JSLT use cnum{32,64}_from_srange
constructors instead of unsigned variants (sashiko).
- cnum{32,64}_intersect_with{,_urange,_srange}() helpers added
(Alexei)
[RFCv1] https://lore.kernel.org/bpf/0c47b0b7ea476647746806c46fded4353be885f7.camel@gmail.com/T/
[v2] https://lore.kernel.org/bpf/c4fb60bafd526ae2d92b86e2250255aef0ba5ee1.camel@gmail.com/T/
Patch set layout
================
Patch #1 introduces the cnum (circular number) data type and basic
operations: intersection, addition, multiplication, negation, 32-to-64
and 64-to-32 projections.
CBMC based proofs for these operations are available at [2].
(The proofs lag behind the current patch set and need an update if
this RFC moves further. Although, I don't expect any significant
changes).
[2] https://github.com/eddyz87/cnum-verif
Patch #2 mechanically converts all direct accesses to bpf_reg_state's
min/max fields to use accessor functions, preparing for the field
replacement.
Patch #3 replaces the eight independent min/max fields in
bpf_reg_state with two cnum fields. The signed and unsigned bounds are
derived on demand from the cnum via the accessor functions.
This eliminates the separate signed<->unsigned cross-deduction logic,
simplifies reg_bounds_sync(), regs_refine_cond_op() and some
arithmetic operations.
Patch #4 adds selftest cases for improved 32->64 range refinements
enabled by cnum64_cnum32_intersect().
Precision trade-offs
====================
A cnum represents a contiguous arc on the circular number line.
Current master tracks four independent ranges (s64, u64, s32, u32)
whose intersection could implicitly represent value sets that are
two disjoint arcs on the circle - something a single cnum cannot.
Cnums lose precision when a cross-domain conditional comparison
(e.g., unsigned comparison on a register with a signed-derived range)
produces an intersection that would be two disjoint arcs.
The cnum must over-approximate by returning one of the input arcs.
E.g. a pair of ranges u64:[1,U64_MAX] s:[-1,1] represents a set of two
values: {-1,1}, this can only be approximated as a set of three values
by cnum: {-1,0,1}.
Cnums gain precision in arithmetic: when addition, subtraction or
multiplication causes register value to cross both signed and unsigned
min/max boundaries. Current master resets the register as unbound in
such cases, while cnums preserve the exact wrapping arc.
In practice precision loss turned out to matter only for a set of
specifically crafted selftests from reg_bounds.c (see patch #3 for
more details). On the other hand, precision gains are observable for a
set real-life programs.
Verification performance measurements
=====================================
Tested on 6683 programs across four corpora: Cilium (134 programs),
selftests (4635), sched_ext (376), and Meta internal BPF corpus
(1540). Program verification verdicts are identical across all four
program sets - no program changes between success and failure.
Instruction count comparison (cnum vs signed/unsigned pair):
- 83 programs improve, 44 regress, 6596 unchanged
- Total saved: ~290K insns, total added: 10K insns
- Largest improvements:
-26% insns in a internal network firewall program;
-47% insns in rusty/wd40 sched_ext schedulers.
- Largest regression: +24% insns in one Cilium program (5062 insns
added).
Raw performance data is at the bottom of the email.
Raw verification performance metrics
====================================
Filtered out by -f insns_pct>5 -f !insns<10000
========= selftests: master vs cnums-everywhere-v2 =========
File Program Insns (A) Insns (B) Insns (DIFF)
---- ------- --------- --------- ------------
Total progs: 4665
total_insns diff min: -15.71%
total_insns diff max: 45.45%
-20 .. -10 %: 3
-5 .. 0 %: 6
0 .. 5 %: 4654
45 .. 50 %: 2
========= scx: master vs cnums-everywhere-v2 =========
File Program Insns (A) Insns (B) Insns (DIFF)
--------------- -------------- --------- --------- ----------------
scx_rusty.bpf.o rusty_enqueue 39842 22053 -17789 (-44.65%)
scx_rusty.bpf.o rusty_stopping 37738 19949 -17789 (-47.14%)
scx_wd40.bpf.o wd40_stopping 37729 19880 -17849 (-47.31%)
Total progs: 376
total_insns diff min: -47.31%
total_insns diff max: 19.61%
-50 .. -40 %: 3
-5 .. 0 %: 5
0 .. 5 %: 366
5 .. 15 %: 1
15 .. 20 %: 1
========= meta: master vs cnums-everywhere-v2 =========
File Program Insns (A) Insns (B) Insns (DIFF)
--------------------------------- ---------- --------- --------- ----------------
<meta-internal-firewall-program> ...egress 222327 164648 -57679 (-25.94%)
<meta-internal-firewall-program> ..._tc_eg 222839 164772 -58067 (-26.06%)
<meta-internal-firewall-program> ...egress 222327 164648 -57679 (-25.94%)
<meta-internal-firewall-program> ..._tc_eg 222839 164772 -58067 (-26.06%)
Total progs: 1540
total_insns diff min: -26.06%
total_insns diff max: 6.69%
-30 .. -20 %: 4
-15 .. -5 %: 6
-5 .. 0 %: 36
0 .. 5 %: 1493
5 .. 10 %: 1
========= cilium: master vs cnums-everywhere-v2 =========
File Program Insns (A) Insns (B) Insns (DIFF)
---------- ------------------------------- --------- --------- ---------------
bpf_host.o tail_handle_ipv4_cont_from_host 20962 26024 +5062 (+24.15%)
bpf_host.o tail_handle_ipv6_cont_from_host 17036 18672 +1636 (+9.60%)
Total progs: 134
total_insns diff min: -3.32%
total_insns diff max: 24.15%
-5 .. 0 %: 12
0 .. 5 %: 120
5 .. 15 %: 1
20 .. 25 %: 1
---
====================
Link: https://patch.msgid.link/20260424-cnums-everywhere-rfc-v1-v3-0-ca434b39a486@gmail.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
1. 32-bit range starts before 64-bit range's low bits in each block,
causing intersection to skip entire blocks.
2. 32-bit range crosses the U32_MAX/0 boundary, represented as s32
range crossing sign boundary.
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://lore.kernel.org/r/20260424-cnums-everywhere-rfc-v1-v3-4-ca434b39a486@gmail.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Replace eight independent s64, u64, s32, u32 min/max fields in
bpf_reg_state with two circular number fields:
- cnum64 for a unified signed/unsigned 64-bit range tracking;
- cnum32 for a unified signed/unsigned 32-bit range tracking.
Each cnum represents a range as a single arc on the circular number
line (base + size), from which signed and unsigned bounds are derived
on demand via accessor functions introduced in the preceding commit.
Notable changes:
- Signed<->unsigned deductions in __reg_deduce_bounds() are removed.
- 64<->32 bit deductions are replaced with:
- reg->r32 = cnum32_intersect(reg->r32, cnum32_from_cnum64(reg->r64));
this is functionally equivalent to the old code.
- reg->r64 = cnum64_cnum32_intersect(reg->r64, reg->r32);
this handles a few additional cases, see commit message for
"bpf: representation and basic operations on circular numbers".
- regs_refine_cond_op() now computes results in terms of operations on
sets, e.g. for JNE:
/* Complement of the range [val, val] as cnum64. */
lo = (struct cnum64){ val + 1, U64_MAX - 1 };
reg1->r64 = cnum64_intersect(reg1->r64, lo);
- For add, sub operations on scalars replace explicit bounds
computations with cnum{32,64}_{add,negate}.
- For add, sub operations on pointers deduplicate with arithmetic
operations on scalars and use cnum{32,64}_{add,negate}.
- For and, or, xor operations on scalars remove explicit signed bounds
computations.
- range_bounds_violation() reduces to checking cnum_is_empty().
- const_tnum_range_mismatch() reduces to checking cnum_is_const().
Selftest adjustments: a few existing tests are updated because a
single cnum arc cannot always represent what the old system expressed
as the intersection of independent signed and unsigned ranges.
For example, if the old system tracked u64=[0, U64_MAX-U32_MAX+2] and
s64=[S64_MIN+2, 2] independently, their intersection is a tight
two-point set. A single cnum must pick the shorter arc, losing the
other constraint. These cases are documented with comments in the
adjusted tests.
reg_bounds.c is updated with logic similar to
cnum64_cnum32_intersect(). Instead of using cnums it inspects
intersection between 'b' and first / last / next-after-first /
previous-before-last sub-ranges of 'a'.
reg_bounds.c is also updated to skip test cases that rely
in signed and unsigned ranges intersecting in two intervals,
as such cases are not representable by a single cnum.
The following "crafted" test cases are affected:
- reg_bounds_crafted/(s64)[0xffffffffffff8000; 0x7fff] (u32)<op> [0; 0x1f]
- reg_bounds_crafted/(s64)[0; 0x1f] (u32)<op> [0xffffffffffffff80; 0x7f]
- reg_bounds_crafted/(s64)[0xffffffffffffff80; 0x7f] (u32)<op> [0; 0x1f]
- reg_bounds_crafted/(u64)[0; 1] (s32)<op> [1; 2147483648]
- reg_bounds_crafted/(u64)[1; 2147483648] (s32)<op> [0; 1]
- reg_bounds_crafted/(u64)[0; 0xffffffff00000000] (s64)<op> 0
- reg_bounds_crafted/(u64)0 (s64)<op> [0; 0xffffffff00000000]
- reg_bounds_crafted/(u64)[0; 0xffffffff00000000] (s32)<op> 0
- reg_bounds_crafted/(u64)0 (s32)<op> [0; 0xffffffff00000000]
- reg_bounds_crafted/(s64)[S64_MIN; 0] (u64)<op> S64_MIN
- reg_bounds_crafted/(s64)S64_MIN (u64)<op> [S64_MIN; 0]
- reg_bounds_crafted/(s32)[S32_MIN; 0] (u32)<op> S32_MIN
- reg_bounds_crafted/(s32)S32_MIN (u32)<op> [S32_MIN; 0]
- reg_bounds_crafted/(s64)[0; 0x1f] (u32)<op> [0xffffffff80000000; 0x7fffffff]
- reg_bounds_crafted/(s64)[0xffffffff80000000; 0x7fffffff] (u32)<op> [0; 0x1f]
- reg_bounds_crafted/(s64)[0; 0x1f] (u32)<op> [0xffffffffffff8000; 0x7fff]
As well as some reg_bounds_roand_{consts,ranges}_A_B, where A and B
differ in sign domain.
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://lore.kernel.org/r/20260424-cnums-everywhere-rfc-v1-v3-3-ca434b39a486@gmail.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Replace direct access to bpf_reg_state->{smin,smax,umin,umax,
s32_min,s32_max,u32_min,u32_max}_value with getter/setter inline
functions, preparing for future switch to cnum-based internal
representation.
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://lore.kernel.org/r/20260424-cnums-everywhere-rfc-v1-v3-2-ca434b39a486@gmail.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
This commit adds basic definitions for cnum32/cnum64.
This is a unified numeric range representation for signed and unsigned
domains. Inspired by an old post from Shung-Hsi Yu [1] and paper [2].
Operations correctness is verified using cbmc model checker,
tests source code can be found in a separate repo [3].
The cnum64_cnum32_intersect() function is notable, because it handled
several cases verifier.c:deduce_bounds_64_from_32() does not.
Given:
- a is a 64-bit range
- b is a 32-bit range
- t is a refined 64-bit range, such that ∀ v ∈ a, (u32)v ∈ b: v ∈ t.
cnum64_cnum32_intersect() makes the following deductions:
(A): 'b' is a sub-range of the first or the last 32-bit
sub-range of 'a':
64-bit number axis --->
N*2^32 (N+1)*2^32 (N+2)*2^32 (N+3)*2^32
||------|---|=====|-------||----------|=====|-------||----------|=====|----|--||
| |< b >| |< b >| |< b >| |
| | | |
|<--+--------------------------- a ---------------------------+--->|
| |
|<-------------------------- t -------------------------->|
(B) 'b' does not intersect with the first of the last 32-bit
sub-range of 'a':
N*2^32 (N+1)*2^32 (N+2)*2^32 (N+3)*2^32
||--|=====|----|----------||--|=====|---------------||--|=====|------------|--||
|< b >| | |< b >| |< b >| |
| | | |
|<-------------+--------- a -------------------|----------->|
| |
|<-------- t ------------------>|
(C) 'b' crosses 0/U32_MAX boundary:
N*2^32 (N+1)*2^32 (N+2)*2^32 (N+3)*2^32
||===|---------|------|===||===|----------------|===||===|---------|------|===||
|b >| | |< b||b >| |< b||b >| | |< b|
| | | |
|<-----+----------------- a --------------+-------->|
| |
|<---------------- t ------------->|
Current implementation of deduce_bounds_64_from_32() only handles
case (A).
[1] https://lore.kernel.org/all/ZTZxoDJJbX9mrQ9w@u94a/
[2] https://jorgenavas.github.io/papers/ACM-TOPLAS-wrapped.pdf
[3] https://github.com/eddyz87/cnum-verif/tree/master
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://lore.kernel.org/r/20260424-cnums-everywhere-rfc-v1-v3-1-ca434b39a486@gmail.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Alexis Lothoré says:
====================
selftests/bpf: drop xdping tool
this is the v2 of the small series dropping xdping tool. This removal is
part of the larger effort aiming to tidy the bpf selftests directory.
This new revision updates btf_dump test to make it use another bpf
program rather than xdping_kern so that we can drop it as well.
====================
Link: https://patch.msgid.link/20260422-xdping-v2-0-c0f8ccedcf91@bootlin.com
Signed-off-by: Martin KaFai Lau <martin.lau@kernel.org>
As part of a larger cleanup effort in the bpf selftests directory,
tests and scripts are either being converted to the test_progs framework
(so they are executed automatically in bpf CI), or removed if not
relevant for such integration.
The test_xdping.sh script (with the associated xdping.c) acts as a RTT
measurement tool, by attaching two small xdp programs to two interfaces.
Converting this test to test_progs may not make much sense:
- RTT measurement does not really fit in the scope of a functional test,
this is rather about measuring some performance level.
- there are other existing tests in test_progs that actively validate
XDP features like program attachment, return value processing, packet
modification, etc
Drop test_xdping.sh, the corresponding xdping.c userspace part, the
xdping_kern.c program, and the shared header, xdping.h
Signed-off-by: Alexis Lothoré (eBPF Foundation) <alexis.lothore@bootlin.com>
Signed-off-by: Martin KaFai Lau <martin.lau@kernel.org>
Reviewed-by: Alan Maguire <alan.maguire@oracle.com>
Acked-by: Paul Chaignon <paul.chaignon@gmail.com>
Link: https://patch.msgid.link/20260422-xdping-v2-2-c0f8ccedcf91@bootlin.com
In order to prepare xdping tool removal from the BPF selftests
directory, make the btf_dump test use another BPF program for the btf
datasec dump test. Use xdp_dummy.bpf.o, as it is already used by various
other tests.
Signed-off-by: Alexis Lothoré (eBPF Foundation) <alexis.lothore@bootlin.com>
Signed-off-by: Martin KaFai Lau <martin.lau@kernel.org>
Acked-by: Paul Chaignon <paul.chaignon@gmail.com>
Link: https://patch.msgid.link/20260422-xdping-v2-1-c0f8ccedcf91@bootlin.com
Yonghong Song says:
====================
bpf: Prepare to support stack arguments
The patch set prepares to support stack arguments for bpf functions
and kfuncs. The major changes include:
- Avoid redundant calculation of bpf_reg_state. For stack
arguments, there exists no corresponding register number.
- Refactor check_kfunc_mem_size_reg() to have bpf_reg_state's
for both mem_reg and size_reg.
- Allow verifier logs to print stack arguments if there is no
corresponding register.
Please see individual patches for details.
Changelogs:
v2 -> v3:
- v2: https://lore.kernel.org/bpf/20260422054149.3124342-1-yonghong.song@linux.dev/
- Fix a mark_chain_precision issue by rewriting helper reg_from_argno().
v1 -> v2:
- v1: https://lore.kernel.org/bpf/20260421171927.3507554-1-yonghong.song@linux.dev/
- Major change to patch 7. In v1, the argno has type u32 to
represent registers and arguments. This works but error prone
as u32 is too easy to mess and leak. This version uses a struct
type to represent argno which makes things more explicit and easy
to reason.
====================
Link: https://patch.msgid.link/20260423033425.2536883-1-yonghong.song@linux.dev
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Currently, MAX_BPF_FUNC_ARGS is used for tracepoint related progs where
the number of parameters cannot exceed MAX_BPF_FUNC_ARGS.
Here, MAX_BPF_FUNC_ARGS is reused to set a limit of the number of arguments
for bpf functions and kfuncs. The current value for MAX_BPF_FUNC_ARGS
is 12 which should be sufficient for majority of bpf functions and
kfuncs.
Acked-by: Puranjay Mohan <puranjay@kernel.org>
Acked-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
Link: https://lore.kernel.org/r/20260423033511.2542870-1-yonghong.song@linux.dev
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Introduce BPF_REG_PARAMS as a dedicated BPF register for stack
argument accesses. It occupies the BPF register number 11 (R11),
which is used as the base pointer for the stack argument area,
keeping it separate from the R10-based (BPF_REG_FP) program stack.
The kernel-internal hidden register BPF_REG_AX previously occupied
slot 11 (MAX_BPF_REG). With BPF_REG_PARAMS taking that slot,
BPF_REG_AX moves to slot 12 and MAX_BPF_EXT_REG increases
accordingly.
Acked-by: Puranjay Mohan <puranjay@kernel.org>
Acked-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
Link: https://lore.kernel.org/r/20260423033506.2542005-1-yonghong.song@linux.dev
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
This change prepares verifier log reporting for upcoming kfunc stack
argument support.
Currently verifier log code mostly assumes that an argument can be
described directly by a register number. That works for arguments
passed in `R1` to `R5`, but it does not work once kfunc arguments
can also be passed on the stack.
Introduce an opaque `argno_t` type that encodes both register-based
and arg-based references. Four helpers form the interface:
- argno_from_reg(regno): create from a register number
- argno_from_arg(arg): create from a 1-based arg number
- reg_from_argno(a): extract register number, or -1
- arg_from_argno(a): extract arg number, or -1
reg_arg_name() converts an argno_t to a human-readable string for
verifier logs: "R%d" for register arguments, or "*(R11-off)" for
stack arguments beyond R5.
Update selftests accordingly.
Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
Link: https://lore.kernel.org/r/20260423033501.2539667-1-yonghong.song@linux.dev
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
To support stack arguments, in later patches, argno will represent
both registers and stack arguments. To avoid confusion, rename
existing argno to arg.
Acked-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
Link: https://lore.kernel.org/r/20260423033456.2539340-1-yonghong.song@linux.dev
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Similar to the previous patch, try to pass bpf_reg_state from caller
to callee. Both mem_reg and size_reg are passed to helper functions.
This is important for stack arguments as they may be beyond registers 1-5.
Acked-by: Puranjay Mohan <puranjay@kernel.org>
Acked-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
Link: https://lore.kernel.org/r/20260423033451.2539065-1-yonghong.song@linux.dev
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
In many cases, once a bpf_reg_state is defined, it can pass to
callee's. Otherwise, callee will need to get bpf_reg_state again
based on regno. More importantly, this is needed for later stack
arguments for kfuncs since the register state for stack arguments does
not have a corresponding regno. So it makes sense to pass reg state
for callee's.
The following is the only change to avoid compilation warning:
static int sanitize_check_bounds(struct bpf_verifier_env *env,
const struct bpf_insn *insn,
- const struct bpf_reg_state *dst_reg)
+ struct bpf_reg_state *dst_reg)
Acked-by: Puranjay Mohan <puranjay@kernel.org>
Acked-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Reviewed-by: Amery Hung <ameryhung@gmail.com>
Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
Link: https://lore.kernel.org/r/20260423033446.2538321-1-yonghong.song@linux.dev
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
In check_max_stack_depth_subprog(), the local variable
tail_call_reachable is set when entering a callee that has a tail
call, but never reset when popping back to the parent. This causes
the flag to leak across sibling subprogs in the DFS traversal.
This results in unnecessary JIT overhead: the JIT emits tail call
counter preservation code for subprogs that can never be reached
via a tail call path.
Fix this by resetting tail_call_reachable to the parent's actual
per-subprog flag when popping a frame. If the parent was already
marked tail_call_reachable by a previous sibling's traversal, the
local variable stays true. Otherwise it resets to false, so
subsequent siblings start with a clean state.
Acked-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
Link: https://lore.kernel.org/r/20260423033435.2538013-1-yonghong.song@linux.dev
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
uprobe_multi usage message not in sync with the list of subtests it
actually supports.
Add the missing subtests in the help message.
Signed-off-by: Alexis Lothoré (eBPF Foundation) <alexis.lothore@bootlin.com>
Link: https://lore.kernel.org/bpf/20260421-uprobe_multi_usage-v1-1-4c51675955e6@bootlin.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
If a typedef is defined both in a base and in a split BTF, after
deduplication a single instance should be found in the base BTF.
Suggested-by: Alan Maguire <alan.maguire@oracle.com>
Signed-off-by: Antoine Tenart <atenart@kernel.org>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Link: https://lore.kernel.org/bpf/20260417083319.32716-2-atenart@kernel.org
When deduplicating definitions for a module, typedef defined in the base
are not removed. This is because the hash used for base types differs
from the one used in the deduplication logic in btf_dedup_struct_type.
This was introduced by the referenced commit when moving the typedef
deduplication logic handling from btf_dedup_ref_type to
btf_dedup_struct_type, as this also changed the hash logic
(btf_hash_common to btf_hash_typedef).
This also impacts other types referencing those typedef (e.g. const). In
my test, the BTF section size of the openvswitch module went from 31KB
to 45KB.
Fixes: 3781413465 ("libbpf: Fix BTF dedup to support recursive typedef definitions").
Signed-off-by: Antoine Tenart <atenart@kernel.org>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Tested-by: Alan Maguire <alan.maguire@oracle.com>
Reviewed-by: Alan Maguire <alan.maguire@oracle.com>
Link: https://lore.kernel.org/bpf/20260417083319.32716-1-atenart@kernel.org
The file_reader/on_open_expect_fault fails consistently on my system.
It expects a page fault on first dynptr read of some range the exe
file of the current process because it has paged out that page range
earlier. However a lot can happen to that range (which depending on
the actual memory layout could contain text section, data section,
sections )related to dynamic linking...) between the moment it was
paged out and the moment the bpf program expected to hit a pagefault
actually run.
A bit of instrumentation with mincore() shows that pages from that
range were accessed several times before the program is run. In
particular the call of file_reader__load() seems to fault all the
range in.
Move the call to madvise(MADV_PAGEOUT) to just before attaching the
program to minimize the risk of having those page pulled back in from
under our feet.
Signed-off-by: Jerome Marchand <jmarchan@redhat.com>
Acked-by: Mykyta Yatsenko <yatsenko@meta.com>
Acked-by: Jiri Olsa <jolsa@kernel.org>
Link: https://lore.kernel.org/bpf/20260420134637.2513867-1-jmarchan@redhat.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Mykyta Yatsenko says:
====================
bpf: Add support for sleepable tracepoint programs
This series adds support for sleepable BPF programs attached to raw
tracepoints (tp_btf, raw_tp) and classic tracepoints (tp).
The motivation is to allow BPF programs on syscall
tracepoints to use sleepable helpers such as bpf_copy_from_user(),
enabling reliable user memory reads that can page-fault.
This series removes restriction for faultable tracepoints:
Patch 1 modifies __bpf_trace_run() to support sleepable programs.
Patch 2 introduces bpf_prog_run_array_sleepable() to support new usecase.
Patch 3 adds sleepable support for classic tracepoints
(BPF_PROG_TYPE_TRACEPOINT) by introducing trace_call_bpf_faultable()
and restructuring perf_syscall_enter/exit() to run BPF programs in
faultable context.
Patch 4 allows BPF_TRACE_RAW_TP, BPF_PROG_TYPE_RAW_TRACEPOINT, and
BPF_PROG_TYPE_TRACEPOINT programs to be loaded as sleepable, with
load-time and attach-time checks to reject sleepable programs on
non-faultable tracepoints.
Patch 5 adds libbpf SEC_DEF handlers: tp_btf.s, raw_tp.s,
raw_tracepoint.s, tp.s, and tracepoint.s.
Patch 6 adds selftests covering tp_btf.s, raw_tp.s, and tp.s positive
cases using bpf_copy_from_user() plus negative tests for non-faultable
tracepoints.
Signed-off-by: Mykyta Yatsenko <yatsenko@meta.com>
---
Changes in v13:
- Invert if (prog->sleepable) check in bpf_prog_run_array_sleepable()
- Link to v12: https://patch.msgid.link/20260422-sleepable_tracepoints-v12-0-744bf0e3b311@meta.com
Changes in v12:
- Style improvement in the bpf_prog_run_array_sleepable(): use guard(rcu)(), remove
unnecessary defensive programming artifacts.
- Link to v11: https://patch.msgid.link/20260421-sleepable_tracepoints-v11-0-d8ff138d6f05@meta.com
Changes in v11:
- Avoid running dummy prog in the bpf_prog_run_array_sleepable()
- Migrate selftests from nanosleep() to getcwd() to avoid issues with the
different struct layouts.
- Link to v10: https://patch.msgid.link/20260415-sleepable_tracepoints-v10-0-161f40b33dd7@meta.com
Changes in v10:
- Guard per-prog recursion check in bpf_prog_run_array_sleepable()
with prog->active NULL check, following the same pattern as
commit 7dc211c115 for prog->stats. dummy_bpf_prog has NULL
active field and can appear in the array via
bpf_prog_array_delete_safe() fallback on allocation failure.
- Link to v9: https://patch.msgid.link/20260410-sleepable_tracepoints-v9-0-e719e664e84c@meta.com
Changes in v9:
- Fixed "classic raw tracepoints" to "raw tracepoints (tp_btf, raw_tp)"
in commit message
- Added bpf_prog_get_recursion_context() guard to
__bpf_prog_test_run_raw_tp() to protect per-CPU private stack from
concurrent sleepable test runs
- Added new bpf_prog_run_array_sleepable() without is_uprobe parameter,
remove all changes in bpf_prog_run_array_uprobe()
- Refactored attach_tp() to use prefix array uniformly (matching
attach_raw_tp() pattern), removing hardcoded strcmp() bare-name checks.
- Recursion check in __bpf_prog_test_run_raw_tp()
- Refactored selftests
- Link to v8: https://patch.msgid.link/20260330-sleepable_tracepoints-v8-0-2e323467f3a0@meta.com
Changes in v8:
- Fix sleepable tracepoint support in bpf_prog_test_run() (Kumar, sashiko)
- Link to v7: https://patch.msgid.link/20260325-sleepable_tracepoints-v6-0-2b182dacea13@meta.com
Changes in v7:
- Add recursion check (bpf_prog_get_recursion_context()) to make sure
private stack is safe when sleepable program is preempted by itself
(Alexei, Kumar)
- Use combined rcu_read_lock_dont_migrate() instead of separate
rcu_read_lock()/migrate_disable() calls for non-sleepable path (Alexei)
- Link to v6: https://lore.kernel.org/bpf/20260324-sleepable_tracepoints-v6-0-81bab3a43f25@meta.com/
Changes in v6:
- Remove recursion check from trace_call_bpf_faultable(), sleepable
tracepoints are called from syscall enter/exit, no recursion is
possible.(Kumar)
- Refactor bpf_prog_run_array_uprobe() to support tracepoints
usecase cleanly (Kumar)
- Link to v5: https://lore.kernel.org/r/20260316-sleepable_tracepoints-v5-0-85525de71d25@meta.com
Changes in v5:
- Addressed AI review: zero initialize struct pt_regs in
perf_call_bpf_enter(); changed handling tp.s and tracepoint.s in
attach_tp() in libbpf.
- Updated commit messages
- Link to v4: https://lore.kernel.org/r/20260313-sleepable_tracepoints-v4-0-debc688a66b3@meta.com
Changes in v4:
- Follow uprobe_prog_run() pattern with explicit rcu_read_lock_trace()
instead of relying on outer rcu_tasks_trace lock
- Add sleepable support for classic raw tracepoints (raw_tp.s)
- Add sleepable support for classic tracepoints (tp.s) with new
trace_call_bpf_faultable() and restructured perf_syscall_enter/exit()
- Add raw_tp.s, raw_tracepoint.s, tp.s, tracepoint.s SEC_DEF handlers
- Replace growing type enumeration in error message with generic
"program of this type cannot be sleepable"
- Use PT_REGS_PARM1_SYSCALL (non-CO-RE) in BTF test
- Add classic raw_tp and classic tracepoint sleepable tests
- Link to v3: https://lore.kernel.org/r/20260311-sleepable_tracepoints-v3-0-3e9bbde5bd22@meta.com
Changes in v3:
- Moved faultable tracepoint check from attach time to load time in
bpf_check_attach_target(), providing a clear verifier error message
- Folded preempt_disable removal into the sleepable execution path
patch
- Used RUN_TESTS() with __failure/__msg for negative test case instead
of explicit userspace program
- Reduced series from 6 patches to 4
- Link to v2: https://lore.kernel.org/r/20260225-sleepable_tracepoints-v2-0-0330dafd650f@meta.com
Changes in v2:
- Address AI review points - modified the order of the patches
- Link to v1: https://lore.kernel.org/bpf/20260218-sleepable_tracepoints-v1-0-ec2705497208@meta.com/
---
====================
Link: https://patch.msgid.link/20260422-sleepable_tracepoints-v13-0-99005dff21ef@meta.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Cover all three sleepable tracepoint types (tp_btf.s, raw_tp.s, tp.s)
and sys_exit (via bpf_task_pt_regs) with functional tests using
bpf_copy_from_user() on getcwd. Verify alias and bare SEC variants,
bpf_prog_test_run_raw_tp() with BPF_F_TEST_RUN_ON_CPU rejection,
attach-time rejection on non-faultable tracepoints, and load-time
rejection for sleepable tp_btf on non-faultable tracepoints.
Signed-off-by: Mykyta Yatsenko <yatsenko@meta.com>
Acked-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Link: https://lore.kernel.org/bpf/20260422-sleepable_tracepoints-v13-6-99005dff21ef@meta.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Allow BPF_PROG_TYPE_RAW_TRACEPOINT, BPF_PROG_TYPE_TRACEPOINT, and
BPF_TRACE_RAW_TP (tp_btf) programs to be sleepable by adding them
to can_be_sleepable().
For BTF-based raw tracepoints (tp_btf), add a load-time check in
bpf_check_attach_target() that rejects sleepable programs attaching
to non-faultable tracepoints with a descriptive error message.
For raw tracepoints (raw_tp), add an attach-time check in
bpf_raw_tp_link_attach() that rejects sleepable programs on
non-faultable tracepoints. The attach-time check is needed because
the tracepoint name is not known at load time for raw_tp.
The attach-time check for classic tracepoints (tp) in
__perf_event_set_bpf_prog() was added in the previous patch.
Replace the verbose error message that enumerates allowed program
types with a generic "Program of this type cannot be sleepable"
message, since the list of sleepable-capable types keeps growing.
Signed-off-by: Mykyta Yatsenko <yatsenko@meta.com>
Acked-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Link: https://lore.kernel.org/bpf/20260422-sleepable_tracepoints-v13-4-99005dff21ef@meta.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Add trace_call_bpf_faultable(), a variant of trace_call_bpf() for
faultable tracepoints that supports sleepable BPF programs. It uses
rcu_tasks_trace for lifetime protection and
bpf_prog_run_array_sleepable() for per-program RCU flavor selection,
following the uprobe_prog_run() pattern.
Restructure perf_syscall_enter() and perf_syscall_exit() to run BPF
programs before perf event processing. Previously, BPF ran after the
per-cpu perf trace buffer was allocated under preempt_disable,
requiring cleanup via perf_swevent_put_recursion_context() on filter.
Now BPF runs in faultable context before preempt_disable, reading
syscall arguments from local variables instead of the per-cpu trace
record, removing the dependency on buffer allocation. This allows
sleepable BPF programs to execute and avoids unnecessary buffer
allocation when BPF filters the event. The perf event submission
path (buffer allocation, fill, submit) remains under preempt_disable
as before. Since BPF no longer runs within the buffer allocation
context, the fake_regs output parameter to perf_trace_buf_alloc()
is no longer needed and is replaced with NULL.
Add an attach-time check in __perf_event_set_bpf_prog() to reject
sleepable BPF_PROG_TYPE_TRACEPOINT programs on non-syscall
tracepoints, since only syscall tracepoints run in faultable context.
This prepares the classic tracepoint runtime and attach paths for
sleepable programs. The verifier changes to allow loading sleepable
BPF_PROG_TYPE_TRACEPOINT programs are in a subsequent patch.
To: Peter Zijlstra <peterz@infradead.org>
To: Steven Rostedt <rostedt@goodmis.org>
Signed-off-by: Mykyta Yatsenko <yatsenko@meta.com>
Acked-by: Kumar Kartikeya Dwivedi <memxor@gmail.com> # for BPF bits
Acked-by: Steven Rostedt <rostedt@goodmis.org>
Link: https://lore.kernel.org/bpf/20260422-sleepable_tracepoints-v13-3-99005dff21ef@meta.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Add bpf_prog_run_array_sleepable() for running BPF program arrays
on faultable tracepoints. Unlike bpf_prog_run_array_uprobe(), it
includes per-program recursion checking for private stack safety
and hardcodes is_uprobe to false.
Skip dummy_bpf_prog at the top of the loop. When
bpf_prog_array_delete_safe() replaces a detached program with
dummy_bpf_prog on allocation failure, the dummy is statically
allocated and has NULL active, stats, and aux fields. Identify
it by prog->len == 0, since every real program has at least one
instruction.
Keep bpf_prog_run_array_uprobe() unchanged for uprobe callers.
Signed-off-by: Mykyta Yatsenko <yatsenko@meta.com>
Acked-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Link: https://lore.kernel.org/bpf/20260422-sleepable_tracepoints-v13-2-99005dff21ef@meta.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Rework __bpf_trace_run() to support sleepable BPF programs by using
explicit RCU flavor selection, following the uprobe_prog_run() pattern.
For sleepable programs, use rcu_read_lock_tasks_trace() for lifetime
protection with migrate_disable(). For non-sleepable programs, use the
regular rcu_read_lock_dont_migrate().
Remove the preempt_disable_notrace/preempt_enable_notrace pair from
the faultable tracepoint BPF probe wrapper in bpf_probe.h, since
migration protection and RCU locking are now handled per-program
inside __bpf_trace_run().
Adapt bpf_prog_test_run_raw_tp() for sleepable programs: reject
BPF_F_TEST_RUN_ON_CPU since sleepable programs cannot run in hardirq
or preempt-disabled context, and call __bpf_prog_test_run_raw_tp()
directly instead of via smp_call_function_single(). Rework
__bpf_prog_test_run_raw_tp() to select RCU flavor per-program and
add per-program recursion context guard for private stack safety.
Signed-off-by: Mykyta Yatsenko <yatsenko@meta.com>
Acked-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Link: https://lore.kernel.org/bpf/20260422-sleepable_tracepoints-v13-1-99005dff21ef@meta.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Inline these calls in bpf jit:
- bpf_get_smp_processor_id()
- bpf_get_current_task()
- bpf_get_current_task_btf()
s390 has a 8 KiB per-CPU prefix area in the CPU's
virtual address space, called the lowcore. It is a
struct that contains the cpu number and a pointer
to the current task. These are exactly the values
returned by the BPF helpers.
Emit a load from the lowcore instead of a helper
function call.
JIT output for `bpf_get_smp_processor_id`:
Before: After:
--------------- ----------------
brasl %r14,0x3ffe0385460 ly %r14,928
lgr %r14,%r2
JIT output for `bpf_get_current_task`:
Before: After:
--------------- ----------------
brasl %r14,0x3ffe0362a90 lg %r14,832
lgr %r14,%r2
Benchmark using [1] on KVM(virtme-ng).
./benchs/run_bench_trigger.sh glob-arr-inc arr-inc hash-inc
+---------------+--------------------+--------------------+--------------+
| Name | Before | After | % change |
|---------------+--------------------+--------------------+--------------|
| glob-arr-inc | 244.954 ± 0.654M/s | 278.501 ± 0.834M/s | + 13.70% |
| arr-inc | 311.597 ± 1.016M/s | 313.610 ± 0.331M/s | + 0.65% |
| hash-inc | 47.421 ± 0.017M/s | 47.600 ± 0.004M/s | + 0.38% |
+---------------+--------------------+--------------------+--------------+
[1] https://github.com/anakryiko/linux/commit/8dec900975ef
Signed-off-by: Maxim Khmelevskii <max@linux.ibm.com>
Reviewed-by: Ilya Leoshkevich <iii@linux.ibm.com>
Link: https://lore.kernel.org/r/20260414142930.528751-1-max@linux.ibm.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Extend lib/test_bpf.c to provide comprehensive test coverage for BPF
signed division (SDIV) and signed modulo (SMOD) instructions, both
32-bit and 64-bit variants with immediate operands.
Introduce F_ALU32 and F_SIGNED flags to replace the less readable
bool alu32 and s16 off parameters throughout the test helpers. The
BPF instruction 'off' field is derived from flags only at the point
of instruction encoding.
Changes:
- Add enum { F_ALU32 = 1, F_SIGNED = 2 } for readable test flags.
- __bpf_alu_result(): take u32 flags instead of separate signed/alu32
parameters. Narrows operands internally for ALU32 (unsigned via u32
cast, signed via s32 cast) before computing the reference result.
- __bpf_emit_alu64_imm(), __bpf_emit_alu32_imm(): pass flags through
to __bpf_alu_result, derive 'off' for instruction encoding locally.
- __bpf_fill_alu_imm_regs(): take u32 flags, use F_ALU32/F_SIGNED for
operand setup and single-line __bpf_alu_result() call.
- __bpf_fill_alu_shift(), __bpf_fill_alu_shift_same_reg(): convert
bool alu32 parameter to u32 flags for consistency.
- New test fill functions: bpf_fill_alu{32,64}_{sdiv,smod}_imm() and
bpf_fill_alu{32,64}_{sdiv,smod}_imm_regs(), each testing all
immediate value magnitudes and all register pair combinations.
- All existing unsigned tests updated to use flags (0 or F_ALU32),
preserving backward compatibility.
8 new test cases added:
ALU64_SDIV_K, ALU64_SMOD_K (immediate magnitudes + register combos)
ALU32_SDIV_K, ALU32_SMOD_K (immediate magnitudes + register combos)
Test results:
test_bpf: Summary: 1061 PASSED, 0 FAILED, [1049/1049 JIT'ed]
test_bpf: test_tail_calls: Summary: 10 PASSED, 0 FAILED, [10/10 JIT'ed]
test_bpf: test_skb_segment: Summary: 2 PASSED, 0 FAILED
Assisted-by: Claude:claude-opus-4-6
Signed-off-by: Jie Meng <jmeng@fb.com>
Link: https://lore.kernel.org/r/20260413172311.3918767-1-jmeng@fb.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Alan Maguire says:
====================
bpf: Extend BTF UAPI vlen, kinds to use unused bits
Currently BTF types can have a maximum of 65535 vlen-specified
objects. While this limit has not yet been surpassed for existing
cases (struct/union fields, enum values, function arguments and
datasec), upcoming BTF location information - specifically inline
sites - will hit that limit. Utilize unused BTF info bits in
struct btf_type to increase limit to 24-bits (over 16 million).
This is more than an order of magnitude greater than inline
site counts for the kernel (~400,000) so should be enough for
the near future at least.
Similarly, struct btf_type uses 5 bits for BTF kind values;
currently we use 20 kinds, but BTF location information will
take another 3, and conceivably providing better support for Rust
types in BTF (now that modules can be built using Rust) could
push us close to the max of 31. Use 2 unused bits to provide
a max possible kind of 127.
Patch 1 handles UAPI and kernel-related changes.
Patch 2 updates libbpf to have btf_vlen() return a __u32 instead
of __u16 and updates consumers in libbpf accordingly. Also adds
bounds checking for max vlen.
Patch 3 cleans up __u16 vlen usage in bpftool.
Patch 4 fixes up a test that relies on BTF info overflowing
maximum kind value; fix up the expected error to be an
invalid kind rather than (now impossible) invalid btf_info.
Patch 5 fixes up __u16 vlen usage in selftests/bpf.
Patch 6 syncs BTF documentation with UAPI changes.
Changes since v2 [1]:
- dropped backwards-compatibility macros (Alexei, patch 1)
- added additional vlen bounds check in btf__add_field()
(sashiko, bpf bot, patch 2)
- dropped sanitization check in libbpf as older kernels will
reject BTF with extended vlen/kinds using BTF info mask
(Alexei)
- fix more vlen assumptions in prog_test/ctx_rewrite.c
(sashiko, patch 5)
Changes since v1 [2]:
- added redundant macros to UAPI in case enum switch for BTF_MAX*
values causes problems (sashiko, patch 1)
- updated feature test to use BTF loading rather than vmlinux BTF
lookup, in line with other BTF feature tests (sashiko, patch 3)
- fixed up a few more kernel, libbpf, bpftool instances of __u16 usage
(bpf bot, sashiko, patches 1, 2, 4)
- fixed up sanitize selftest cleanup (Mykyta, patch 5)
- used unlikely-to-be-used kind (sashiko, patch 6)
- fixed a few lingering selftests uses of __u16 for kind (patch 7)
- update btf.rst with new vlen, kind sizes (patch 8)
[1] https://lore.kernel.org/bpf/20260416143904.1220662-1-alan.maguire@oracle.com/
[2] https://lore.kernel.org/bpf/20260414195019.684531-1-alan.maguire@oracle.com/
====================
Link: https://patch.msgid.link/20260417143023.1551481-1-alan.maguire@oracle.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
With extended kinds, 32 becomes a valid (but not used)
BTF info kind value; fix up the test to check for the
"Invalid kind" rather than "Invalid btf_info" message.
Since all bits are used in BTF info, it is no longer
possible to craft an invalid BTF info value. Use
127 (new maximum possible kind value).
Signed-off-by: Alan Maguire <alan.maguire@oracle.com>
Acked-by: Mykyta Yatsenko <yatsenko@meta.com>
Link: https://lore.kernel.org/r/20260417143023.1551481-5-alan.maguire@oracle.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
