github-mirror/linux
1
0
Fork 0
mirror of https://github.com/torvalds/linux.git synced 2026-05-30 10:04:04 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity Graph

Merge branch 'bpf-replace-path-sensitive-with-path-insensitive-live-stack-analysis'

Browse Source 
Eduard Zingerman says:

====================
bpf: replace path-sensitive with path-insensitive live stack analysis

Consider the following program, assuming checkpoint is created for a
state at instruction (3):

  1: call bpf_get_prandom_u32()
  2: *(u64 *)(r10 - 8) = 42
  -- checkpoint #1 --
  3: if r0 != 0 goto +1
  4: exit;
  5: r0 = *(u64 *)(r10 - 8)
  6: exit

The verifier processes this program by exploring two paths:
 - 1 -> 2 -> 3 -> 4
 - 1 -> 2 -> 3 -> 5 -> 6

When instruction (5) is processed, the current liveness tracking
mechanism moves up the register parent links and records a "read" mark
for stack slot -8 at checkpoint #1, stopping because of the "write"
mark recorded at instruction (2).

This patch set replaces the existing liveness tracking mechanism with
a path-insensitive data flow analysis. The program above is processed
as follows:
 - a data structure representing live stack slots for
   instructions 1-6 in frame #0 is allocated;
 - when instruction (2) is processed, record that slot -8 is written at
   instruction (2) in frame #0;
 - when instruction (5) is processed, record that slot -8 is read at
   instruction (5) in frame #0;
 - when instruction (6) is processed, propagate read mark for slot -8
   up the control flow graph to instructions 3 and 2.

The key difference is that the new mechanism operates on a control
flow graph and associates read and write marks with pairs of (call
chain, instruction index). In contrast, the old mechanism operates on
verifier states and register parent links, associating read and write
marks with verifier states.

Motivation
==========

As it stands, this patch set makes liveness tracking slightly less
precise, as it no longer distinguishes individual program paths taken
by the verifier during symbolic execution.
See the "Impact on verification performance" section for details.

However, this change is intended as a stepping stone toward the
following goals:
 - Short term, integrate precision tracking into liveness analysis and
   remove the following code:
   - verifier backedge states accumulation in is_state_visited();
   - most of the logic for precision tracking;
   - jump history tracking.
 - Long term, help with more efficient loop verification handling.

 Why integrating precision tracking?
 -----------------------------------

In a sense, precision tracking is very similar to liveness tracking.
The data flow equations for liveness tracking look as follows:

  live_after =
    U [state[s].live_before for s in insn_successors(i)]

  state[i].live_before =
    (live_after / state[i].must_write) U state[i].may_read

While data flow equations for precision tracking look as follows:

  precise_after =
    U [state[s].precise_before for s in insn_successors(i)]

  // if some of the instruction outputs are precise,
  // assume its inputs to be precise
  induced_precise =
    ⎧ state[i].may_read   if (state[i].may_write ∩ precise_after) ≠ ∅
    ⎨
    ⎩ ∅                   otherwise

  state[i].precise_before =
    (precise_after / state[i].must_write) ∩ induced_precise

Where:
 - `may_read` set represents a union of all possibly read slots
   (any slot in `may_read` set might be by the instruction);
 - `must_write` set represents an intersection of all possibly written slots
   (any slot in `must_write` set is guaranteed to be written by the instruction).
 - `may_write` set represents a union of all possibly written slots
   (any slot in `may_write` set might be written by the instruction).

This means that precision tracking can be implemented as a logical
extension of liveness tracking:
 - track registers as well as stack slots;
 - add bit masks to represent `precise_before` and `may_write`;
 - add above equations for `precise_before` computation;
 - (linked registers require some additional consideration).

Such extension would allow removal of:
 - precision propagation logic in verifier.c:
   - backtrack_insn()
   - mark_chain_precision()
   - propagate_{precision,backedges}()
 - push_jmp_history() and related data structures, which are only used
   by precision tracking;
 - add_scc_backedge() and related backedge state accumulation in
   is_state_visited(), superseded by per-callchain function state
   accumulated by liveness analysis.

The hope here is that unifying liveness and precision tracking will
reduce overall amount of code and make it easier to reason about.

 How this helps with loops?
 --------------------------

As it stands, this patch set shares the same deficiency as the current
liveness tracking mechanism. Liveness marks on stack slots cannot be
used to prune states when processing iterator-based loops:
 - such states still have branches to be explored;
 - meaning that not all stack slot reads have been discovered.

For example:

  1: while(iter_next()) {
  2:   if (...)
  3:     r0 = *(u64 *)(r10 - 8)
  4:   if (...)
  5:     r0 = *(u64 *)(r10 - 16)
  6:   ...
  7: }

For any checkpoint state created at instruction (1), it is only
possible to rely on read marks for slots fp[-8] and fp[-16] once all
child states of (1) have been explored. Thus, when the verifier
transitions from (7) to (1), it cannot rely on read marks.

However, sacrificing path-sensitivity makes it possible to run
analysis defined in this patch set before main verification pass,
if estimates for value ranges are available.
E.g. for the following program:

  1: while(iter_next()) {
  2:   r0 = r10
  3:   r0 += r2
  4:   r0 = *(u64 *)(r2 + 0)
  5:   ...
  6: }

If an estimate for `r2` range is available before the main
verification pass, it can be used to populate read marks at
instruction (4) and run the liveness analysis. Thus making
conservative liveness information available during loops verification.

Such estimates can be provided by some form of value range analysis.
Value range analysis is also necessary to address loop verification
from another angle: computing boundaries for loop induction variables
and iteration counts.

The hope here is that the new liveness tracking mechanism will support
the broader goal of making loop verification more efficient.

Validation
==========

The change was tested on three program sets:
 - bpf selftests
 - sched_ext
 - Meta's internal set of programs

Commit [#8] enables a special mode where both the current and new
liveness analyses are enabled simultaneously. This mode signals an
error if the new algorithm considers a stack slot dead while the
current algorithm assumes it is alive. This mode was very useful for
debugging. At the time of posting, no such errors have been reported
for the above program sets.

[#8] "bpf: signal error if old liveness is more conservative than new"

Impact on memory consumption
============================

Debug patch [1] extends the kernel and veristat to count the amount of
memory allocated for storing analysis data. This patch is not included
in the submission. The maximal observed impact for the above program
sets is 2.6Mb.

Data below is shown in bytes.

For bpf selftests top 5 consumers look as follows:

  File                     Program           liveness mem
  -----------------------  ----------------  ------------
  pyperf180.bpf.o          on_event               2629740
  pyperf600.bpf.o          on_event               2287662
  pyperf100.bpf.o          on_event               1427022
  test_verif_scale3.bpf.o  balancer_ingress       1121283
  pyperf_subprogs.bpf.o    on_event                756900

For sched_ext top 5 consumers loog as follows:

  File       Program                          liveness mem
  ---------  -------------------------------  ------------
  bpf.bpf.o  lavd_enqueue                           164686
  bpf.bpf.o  lavd_select_cpu                        157393
  bpf.bpf.o  layered_enqueue                        154817
  bpf.bpf.o  lavd_init                              127865
  bpf.bpf.o  layered_dispatch                       110129

For Meta's internal set of programs top consumer is 1Mb.

[1] 085588e787

Impact on verification performance
==================================

Veristat results below are reported using
`-f insns_pct>1 -f !insns<500` filter and -t option
(BPF_F_TEST_STATE_FREQ flag).

 master vs patch-set, selftests (out of ~4K programs)
 ----------------------------------------------------

  File                              Program                                 Insns (A)  Insns (B)  Insns    (DIFF)
  --------------------------------  --------------------------------------  ---------  ---------  ---------------
  cpumask_success.bpf.o             test_global_mask_nested_deep_array_rcu       1622       1655     +33 (+2.03%)
  strobemeta_bpf_loop.bpf.o         on_event                                     2163       2684   +521 (+24.09%)
  test_cls_redirect.bpf.o           cls_redirect                                36001      42515  +6514 (+18.09%)
  test_cls_redirect_dynptr.bpf.o    cls_redirect                                 2299       2339     +40 (+1.74%)
  test_cls_redirect_subprogs.bpf.o  cls_redirect                                69545      78497  +8952 (+12.87%)
  test_l4lb_noinline.bpf.o          balancer_ingress                             2993       3084     +91 (+3.04%)
  test_xdp_noinline.bpf.o           balancer_ingress_v4                          3539       3616     +77 (+2.18%)
  test_xdp_noinline.bpf.o           balancer_ingress_v6                          3608       3685     +77 (+2.13%)

 master vs patch-set, sched_ext (out of 148 programs)
 ----------------------------------------------------

  File       Program           Insns (A)  Insns (B)  Insns    (DIFF)
  ---------  ----------------  ---------  ---------  ---------------
  bpf.bpf.o  chaos_dispatch         2257       2287     +30 (+1.33%)
  bpf.bpf.o  lavd_enqueue          20735      22101   +1366 (+6.59%)
  bpf.bpf.o  lavd_select_cpu       22100      24409  +2309 (+10.45%)
  bpf.bpf.o  layered_dispatch      25051      25606    +555 (+2.22%)
  bpf.bpf.o  p2dq_dispatch           961        990     +29 (+3.02%)
  bpf.bpf.o  rusty_quiescent         526        534      +8 (+1.52%)
  bpf.bpf.o  rusty_runnable          541        547      +6 (+1.11%)

Perf report
===========

In relative terms, the analysis does not consume much CPU time.
For example, here is a perf report collected for pyperf180 selftest:

 # Children      Self  Command   Shared Object         Symbol
 # ........  ........  ........  ....................  ........................................
        ...
      1.22%     1.22%  veristat  [kernel.kallsyms]     [k] bpf_update_live_stack
        ...

Changelog
=========

v1: https://lore.kernel.org/bpf/20250911010437.2779173-1-eddyz87@gmail.com/T/
v1 -> v2:
 - compute_postorder() fixed to handle jumps with offset -1 (syzbot).
 - is_state_visited() in patch #9 fixed access to uninitialized `err`
   (kernel test robot, Dan Carpenter).
 - Selftests added.
 - Fixed bug with write marks propagation from callee to caller,
   see verifier_live_stack.c:caller_stack_write() test case.
 - Added a patch for __not_msg() annotation for test_loader based
   tests.

v2: https://lore.kernel.org/bpf/20250918-callchain-sensitive-liveness-v2-0-214ed2653eee@gmail.com/
v2 -> v3:
 - Added __diag_ignore_all("-Woverride-init", ...) in liveness.c for
   bpf_insn_successors() (suggested by Alexei).

Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
====================

Link: https://patch.msgid.link/20250918-callchain-sensitive-liveness-v3-0-c3cd27bacc60@gmail.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
This commit is contained in:
Alexei Starovoitov 2025-09-19 09:27:24 -07:00
commit 815276dbfb
27 changed files with 1717 additions and 978 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

264
Documentation/bpf/verifier.rst
View File

@ -347,270 +347,6 @@ However, only the value of register ``r1`` is important to successfully finish
verification. The goal of the liveness tracking algorithm is to spot this fact
and figure out that both states are actually equivalent.
Data structures
~~~~~~~~~~~~~~~
Liveness is tracked using the following data structures::
enum bpf_reg_liveness {
REG_LIVE_NONE = 0,
REG_LIVE_READ32 = 0x1,
REG_LIVE_READ64 = 0x2,
REG_LIVE_READ = REG_LIVE_READ32 | REG_LIVE_READ64,
REG_LIVE_WRITTEN = 0x4,
REG_LIVE_DONE = 0x8,
};
struct bpf_reg_state {
...
struct bpf_reg_state *parent;
...
enum bpf_reg_liveness live;
...
};
struct bpf_stack_state {
struct bpf_reg_state spilled_ptr;
...
};
struct bpf_func_state {
struct bpf_reg_state regs[MAX_BPF_REG];
...
struct bpf_stack_state *stack;
}
struct bpf_verifier_state {
struct bpf_func_state *frame[MAX_CALL_FRAMES];
struct bpf_verifier_state *parent;
...
}
* ``REG_LIVE_NONE`` is an initial value assigned to ``->live`` fields upon new
verifier state creation;
* ``REG_LIVE_WRITTEN`` means that the value of the register (or stack slot) is
defined by some instruction verified between this verifier state's parent and
verifier state itself;
* ``REG_LIVE_READ{32,64}`` means that the value of the register (or stack slot)
is read by a some child state of this verifier state;
* ``REG_LIVE_DONE`` is a marker used by ``clean_verifier_state()`` to avoid
processing same verifier state multiple times and for some sanity checks;
* ``->live`` field values are formed by combining ``enum bpf_reg_liveness``
values using bitwise or.
Register parentage chains
~~~~~~~~~~~~~~~~~~~~~~~~~
In order to propagate information between parent and child states, a *register
parentage chain* is established. Each register or stack slot is linked to a
corresponding register or stack slot in its parent state via a ``->parent``
pointer. This link is established upon state creation in ``is_state_visited()``
and might be modified by ``set_callee_state()`` called from
``__check_func_call()``.
The rules for correspondence between registers / stack slots are as follows:
* For the current stack frame, registers and stack slots of the new state are
linked to the registers and stack slots of the parent state with the same
indices.
* For the outer stack frames, only callee saved registers (r6-r9) and stack
slots are linked to the registers and stack slots of the parent state with the
same indices.
* When function call is processed a new ``struct bpf_func_state`` instance is
allocated, it encapsulates a new set of registers and stack slots. For this
new frame, parent links for r6-r9 and stack slots are set to nil, parent links
for r1-r5 are set to match caller r1-r5 parent links.
This could be illustrated by the following diagram (arrows stand for
``->parent`` pointers)::
... ; Frame #0, some instructions
--- checkpoint #0 ---
1 : r6 = 42 ; Frame #0
--- checkpoint #1 ---
2 : call foo() ; Frame #0
... ; Frame #1, instructions from foo()
--- checkpoint #2 ---
... ; Frame #1, instructions from foo()
--- checkpoint #3 ---
exit ; Frame #1, return from foo()
3 : r1 = r6 ; Frame #0 <- current state
+-------------------------------+-------------------------------+
| Frame #0 | Frame #1 |
Checkpoint +-------------------------------+-------------------------------+
#0 | r0 | r1-r5 | r6-r9 | fp-8 ... |
+-------------------------------+
^ ^ ^ ^
| | | |
Checkpoint +-------------------------------+
#1 | r0 | r1-r5 | r6-r9 | fp-8 ... |
+-------------------------------+
^ ^ ^
|_______|_______|_______________
| | |
nil nil | | | nil nil
| | | | | | |
Checkpoint +-------------------------------+-------------------------------+
#2 | r0 | r1-r5 | r6-r9 | fp-8 ... | r0 | r1-r5 | r6-r9 | fp-8 ... |
+-------------------------------+-------------------------------+
^ ^ ^ ^ ^
nil nil | | | | |
| | | | | | |
Checkpoint +-------------------------------+-------------------------------+
#3 | r0 | r1-r5 | r6-r9 | fp-8 ... | r0 | r1-r5 | r6-r9 | fp-8 ... |
+-------------------------------+-------------------------------+
^ ^
nil nil | |
| | | |
Current +-------------------------------+
state | r0 | r1-r5 | r6-r9 | fp-8 ... |
+-------------------------------+
\
r6 read mark is propagated via these links
all the way up to checkpoint #1.
The checkpoint #1 contains a write mark for r6
because of instruction (1), thus read propagation
does not reach checkpoint #0 (see section below).
Liveness marks tracking
~~~~~~~~~~~~~~~~~~~~~~~
For each processed instruction, the verifier tracks read and written registers
and stack slots. The main idea of the algorithm is that read marks propagate
back along the state parentage chain until they hit a write mark, which 'screens
off' earlier states from the read. The information about reads is propagated by
function ``mark_reg_read()`` which could be summarized as follows::
mark_reg_read(struct bpf_reg_state *state, ...):
parent = state->parent
while parent:
if state->live & REG_LIVE_WRITTEN:
break
if parent->live & REG_LIVE_READ64:
break
parent->live |= REG_LIVE_READ64
state = parent
parent = state->parent
Notes:
* The read marks are applied to the **parent** state while write marks are
applied to the **current** state. The write mark on a register or stack slot
means that it is updated by some instruction in the straight-line code leading
from the parent state to the current state.
* Details about REG_LIVE_READ32 are omitted.
* Function ``propagate_liveness()`` (see section :ref:`read_marks_for_cache_hits`)
might override the first parent link. Please refer to the comments in the
``propagate_liveness()`` and ``mark_reg_read()`` source code for further
details.
Because stack writes could have different sizes ``REG_LIVE_WRITTEN`` marks are
applied conservatively: stack slots are marked as written only if write size
corresponds to the size of the register, e.g. see function ``save_register_state()``.
Consider the following example::
0: (*u64)(r10 - 8) = 0 ; define 8 bytes of fp-8
--- checkpoint #0 ---
1: (*u32)(r10 - 8) = 1 ; redefine lower 4 bytes
2: r1 = (*u32)(r10 - 8) ; read lower 4 bytes defined at (1)
3: r2 = (*u32)(r10 - 4) ; read upper 4 bytes defined at (0)
As stated above, the write at (1) does not count as ``REG_LIVE_WRITTEN``. Should
it be otherwise, the algorithm above wouldn't be able to propagate the read mark
from (3) to checkpoint #0.
Once the ``BPF_EXIT`` instruction is reached ``update_branch_counts()`` is
called to update the ``->branches`` counter for each verifier state in a chain
of parent verifier states. When the ``->branches`` counter reaches zero the
verifier state becomes a valid entry in a set of cached verifier states.
Each entry of the verifier states cache is post-processed by a function
``clean_live_states()``. This function marks all registers and stack slots
without ``REG_LIVE_READ{32,64}`` marks as ``NOT_INIT`` or ``STACK_INVALID``.
Registers/stack slots marked in this way are ignored in function ``stacksafe()``
called from ``states_equal()`` when a state cache entry is considered for
equivalence with a current state.
Now it is possible to explain how the example from the beginning of the section
works::
0: call bpf_get_prandom_u32()
1: r1 = 0
2: if r0 == 0 goto +1
3: r0 = 1
--- checkpoint[0] ---
4: r0 = r1
5: exit
* At instruction #2 branching point is reached and state ``{ r0 == 0, r1 == 0, pc == 4 }``
is pushed to states processing queue (pc stands for program counter).
* At instruction #4:
* ``checkpoint[0]`` states cache entry is created: ``{ r0 == 1, r1 == 0, pc == 4 }``;
* ``checkpoint[0].r0`` is marked as written;
* ``checkpoint[0].r1`` is marked as read;
* At instruction #5 exit is reached and ``checkpoint[0]`` can now be processed
by ``clean_live_states()``. After this processing ``checkpoint[0].r1`` has a
read mark and all other registers and stack slots are marked as ``NOT_INIT``
or ``STACK_INVALID``
* The state ``{ r0 == 0, r1 == 0, pc == 4 }`` is popped from the states queue
and is compared against a cached state ``{ r1 == 0, pc == 4 }``, the states
are considered equivalent.
.. _read_marks_for_cache_hits:
Read marks propagation for cache hits
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Another point is the handling of read marks when a previously verified state is
found in the states cache. Upon cache hit verifier must behave in the same way
as if the current state was verified to the program exit. This means that all
read marks, present on registers and stack slots of the cached state, must be
propagated over the parentage chain of the current state. Example below shows
why this is important. Function ``propagate_liveness()`` handles this case.
Consider the following state parentage chain (S is a starting state, A-E are
derived states, -> arrows show which state is derived from which)::
r1 read
<------------- A[r1] == 0
C[r1] == 0
S ---> A ---> B ---> exit E[r1] == 1
|
` ---> C ---> D
|
` ---> E ^
|___ suppose all these
^ states are at insn #Y
|
suppose all these
states are at insn #X
* Chain of states ``S -> A -> B -> exit`` is verified first.
* While ``B -> exit`` is verified, register ``r1`` is read and this read mark is
propagated up to state ``A``.
* When chain of states ``C -> D`` is verified the state ``D`` turns out to be
equivalent to state ``B``.
* The read mark for ``r1`` has to be propagated to state ``C``, otherwise state
``C`` might get mistakenly marked as equivalent to state ``E`` even though
values for register ``r1`` differ between ``C`` and ``E``.
Understanding eBPF verifier messages
====================================

52
include/linux/bpf_verifier.h
View File

@ -26,28 +26,6 @@
/* Patch buffer size */
#define INSN_BUF_SIZE 32
/* Liveness marks, used for registers and spilled-regs (in stack slots).
* Read marks propagate upwards until they find a write mark; they record that
* "one of this state's descendants read this reg" (and therefore the reg is
* relevant for states_equal() checks).
* Write marks collect downwards and do not propagate; they record that "the
* straight-line code that reached this state (from its parent) wrote this reg"
* (and therefore that reads propagated from this state or its descendants
* should not propagate to its parent).
* A state with a write mark can receive read marks; it just won't propagate
* them to its parent, since the write mark is a property, not of the state,
* but of the link between it and its parent. See mark_reg_read() and
* mark_stack_slot_read() in kernel/bpf/verifier.c.
*/
enum bpf_reg_liveness {
REG_LIVE_NONE = 0, /* reg hasn't been read or written this branch */
REG_LIVE_READ32 = 0x1, /* reg was read, so we're sensitive to initial value */
REG_LIVE_READ64 = 0x2, /* likewise, but full 64-bit content matters */
REG_LIVE_READ = REG_LIVE_READ32 | REG_LIVE_READ64,
REG_LIVE_WRITTEN = 0x4, /* reg was written first, screening off later reads */
REG_LIVE_DONE = 0x8, /* liveness won't be updating this register anymore */
};
#define ITER_PREFIX "bpf_iter_"
enum bpf_iter_state {
@ -212,8 +190,6 @@ struct bpf_reg_state {
* allowed and has the same effect as bpf_sk_release(sk).
*/
u32 ref_obj_id;
/* parentage chain for liveness checking */
struct bpf_reg_state *parent;
/* Inside the callee two registers can be both PTR_TO_STACK like
* R1=fp-8 and R2=fp-8, but one of them points to this function stack
* while another to the caller's stack. To differentiate them 'frameno'
@ -226,7 +202,6 @@ struct bpf_reg_state {
* patching which only happens after main verification finished.
*/
s32 subreg_def;
enum bpf_reg_liveness live;
/* if (!precise && SCALAR_VALUE) min/max/tnum don't affect safety */
bool precise;
};
@ -445,6 +420,7 @@ struct bpf_verifier_state {
bool speculative;
bool in_sleepable;
bool cleaned;
/* first and last insn idx of this verifier state */
u32 first_insn_idx;
@ -665,6 +641,7 @@ struct bpf_subprog_info {
/* 'start' has to be the first field otherwise find_subprog() won't work */
u32 start; /* insn idx of function entry point */
u32 linfo_idx; /* The idx to the main_prog->aux->linfo */
u32 postorder_start; /* The idx to the env->cfg.insn_postorder */
u16 stack_depth; /* max. stack depth used by this function */
u16 stack_extra;
/* offsets in range [stack_depth .. fastcall_stack_off)
@ -744,6 +721,8 @@ struct bpf_scc_info {
struct bpf_scc_visit visits[];
};
struct bpf_liveness;
/* single container for all structs
* one verifier_env per bpf_check() call
*/
@ -794,7 +773,10 @@ struct bpf_verifier_env {
struct {
int *insn_state;
int *insn_stack;
/* vector of instruction indexes sorted in post-order */
/*
* vector of instruction indexes sorted in post-order, grouped by subprogram,
* see bpf_subprog_info->postorder_start.
*/
int *insn_postorder;
int cur_stack;
/* current position in the insn_postorder vector */
@ -842,6 +824,7 @@ struct bpf_verifier_env {
struct bpf_insn insn_buf[INSN_BUF_SIZE];
struct bpf_insn epilogue_buf[INSN_BUF_SIZE];
struct bpf_scc_callchain callchain_buf;
struct bpf_liveness *liveness;
/* array of pointers to bpf_scc_info indexed by SCC id */
struct bpf_scc_info **scc_info;
u32 scc_cnt;
@ -1065,4 +1048,21 @@ void print_verifier_state(struct bpf_verifier_env *env, const struct bpf_verifie
void print_insn_state(struct bpf_verifier_env *env, const struct bpf_verifier_state *vstate,
u32 frameno);
struct bpf_subprog_info *bpf_find_containing_subprog(struct bpf_verifier_env *env, int off);
int bpf_jmp_offset(struct bpf_insn *insn);
int bpf_insn_successors(struct bpf_prog *prog, u32 idx, u32 succ[2]);
void bpf_fmt_stack_mask(char *buf, ssize_t buf_sz, u64 stack_mask);
bool bpf_calls_callback(struct bpf_verifier_env *env, int insn_idx);
int bpf_stack_liveness_init(struct bpf_verifier_env *env);
void bpf_stack_liveness_free(struct bpf_verifier_env *env);
int bpf_update_live_stack(struct bpf_verifier_env *env);
int bpf_mark_stack_read(struct bpf_verifier_env *env, u32 frameno, u32 insn_idx, u64 mask);
void bpf_mark_stack_write(struct bpf_verifier_env *env, u32 frameno, u64 mask);
int bpf_reset_stack_write_marks(struct bpf_verifier_env *env, u32 insn_idx);
int bpf_commit_stack_write_marks(struct bpf_verifier_env *env);
int bpf_live_stack_query_init(struct bpf_verifier_env *env, struct bpf_verifier_state *st);
bool bpf_stack_slot_alive(struct bpf_verifier_env *env, u32 frameno, u32 spi);
void bpf_reset_live_stack_callchain(struct bpf_verifier_env *env);
#endif /* _LINUX_BPF_VERIFIER_H */

2
kernel/bpf/Makefile
View File

@ -6,7 +6,7 @@ cflags-nogcse-$(CONFIG_X86)$(CONFIG_CC_IS_GCC) := -fno-gcse
endif
CFLAGS_core.o += -Wno-override-init $(cflags-nogcse-yy)
obj-$(CONFIG_BPF_SYSCALL) += syscall.o verifier.o inode.o helpers.o tnum.o log.o token.o
obj-$(CONFIG_BPF_SYSCALL) += syscall.o verifier.o inode.o helpers.o tnum.o log.o token.o liveness.o
obj-$(CONFIG_BPF_SYSCALL) += bpf_iter.o map_iter.o task_iter.o prog_iter.o link_iter.o
obj-$(CONFIG_BPF_SYSCALL) += hashtab.o arraymap.o percpu_freelist.o bpf_lru_list.o lpm_trie.o map_in_map.o bloom_filter.o
obj-$(CONFIG_BPF_SYSCALL) += local_storage.o queue_stack_maps.o ringbuf.o

733
kernel/bpf/liveness.c Normal file
View File

@ -0,0 +1,733 @@
// SPDX-License-Identifier: GPL-2.0-only
/* Copyright (c) 2025 Meta Platforms, Inc. and affiliates. */
#include <linux/bpf_verifier.h>
#include <linux/hashtable.h>
#include <linux/jhash.h>
#include <linux/slab.h>
/*
* This file implements live stack slots analysis. After accumulating
* stack usage data, the analysis answers queries about whether a
* particular stack slot may be read by an instruction or any of it's
* successors. This data is consumed by the verifier states caching
* mechanism to decide which stack slots are important when looking for a
* visited state corresponding to the current state.
*
* The analysis is call chain sensitive, meaning that data is collected
* and queried for tuples (call chain, subprogram instruction index).
* Such sensitivity allows identifying if some subprogram call always
* leads to writes in the caller's stack.
*
* The basic idea is as follows:
* - As the verifier accumulates a set of visited states, the analysis instance
* accumulates a conservative estimate of stack slots that can be read
* or must be written for each visited tuple (call chain, instruction index).
* - If several states happen to visit the same instruction with the same
* call chain, stack usage information for the corresponding tuple is joined:
* - "may_read" set represents a union of all possibly read slots
* (any slot in "may_read" set might be read at or after the instruction);
* - "must_write" set represents an intersection of all possibly written slots
* (any slot in "must_write" set is guaranteed to be written by the instruction).
* - The analysis is split into two phases:
* - read and write marks accumulation;
* - read and write marks propagation.
* - The propagation phase is a textbook live variable data flow analysis:
*
* state[cc, i].live_after = U [state[cc, s].live_before for s in insn_successors(i)]
* state[cc, i].live_before =
* (state[cc, i].live_after / state[cc, i].must_write) U state[i].may_read
*
* Where:
* - `U` stands for set union
* - `/` stands for set difference;
* - `cc` stands for a call chain;
* - `i` and `s` are instruction indexes;
*
* The above equations are computed for each call chain and instruction
* index until state stops changing.
* - Additionally, in order to transfer "must_write" information from a
* subprogram to call instructions invoking this subprogram,
* the "must_write_acc" set is tracked for each (cc, i) tuple.
* A set of stack slots that are guaranteed to be written by this
* instruction or any of its successors (within the subprogram).
* The equation for "must_write_acc" propagation looks as follows:
*
* state[cc, i].must_write_acc =
* ∩ [state[cc, s].must_write_acc for s in insn_successors(i)]
* U state[cc, i].must_write
*
* (An intersection of all "must_write_acc" for instruction successors
* plus all "must_write" slots for the instruction itself).
* - After the propagation phase completes for a subprogram, information from
* (cc, 0) tuple (subprogram entry) is transferred to the caller's call chain:
* - "must_write_acc" set is intersected with the call site's "must_write" set;
* - "may_read" set is added to the call site's "may_read" set.
* - Any live stack queries must be taken after the propagation phase.
* - Accumulation and propagation phases can be entered multiple times,
* at any point in time:
* - "may_read" set only grows;
* - "must_write" set only shrinks;
* - for each visited verifier state with zero branches, all relevant
* read and write marks are already recorded by the analysis instance.
*
* Technically, the analysis is facilitated by the following data structures:
* - Call chain: for given verifier state, the call chain is a tuple of call
* instruction indexes leading to the current subprogram plus the subprogram
* entry point index.
* - Function instance: for a given call chain, for each instruction in
* the current subprogram, a mapping between instruction index and a
* set of "may_read", "must_write" and other marks accumulated for this
* instruction.
* - A hash table mapping call chains to function instances.
*/
struct callchain {
u32 callsites[MAX_CALL_FRAMES]; /* instruction pointer for each frame */
/* cached subprog_info[*].start for functions owning the frames:
* - sp_starts[curframe] used to get insn relative index within current function;
* - sp_starts[0..current-1] used for fast callchain_frame_up().
*/
u32 sp_starts[MAX_CALL_FRAMES];
u32 curframe; /* depth of callsites and sp_starts arrays */
};
struct per_frame_masks {
u64 may_read; /* stack slots that may be read by this instruction */
u64 must_write; /* stack slots written by this instruction */
u64 must_write_acc; /* stack slots written by this instruction and its successors */
u64 live_before; /* stack slots that may be read by this insn and its successors */
};
/*
* A function instance created for a specific callchain.
* Encapsulates read and write marks for each instruction in the function.
* Marks are tracked for each frame in the callchain.
*/
struct func_instance {
struct hlist_node hl_node;
struct callchain callchain;
u32 insn_cnt; /* cached number of insns in the function */
bool updated;
bool must_write_dropped;
/* Per frame, per instruction masks, frames allocated lazily. */
struct per_frame_masks *frames[MAX_CALL_FRAMES];
/* For each instruction a flag telling if "must_write" had been initialized for it. */
bool *must_write_set;
};
struct live_stack_query {
struct func_instance *instances[MAX_CALL_FRAMES]; /* valid in range [0..curframe] */
u32 curframe;
u32 insn_idx;
};
struct bpf_liveness {
DECLARE_HASHTABLE(func_instances, 8); /* maps callchain to func_instance */
struct live_stack_query live_stack_query; /* cache to avoid repetitive ht lookups */
/* Cached instance corresponding to env->cur_state, avoids per-instruction ht lookup */
struct func_instance *cur_instance;
/*
* Below fields are used to accumulate stack write marks for instruction at
* @write_insn_idx before submitting the marks to @cur_instance.
*/
u64 write_masks_acc[MAX_CALL_FRAMES];
u32 write_insn_idx;
};
/* Compute callchain corresponding to state @st at depth @frameno */
static void compute_callchain(struct bpf_verifier_env *env, struct bpf_verifier_state *st,
struct callchain *callchain, u32 frameno)
{
struct bpf_subprog_info *subprog_info = env->subprog_info;
u32 i;
memset(callchain, 0, sizeof(*callchain));
for (i = 0; i <= frameno; i++) {
callchain->sp_starts[i] = subprog_info[st->frame[i]->subprogno].start;
if (i < st->curframe)
callchain->callsites[i] = st->frame[i + 1]->callsite;
}
callchain->curframe = frameno;
callchain->callsites[callchain->curframe] = callchain->sp_starts[callchain->curframe];
}
static u32 hash_callchain(struct callchain *callchain)
{
return jhash2(callchain->callsites, callchain->curframe, 0);
}
static bool same_callsites(struct callchain *a, struct callchain *b)
{
int i;
if (a->curframe != b->curframe)
return false;
for (i = a->curframe; i >= 0; i--)
if (a->callsites[i] != b->callsites[i])
return false;
return true;
}
/*
* Find existing or allocate new function instance corresponding to @callchain.
* Instances are accumulated in env->liveness->func_instances and persist
* until the end of the verification process.
*/
static struct func_instance *__lookup_instance(struct bpf_verifier_env *env,
struct callchain *callchain)
{
struct bpf_liveness *liveness = env->liveness;
struct bpf_subprog_info *subprog;
struct func_instance *result;
u32 subprog_sz, size, key;
key = hash_callchain(callchain);
hash_for_each_possible(liveness->func_instances, result, hl_node, key)
if (same_callsites(&result->callchain, callchain))
return result;
subprog = bpf_find_containing_subprog(env, callchain->sp_starts[callchain->curframe]);
subprog_sz = (subprog + 1)->start - subprog->start;
size = sizeof(struct func_instance);
result = kvzalloc(size, GFP_KERNEL_ACCOUNT);
if (!result)
return ERR_PTR(-ENOMEM);
result->must_write_set = kvcalloc(subprog_sz, sizeof(*result->must_write_set),
GFP_KERNEL_ACCOUNT);
if (!result->must_write_set)
return ERR_PTR(-ENOMEM);
memcpy(&result->callchain, callchain, sizeof(*callchain));
result->insn_cnt = subprog_sz;
hash_add(liveness->func_instances, &result->hl_node, key);
return result;
}
static struct func_instance *lookup_instance(struct bpf_verifier_env *env,
struct bpf_verifier_state *st,
u32 frameno)
{
struct callchain callchain;
compute_callchain(env, st, &callchain, frameno);
return __lookup_instance(env, &callchain);
}
int bpf_stack_liveness_init(struct bpf_verifier_env *env)
{
env->liveness = kvzalloc(sizeof(*env->liveness), GFP_KERNEL_ACCOUNT);
if (!env->liveness)
return -ENOMEM;
hash_init(env->liveness->func_instances);
return 0;
}
void bpf_stack_liveness_free(struct bpf_verifier_env *env)
{
struct func_instance *instance;
struct hlist_node *tmp;
int bkt, i;
if (!env->liveness)
return;
hash_for_each_safe(env->liveness->func_instances, bkt, tmp, instance, hl_node) {
for (i = 0; i <= instance->callchain.curframe; i++)
kvfree(instance->frames[i]);
kvfree(instance->must_write_set);
kvfree(instance);
}
kvfree(env->liveness);
}
/*
* Convert absolute instruction index @insn_idx to an index relative
* to start of the function corresponding to @instance.
*/
static int relative_idx(struct func_instance *instance, u32 insn_idx)
{
return insn_idx - instance->callchain.sp_starts[instance->callchain.curframe];
}
static struct per_frame_masks *get_frame_masks(struct func_instance *instance,
u32 frame, u32 insn_idx)
{
if (!instance->frames[frame])
return NULL;
return &instance->frames[frame][relative_idx(instance, insn_idx)];
}
static struct per_frame_masks *alloc_frame_masks(struct bpf_verifier_env *env,
struct func_instance *instance,
u32 frame, u32 insn_idx)
{
struct per_frame_masks *arr;
if (!instance->frames[frame]) {
arr = kvcalloc(instance->insn_cnt, sizeof(*arr), GFP_KERNEL_ACCOUNT);
instance->frames[frame] = arr;
if (!arr)
return ERR_PTR(-ENOMEM);
}
return get_frame_masks(instance, frame, insn_idx);
}
void bpf_reset_live_stack_callchain(struct bpf_verifier_env *env)
{
env->liveness->cur_instance = NULL;
}
/* If @env->liveness->cur_instance is null, set it to instance corresponding to @env->cur_state. */
static int ensure_cur_instance(struct bpf_verifier_env *env)
{
struct bpf_liveness *liveness = env->liveness;
struct func_instance *instance;
if (liveness->cur_instance)
return 0;
instance = lookup_instance(env, env->cur_state, env->cur_state->curframe);
if (IS_ERR(instance))
return PTR_ERR(instance);
liveness->cur_instance = instance;
return 0;
}
/* Accumulate may_read masks for @frame at @insn_idx */
static int mark_stack_read(struct bpf_verifier_env *env,
struct func_instance *instance, u32 frame, u32 insn_idx, u64 mask)
{
struct per_frame_masks *masks;
u64 new_may_read;
masks = alloc_frame_masks(env, instance, frame, insn_idx);
if (IS_ERR(masks))
return PTR_ERR(masks);
new_may_read = masks->may_read | mask;
if (new_may_read != masks->may_read &&
((new_may_read | masks->live_before) != masks->live_before))
instance->updated = true;
masks->may_read |= mask;
return 0;
}
int bpf_mark_stack_read(struct bpf_verifier_env *env, u32 frame, u32 insn_idx, u64 mask)
{
int err;
err = ensure_cur_instance(env);
err = err ?: mark_stack_read(env, env->liveness->cur_instance, frame, insn_idx, mask);
return err;
}
static void reset_stack_write_marks(struct bpf_verifier_env *env,
struct func_instance *instance, u32 insn_idx)
{
struct bpf_liveness *liveness = env->liveness;
int i;
liveness->write_insn_idx = insn_idx;
for (i = 0; i <= instance->callchain.curframe; i++)
liveness->write_masks_acc[i] = 0;
}
int bpf_reset_stack_write_marks(struct bpf_verifier_env *env, u32 insn_idx)
{
struct bpf_liveness *liveness = env->liveness;
int err;
err = ensure_cur_instance(env);
if (err)
return err;
reset_stack_write_marks(env, liveness->cur_instance, insn_idx);
return 0;
}
void bpf_mark_stack_write(struct bpf_verifier_env *env, u32 frame, u64 mask)
{
env->liveness->write_masks_acc[frame] |= mask;
}
static int commit_stack_write_marks(struct bpf_verifier_env *env,
struct func_instance *instance)
{
struct bpf_liveness *liveness = env->liveness;
u32 idx, frame, curframe, old_must_write;
struct per_frame_masks *masks;
u64 mask;
if (!instance)
return 0;
curframe = instance->callchain.curframe;
idx = relative_idx(instance, liveness->write_insn_idx);
for (frame = 0; frame <= curframe; frame++) {
mask = liveness->write_masks_acc[frame];
/* avoid allocating frames for zero masks */
if (mask == 0 && !instance->must_write_set[idx])
continue;
masks = alloc_frame_masks(env, instance, frame, liveness->write_insn_idx);
if (IS_ERR(masks))
return PTR_ERR(masks);
old_must_write = masks->must_write;
/*
* If instruction at this callchain is seen for a first time, set must_write equal
* to @mask. Otherwise take intersection with the previous value.
*/
if (instance->must_write_set[idx])
mask &= old_must_write;
if (old_must_write != mask) {
masks->must_write = mask;
instance->updated = true;
}
if (old_must_write & ~mask)
instance->must_write_dropped = true;
}
instance->must_write_set[idx] = true;
liveness->write_insn_idx = 0;
return 0;
}
/*
* Merge stack writes marks in @env->liveness->write_masks_acc
* with information already in @env->liveness->cur_instance.
*/
int bpf_commit_stack_write_marks(struct bpf_verifier_env *env)
{
return commit_stack_write_marks(env, env->liveness->cur_instance);
}
static char *fmt_callchain(struct bpf_verifier_env *env, struct callchain *callchain)
{
char *buf_end = env->tmp_str_buf + sizeof(env->tmp_str_buf);
char *buf = env->tmp_str_buf;
int i;
buf += snprintf(buf, buf_end - buf, "(");
for (i = 0; i <= callchain->curframe; i++)
buf += snprintf(buf, buf_end - buf, "%s%d", i ? "," : "", callchain->callsites[i]);
snprintf(buf, buf_end - buf, ")");
return env->tmp_str_buf;
}
static void log_mask_change(struct bpf_verifier_env *env, struct callchain *callchain,
char *pfx, u32 frame, u32 insn_idx, u64 old, u64 new)
{
u64 changed_bits = old ^ new;
u64 new_ones = new & changed_bits;
u64 new_zeros = ~new & changed_bits;
if (!changed_bits)
return;
bpf_log(&env->log, "%s frame %d insn %d ", fmt_callchain(env, callchain), frame, insn_idx);
if (new_ones) {
bpf_fmt_stack_mask(env->tmp_str_buf, sizeof(env->tmp_str_buf), new_ones);
bpf_log(&env->log, "+%s %s ", pfx, env->tmp_str_buf);
}
if (new_zeros) {
bpf_fmt_stack_mask(env->tmp_str_buf, sizeof(env->tmp_str_buf), new_zeros);
bpf_log(&env->log, "-%s %s", pfx, env->tmp_str_buf);
}
bpf_log(&env->log, "\n");
}
int bpf_jmp_offset(struct bpf_insn *insn)
{
u8 code = insn->code;
if (code == (BPF_JMP32 | BPF_JA))
return insn->imm;
return insn->off;
}
__diag_push();
__diag_ignore_all("-Woverride-init", "Allow field initialization overrides for opcode_info_tbl");
inline int bpf_insn_successors(struct bpf_prog *prog, u32 idx, u32 succ[2])
{
static const struct opcode_info {
bool can_jump;
bool can_fallthrough;
} opcode_info_tbl[256] = {
[0 ... 255] = {.can_jump = false, .can_fallthrough = true},
#define _J(code, ...) \
[BPF_JMP | code] = __VA_ARGS__, \
[BPF_JMP32 | code] = __VA_ARGS__
_J(BPF_EXIT, {.can_jump = false, .can_fallthrough = false}),
_J(BPF_JA, {.can_jump = true, .can_fallthrough = false}),
_J(BPF_JEQ, {.can_jump = true, .can_fallthrough = true}),
_J(BPF_JNE, {.can_jump = true, .can_fallthrough = true}),
_J(BPF_JLT, {.can_jump = true, .can_fallthrough = true}),
_J(BPF_JLE, {.can_jump = true, .can_fallthrough = true}),
_J(BPF_JGT, {.can_jump = true, .can_fallthrough = true}),
_J(BPF_JGE, {.can_jump = true, .can_fallthrough = true}),
_J(BPF_JSGT, {.can_jump = true, .can_fallthrough = true}),
_J(BPF_JSGE, {.can_jump = true, .can_fallthrough = true}),
_J(BPF_JSLT, {.can_jump = true, .can_fallthrough = true}),
_J(BPF_JSLE, {.can_jump = true, .can_fallthrough = true}),
_J(BPF_JCOND, {.can_jump = true, .can_fallthrough = true}),
_J(BPF_JSET, {.can_jump = true, .can_fallthrough = true}),
#undef _J
};
struct bpf_insn *insn = &prog->insnsi[idx];
const struct opcode_info *opcode_info;
int i = 0, insn_sz;
opcode_info = &opcode_info_tbl[BPF_CLASS(insn->code) | BPF_OP(insn->code)];
insn_sz = bpf_is_ldimm64(insn) ? 2 : 1;
if (opcode_info->can_fallthrough)
succ[i++] = idx + insn_sz;
if (opcode_info->can_jump)
succ[i++] = idx + bpf_jmp_offset(insn) + 1;
return i;
}
__diag_pop();
static struct func_instance *get_outer_instance(struct bpf_verifier_env *env,
struct func_instance *instance)
{
struct callchain callchain = instance->callchain;
/* Adjust @callchain to represent callchain one frame up */
callchain.callsites[callchain.curframe] = 0;
callchain.sp_starts[callchain.curframe] = 0;
callchain.curframe--;
callchain.callsites[callchain.curframe] = callchain.sp_starts[callchain.curframe];
return __lookup_instance(env, &callchain);
}
static u32 callchain_subprog_start(struct callchain *callchain)
{
return callchain->sp_starts[callchain->curframe];
}
/*
* Transfer @may_read and @must_write_acc marks from the first instruction of @instance,
* to the call instruction in function instance calling @instance.
*/
static int propagate_to_outer_instance(struct bpf_verifier_env *env,
struct func_instance *instance)
{
struct callchain *callchain = &instance->callchain;
u32 this_subprog_start, callsite, frame;
struct func_instance *outer_instance;
struct per_frame_masks *insn;
int err;
this_subprog_start = callchain_subprog_start(callchain);
outer_instance = get_outer_instance(env, instance);
callsite = callchain->callsites[callchain->curframe - 1];
reset_stack_write_marks(env, outer_instance, callsite);
for (frame = 0; frame < callchain->curframe; frame++) {
insn = get_frame_masks(instance, frame, this_subprog_start);
if (!insn)
continue;
bpf_mark_stack_write(env, frame, insn->must_write_acc);
err = mark_stack_read(env, outer_instance, frame, callsite, insn->live_before);
if (err)
return err;
}
commit_stack_write_marks(env, outer_instance);
return 0;
}
static inline bool update_insn(struct bpf_verifier_env *env,
struct func_instance *instance, u32 frame, u32 insn_idx)
{
struct bpf_insn_aux_data *aux = env->insn_aux_data;
u64 new_before, new_after, must_write_acc;
struct per_frame_masks *insn, *succ_insn;
u32 succ_num, s, succ[2];
bool changed;
succ_num = bpf_insn_successors(env->prog, insn_idx, succ);
if (unlikely(succ_num == 0))
return false;
changed = false;
insn = get_frame_masks(instance, frame, insn_idx);
new_before = 0;
new_after = 0;
/*
* New "must_write_acc" is an intersection of all "must_write_acc"
* of successors plus all "must_write" slots of instruction itself.
*/
must_write_acc = U64_MAX;
for (s = 0; s < succ_num; ++s) {
succ_insn = get_frame_masks(instance, frame, succ[s]);
new_after |= succ_insn->live_before;
must_write_acc &= succ_insn->must_write_acc;
}
must_write_acc |= insn->must_write;
/*
* New "live_before" is a union of all "live_before" of successors
* minus slots written by instruction plus slots read by instruction.
*/
new_before = (new_after & ~insn->must_write) | insn->may_read;
changed |= new_before != insn->live_before;
changed |= must_write_acc != insn->must_write_acc;
if (unlikely(env->log.level & BPF_LOG_LEVEL2) &&
(insn->may_read || insn->must_write ||
insn_idx == callchain_subprog_start(&instance->callchain) ||
aux[insn_idx].prune_point)) {
log_mask_change(env, &instance->callchain, "live",
frame, insn_idx, insn->live_before, new_before);
log_mask_change(env, &instance->callchain, "written",
frame, insn_idx, insn->must_write_acc, must_write_acc);
}
insn->live_before = new_before;
insn->must_write_acc = must_write_acc;
return changed;
}
/* Fixed-point computation of @live_before and @must_write_acc marks */
static int update_instance(struct bpf_verifier_env *env, struct func_instance *instance)
{
u32 i, frame, po_start, po_end, cnt, this_subprog_start;
struct callchain *callchain = &instance->callchain;
int *insn_postorder = env->cfg.insn_postorder;
struct bpf_subprog_info *subprog;
struct per_frame_masks *insn;
bool changed;
int err;
this_subprog_start = callchain_subprog_start(callchain);
/*
* If must_write marks were updated must_write_acc needs to be reset
* (to account for the case when new must_write sets became smaller).
*/
if (instance->must_write_dropped) {
for (frame = 0; frame <= callchain->curframe; frame++) {
if (!instance->frames[frame])
continue;
for (i = 0; i < instance->insn_cnt; i++) {
insn = get_frame_masks(instance, frame, this_subprog_start + i);
insn->must_write_acc = 0;
}
}
}
subprog = bpf_find_containing_subprog(env, this_subprog_start);
po_start = subprog->postorder_start;
po_end = (subprog + 1)->postorder_start;
cnt = 0;
/* repeat until fixed point is reached */
do {
cnt++;
changed = false;
for (frame = 0; frame <= instance->callchain.curframe; frame++) {
if (!instance->frames[frame])
continue;
for (i = po_start; i < po_end; i++)
changed |= update_insn(env, instance, frame, insn_postorder[i]);
}
} while (changed);
if (env->log.level & BPF_LOG_LEVEL2)
bpf_log(&env->log, "%s live stack update done in %d iterations\n",
fmt_callchain(env, callchain), cnt);
/* transfer marks accumulated for outer frames to outer func instance (caller) */
if (callchain->curframe > 0) {
err = propagate_to_outer_instance(env, instance);
if (err)
return err;
}
return 0;
}
/*
* Prepare all callchains within @env->cur_state for querying.
* This function should be called after each verifier.c:pop_stack()
* and whenever verifier.c:do_check_insn() processes subprogram exit.
* This would guarantee that visited verifier states with zero branches
* have their bpf_mark_stack_{read,write}() effects propagated in
* @env->liveness.
*/
int bpf_update_live_stack(struct bpf_verifier_env *env)
{
struct func_instance *instance;
int err, frame;
bpf_reset_live_stack_callchain(env);
for (frame = env->cur_state->curframe; frame >= 0; --frame) {
instance = lookup_instance(env, env->cur_state, frame);
if (IS_ERR(instance))
return PTR_ERR(instance);
if (instance->updated) {
err = update_instance(env, instance);
if (err)
return err;
instance->updated = false;
instance->must_write_dropped = false;
}
}
return 0;
}
static bool is_live_before(struct func_instance *instance, u32 insn_idx, u32 frameno, u32 spi)
{
struct per_frame_masks *masks;
masks = get_frame_masks(instance, frameno, insn_idx);
return masks && (masks->live_before & BIT(spi));
}
int bpf_live_stack_query_init(struct bpf_verifier_env *env, struct bpf_verifier_state *st)
{
struct live_stack_query *q = &env->liveness->live_stack_query;
struct func_instance *instance;
u32 frame;
memset(q, 0, sizeof(*q));
for (frame = 0; frame <= st->curframe; frame++) {
instance = lookup_instance(env, st, frame);
if (IS_ERR(instance))
return PTR_ERR(instance);
q->instances[frame] = instance;
}
q->curframe = st->curframe;
q->insn_idx = st->insn_idx;
return 0;
}
bool bpf_stack_slot_alive(struct bpf_verifier_env *env, u32 frameno, u32 spi)
{
/*
* Slot is alive if it is read before q->st->insn_idx in current func instance,
* or if for some outer func instance:
* - alive before callsite if callsite calls callback, otherwise
* - alive after callsite
*/
struct live_stack_query *q = &env->liveness->live_stack_query;
struct func_instance *instance, *curframe_instance;
u32 i, callsite;
bool alive;
curframe_instance = q->instances[q->curframe];
if (is_live_before(curframe_instance, q->insn_idx, frameno, spi))
return true;
for (i = frameno; i < q->curframe; i++) {
callsite = curframe_instance->callchain.callsites[i];
instance = q->instances[i];
alive = bpf_calls_callback(env, callsite)
? is_live_before(instance, callsite, frameno, spi)
: is_live_before(instance, callsite + 1, frameno, spi);
if (alive)
return true;
}
return false;
}

28
kernel/bpf/log.c
View File

@ -542,19 +542,6 @@ static char slot_type_char[] = {
[STACK_IRQ_FLAG] = 'f'
};
static void print_liveness(struct bpf_verifier_env *env,
enum bpf_reg_liveness live)
{
if (live & (REG_LIVE_READ | REG_LIVE_WRITTEN | REG_LIVE_DONE))
verbose(env, "_");
if (live & REG_LIVE_READ)
verbose(env, "r");
if (live & REG_LIVE_WRITTEN)
verbose(env, "w");
if (live & REG_LIVE_DONE)
verbose(env, "D");
}
#define UNUM_MAX_DECIMAL U16_MAX
#define SNUM_MAX_DECIMAL S16_MAX
#define SNUM_MIN_DECIMAL S16_MIN
@ -772,7 +759,6 @@ void print_verifier_state(struct bpf_verifier_env *env, const struct bpf_verifie
if (!print_all && !reg_scratched(env, i))
continue;
verbose(env, " R%d", i);
print_liveness(env, reg->live);
verbose(env, "=");
print_reg_state(env, state, reg);
}
@ -805,9 +791,7 @@ void print_verifier_state(struct bpf_verifier_env *env, const struct bpf_verifie
break;
types_buf[j] = '\0';
verbose(env, " fp%d", (-i - 1) * BPF_REG_SIZE);
print_liveness(env, reg->live);
verbose(env, "=%s", types_buf);
verbose(env, " fp%d=%s", (-i - 1) * BPF_REG_SIZE, types_buf);
print_reg_state(env, state, reg);
break;
case STACK_DYNPTR:
@ -816,7 +800,6 @@ void print_verifier_state(struct bpf_verifier_env *env, const struct bpf_verifie
reg = &state->stack[i].spilled_ptr;
verbose(env, " fp%d", (-i - 1) * BPF_REG_SIZE);
print_liveness(env, reg->live);
verbose(env, "=dynptr_%s(", dynptr_type_str(reg->dynptr.type));
if (reg->id)
verbose_a("id=%d", reg->id);
@ -831,9 +814,8 @@ void print_verifier_state(struct bpf_verifier_env *env, const struct bpf_verifie
if (!reg->ref_obj_id)
continue;
verbose(env, " fp%d", (-i - 1) * BPF_REG_SIZE);
print_liveness(env, reg->live);
verbose(env, "=iter_%s(ref_id=%d,state=%s,depth=%u)",
verbose(env, " fp%d=iter_%s(ref_id=%d,state=%s,depth=%u)",
(-i - 1) * BPF_REG_SIZE,
iter_type_str(reg->iter.btf, reg->iter.btf_id),
reg->ref_obj_id, iter_state_str(reg->iter.state),
reg->iter.depth);
@ -841,9 +823,7 @@ void print_verifier_state(struct bpf_verifier_env *env, const struct bpf_verifie
case STACK_MISC:
case STACK_ZERO:
default:
verbose(env, " fp%d", (-i - 1) * BPF_REG_SIZE);
print_liveness(env, reg->live);
verbose(env, "=%s", types_buf);
verbose(env, " fp%d=%s", (-i - 1) * BPF_REG_SIZE, types_buf);
break;
}
}

568
kernel/bpf/verifier.c
View File

File diff suppressed because it is too large Load Diff

178
tools/testing/selftests/bpf/prog_tests/align.c
View File

@ -42,11 +42,11 @@ static struct bpf_align_test tests[] = {
.matches = {
{0, "R1", "ctx()"},
{0, "R10", "fp0"},
{0, "R3_w", "2"},
{1, "R3_w", "4"},
{2, "R3_w", "8"},
{3, "R3_w", "16"},
{4, "R3_w", "32"},
{0, "R3", "2"},
{1, "R3", "4"},
{2, "R3", "8"},
{3, "R3", "16"},
{4, "R3", "32"},
},
},
{
@ -70,17 +70,17 @@ static struct bpf_align_test tests[] = {
.matches = {
{0, "R1", "ctx()"},
{0, "R10", "fp0"},
{0, "R3_w", "1"},
{1, "R3_w", "2"},
{2, "R3_w", "4"},
{3, "R3_w", "8"},
{4, "R3_w", "16"},
{5, "R3_w", "1"},
{6, "R4_w", "32"},
{7, "R4_w", "16"},
{8, "R4_w", "8"},
{9, "R4_w", "4"},
{10, "R4_w", "2"},
{0, "R3", "1"},
{1, "R3", "2"},
{2, "R3", "4"},
{3, "R3", "8"},
{4, "R3", "16"},
{5, "R3", "1"},
{6, "R4", "32"},
{7, "R4", "16"},
{8, "R4", "8"},
{9, "R4", "4"},
{10, "R4", "2"},
},
},
{
@ -99,12 +99,12 @@ static struct bpf_align_test tests[] = {
.matches = {
{0, "R1", "ctx()"},
{0, "R10", "fp0"},
{0, "R3_w", "4"},
{1, "R3_w", "8"},
{2, "R3_w", "10"},
{3, "R4_w", "8"},
{4, "R4_w", "12"},
{5, "R4_w", "14"},
{0, "R3", "4"},
{1, "R3", "8"},
{2, "R3", "10"},
{3, "R4", "8"},
{4, "R4", "12"},
{5, "R4", "14"},
},
},
{
@ -121,10 +121,10 @@ static struct bpf_align_test tests[] = {
.matches = {
{0, "R1", "ctx()"},
{0, "R10", "fp0"},
{0, "R3_w", "7"},
{1, "R3_w", "7"},
{2, "R3_w", "14"},
{3, "R3_w", "56"},
{0, "R3", "7"},
{1, "R3", "7"},
{2, "R3", "14"},
{3, "R3", "56"},
},
},
@ -162,19 +162,19 @@ static struct bpf_align_test tests[] = {
},
.prog_type = BPF_PROG_TYPE_SCHED_CLS,
.matches = {
{6, "R0_w", "pkt(off=8,r=8)"},
{6, "R3_w", "var_off=(0x0; 0xff)"},
{7, "R3_w", "var_off=(0x0; 0x1fe)"},
{8, "R3_w", "var_off=(0x0; 0x3fc)"},
{9, "R3_w", "var_off=(0x0; 0x7f8)"},
{10, "R3_w", "var_off=(0x0; 0xff0)"},
{12, "R3_w", "pkt_end()"},
{17, "R4_w", "var_off=(0x0; 0xff)"},
{18, "R4_w", "var_off=(0x0; 0x1fe0)"},
{19, "R4_w", "var_off=(0x0; 0xff0)"},
{20, "R4_w", "var_off=(0x0; 0x7f8)"},
{21, "R4_w", "var_off=(0x0; 0x3fc)"},
{22, "R4_w", "var_off=(0x0; 0x1fe)"},
{6, "R0", "pkt(off=8,r=8)"},
{6, "R3", "var_off=(0x0; 0xff)"},
{7, "R3", "var_off=(0x0; 0x1fe)"},
{8, "R3", "var_off=(0x0; 0x3fc)"},
{9, "R3", "var_off=(0x0; 0x7f8)"},
{10, "R3", "var_off=(0x0; 0xff0)"},
{12, "R3", "pkt_end()"},
{17, "R4", "var_off=(0x0; 0xff)"},
{18, "R4", "var_off=(0x0; 0x1fe0)"},
{19, "R4", "var_off=(0x0; 0xff0)"},
{20, "R4", "var_off=(0x0; 0x7f8)"},
{21, "R4", "var_off=(0x0; 0x3fc)"},
{22, "R4", "var_off=(0x0; 0x1fe)"},
},
},
{
@ -195,16 +195,16 @@ static struct bpf_align_test tests[] = {
},
.prog_type = BPF_PROG_TYPE_SCHED_CLS,
.matches = {
{6, "R3_w", "var_off=(0x0; 0xff)"},
{7, "R4_w", "var_off=(0x0; 0xff)"},
{8, "R4_w", "var_off=(0x0; 0xff)"},
{9, "R4_w", "var_off=(0x0; 0xff)"},
{10, "R4_w", "var_off=(0x0; 0x1fe)"},
{11, "R4_w", "var_off=(0x0; 0xff)"},
{12, "R4_w", "var_off=(0x0; 0x3fc)"},
{13, "R4_w", "var_off=(0x0; 0xff)"},
{14, "R4_w", "var_off=(0x0; 0x7f8)"},
{15, "R4_w", "var_off=(0x0; 0xff0)"},
{6, "R3", "var_off=(0x0; 0xff)"},
{7, "R4", "var_off=(0x0; 0xff)"},
{8, "R4", "var_off=(0x0; 0xff)"},
{9, "R4", "var_off=(0x0; 0xff)"},
{10, "R4", "var_off=(0x0; 0x1fe)"},
{11, "R4", "var_off=(0x0; 0xff)"},
{12, "R4", "var_off=(0x0; 0x3fc)"},
{13, "R4", "var_off=(0x0; 0xff)"},
{14, "R4", "var_off=(0x0; 0x7f8)"},
{15, "R4", "var_off=(0x0; 0xff0)"},
},
},
{
@ -235,14 +235,14 @@ static struct bpf_align_test tests[] = {
},
.prog_type = BPF_PROG_TYPE_SCHED_CLS,
.matches = {
{2, "R5_w", "pkt(r=0)"},
{4, "R5_w", "pkt(off=14,r=0)"},
{5, "R4_w", "pkt(off=14,r=0)"},
{2, "R5", "pkt(r=0)"},
{4, "R5", "pkt(off=14,r=0)"},
{5, "R4", "pkt(off=14,r=0)"},
{9, "R2", "pkt(r=18)"},
{10, "R5", "pkt(off=14,r=18)"},
{10, "R4_w", "var_off=(0x0; 0xff)"},
{13, "R4_w", "var_off=(0x0; 0xffff)"},
{14, "R4_w", "var_off=(0x0; 0xffff)"},
{10, "R4", "var_off=(0x0; 0xff)"},
{13, "R4", "var_off=(0x0; 0xffff)"},
{14, "R4", "var_off=(0x0; 0xffff)"},
},
},
{
@ -299,12 +299,12 @@ static struct bpf_align_test tests[] = {
/* Calculated offset in R6 has unknown value, but known
* alignment of 4.
*/
{6, "R2_w", "pkt(r=8)"},
{7, "R6_w", "var_off=(0x0; 0x3fc)"},
{6, "R2", "pkt(r=8)"},
{7, "R6", "var_off=(0x0; 0x3fc)"},
/* Offset is added to packet pointer R5, resulting in
* known fixed offset, and variable offset from R6.
*/
{11, "R5_w", "pkt(id=1,off=14,"},
{11, "R5", "pkt(id=1,off=14,"},
/* At the time the word size load is performed from R5,
* it's total offset is NET_IP_ALIGN + reg->off (0) +
* reg->aux_off (14) which is 16. Then the variable
@ -320,12 +320,12 @@ static struct bpf_align_test tests[] = {
* instruction to validate R5 state. We also check
* that R4 is what it should be in such case.
*/
{18, "R4_w", "var_off=(0x0; 0x3fc)"},
{18, "R5_w", "var_off=(0x0; 0x3fc)"},
{18, "R4", "var_off=(0x0; 0x3fc)"},
{18, "R5", "var_off=(0x0; 0x3fc)"},
/* Constant offset is added to R5, resulting in
* reg->off of 14.
*/
{19, "R5_w", "pkt(id=2,off=14,"},
{19, "R5", "pkt(id=2,off=14,"},
/* At the time the word size load is performed from R5,
* its total fixed offset is NET_IP_ALIGN + reg->off
* (14) which is 16. Then the variable offset is 4-byte
@ -337,21 +337,21 @@ static struct bpf_align_test tests[] = {
/* Constant offset is added to R5 packet pointer,
* resulting in reg->off value of 14.
*/
{26, "R5_w", "pkt(off=14,r=8)"},
{26, "R5", "pkt(off=14,r=8)"},
/* Variable offset is added to R5, resulting in a
* variable offset of (4n). See comment for insn #18
* for R4 = R5 trick.
*/
{28, "R4_w", "var_off=(0x0; 0x3fc)"},
{28, "R5_w", "var_off=(0x0; 0x3fc)"},
{28, "R4", "var_off=(0x0; 0x3fc)"},
{28, "R5", "var_off=(0x0; 0x3fc)"},
/* Constant is added to R5 again, setting reg->off to 18. */
{29, "R5_w", "pkt(id=3,off=18,"},
{29, "R5", "pkt(id=3,off=18,"},
/* And once more we add a variable; resulting var_off
* is still (4n), fixed offset is not changed.
* Also, we create a new reg->id.
*/
{31, "R4_w", "var_off=(0x0; 0x7fc)"},
{31, "R5_w", "var_off=(0x0; 0x7fc)"},
{31, "R4", "var_off=(0x0; 0x7fc)"},
{31, "R5", "var_off=(0x0; 0x7fc)"},
/* At the time the word size load is performed from R5,
* its total fixed offset is NET_IP_ALIGN + reg->off (18)
* which is 20. Then the variable offset is (4n), so
@ -397,12 +397,12 @@ static struct bpf_align_test tests[] = {
/* Calculated offset in R6 has unknown value, but known
* alignment of 4.
*/
{6, "R2_w", "pkt(r=8)"},
{7, "R6_w", "var_off=(0x0; 0x3fc)"},
{6, "R2", "pkt(r=8)"},
{7, "R6", "var_off=(0x0; 0x3fc)"},
/* Adding 14 makes R6 be (4n+2) */
{8, "R6_w", "var_off=(0x2; 0x7fc)"},
{8, "R6", "var_off=(0x2; 0x7fc)"},
/* Packet pointer has (4n+2) offset */
{11, "R5_w", "var_off=(0x2; 0x7fc)"},
{11, "R5", "var_off=(0x2; 0x7fc)"},
{12, "R4", "var_off=(0x2; 0x7fc)"},
/* At the time the word size load is performed from R5,
* its total fixed offset is NET_IP_ALIGN + reg->off (0)
@ -414,11 +414,11 @@ static struct bpf_align_test tests[] = {
/* Newly read value in R6 was shifted left by 2, so has
* known alignment of 4.
*/
{17, "R6_w", "var_off=(0x0; 0x3fc)"},
{17, "R6", "var_off=(0x0; 0x3fc)"},
/* Added (4n) to packet pointer's (4n+2) var_off, giving
* another (4n+2).
*/
{19, "R5_w", "var_off=(0x2; 0xffc)"},
{19, "R5", "var_off=(0x2; 0xffc)"},
{20, "R4", "var_off=(0x2; 0xffc)"},
/* At the time the word size load is performed from R5,
* its total fixed offset is NET_IP_ALIGN + reg->off (0)
@ -459,18 +459,18 @@ static struct bpf_align_test tests[] = {
.prog_type = BPF_PROG_TYPE_SCHED_CLS,
.result = REJECT,
.matches = {
{3, "R5_w", "pkt_end()"},
{3, "R5", "pkt_end()"},
/* (ptr - ptr) << 2 == unknown, (4n) */
{5, "R5_w", "var_off=(0x0; 0xfffffffffffffffc)"},
{5, "R5", "var_off=(0x0; 0xfffffffffffffffc)"},
/* (4n) + 14 == (4n+2). We blow our bounds, because
* the add could overflow.
*/
{6, "R5_w", "var_off=(0x2; 0xfffffffffffffffc)"},
{6, "R5", "var_off=(0x2; 0xfffffffffffffffc)"},
/* Checked s>=0 */
{9, "R5", "var_off=(0x2; 0x7ffffffffffffffc)"},
/* packet pointer + nonnegative (4n+2) */
{11, "R6_w", "var_off=(0x2; 0x7ffffffffffffffc)"},
{12, "R4_w", "var_off=(0x2; 0x7ffffffffffffffc)"},
{11, "R6", "var_off=(0x2; 0x7ffffffffffffffc)"},
{12, "R4", "var_off=(0x2; 0x7ffffffffffffffc)"},
/* NET_IP_ALIGN + (4n+2) == (4n), alignment is fine.
* We checked the bounds, but it might have been able
* to overflow if the packet pointer started in the
@ -478,7 +478,7 @@ static struct bpf_align_test tests[] = {
* So we did not get a 'range' on R6, and the access
* attempt will fail.
*/
{15, "R6_w", "var_off=(0x2; 0x7ffffffffffffffc)"},
{15, "R6", "var_off=(0x2; 0x7ffffffffffffffc)"},
}
},
{
@ -513,12 +513,12 @@ static struct bpf_align_test tests[] = {
/* Calculated offset in R6 has unknown value, but known
* alignment of 4.
*/
{6, "R2_w", "pkt(r=8)"},
{8, "R6_w", "var_off=(0x0; 0x3fc)"},
{6, "R2", "pkt(r=8)"},
{8, "R6", "var_off=(0x0; 0x3fc)"},
/* Adding 14 makes R6 be (4n+2) */
{9, "R6_w", "var_off=(0x2; 0x7fc)"},
{9, "R6", "var_off=(0x2; 0x7fc)"},
/* New unknown value in R7 is (4n) */
{10, "R7_w", "var_off=(0x0; 0x3fc)"},
{10, "R7", "var_off=(0x0; 0x3fc)"},
/* Subtracting it from R6 blows our unsigned bounds */
{11, "R6", "var_off=(0x2; 0xfffffffffffffffc)"},
/* Checked s>= 0 */
@ -566,16 +566,16 @@ static struct bpf_align_test tests[] = {
/* Calculated offset in R6 has unknown value, but known
* alignment of 4.
*/
{6, "R2_w", "pkt(r=8)"},
{9, "R6_w", "var_off=(0x0; 0x3c)"},
{6, "R2", "pkt(r=8)"},
{9, "R6", "var_off=(0x0; 0x3c)"},
/* Adding 14 makes R6 be (4n+2) */
{10, "R6_w", "var_off=(0x2; 0x7c)"},
{10, "R6", "var_off=(0x2; 0x7c)"},
/* Subtracting from packet pointer overflows ubounds */
{13, "R5_w", "var_off=(0xffffffffffffff82; 0x7c)"},
{13, "R5", "var_off=(0xffffffffffffff82; 0x7c)"},
/* New unknown value in R7 is (4n), >= 76 */
{14, "R7_w", "var_off=(0x0; 0x7fc)"},
{14, "R7", "var_off=(0x0; 0x7fc)"},
/* Adding it to packet pointer gives nice bounds again */
{16, "R5_w", "var_off=(0x2; 0x7fc)"},
{16, "R5", "var_off=(0x2; 0x7fc)"},
/* At the time the word size load is performed from R5,
* its total fixed offset is NET_IP_ALIGN + reg->off (0)
* which is 2. Then the variable offset is (4n+2), so

125
tools/testing/selftests/bpf/prog_tests/prog_tests_framework.c
View File

@ -54,3 +54,128 @@ void test_prog_tests_framework(void)
return;
clear_test_state(state);
}
static void dummy_emit(const char *buf, bool force) {}
void test_prog_tests_framework_expected_msgs(void)
{
struct expected_msgs msgs;
int i, j, error_cnt;
const struct {
const char *name;
const char *log;
const char *expected;
struct expect_msg *pats;
} cases[] = {
{
.name = "simple-ok",
.log = "aaabbbccc",
.pats = (struct expect_msg[]) {
{ .substr = "aaa" },
{ .substr = "ccc" },
{}
}
},
{
.name = "simple-fail",
.log = "aaabbbddd",
.expected = "MATCHED SUBSTR: 'aaa'\n"
"EXPECTED SUBSTR: 'ccc'\n",
.pats = (struct expect_msg[]) {
{ .substr = "aaa" },
{ .substr = "ccc" },
{}
}
},
{
.name = "negative-ok-mid",
.log = "aaabbbccc",
.pats = (struct expect_msg[]) {
{ .substr = "aaa" },
{ .substr = "foo", .negative = true },
{ .substr = "bar", .negative = true },
{ .substr = "ccc" },
{}
}
},
{
.name = "negative-ok-tail",
.log = "aaabbbccc",
.pats = (struct expect_msg[]) {
{ .substr = "aaa" },
{ .substr = "foo", .negative = true },
{}
}
},
{
.name = "negative-ok-head",
.log = "aaabbbccc",
.pats = (struct expect_msg[]) {
{ .substr = "foo", .negative = true },
{ .substr = "ccc" },
{}
}
},
{
.name = "negative-fail-head",
.log = "aaabbbccc",
.expected = "UNEXPECTED SUBSTR: 'aaa'\n",
.pats = (struct expect_msg[]) {
{ .substr = "aaa", .negative = true },
{ .substr = "bbb" },
{}
}
},
{
.name = "negative-fail-tail",
.log = "aaabbbccc",
.expected = "UNEXPECTED SUBSTR: 'ccc'\n",
.pats = (struct expect_msg[]) {
{ .substr = "bbb" },
{ .substr = "ccc", .negative = true },
{}
}
},
{
.name = "negative-fail-mid-1",
.log = "aaabbbccc",
.expected = "UNEXPECTED SUBSTR: 'bbb'\n",
.pats = (struct expect_msg[]) {
{ .substr = "aaa" },
{ .substr = "bbb", .negative = true },
{ .substr = "ccc" },
{}
}
},
{
.name = "negative-fail-mid-2",
.log = "aaabbb222ccc",
.expected = "UNEXPECTED SUBSTR: '222'\n",
.pats = (struct expect_msg[]) {
{ .substr = "aaa" },
{ .substr = "222", .negative = true },
{ .substr = "bbb", .negative = true },
{ .substr = "ccc" },
{}
}
}
};
for (i = 0; i < ARRAY_SIZE(cases); i++) {
if (test__start_subtest(cases[i].name)) {
error_cnt = env.subtest_state->error_cnt;
msgs.patterns = cases[i].pats;
msgs.cnt = 0;
for (j = 0; cases[i].pats[j].substr; j++)
msgs.cnt++;
validate_msgs(cases[i].log, &msgs, dummy_emit);
fflush(stderr);
env.subtest_state->error_cnt = error_cnt;
if (cases[i].expected)
ASSERT_HAS_SUBSTR(env.subtest_state->log_buf, cases[i].expected, "expected output");
else
ASSERT_STREQ(env.subtest_state->log_buf, "", "expected no output");
test__end_subtest();
}
}
}

12
tools/testing/selftests/bpf/prog_tests/spin_lock.c
View File

@ -13,22 +13,22 @@ static struct {
const char *err_msg;
} spin_lock_fail_tests[] = {
{ "lock_id_kptr_preserve",
"5: (bf) r1 = r0 ; R0_w=ptr_foo(id=2,ref_obj_id=2) "
"R1_w=ptr_foo(id=2,ref_obj_id=2) refs=2\n6: (85) call bpf_this_cpu_ptr#154\n"
"5: (bf) r1 = r0 ; R0=ptr_foo(id=2,ref_obj_id=2) "
"R1=ptr_foo(id=2,ref_obj_id=2) refs=2\n6: (85) call bpf_this_cpu_ptr#154\n"
"R1 type=ptr_ expected=percpu_ptr_" },
{ "lock_id_global_zero",
"; R1_w=map_value(map=.data.A,ks=4,vs=4)\n2: (85) call bpf_this_cpu_ptr#154\n"
"; R1=map_value(map=.data.A,ks=4,vs=4)\n2: (85) call bpf_this_cpu_ptr#154\n"
"R1 type=map_value expected=percpu_ptr_" },
{ "lock_id_mapval_preserve",
"[0-9]\\+: (bf) r1 = r0 ;"
" R0_w=map_value(id=1,map=array_map,ks=4,vs=8)"
" R1_w=map_value(id=1,map=array_map,ks=4,vs=8)\n"
" R0=map_value(id=1,map=array_map,ks=4,vs=8)"
" R1=map_value(id=1,map=array_map,ks=4,vs=8)\n"
"[0-9]\\+: (85) call bpf_this_cpu_ptr#154\n"
"R1 type=map_value expected=percpu_ptr_" },
{ "lock_id_innermapval_preserve",
"[0-9]\\+: (bf) r1 = r0 ;"
" R0=map_value(id=2,ks=4,vs=8)"
" R1_w=map_value(id=2,ks=4,vs=8)\n"
" R1=map_value(id=2,ks=4,vs=8)\n"
"[0-9]\\+: (85) call bpf_this_cpu_ptr#154\n"
"R1 type=map_value expected=percpu_ptr_" },
{ "lock_id_mismatch_kptr_kptr", "bpf_spin_unlock of different lock" },

44
tools/testing/selftests/bpf/prog_tests/test_veristat.c
View File

@ -75,26 +75,26 @@ static void test_set_global_vars_succeeds(void)
" -vl2 > %s", fix->veristat, fix->tmpfile);
read(fix->fd, fix->output, fix->sz);
__CHECK_STR("_w=0xf000000000000001 ", "var_s64 = 0xf000000000000001");
__CHECK_STR("_w=0xfedcba9876543210 ", "var_u64 = 0xfedcba9876543210");
__CHECK_STR("_w=0x80000000 ", "var_s32 = -0x80000000");
__CHECK_STR("_w=0x76543210 ", "var_u32 = 0x76543210");
__CHECK_STR("_w=0x8000 ", "var_s16 = -32768");
__CHECK_STR("_w=0xecec ", "var_u16 = 60652");
__CHECK_STR("_w=128 ", "var_s8 = -128");
__CHECK_STR("_w=255 ", "var_u8 = 255");
__CHECK_STR("_w=11 ", "var_ea = EA2");
__CHECK_STR("_w=12 ", "var_eb = EB2");
__CHECK_STR("_w=13 ", "var_ec = EC2");
__CHECK_STR("_w=1 ", "var_b = 1");
__CHECK_STR("_w=170 ", "struct1[2].struct2[1][2].u.var_u8[2]=170");
__CHECK_STR("_w=0xaaaa ", "union1.var_u16 = 0xaaaa");
__CHECK_STR("_w=171 ", "arr[3]= 171");
__CHECK_STR("_w=172 ", "arr[EA2] =172");
__CHECK_STR("_w=10 ", "enum_arr[EC2]=EA3");
__CHECK_STR("_w=173 ", "matrix[31][7][11]=173");
__CHECK_STR("_w=174 ", "struct1[2].struct2[1][2].u.mat[5][3]=174");
__CHECK_STR("_w=175 ", "struct11[7][5].struct2[0][1].u.mat[3][0]=175");
__CHECK_STR("=0xf000000000000001 ", "var_s64 = 0xf000000000000001");
__CHECK_STR("=0xfedcba9876543210 ", "var_u64 = 0xfedcba9876543210");
__CHECK_STR("=0x80000000 ", "var_s32 = -0x80000000");
__CHECK_STR("=0x76543210 ", "var_u32 = 0x76543210");
__CHECK_STR("=0x8000 ", "var_s16 = -32768");
__CHECK_STR("=0xecec ", "var_u16 = 60652");
__CHECK_STR("=128 ", "var_s8 = -128");
__CHECK_STR("=255 ", "var_u8 = 255");
__CHECK_STR("=11 ", "var_ea = EA2");
__CHECK_STR("=12 ", "var_eb = EB2");
__CHECK_STR("=13 ", "var_ec = EC2");
__CHECK_STR("=1 ", "var_b = 1");
__CHECK_STR("=170 ", "struct1[2].struct2[1][2].u.var_u8[2]=170");
__CHECK_STR("=0xaaaa ", "union1.var_u16 = 0xaaaa");
__CHECK_STR("=171 ", "arr[3]= 171");
__CHECK_STR("=172 ", "arr[EA2] =172");
__CHECK_STR("=10 ", "enum_arr[EC2]=EA3");
__CHECK_STR("=173 ", "matrix[31][7][11]=173");
__CHECK_STR("=174 ", "struct1[2].struct2[1][2].u.mat[5][3]=174");
__CHECK_STR("=175 ", "struct11[7][5].struct2[0][1].u.mat[3][0]=175");
out:
teardown_fixture(fix);
@ -117,8 +117,8 @@ static void test_set_global_vars_from_file_succeeds(void)
SYS(out, "%s set_global_vars.bpf.o -G \"@%s\" -vl2 > %s",
fix->veristat, input_file, fix->tmpfile);
read(fix->fd, fix->output, fix->sz);
__CHECK_STR("_w=0x8000 ", "var_s16 = -32768");
__CHECK_STR("_w=0xecec ", "var_u16 = 60652");
__CHECK_STR("=0x8000 ", "var_s16 = -32768");
__CHECK_STR("=0xecec ", "var_u16 = 60652");
out:
close(fd);

2
tools/testing/selftests/bpf/prog_tests/verifier.c
View File

@ -46,6 +46,7 @@
#include "verifier_ldsx.skel.h"
#include "verifier_leak_ptr.skel.h"
#include "verifier_linked_scalars.skel.h"
#include "verifier_live_stack.skel.h"
#include "verifier_load_acquire.skel.h"
#include "verifier_loops1.skel.h"
#include "verifier_lwt.skel.h"
@ -184,6 +185,7 @@ void test_verifier_ld_ind(void) { RUN(verifier_ld_ind); }
void test_verifier_ldsx(void) { RUN(verifier_ldsx); }
void test_verifier_leak_ptr(void) { RUN(verifier_leak_ptr); }
void test_verifier_linked_scalars(void) { RUN(verifier_linked_scalars); }
void test_verifier_live_stack(void) { RUN(verifier_live_stack); }
void test_verifier_loops1(void) { RUN(verifier_loops1); }
void test_verifier_lwt(void) { RUN(verifier_lwt); }
void test_verifier_map_in_map(void) { RUN(verifier_map_in_map); }

9
tools/testing/selftests/bpf/progs/bpf_misc.h
View File

@ -33,7 +33,14 @@
* e.g. "foo{{[0-9]+}}" matches strings like "foo007".
* Extended POSIX regular expression syntax is allowed
* inside the brackets.
* __not_msg Message not expected to be found in verifier log.
* If __msg_not is situated between __msg tags
* framework matches __msg tags first, and then
* checks that __msg_not is not present in a portion of
* a log between bracketing __msg tags.
* Same regex syntax as for __msg is supported.
* __msg_unpriv Same as __msg but for unprivileged mode.
* __not_msg_unpriv Same as __not_msg but for unprivileged mode.
*
* __stderr Message expected to be found in bpf stderr stream. The
* same regex rules apply like __msg.
@ -121,12 +128,14 @@
* __caps_unpriv Specify the capabilities that should be set when running the test.
*/
#define __msg(msg) __attribute__((btf_decl_tag("comment:test_expect_msg=" XSTR(__COUNTER__) "=" msg)))
#define __not_msg(msg) __attribute__((btf_decl_tag("comment:test_expect_not_msg=" XSTR(__COUNTER__) "=" msg)))
#define __xlated(msg) __attribute__((btf_decl_tag("comment:test_expect_xlated=" XSTR(__COUNTER__) "=" msg)))
#define __jited(msg) __attribute__((btf_decl_tag("comment:test_jited=" XSTR(__COUNTER__) "=" msg)))
#define __failure __attribute__((btf_decl_tag("comment:test_expect_failure")))
#define __success __attribute__((btf_decl_tag("comment:test_expect_success")))
#define __description(desc) __attribute__((btf_decl_tag("comment:test_description=" desc)))
#define __msg_unpriv(msg) __attribute__((btf_decl_tag("comment:test_expect_msg_unpriv=" XSTR(__COUNTER__) "=" msg)))
#define __not_msg_unpriv(msg) __attribute__((btf_decl_tag("comment:test_expect_not_msg_unpriv=" XSTR(__COUNTER__) "=" msg)))
#define __xlated_unpriv(msg) __attribute__((btf_decl_tag("comment:test_expect_xlated_unpriv=" XSTR(__COUNTER__) "=" msg)))
#define __jited_unpriv(msg) __attribute__((btf_decl_tag("comment:test_jited=" XSTR(__COUNTER__) "=" msg)))
#define __failure_unpriv __attribute__((btf_decl_tag("comment:test_expect_failure_unpriv")))

34
tools/testing/selftests/bpf/progs/exceptions_assert.c
View File

@ -18,43 +18,43 @@
return *(u64 *)num; \
}
__msg(": R0_w=0xffffffff80000000")
__msg(": R0=0xffffffff80000000")
check_assert(s64, ==, eq_int_min, INT_MIN);
__msg(": R0_w=0x7fffffff")
__msg(": R0=0x7fffffff")
check_assert(s64, ==, eq_int_max, INT_MAX);
__msg(": R0_w=0")
__msg(": R0=0")
check_assert(s64, ==, eq_zero, 0);
__msg(": R0_w=0x8000000000000000 R1_w=0x8000000000000000")
__msg(": R0=0x8000000000000000 R1=0x8000000000000000")
check_assert(s64, ==, eq_llong_min, LLONG_MIN);
__msg(": R0_w=0x7fffffffffffffff R1_w=0x7fffffffffffffff")
__msg(": R0=0x7fffffffffffffff R1=0x7fffffffffffffff")
check_assert(s64, ==, eq_llong_max, LLONG_MAX);
__msg(": R0_w=scalar(id=1,smax=0x7ffffffe)")
__msg(": R0=scalar(id=1,smax=0x7ffffffe)")
check_assert(s64, <, lt_pos, INT_MAX);
__msg(": R0_w=scalar(id=1,smax=-1,umin=0x8000000000000000,var_off=(0x8000000000000000; 0x7fffffffffffffff))")
__msg(": R0=scalar(id=1,smax=-1,umin=0x8000000000000000,var_off=(0x8000000000000000; 0x7fffffffffffffff))")
check_assert(s64, <, lt_zero, 0);
__msg(": R0_w=scalar(id=1,smax=0xffffffff7fffffff")
__msg(": R0=scalar(id=1,smax=0xffffffff7fffffff")
check_assert(s64, <, lt_neg, INT_MIN);
__msg(": R0_w=scalar(id=1,smax=0x7fffffff)")
__msg(": R0=scalar(id=1,smax=0x7fffffff)")
check_assert(s64, <=, le_pos, INT_MAX);
__msg(": R0_w=scalar(id=1,smax=0)")
__msg(": R0=scalar(id=1,smax=0)")
check_assert(s64, <=, le_zero, 0);
__msg(": R0_w=scalar(id=1,smax=0xffffffff80000000")
__msg(": R0=scalar(id=1,smax=0xffffffff80000000")
check_assert(s64, <=, le_neg, INT_MIN);
__msg(": R0_w=scalar(id=1,smin=umin=0x80000000,umax=0x7fffffffffffffff,var_off=(0x0; 0x7fffffffffffffff))")
__msg(": R0=scalar(id=1,smin=umin=0x80000000,umax=0x7fffffffffffffff,var_off=(0x0; 0x7fffffffffffffff))")
check_assert(s64, >, gt_pos, INT_MAX);
__msg(": R0_w=scalar(id=1,smin=umin=1,umax=0x7fffffffffffffff,var_off=(0x0; 0x7fffffffffffffff))")
__msg(": R0=scalar(id=1,smin=umin=1,umax=0x7fffffffffffffff,var_off=(0x0; 0x7fffffffffffffff))")
check_assert(s64, >, gt_zero, 0);
__msg(": R0_w=scalar(id=1,smin=0xffffffff80000001")
__msg(": R0=scalar(id=1,smin=0xffffffff80000001")
check_assert(s64, >, gt_neg, INT_MIN);
__msg(": R0_w=scalar(id=1,smin=umin=0x7fffffff,umax=0x7fffffffffffffff,var_off=(0x0; 0x7fffffffffffffff))")
__msg(": R0=scalar(id=1,smin=umin=0x7fffffff,umax=0x7fffffffffffffff,var_off=(0x0; 0x7fffffffffffffff))")
check_assert(s64, >=, ge_pos, INT_MAX);
__msg(": R0_w=scalar(id=1,smin=0,umax=0x7fffffffffffffff,var_off=(0x0; 0x7fffffffffffffff))")
__msg(": R0=scalar(id=1,smin=0,umax=0x7fffffffffffffff,var_off=(0x0; 0x7fffffffffffffff))")
check_assert(s64, >=, ge_zero, 0);
__msg(": R0_w=scalar(id=1,smin=0xffffffff80000000")
__msg(": R0=scalar(id=1,smin=0xffffffff80000000")
check_assert(s64, >=, ge_neg, INT_MIN);
SEC("?tc")

4
tools/testing/selftests/bpf/progs/iters_state_safety.c
View File

@ -30,7 +30,7 @@ int force_clang_to_emit_btf_for_externs(void *ctx)
SEC("?raw_tp")
__success __log_level(2)
__msg("fp-8_w=iter_num(ref_id=1,state=active,depth=0)")
__msg("fp-8=iter_num(ref_id=1,state=active,depth=0)")
int create_and_destroy(void *ctx)
{
struct bpf_iter_num iter;
@ -196,7 +196,7 @@ int leak_iter_from_subprog_fail(void *ctx)
SEC("?raw_tp")
__success __log_level(2)
__msg("fp-8_w=iter_num(ref_id=1,state=active,depth=0)")
__msg("fp-8=iter_num(ref_id=1,state=active,depth=0)")
int valid_stack_reuse(void *ctx)
{
struct bpf_iter_num iter;

6
tools/testing/selftests/bpf/progs/iters_testmod_seq.c
View File

@ -20,7 +20,7 @@ __s64 res_empty;
SEC("raw_tp/sys_enter")
__success __log_level(2)
__msg("fp-16_w=iter_testmod_seq(ref_id=1,state=active,depth=0)")
__msg("fp-16=iter_testmod_seq(ref_id=1,state=active,depth=0)")
__msg("fp-16=iter_testmod_seq(ref_id=1,state=drained,depth=0)")
__msg("call bpf_iter_testmod_seq_destroy")
int testmod_seq_empty(const void *ctx)
@ -38,7 +38,7 @@ __s64 res_full;
SEC("raw_tp/sys_enter")
__success __log_level(2)
__msg("fp-16_w=iter_testmod_seq(ref_id=1,state=active,depth=0)")
__msg("fp-16=iter_testmod_seq(ref_id=1,state=active,depth=0)")
__msg("fp-16=iter_testmod_seq(ref_id=1,state=drained,depth=0)")
__msg("call bpf_iter_testmod_seq_destroy")
int testmod_seq_full(const void *ctx)
@ -58,7 +58,7 @@ static volatile int zero = 0;
SEC("raw_tp/sys_enter")
__success __log_level(2)
__msg("fp-16_w=iter_testmod_seq(ref_id=1,state=active,depth=0)")
__msg("fp-16=iter_testmod_seq(ref_id=1,state=active,depth=0)")
__msg("fp-16=iter_testmod_seq(ref_id=1,state=drained,depth=0)")
__msg("call bpf_iter_testmod_seq_destroy")
int testmod_seq_truncated(const void *ctx)

4
tools/testing/selftests/bpf/progs/mem_rdonly_untrusted.c
View File

@ -8,8 +8,8 @@
SEC("tp_btf/sys_enter")
__success
__log_level(2)
__msg("r8 = *(u64 *)(r7 +0) ; R7_w=ptr_nameidata(off={{[0-9]+}}) R8_w=rdonly_untrusted_mem(sz=0)")
__msg("r9 = *(u8 *)(r8 +0) ; R8_w=rdonly_untrusted_mem(sz=0) R9_w=scalar")
__msg("r8 = *(u64 *)(r7 +0) ; R7=ptr_nameidata(off={{[0-9]+}}) R8=rdonly_untrusted_mem(sz=0)")
__msg("r9 = *(u8 *)(r8 +0) ; R8=rdonly_untrusted_mem(sz=0) R9=scalar")
int btf_id_to_ptr_mem(void *ctx)
{
struct task_struct *task;

38
tools/testing/selftests/bpf/progs/verifier_bounds.c
View File

@ -926,7 +926,7 @@ l1_%=: r0 = 0; \
SEC("socket")
__description("bounds check for non const xor src dst")
__success __log_level(2)
__msg("5: (af) r0 ^= r6 ; R0_w=scalar(smin=smin32=0,smax=umax=smax32=umax32=431,var_off=(0x0; 0x1af))")
__msg("5: (af) r0 ^= r6 ; R0=scalar(smin=smin32=0,smax=umax=smax32=umax32=431,var_off=(0x0; 0x1af))")
__naked void non_const_xor_src_dst(void)
{
asm volatile (" \
@ -947,7 +947,7 @@ __naked void non_const_xor_src_dst(void)
SEC("socket")
__description("bounds check for non const or src dst")
__success __log_level(2)
__msg("5: (4f) r0 |= r6 ; R0_w=scalar(smin=smin32=0,smax=umax=smax32=umax32=431,var_off=(0x0; 0x1af))")
__msg("5: (4f) r0 |= r6 ; R0=scalar(smin=smin32=0,smax=umax=smax32=umax32=431,var_off=(0x0; 0x1af))")
__naked void non_const_or_src_dst(void)
{
asm volatile (" \
@ -968,7 +968,7 @@ __naked void non_const_or_src_dst(void)
SEC("socket")
__description("bounds check for non const mul regs")
__success __log_level(2)
__msg("5: (2f) r0 *= r6 ; R0_w=scalar(smin=smin32=0,smax=umax=smax32=umax32=3825,var_off=(0x0; 0xfff))")
__msg("5: (2f) r0 *= r6 ; R0=scalar(smin=smin32=0,smax=umax=smax32=umax32=3825,var_off=(0x0; 0xfff))")
__naked void non_const_mul_regs(void)
{
asm volatile (" \
@ -1241,7 +1241,7 @@ l0_%=: r0 = 0; \
SEC("tc")
__description("multiply mixed sign bounds. test 1")
__success __log_level(2)
__msg("r6 *= r7 {{.*}}; R6_w=scalar(smin=umin=0x1bc16d5cd4927ee1,smax=umax=0x1bc16d674ec80000,smax32=0x7ffffeff,umax32=0xfffffeff,var_off=(0x1bc16d4000000000; 0x3ffffffeff))")
__msg("r6 *= r7 {{.*}}; R6=scalar(smin=umin=0x1bc16d5cd4927ee1,smax=umax=0x1bc16d674ec80000,smax32=0x7ffffeff,umax32=0xfffffeff,var_off=(0x1bc16d4000000000; 0x3ffffffeff))")
__naked void mult_mixed0_sign(void)
{
asm volatile (
@ -1264,7 +1264,7 @@ __naked void mult_mixed0_sign(void)
SEC("tc")
__description("multiply mixed sign bounds. test 2")
__success __log_level(2)
__msg("r6 *= r7 {{.*}}; R6_w=scalar(smin=smin32=-100,smax=smax32=200)")
__msg("r6 *= r7 {{.*}}; R6=scalar(smin=smin32=-100,smax=smax32=200)")
__naked void mult_mixed1_sign(void)
{
asm volatile (
@ -1287,7 +1287,7 @@ __naked void mult_mixed1_sign(void)
SEC("tc")
__description("multiply negative bounds")
__success __log_level(2)
__msg("r6 *= r7 {{.*}}; R6_w=scalar(smin=umin=smin32=umin32=0x3ff280b0,smax=umax=smax32=umax32=0x3fff0001,var_off=(0x3ff00000; 0xf81ff))")
__msg("r6 *= r7 {{.*}}; R6=scalar(smin=umin=smin32=umin32=0x3ff280b0,smax=umax=smax32=umax32=0x3fff0001,var_off=(0x3ff00000; 0xf81ff))")
__naked void mult_sign_bounds(void)
{
asm volatile (
@ -1311,7 +1311,7 @@ __naked void mult_sign_bounds(void)
SEC("tc")
__description("multiply bounds that don't cross signed boundary")
__success __log_level(2)
__msg("r8 *= r6 {{.*}}; R6_w=scalar(smin=smin32=0,smax=umax=smax32=umax32=11,var_off=(0x0; 0xb)) R8_w=scalar(smin=0,smax=umax=0x7b96bb0a94a3a7cd,var_off=(0x0; 0x7fffffffffffffff))")
__msg("r8 *= r6 {{.*}}; R6=scalar(smin=smin32=0,smax=umax=smax32=umax32=11,var_off=(0x0; 0xb)) R8=scalar(smin=0,smax=umax=0x7b96bb0a94a3a7cd,var_off=(0x0; 0x7fffffffffffffff))")
__naked void mult_no_sign_crossing(void)
{
asm volatile (
@ -1331,7 +1331,7 @@ __naked void mult_no_sign_crossing(void)
SEC("tc")
__description("multiplication overflow, result in unbounded reg. test 1")
__success __log_level(2)
__msg("r6 *= r7 {{.*}}; R6_w=scalar()")
__msg("r6 *= r7 {{.*}}; R6=scalar()")
__naked void mult_unsign_ovf(void)
{
asm volatile (
@ -1353,7 +1353,7 @@ __naked void mult_unsign_ovf(void)
SEC("tc")
__description("multiplication overflow, result in unbounded reg. test 2")
__success __log_level(2)
__msg("r6 *= r7 {{.*}}; R6_w=scalar()")
__msg("r6 *= r7 {{.*}}; R6=scalar()")
__naked void mult_sign_ovf(void)
{
asm volatile (
@ -1376,7 +1376,7 @@ __naked void mult_sign_ovf(void)
SEC("socket")
__description("64-bit addition, all outcomes overflow")
__success __log_level(2)
__msg("5: (0f) r3 += r3 {{.*}} R3_w=scalar(umin=0x4000000000000000,umax=0xfffffffffffffffe)")
__msg("5: (0f) r3 += r3 {{.*}} R3=scalar(umin=0x4000000000000000,umax=0xfffffffffffffffe)")
__retval(0)
__naked void add64_full_overflow(void)
{
@ -1396,7 +1396,7 @@ __naked void add64_full_overflow(void)
SEC("socket")
__description("64-bit addition, partial overflow, result in unbounded reg")
__success __log_level(2)
__msg("4: (0f) r3 += r3 {{.*}} R3_w=scalar()")
__msg("4: (0f) r3 += r3 {{.*}} R3=scalar()")
__retval(0)
__naked void add64_partial_overflow(void)
{
@ -1416,7 +1416,7 @@ __naked void add64_partial_overflow(void)
SEC("socket")
__description("32-bit addition overflow, all outcomes overflow")
__success __log_level(2)
__msg("4: (0c) w3 += w3 {{.*}} R3_w=scalar(smin=umin=umin32=0x40000000,smax=umax=umax32=0xfffffffe,var_off=(0x0; 0xffffffff))")
__msg("4: (0c) w3 += w3 {{.*}} R3=scalar(smin=umin=umin32=0x40000000,smax=umax=umax32=0xfffffffe,var_off=(0x0; 0xffffffff))")
__retval(0)
__naked void add32_full_overflow(void)
{
@ -1436,7 +1436,7 @@ __naked void add32_full_overflow(void)
SEC("socket")
__description("32-bit addition, partial overflow, result in unbounded u32 bounds")
__success __log_level(2)
__msg("4: (0c) w3 += w3 {{.*}} R3_w=scalar(smin=0,smax=umax=0xffffffff,var_off=(0x0; 0xffffffff))")
__msg("4: (0c) w3 += w3 {{.*}} R3=scalar(smin=0,smax=umax=0xffffffff,var_off=(0x0; 0xffffffff))")
__retval(0)
__naked void add32_partial_overflow(void)
{
@ -1456,7 +1456,7 @@ __naked void add32_partial_overflow(void)
SEC("socket")
__description("64-bit subtraction, all outcomes underflow")
__success __log_level(2)
__msg("6: (1f) r3 -= r1 {{.*}} R3_w=scalar(umin=1,umax=0x8000000000000000)")
__msg("6: (1f) r3 -= r1 {{.*}} R3=scalar(umin=1,umax=0x8000000000000000)")
__retval(0)
__naked void sub64_full_overflow(void)
{
@ -1477,7 +1477,7 @@ __naked void sub64_full_overflow(void)
SEC("socket")
__description("64-bit subtraction, partial overflow, result in unbounded reg")
__success __log_level(2)
__msg("3: (1f) r3 -= r2 {{.*}} R3_w=scalar()")
__msg("3: (1f) r3 -= r2 {{.*}} R3=scalar()")
__retval(0)
__naked void sub64_partial_overflow(void)
{
@ -1496,7 +1496,7 @@ __naked void sub64_partial_overflow(void)
SEC("socket")
__description("32-bit subtraction overflow, all outcomes underflow")
__success __log_level(2)
__msg("5: (1c) w3 -= w1 {{.*}} R3_w=scalar(smin=umin=umin32=1,smax=umax=umax32=0x80000000,var_off=(0x0; 0xffffffff))")
__msg("5: (1c) w3 -= w1 {{.*}} R3=scalar(smin=umin=umin32=1,smax=umax=umax32=0x80000000,var_off=(0x0; 0xffffffff))")
__retval(0)
__naked void sub32_full_overflow(void)
{
@ -1517,7 +1517,7 @@ __naked void sub32_full_overflow(void)
SEC("socket")
__description("32-bit subtraction, partial overflow, result in unbounded u32 bounds")
__success __log_level(2)
__msg("3: (1c) w3 -= w2 {{.*}} R3_w=scalar(smin=0,smax=umax=0xffffffff,var_off=(0x0; 0xffffffff))")
__msg("3: (1c) w3 -= w2 {{.*}} R3=scalar(smin=0,smax=umax=0xffffffff,var_off=(0x0; 0xffffffff))")
__retval(0)
__naked void sub32_partial_overflow(void)
{
@ -1617,7 +1617,7 @@ l0_%=: r0 = 0; \
SEC("socket")
__description("bounds deduction cross sign boundary, positive overlap")
__success __log_level(2) __flag(BPF_F_TEST_REG_INVARIANTS)
__msg("3: (2d) if r0 > r1 {{.*}} R0_w=scalar(smin=smin32=0,smax=umax=smax32=umax32=127,var_off=(0x0; 0x7f))")
__msg("3: (2d) if r0 > r1 {{.*}} R0=scalar(smin=smin32=0,smax=umax=smax32=umax32=127,var_off=(0x0; 0x7f))")
__retval(0)
__naked void bounds_deduct_positive_overlap(void)
{
@ -1650,7 +1650,7 @@ l0_%=: r0 = 0; \
SEC("socket")
__description("bounds deduction cross sign boundary, two overlaps")
__failure __flag(BPF_F_TEST_REG_INVARIANTS)
__msg("3: (2d) if r0 > r1 {{.*}} R0_w=scalar(smin=smin32=-128,smax=smax32=127,umax=0xffffffffffffff80)")
__msg("3: (2d) if r0 > r1 {{.*}} R0=scalar(smin=smin32=-128,smax=smax32=127,umax=0xffffffffffffff80)")
__msg("frame pointer is read only")
__naked void bounds_deduct_two_overlaps(void)
{

4
tools/testing/selftests/bpf/progs/verifier_global_ptr_args.c
View File

@ -215,7 +215,7 @@ __weak int subprog_untrusted(const volatile struct task_struct *restrict task __
SEC("tp_btf/sys_enter")
__success
__log_level(2)
__msg("r1 = {{.*}}; {{.*}}R1_w=trusted_ptr_task_struct()")
__msg("r1 = {{.*}}; {{.*}}R1=trusted_ptr_task_struct()")
__msg("Func#1 ('subprog_untrusted') is global and assumed valid.")
__msg("Validating subprog_untrusted() func#1...")
__msg(": R1=untrusted_ptr_task_struct")
@ -278,7 +278,7 @@ __weak int subprog_enum_untrusted(enum bpf_attach_type *p __arg_untrusted)
SEC("tp_btf/sys_enter")
__success
__log_level(2)
__msg("r1 = {{.*}}; {{.*}}R1_w=trusted_ptr_task_struct()")
__msg("r1 = {{.*}}; {{.*}}R1=trusted_ptr_task_struct()")
__msg("Func#1 ('subprog_void_untrusted') is global and assumed valid.")
__msg("Validating subprog_void_untrusted() func#1...")
__msg(": R1=rdonly_untrusted_mem(sz=0)")

2
tools/testing/selftests/bpf/progs/verifier_ldsx.c
View File

@ -65,7 +65,7 @@ __naked void ldsx_s32(void)
SEC("socket")
__description("LDSX, S8 range checking, privileged")
__log_level(2) __success __retval(1)
__msg("R1_w=scalar(smin=smin32=-128,smax=smax32=127)")
__msg("R1=scalar(smin=smin32=-128,smax=smax32=127)")
__naked void ldsx_s8_range_priv(void)
{
asm volatile (

294
tools/testing/selftests/bpf/progs/verifier_live_stack.c Normal file
View File

@ -0,0 +1,294 @@
// SPDX-License-Identifier: GPL-2.0
/* Copyright (c) 2025 Meta Platforms, Inc. and affiliates. */
#include <linux/bpf.h>
#include <bpf/bpf_helpers.h>
#include "bpf_misc.h"
struct {
__uint(type, BPF_MAP_TYPE_HASH);
__uint(max_entries, 1);
__type(key, int);
__type(value, long long);
} map SEC(".maps");
SEC("socket")
__log_level(2)
__msg("(0) frame 0 insn 2 +written -8")
__msg("(0) frame 0 insn 1 +live -24")
__msg("(0) frame 0 insn 1 +written -8")
__msg("(0) frame 0 insn 0 +live -8,-24")
__msg("(0) frame 0 insn 0 +written -8")
__msg("(0) live stack update done in 2 iterations")
__naked void simple_read_simple_write(void)
{
asm volatile (
"r1 = *(u64 *)(r10 - 8);"
"r2 = *(u64 *)(r10 - 24);"
"*(u64 *)(r10 - 8) = r1;"
"r0 = 0;"
"exit;"
::: __clobber_all);
}
SEC("socket")
__log_level(2)
__msg("(0) frame 0 insn 1 +live -8")
__not_msg("(0) frame 0 insn 1 +written")
__msg("(0) live stack update done in 2 iterations")
__msg("(0) frame 0 insn 1 +live -16")
__msg("(0) frame 0 insn 1 +written -32")
__msg("(0) live stack update done in 2 iterations")
__naked void read_write_join(void)
{
asm volatile (
"call %[bpf_get_prandom_u32];"
"if r0 > 42 goto 1f;"
"r0 = *(u64 *)(r10 - 8);"
"*(u64 *)(r10 - 32) = r0;"
"*(u64 *)(r10 - 40) = r0;"
"exit;"
"1:"
"r0 = *(u64 *)(r10 - 16);"
"*(u64 *)(r10 - 32) = r0;"
"exit;"
:: __imm(bpf_get_prandom_u32)
: __clobber_all);
}
SEC("socket")
__log_level(2)
__msg("2: (25) if r0 > 0x2a goto pc+1")
__msg("7: (95) exit")
__msg("(0) frame 0 insn 2 +written -16")
__msg("(0) live stack update done in 2 iterations")
__msg("7: (95) exit")
__not_msg("(0) frame 0 insn 2")
__msg("(0) live stack update done in 1 iterations")
__naked void must_write_not_same_slot(void)
{
asm volatile (
"call %[bpf_get_prandom_u32];"
"r1 = -8;"
"if r0 > 42 goto 1f;"
"r1 = -16;"
"1:"
"r2 = r10;"
"r2 += r1;"
"*(u64 *)(r2 + 0) = r0;"
"exit;"
:: __imm(bpf_get_prandom_u32)
: __clobber_all);
}
SEC("socket")
__log_level(2)
__msg("(0) frame 0 insn 0 +written -8,-16")
__msg("(0) live stack update done in 2 iterations")
__msg("(0) frame 0 insn 0 +written -8")
__msg("(0) live stack update done in 2 iterations")
__naked void must_write_not_same_type(void)
{
asm volatile (
"*(u64*)(r10 - 8) = 0;"
"r2 = r10;"
"r2 += -8;"
"r1 = %[map] ll;"
"call %[bpf_map_lookup_elem];"
"if r0 != 0 goto 1f;"
"r0 = r10;"
"r0 += -16;"
"1:"
"*(u64 *)(r0 + 0) = 42;"
"exit;"
:
: __imm(bpf_get_prandom_u32),
__imm(bpf_map_lookup_elem),
__imm_addr(map)
: __clobber_all);
}
SEC("socket")
__log_level(2)
__msg("(2,4) frame 0 insn 4 +written -8")
__msg("(2,4) live stack update done in 2 iterations")
__msg("(0) frame 0 insn 2 +written -8")
__msg("(0) live stack update done in 2 iterations")
__naked void caller_stack_write(void)
{
asm volatile (
"r1 = r10;"
"r1 += -8;"
"call write_first_param;"
"exit;"
::: __clobber_all);
}
static __used __naked void write_first_param(void)
{
asm volatile (
"*(u64 *)(r1 + 0) = 7;"
"r0 = 0;"
"exit;"
::: __clobber_all);
}
SEC("socket")
__log_level(2)
/* caller_stack_read() function */
__msg("2: .12345.... (85) call pc+4")
__msg("5: .12345.... (85) call pc+1")
__msg("6: 0......... (95) exit")
/* read_first_param() function */
__msg("7: .1........ (79) r0 = *(u64 *)(r1 +0)")
__msg("8: 0......... (95) exit")
/* update for callsite at (2) */
__msg("(2,7) frame 0 insn 7 +live -8")
__msg("(2,7) live stack update done in 2 iterations")
__msg("(0) frame 0 insn 2 +live -8")
__msg("(0) live stack update done in 2 iterations")
/* update for callsite at (5) */
__msg("(5,7) frame 0 insn 7 +live -16")
__msg("(5,7) live stack update done in 2 iterations")
__msg("(0) frame 0 insn 5 +live -16")
__msg("(0) live stack update done in 2 iterations")
__naked void caller_stack_read(void)
{
asm volatile (
"r1 = r10;"
"r1 += -8;"
"call read_first_param;"
"r1 = r10;"
"r1 += -16;"
"call read_first_param;"
"exit;"
::: __clobber_all);
}
static __used __naked void read_first_param(void)
{
asm volatile (
"r0 = *(u64 *)(r1 + 0);"
"exit;"
::: __clobber_all);
}
SEC("socket")
__flag(BPF_F_TEST_STATE_FREQ)
__log_level(2)
/* read_first_param2() function */
__msg(" 9: .1........ (79) r0 = *(u64 *)(r1 +0)")
__msg("10: .......... (b7) r0 = 0")
__msg("11: 0......... (05) goto pc+0")
__msg("12: 0......... (95) exit")
/*
* The purpose of the test is to check that checkpoint in
* read_first_param2() stops path traversal. This will only happen if
* verifier understands that fp[0]-8 at insn (12) is not alive.
*/
__msg("12: safe")
__msg("processed 20 insns")
__naked void caller_stack_pruning(void)
{
asm volatile (
"call %[bpf_get_prandom_u32];"
"if r0 == 42 goto 1f;"
"r0 = %[map] ll;"
"1:"
"*(u64 *)(r10 - 8) = r0;"
"r1 = r10;"
"r1 += -8;"
/*
* fp[0]-8 is either pointer to map or a scalar,
* preventing state pruning at checkpoint created for call.
*/
"call read_first_param2;"
"exit;"
:
: __imm(bpf_get_prandom_u32),
__imm_addr(map)
: __clobber_all);
}
static __used __naked void read_first_param2(void)
{
asm volatile (
"r0 = *(u64 *)(r1 + 0);"
"r0 = 0;"
/*
* Checkpoint at goto +0 should fire,
* as caller stack fp[0]-8 is not alive at this point.
*/
"goto +0;"
"exit;"
::: __clobber_all);
}
SEC("socket")
__flag(BPF_F_TEST_STATE_FREQ)
__failure
__msg("R1 type=scalar expected=map_ptr")
__naked void caller_stack_pruning_callback(void)
{
asm volatile (
"r0 = %[map] ll;"
"*(u64 *)(r10 - 8) = r0;"
"r1 = 2;"
"r2 = loop_cb ll;"
"r3 = r10;"
"r3 += -8;"
"r4 = 0;"
/*
* fp[0]-8 is either pointer to map or a scalar,
* preventing state pruning at checkpoint created for call.
*/
"call %[bpf_loop];"
"r0 = 42;"
"exit;"
:
: __imm(bpf_get_prandom_u32),
__imm(bpf_loop),
__imm_addr(map)
: __clobber_all);
}
static __used __naked void loop_cb(void)
{
asm volatile (
/*
* Checkpoint at function entry should not fire, as caller
* stack fp[0]-8 is alive at this point.
*/
"r6 = r2;"
"r1 = *(u64 *)(r6 + 0);"
"*(u64*)(r10 - 8) = 7;"
"r2 = r10;"
"r2 += -8;"
"call %[bpf_map_lookup_elem];"
/*
* This should stop verifier on a second loop iteration,
* but only if verifier correctly maintains that fp[0]-8
* is still alive.
*/
"*(u64 *)(r6 + 0) = 0;"
"r0 = 0;"
"exit;"
:
: __imm(bpf_map_lookup_elem),
__imm(bpf_get_prandom_u32)
: __clobber_all);
}
/*
* Because of a bug in verifier.c:compute_postorder()
* the program below overflowed traversal queue in that function.
*/
SEC("socket")
__naked void syzbot_postorder_bug1(void)
{
asm volatile (
"r0 = 0;"
"if r0 != 0 goto -1;"
"exit;"
::: __clobber_all);
}

16
tools/testing/selftests/bpf/progs/verifier_precision.c
View File

@ -144,21 +144,21 @@ SEC("?raw_tp")
__success __log_level(2)
/*
* Without the bug fix there will be no history between "last_idx 3 first_idx 3"
* and "parent state regs=" lines. "R0_w=6" parts are here to help anchor
* and "parent state regs=" lines. "R0=6" parts are here to help anchor
* expected log messages to the one specific mark_chain_precision operation.
*
* This is quite fragile: if verifier checkpointing heuristic changes, this
* might need adjusting.
*/
__msg("2: (07) r0 += 1 ; R0_w=6")
__msg("2: (07) r0 += 1 ; R0=6")
__msg("3: (35) if r0 >= 0xa goto pc+1")
__msg("mark_precise: frame0: last_idx 3 first_idx 3 subseq_idx -1")
__msg("mark_precise: frame0: regs=r0 stack= before 2: (07) r0 += 1")
__msg("mark_precise: frame0: regs=r0 stack= before 1: (07) r0 += 1")
__msg("mark_precise: frame0: regs=r0 stack= before 4: (05) goto pc-4")
__msg("mark_precise: frame0: regs=r0 stack= before 3: (35) if r0 >= 0xa goto pc+1")
__msg("mark_precise: frame0: parent state regs= stack=: R0_rw=P4")
__msg("3: R0_w=6")
__msg("mark_precise: frame0: parent state regs= stack=: R0=P4")
__msg("3: R0=6")
__naked int state_loop_first_last_equal(void)
{
asm volatile (
@ -233,8 +233,8 @@ __naked void bpf_cond_op_not_r10(void)
SEC("lsm.s/socket_connect")
__success __log_level(2)
__msg("0: (b7) r0 = 1 ; R0_w=1")
__msg("1: (84) w0 = -w0 ; R0_w=0xffffffff")
__msg("0: (b7) r0 = 1 ; R0=1")
__msg("1: (84) w0 = -w0 ; R0=0xffffffff")
__msg("mark_precise: frame0: last_idx 2 first_idx 0 subseq_idx -1")
__msg("mark_precise: frame0: regs=r0 stack= before 1: (84) w0 = -w0")
__msg("mark_precise: frame0: regs=r0 stack= before 0: (b7) r0 = 1")
@ -268,8 +268,8 @@ __naked int bpf_neg_3(void)
SEC("lsm.s/socket_connect")
__success __log_level(2)
__msg("0: (b7) r0 = 1 ; R0_w=1")
__msg("1: (87) r0 = -r0 ; R0_w=-1")
__msg("0: (b7) r0 = 1 ; R0=1")
__msg("1: (87) r0 = -r0 ; R0=-1")
__msg("mark_precise: frame0: last_idx 2 first_idx 0 subseq_idx -1")
__msg("mark_precise: frame0: regs=r0 stack= before 1: (87) r0 = -r0")
__msg("mark_precise: frame0: regs=r0 stack= before 0: (b7) r0 = 1")

10
tools/testing/selftests/bpf/progs/verifier_scalar_ids.c
View File

@ -353,7 +353,7 @@ __flag(BPF_F_TEST_STATE_FREQ)
* collect_linked_regs() can't tie more than 6 registers for a single insn.
*/
__msg("8: (25) if r0 > 0x7 goto pc+0 ; R0=scalar(id=1")
__msg("9: (bf) r6 = r6 ; R6_w=scalar(id=2")
__msg("9: (bf) r6 = r6 ; R6=scalar(id=2")
/* check that r{0-5} are marked precise after 'if' */
__msg("frame0: regs=r0 stack= before 8: (25) if r0 > 0x7 goto pc+0")
__msg("frame0: parent state regs=r0,r1,r2,r3,r4,r5 stack=:")
@ -779,12 +779,12 @@ __success
__retval(0)
/* Check that verifier believes r1/r0 are zero at exit */
__log_level(2)
__msg("4: (77) r1 >>= 32 ; R1_w=0")
__msg("5: (bf) r0 = r1 ; R0_w=0 R1_w=0")
__msg("4: (77) r1 >>= 32 ; R1=0")
__msg("5: (bf) r0 = r1 ; R0=0 R1=0")
__msg("6: (95) exit")
__msg("from 3 to 4")
__msg("4: (77) r1 >>= 32 ; R1_w=0")
__msg("5: (bf) r0 = r1 ; R0_w=0 R1_w=0")
__msg("4: (77) r1 >>= 32 ; R1=0")
__msg("5: (bf) r0 = r1 ; R0=0 R1=0")
__msg("6: (95) exit")
/* Verify that statements to randomize upper half of r1 had not been
* generated.

40
tools/testing/selftests/bpf/progs/verifier_spill_fill.c
View File

@ -506,17 +506,17 @@ SEC("raw_tp")
__log_level(2)
__success
/* fp-8 is spilled IMPRECISE value zero (represented by a zero value fake reg) */
__msg("2: (7a) *(u64 *)(r10 -8) = 0 ; R10=fp0 fp-8_w=0")
__msg("2: (7a) *(u64 *)(r10 -8) = 0 ; R10=fp0 fp-8=0")
/* but fp-16 is spilled IMPRECISE zero const reg */
__msg("4: (7b) *(u64 *)(r10 -16) = r0 ; R0_w=0 R10=fp0 fp-16_w=0")
__msg("4: (7b) *(u64 *)(r10 -16) = r0 ; R0=0 R10=fp0 fp-16=0")
/* validate that assigning R2 from STACK_SPILL with zero value doesn't mark register
* precise immediately; if necessary, it will be marked precise later
*/
__msg("6: (71) r2 = *(u8 *)(r10 -1) ; R2_w=0 R10=fp0 fp-8_w=0")
__msg("6: (71) r2 = *(u8 *)(r10 -1) ; R2=0 R10=fp0 fp-8=0")
/* similarly, when R2 is assigned from spilled register, it is initially
* imprecise, but will be marked precise later once it is used in precise context
*/
__msg("10: (71) r2 = *(u8 *)(r10 -9) ; R2_w=0 R10=fp0 fp-16_w=0")
__msg("10: (71) r2 = *(u8 *)(r10 -9) ; R2=0 R10=fp0 fp-16=0")
__msg("11: (0f) r1 += r2")
__msg("mark_precise: frame0: last_idx 11 first_idx 0 subseq_idx -1")
__msg("mark_precise: frame0: regs=r2 stack= before 10: (71) r2 = *(u8 *)(r10 -9)")
@ -598,7 +598,7 @@ __log_level(2)
__success
/* fp-4 is STACK_ZERO */
__msg("2: (62) *(u32 *)(r10 -4) = 0 ; R10=fp0 fp-8=0000????")
__msg("4: (71) r2 = *(u8 *)(r10 -1) ; R2_w=0 R10=fp0 fp-8=0000????")
__msg("4: (71) r2 = *(u8 *)(r10 -1) ; R2=0 R10=fp0 fp-8=0000????")
__msg("5: (0f) r1 += r2")
__msg("mark_precise: frame0: last_idx 5 first_idx 0 subseq_idx -1")
__msg("mark_precise: frame0: regs=r2 stack= before 4: (71) r2 = *(u8 *)(r10 -1)")
@ -640,25 +640,25 @@ SEC("raw_tp")
__log_level(2) __flag(BPF_F_TEST_STATE_FREQ)
__success
/* make sure fp-8 is IMPRECISE fake register spill */
__msg("3: (7a) *(u64 *)(r10 -8) = 1 ; R10=fp0 fp-8_w=1")
__msg("3: (7a) *(u64 *)(r10 -8) = 1 ; R10=fp0 fp-8=1")
/* and fp-16 is spilled IMPRECISE const reg */
__msg("5: (7b) *(u64 *)(r10 -16) = r0 ; R0_w=1 R10=fp0 fp-16_w=1")
__msg("5: (7b) *(u64 *)(r10 -16) = r0 ; R0=1 R10=fp0 fp-16=1")
/* validate load from fp-8, which was initialized using BPF_ST_MEM */
__msg("8: (79) r2 = *(u64 *)(r10 -8) ; R2_w=1 R10=fp0 fp-8=1")
__msg("8: (79) r2 = *(u64 *)(r10 -8) ; R2=1 R10=fp0 fp-8=1")
__msg("9: (0f) r1 += r2")
__msg("mark_precise: frame0: last_idx 9 first_idx 7 subseq_idx -1")
__msg("mark_precise: frame0: regs=r2 stack= before 8: (79) r2 = *(u64 *)(r10 -8)")
__msg("mark_precise: frame0: regs= stack=-8 before 7: (bf) r1 = r6")
/* note, fp-8 is precise, fp-16 is not yet precise, we'll get there */
__msg("mark_precise: frame0: parent state regs= stack=-8: R0_w=1 R1=ctx() R6_r=map_value(map=.data.two_byte_,ks=4,vs=2) R10=fp0 fp-8_rw=P1 fp-16_w=1")
__msg("mark_precise: frame0: parent state regs= stack=-8: R0=1 R1=ctx() R6=map_value(map=.data.two_byte_,ks=4,vs=2) R10=fp0 fp-8=P1 fp-16=1")
__msg("mark_precise: frame0: last_idx 6 first_idx 3 subseq_idx 7")
__msg("mark_precise: frame0: regs= stack=-8 before 6: (05) goto pc+0")
__msg("mark_precise: frame0: regs= stack=-8 before 5: (7b) *(u64 *)(r10 -16) = r0")
__msg("mark_precise: frame0: regs= stack=-8 before 4: (b7) r0 = 1")
__msg("mark_precise: frame0: regs= stack=-8 before 3: (7a) *(u64 *)(r10 -8) = 1")
__msg("10: R1_w=map_value(map=.data.two_byte_,ks=4,vs=2,off=1) R2_w=1")
__msg("10: R1=map_value(map=.data.two_byte_,ks=4,vs=2,off=1) R2=1")
/* validate load from fp-16, which was initialized using BPF_STX_MEM */
__msg("12: (79) r2 = *(u64 *)(r10 -16) ; R2_w=1 R10=fp0 fp-16=1")
__msg("12: (79) r2 = *(u64 *)(r10 -16) ; R2=1 R10=fp0 fp-16=1")
__msg("13: (0f) r1 += r2")
__msg("mark_precise: frame0: last_idx 13 first_idx 7 subseq_idx -1")
__msg("mark_precise: frame0: regs=r2 stack= before 12: (79) r2 = *(u64 *)(r10 -16)")
@ -668,12 +668,12 @@ __msg("mark_precise: frame0: regs= stack=-16 before 9: (0f) r1 += r2")
__msg("mark_precise: frame0: regs= stack=-16 before 8: (79) r2 = *(u64 *)(r10 -8)")
__msg("mark_precise: frame0: regs= stack=-16 before 7: (bf) r1 = r6")
/* now both fp-8 and fp-16 are precise, very good */
__msg("mark_precise: frame0: parent state regs= stack=-16: R0_w=1 R1=ctx() R6_r=map_value(map=.data.two_byte_,ks=4,vs=2) R10=fp0 fp-8_rw=P1 fp-16_rw=P1")
__msg("mark_precise: frame0: parent state regs= stack=-16: R0=1 R1=ctx() R6=map_value(map=.data.two_byte_,ks=4,vs=2) R10=fp0 fp-8=P1 fp-16=P1")
__msg("mark_precise: frame0: last_idx 6 first_idx 3 subseq_idx 7")
__msg("mark_precise: frame0: regs= stack=-16 before 6: (05) goto pc+0")
__msg("mark_precise: frame0: regs= stack=-16 before 5: (7b) *(u64 *)(r10 -16) = r0")
__msg("mark_precise: frame0: regs=r0 stack= before 4: (b7) r0 = 1")
__msg("14: R1_w=map_value(map=.data.two_byte_,ks=4,vs=2,off=1) R2_w=1")
__msg("14: R1=map_value(map=.data.two_byte_,ks=4,vs=2,off=1) R2=1")
__naked void stack_load_preserves_const_precision(void)
{
asm volatile (
@ -719,22 +719,22 @@ __success
/* make sure fp-8 is 32-bit FAKE subregister spill */
__msg("3: (62) *(u32 *)(r10 -8) = 1 ; R10=fp0 fp-8=????1")
/* but fp-16 is spilled IMPRECISE zero const reg */
__msg("5: (63) *(u32 *)(r10 -16) = r0 ; R0_w=1 R10=fp0 fp-16=????1")
__msg("5: (63) *(u32 *)(r10 -16) = r0 ; R0=1 R10=fp0 fp-16=????1")
/* validate load from fp-8, which was initialized using BPF_ST_MEM */
__msg("8: (61) r2 = *(u32 *)(r10 -8) ; R2_w=1 R10=fp0 fp-8=????1")
__msg("8: (61) r2 = *(u32 *)(r10 -8) ; R2=1 R10=fp0 fp-8=????1")
__msg("9: (0f) r1 += r2")
__msg("mark_precise: frame0: last_idx 9 first_idx 7 subseq_idx -1")
__msg("mark_precise: frame0: regs=r2 stack= before 8: (61) r2 = *(u32 *)(r10 -8)")
__msg("mark_precise: frame0: regs= stack=-8 before 7: (bf) r1 = r6")
__msg("mark_precise: frame0: parent state regs= stack=-8: R0_w=1 R1=ctx() R6_r=map_value(map=.data.two_byte_,ks=4,vs=2) R10=fp0 fp-8_r=????P1 fp-16=????1")
__msg("mark_precise: frame0: parent state regs= stack=-8: R0=1 R1=ctx() R6=map_value(map=.data.two_byte_,ks=4,vs=2) R10=fp0 fp-8=????P1 fp-16=????1")
__msg("mark_precise: frame0: last_idx 6 first_idx 3 subseq_idx 7")
__msg("mark_precise: frame0: regs= stack=-8 before 6: (05) goto pc+0")
__msg("mark_precise: frame0: regs= stack=-8 before 5: (63) *(u32 *)(r10 -16) = r0")
__msg("mark_precise: frame0: regs= stack=-8 before 4: (b7) r0 = 1")
__msg("mark_precise: frame0: regs= stack=-8 before 3: (62) *(u32 *)(r10 -8) = 1")
__msg("10: R1_w=map_value(map=.data.two_byte_,ks=4,vs=2,off=1) R2_w=1")
__msg("10: R1=map_value(map=.data.two_byte_,ks=4,vs=2,off=1) R2=1")
/* validate load from fp-16, which was initialized using BPF_STX_MEM */
__msg("12: (61) r2 = *(u32 *)(r10 -16) ; R2_w=1 R10=fp0 fp-16=????1")
__msg("12: (61) r2 = *(u32 *)(r10 -16) ; R2=1 R10=fp0 fp-16=????1")
__msg("13: (0f) r1 += r2")
__msg("mark_precise: frame0: last_idx 13 first_idx 7 subseq_idx -1")
__msg("mark_precise: frame0: regs=r2 stack= before 12: (61) r2 = *(u32 *)(r10 -16)")
@ -743,12 +743,12 @@ __msg("mark_precise: frame0: regs= stack=-16 before 10: (73) *(u8 *)(r1 +0) = r2
__msg("mark_precise: frame0: regs= stack=-16 before 9: (0f) r1 += r2")
__msg("mark_precise: frame0: regs= stack=-16 before 8: (61) r2 = *(u32 *)(r10 -8)")
__msg("mark_precise: frame0: regs= stack=-16 before 7: (bf) r1 = r6")
__msg("mark_precise: frame0: parent state regs= stack=-16: R0_w=1 R1=ctx() R6_r=map_value(map=.data.two_byte_,ks=4,vs=2) R10=fp0 fp-8_r=????P1 fp-16_r=????P1")
__msg("mark_precise: frame0: parent state regs= stack=-16: R0=1 R1=ctx() R6=map_value(map=.data.two_byte_,ks=4,vs=2) R10=fp0 fp-8=????P1 fp-16=????P1")
__msg("mark_precise: frame0: last_idx 6 first_idx 3 subseq_idx 7")
__msg("mark_precise: frame0: regs= stack=-16 before 6: (05) goto pc+0")
__msg("mark_precise: frame0: regs= stack=-16 before 5: (63) *(u32 *)(r10 -16) = r0")
__msg("mark_precise: frame0: regs=r0 stack= before 4: (b7) r0 = 1")
__msg("14: R1_w=map_value(map=.data.two_byte_,ks=4,vs=2,off=1) R2_w=1")
__msg("14: R1=map_value(map=.data.two_byte_,ks=4,vs=2,off=1) R2=1")
__naked void stack_load_preserves_const_precision_subreg(void)
{
asm volatile (

6
tools/testing/selftests/bpf/progs/verifier_subprog_precision.c
View File

@ -105,7 +105,7 @@ __msg("mark_precise: frame0: regs=r0 stack= before 4: (27) r0 *= 4")
__msg("mark_precise: frame0: regs=r0 stack= before 3: (57) r0 &= 3")
__msg("mark_precise: frame0: regs=r0 stack= before 10: (95) exit")
__msg("mark_precise: frame1: regs=r0 stack= before 9: (bf) r0 = (s8)r10")
__msg("7: R0_w=scalar")
__msg("7: R0=scalar")
__naked int fp_precise_subprog_result(void)
{
asm volatile (
@ -141,7 +141,7 @@ __msg("mark_precise: frame1: regs=r0 stack= before 10: (bf) r0 = (s8)r1")
* anyways, at which point we'll break precision chain
*/
__msg("mark_precise: frame1: regs=r1 stack= before 9: (bf) r1 = r10")
__msg("7: R0_w=scalar")
__msg("7: R0=scalar")
__naked int sneaky_fp_precise_subprog_result(void)
{
asm volatile (
@ -681,7 +681,7 @@ __msg("mark_precise: frame0: last_idx 10 first_idx 7 subseq_idx -1")
__msg("mark_precise: frame0: regs=r7 stack= before 9: (bf) r1 = r8")
__msg("mark_precise: frame0: regs=r7 stack= before 8: (27) r7 *= 4")
__msg("mark_precise: frame0: regs=r7 stack= before 7: (79) r7 = *(u64 *)(r10 -8)")
__msg("mark_precise: frame0: parent state regs= stack=-8: R0_w=2 R6_w=1 R8_rw=map_value(map=.data.vals,ks=4,vs=16) R10=fp0 fp-8_rw=P1")
__msg("mark_precise: frame0: parent state regs= stack=-8: R0=2 R6=1 R8=map_value(map=.data.vals,ks=4,vs=16) R10=fp0 fp-8=P1")
__msg("mark_precise: frame0: last_idx 18 first_idx 0 subseq_idx 7")
__msg("mark_precise: frame0: regs= stack=-8 before 18: (95) exit")
__msg("mark_precise: frame1: regs= stack= before 17: (0f) r0 += r2")

199
tools/testing/selftests/bpf/test_loader.c
View File

@ -2,7 +2,6 @@
/* Copyright (c) 2022 Meta Platforms, Inc. and affiliates. */
#include <linux/capability.h>
#include <stdlib.h>
#include <regex.h>
#include <test_progs.h>
#include <bpf/btf.h>
@ -20,10 +19,12 @@
#define TEST_TAG_EXPECT_FAILURE "comment:test_expect_failure"
#define TEST_TAG_EXPECT_SUCCESS "comment:test_expect_success"
#define TEST_TAG_EXPECT_MSG_PFX "comment:test_expect_msg="
#define TEST_TAG_EXPECT_NOT_MSG_PFX "comment:test_expect_not_msg="
#define TEST_TAG_EXPECT_XLATED_PFX "comment:test_expect_xlated="
#define TEST_TAG_EXPECT_FAILURE_UNPRIV "comment:test_expect_failure_unpriv"
#define TEST_TAG_EXPECT_SUCCESS_UNPRIV "comment:test_expect_success_unpriv"
#define TEST_TAG_EXPECT_MSG_PFX_UNPRIV "comment:test_expect_msg_unpriv="
#define TEST_TAG_EXPECT_NOT_MSG_PFX_UNPRIV "comment:test_expect_not_msg_unpriv="
#define TEST_TAG_EXPECT_XLATED_PFX_UNPRIV "comment:test_expect_xlated_unpriv="
#define TEST_TAG_LOG_LEVEL_PFX "comment:test_log_level="
#define TEST_TAG_PROG_FLAGS_PFX "comment:test_prog_flags="
@ -65,18 +66,6 @@ enum load_mode {
NO_JITED = 1 << 1,
};
struct expect_msg {
const char *substr; /* substring match */
regex_t regex;
bool is_regex;
bool on_next_line;
};
struct expected_msgs {
struct expect_msg *patterns;
size_t cnt;
};
struct test_subspec {
char *name;
bool expect_failure;
@ -216,7 +205,8 @@ static int compile_regex(const char *pattern, regex_t *regex)
return 0;
}
static int __push_msg(const char *pattern, bool on_next_line, struct expected_msgs *msgs)
static int __push_msg(const char *pattern, bool on_next_line, bool negative,
struct expected_msgs *msgs)
{
struct expect_msg *msg;
void *tmp;
@ -232,6 +222,7 @@ static int __push_msg(const char *pattern, bool on_next_line, struct expected_ms
msg = &msgs->patterns[msgs->cnt];
msg->on_next_line = on_next_line;
msg->substr = pattern;
msg->negative = negative;
msg->is_regex = false;
if (strstr(pattern, "{{")) {
err = compile_regex(pattern, &msg->regex);
@ -250,16 +241,16 @@ static int clone_msgs(struct expected_msgs *from, struct expected_msgs *to)
for (i = 0; i < from->cnt; i++) {
msg = &from->patterns[i];
err = __push_msg(msg->substr, msg->on_next_line, to);
err = __push_msg(msg->substr, msg->on_next_line, msg->negative, to);
if (err)
return err;
}
return 0;
}
static int push_msg(const char *substr, struct expected_msgs *msgs)
static int push_msg(const char *substr, bool negative, struct expected_msgs *msgs)
{
return __push_msg(substr, false, msgs);
return __push_msg(substr, false, negative, msgs);
}
static int push_disasm_msg(const char *regex_str, bool *on_next_line, struct expected_msgs *msgs)
@ -270,7 +261,7 @@ static int push_disasm_msg(const char *regex_str, bool *on_next_line, struct exp
*on_next_line = false;
return 0;
}
err = __push_msg(regex_str, *on_next_line, msgs);
err = __push_msg(regex_str, *on_next_line, false, msgs);
if (err)
return err;
*on_next_line = true;
@ -482,12 +473,22 @@ static int parse_test_spec(struct test_loader *tester,
spec->auxiliary = true;
spec->mode_mask |= UNPRIV;
} else if ((msg = skip_dynamic_pfx(s, TEST_TAG_EXPECT_MSG_PFX))) {
err = push_msg(msg, &spec->priv.expect_msgs);
err = push_msg(msg, false, &spec->priv.expect_msgs);
if (err)
goto cleanup;
spec->mode_mask |= PRIV;
} else if ((msg = skip_dynamic_pfx(s, TEST_TAG_EXPECT_NOT_MSG_PFX))) {
err = push_msg(msg, true, &spec->priv.expect_msgs);
if (err)
goto cleanup;
spec->mode_mask |= PRIV;
} else if ((msg = skip_dynamic_pfx(s, TEST_TAG_EXPECT_MSG_PFX_UNPRIV))) {
err = push_msg(msg, &spec->unpriv.expect_msgs);
err = push_msg(msg, false, &spec->unpriv.expect_msgs);
if (err)
goto cleanup;
spec->mode_mask |= UNPRIV;
} else if ((msg = skip_dynamic_pfx(s, TEST_TAG_EXPECT_NOT_MSG_PFX_UNPRIV))) {
err = push_msg(msg, true, &spec->unpriv.expect_msgs);
if (err)
goto cleanup;
spec->mode_mask |= UNPRIV;
@ -764,44 +765,141 @@ static void emit_stdout(const char *bpf_stdout, bool force)
fprintf(stdout, "STDOUT:\n=============\n%s=============\n", bpf_stdout);
}
static void validate_msgs(char *log_buf, struct expected_msgs *msgs,
void (*emit_fn)(const char *buf, bool force))
static const char *match_msg(struct expect_msg *msg, const char **log)
{
const char *log = log_buf, *prev_match;
const char *match = NULL;
regmatch_t reg_match[1];
int prev_match_line;
int match_line;
int i, j, err;
int err;
if (!msg->is_regex) {
match = strstr(*log, msg->substr);
if (match)
*log = match + strlen(msg->substr);
} else {
err = regexec(&msg->regex, *log, 1, reg_match, 0);
if (err == 0) {
match = *log + reg_match[0].rm_so;
*log += reg_match[0].rm_eo;
}
}
return match;
}
static int count_lines(const char *start, const char *end)
{
const char *tmp;
int n = 0;
for (tmp = start; tmp < end; ++tmp)
if (*tmp == '\n')
n++;
return n;
}
struct match {
const char *start;
const char *end;
int line;
};
/*
* Positive messages are matched sequentially, each next message
* is looked for starting from the end of a previous matched one.
*/
static void match_positive_msgs(const char *log, struct expected_msgs *msgs, struct match *matches)
{
const char *prev_match;
int i, line;
prev_match_line = -1;
match_line = 0;
prev_match = log;
line = 0;
for (i = 0; i < msgs->cnt; i++) {
struct expect_msg *msg = &msgs->patterns[i];
const char *match = NULL, *pat_status;
bool wrong_line = false;
const char *match = NULL;
if (!msg->is_regex) {
match = strstr(log, msg->substr);
if (match)
log = match + strlen(msg->substr);
} else {
err = regexec(&msg->regex, log, 1, reg_match, 0);
if (err == 0) {
match = log + reg_match[0].rm_so;
log += reg_match[0].rm_eo;
if (msg->negative)
continue;
match = match_msg(msg, &log);
if (match) {
line += count_lines(prev_match, match);
matches[i].start = match;
matches[i].end = log;
matches[i].line = line;
prev_match = match;
}
}
}
/*
* Each negative messages N located between positive messages P1 and P2
* is matched in the span P1.end .. P2.start. Consequently, negative messages
* are unordered within the span.
*/
static void match_negative_msgs(const char *log, struct expected_msgs *msgs, struct match *matches)
{
const char *start = log, *end, *next, *match;
const char *log_end = log + strlen(log);
int i, j, next_positive;
for (i = 0; i < msgs->cnt; i++) {
struct expect_msg *msg = &msgs->patterns[i];
/* positive message bumps span start */
if (!msg->negative) {
start = matches[i].end ?: start;
continue;
}
/* count stride of negative patterns and adjust span end */
end = log_end;
for (next_positive = i + 1; next_positive < msgs->cnt; next_positive++) {
if (!msgs->patterns[next_positive].negative) {
end = matches[next_positive].start;
break;
}
}
if (match) {
for (; prev_match < match; ++prev_match)
if (*prev_match == '\n')
++match_line;
wrong_line = msg->on_next_line && prev_match_line >= 0 &&
prev_match_line + 1 != match_line;
/* try matching negative messages within identified span */
for (j = i; j < next_positive; j++) {
next = start;
match = match_msg(msg, &next);
if (match && next <= end) {
matches[j].start = match;
matches[j].end = next;
}
}
if (!match || wrong_line) {
/* -1 to account for i++ */
i = next_positive - 1;
}
}
void validate_msgs(const char *log_buf, struct expected_msgs *msgs,
void (*emit_fn)(const char *buf, bool force))
{
struct match matches[msgs->cnt];
struct match *prev_match = NULL;
int i, j;
memset(matches, 0, sizeof(*matches) * msgs->cnt);
match_positive_msgs(log_buf, msgs, matches);
match_negative_msgs(log_buf, msgs, matches);
for (i = 0; i < msgs->cnt; i++) {
struct expect_msg *msg = &msgs->patterns[i];
struct match *match = &matches[i];
const char *pat_status;
bool unexpected;
bool wrong_line;
bool no_match;
no_match = !msg->negative && !match->start;
wrong_line = !msg->negative &&
msg->on_next_line &&
prev_match && prev_match->line + 1 != match->line;
unexpected = msg->negative && match->start;
if (no_match || wrong_line || unexpected) {
PRINT_FAIL("expect_msg\n");
if (env.verbosity == VERBOSE_NONE)
emit_fn(log_buf, true /*force*/);
@ -811,8 +909,10 @@ static void validate_msgs(char *log_buf, struct expected_msgs *msgs,
pat_status = "MATCHED ";
else if (wrong_line)
pat_status = "WRONG LINE";
else
else if (no_match)
pat_status = "EXPECTED ";
else
pat_status = "UNEXPECTED";
msg = &msgs->patterns[j];
fprintf(stderr, "%s %s: '%s'\n",
pat_status,
@ -822,12 +922,13 @@ static void validate_msgs(char *log_buf, struct expected_msgs *msgs,
if (wrong_line) {
fprintf(stderr,
"expecting match at line %d, actual match is at line %d\n",
prev_match_line + 1, match_line);
prev_match->line + 1, match->line);
}
break;
}
prev_match_line = match_line;
if (!msg->negative)
prev_match = match;
}
}

17
tools/testing/selftests/bpf/test_progs.h
View File

@ -7,6 +7,7 @@
#include <errno.h>
#include <string.h>
#include <assert.h>
#include <regex.h>
#include <stdlib.h>
#include <stdarg.h>
#include <time.h>
@ -546,4 +547,20 @@ extern void test_loader_fini(struct test_loader *tester);
test_loader_fini(&tester); \
})
struct expect_msg {
const char *substr; /* substring match */
regex_t regex;
bool is_regex;
bool on_next_line;
bool negative;
};
struct expected_msgs {
struct expect_msg *patterns;
size_t cnt;
};
void validate_msgs(const char *log_buf, struct expected_msgs *msgs,
void (*emit_fn)(const char *buf, bool force));
#endif /* __TEST_PROGS_H */

4
tools/testing/selftests/bpf/verifier/bpf_st_mem.c
View File

@ -93,7 +93,7 @@
.expected_attach_type = BPF_SK_LOOKUP,
.result = VERBOSE_ACCEPT,
.runs = -1,
.errstr = "0: (7a) *(u64 *)(r10 -8) = -44 ; R10=fp0 fp-8_w=-44\
.errstr = "0: (7a) *(u64 *)(r10 -8) = -44 ; R10=fp0 fp-8=-44\
2: (c5) if r0 s< 0x0 goto pc+2\
R0_w=-44",
R0=-44",
},