mirror of
https://github.com/torvalds/linux.git
synced 2026-06-02 19:43:40 +02:00
Merge branch 'bpf-static-stack-liveness-data-flow-analysis'
Eduard Zingerman says:
====================
bpf: static stack liveness data flow analysis
This patch set converts current dynamic stack slot liveness tracking
mechanism to a static data flow analysis. The result is used during
state pruning (clean_verifier_state): to zero out dead stack slots,
enabling more aggressive state equivalence and pruning. To improve
analysis precision live stack slot tracking is converted to 4-byte
granularity.
The key ideas and the bulk of the execution behind the series belong
to Alexei Starovoitov. I contributed to patch set integration
with existing liveness tracking mechanism.
Due to complexity of the changes the bisectability property of the
patch set is not preserved. Some selftests may fail between
intermediate patches of the series.
Analysis consists of two passes:
- A forward fixed-point analysis that tracks which frame's FP each
register value is derived from, and at what byte offset. This is
needed because a callee can receive a pointer to its caller's stack
frame (e.g. r1 = fp-16 at the call site), then do *(u64 *)(r1 + 0)
inside the callee - a cross-frame stack access that the callee's
local liveness must attribute to the caller's stack.
- A backward dataflow pass within each callee subprog that computes
live_in = (live_out \ def) ∪ use for both local and non-local
(ancestor) stack slots. The result of the analysis for callee is
propagated up to the callsite.
The key idea making such analysis possible is that limited and
conservative argument tracking pass is sufficient to recover most of
the offsets / stack pointer arguments.
Changelog:
v3 -> v4:
liveness.c:
- fill_from_stack(): correct conservative stack mask for imprecise
result, instead of picking frames from pointer register
(Alexei, sashiko).
- spill_to_stack(): join with existing values instead of
overwriting when dst has multiple offsets (cnt > 1) or imprecise
offset (cnt == 0) (Alexei, sashiko).
- analyze_subprog(): big change, now each analyze_subprog() is
called with a fresh func_instance, once read/write marks are
collected the instance is joined with the one accumulated for
(callsite, depth) and update_instance() is called.
This handles several issues:
- Avoids stale must_write marks when same func_instance is reused
by analyze_subprog() several times.
- Handles potential calls multiple calls for mark_stack_write()
within single instruction.
(Alexei, sashiko).
- analyze_subprog(): added complexity limit to avoid exponential
analysis time blowup for crafted programs with lots of nested
function calls (Alexei, sashiko).
- the patch "bpf: record arg tracking results in bpf_liveness masks"
is reinstated, it was accidentally squashed during v1->v2
transition.
verifier.c:
- clean_live_states() is replaced by a direct call to
clean_verifier_state(), bpf_verifier_state->cleaned is dropped.
verifier_live_stack.c:
- added selftests for arg tracking changes.
v2 -> v3:
liveness.c:
- record_stack_access(): handle S64_MIN (unknown read) with
imprecise offset. Test case can't be created with existing
helpers/kfuncs (sashiko).
- fmt_subprog(): handle NULL name (subprogs without BTF info).
- print_instance(): use u64 for pos/insn_pos avoid truncation
(bot+bpf-ci).
- compute_subprog_args(): return error if
'env->callsite_at_stack[idx] = kvmalloc_objs(...)' fails
(sashiko).
- clear_overlapping_stack_slots(): avoid integer promoting
issues by adding explicit (int) cast (sashiko).
bpf_verifier.h, verifier.c, liveness.c:
- Fixes in comments and commit messages (bot+bpf-ci).
v1 -> v2:
liveness.c:
- Removed func_instance->callsites and replaced it with explicit
spine passed through analys_subprog() calls (sashiko).
- Fixed BPF_LOAD_ACQ handling in arg_track_xfer: don't clear dst
register tracking (sashiko).
- Various error threading nits highlighted by bots
(sashiko, bot+bpf-ci).
- Massaged fmt_spis_mask() to be more concise (Alexei)
verifier.c:
- Move subprog_info[i].name assignment from add_subprog_and_kfunc to
check_btf_func (sashiko, bot+bpf-ci).
- Fixed inverse usage of msb/lsb halves by patch
"bpf: make liveness.c track stack with 4-byte granularity"
(sashiko, bot+bpf-ci).
v1: https://lore.kernel.org/bpf/20260408-patch-set-v1-0-1a666e860d42@gmail.com/
v2: https://lore.kernel.org/bpf/20260409-patch-set-v2-0-651804512349@gmail.com/
v3: https://lore.kernel.org/bpf/20260410-patch-set-v3-0-1f5826dc0ef2@gmail.com/
Verification performance impact (negative % is good):
========= selftests: master vs patch-set =========
File Program Insns (A) Insns (B) Insns (DIFF)
----------------------- ------------- --------- --------- ---------------
xdp_synproxy_kern.bpf.o syncookie_tc 20363 22910 +2547 (+12.51%)
xdp_synproxy_kern.bpf.o syncookie_xdp 20450 23001 +2551 (+12.47%)
Total progs: 4490
Old success: 2856
New success: 2856
total_insns diff min: -80.26%
total_insns diff max: 12.51%
0 -> value: 0
value -> 0: 0
total_insns abs max old: 837,487
total_insns abs max new: 837,487
-85 .. -75 %: 1
-50 .. -40 %: 1
-35 .. -25 %: 1
-20 .. -10 %: 5
-10 .. 0 %: 18
0 .. 5 %: 4458
5 .. 15 %: 6
========= scx: master vs patch-set =========
File Program Insns (A) Insns (B) Insns (DIFF)
-------------- --------- --------- --------- --------------
scx_qmap.bpf.o qmap_init 20230 19022 -1208 (-5.97%)
Total progs: 376
Old success: 351
New success: 351
total_insns diff min: -27.15%
total_insns diff max: 0.50%
0 -> value: 0
value -> 0: 0
total_insns abs max old: 236,251
total_insns abs max new: 233,669
-30 .. -20 %: 8
-20 .. -10 %: 2
-10 .. 0 %: 21
0 .. 5 %: 345
========= meta: master vs patch-set =========
File Program Insns (A) Insns (B) Insns (DIFF)
---------------------------------------------------------------------------- ----------------- --------- --------- -----------------
...
third-party-scx-backports-scheds-rust-scx_layered-bpf_skel_genskel-bpf.bpf.o layered_dispatch 13944 13104 -840 (-6.02%)
third-party-scx-backports-scheds-rust-scx_layered-bpf_skel_genskel-bpf.bpf.o layered_dispatch 13944 13104 -840 (-6.02%)
third-party-scx-gefe21962f49a-__scx_layered_bpf_skel_genskel-bpf.bpf.o layered_dispatch 13825 12985 -840 (-6.08%)
third-party-scx-v1.0.16-__scx_lavd_bpf_skel_genskel-bpf.bpf.o lavd_enqueue 15501 13602 -1899 (-12.25%)
third-party-scx-v1.0.16-__scx_lavd_bpf_skel_genskel-bpf.bpf.o lavd_select_cpu 19814 16231 -3583 (-18.08%)
third-party-scx-v1.0.17-__scx_lavd_bpf_skel_genskel-bpf.bpf.o lavd_enqueue 15501 13602 -1899 (-12.25%)
third-party-scx-v1.0.17-__scx_lavd_bpf_skel_genskel-bpf.bpf.o lavd_select_cpu 19814 16231 -3583 (-18.08%)
third-party-scx-v1.0.17-__scx_layered_bpf_skel_genskel-bpf.bpf.o layered_dispatch 13976 13151 -825 (-5.90%)
third-party-scx-v1.0.18-__scx_lavd_bpf_skel_genskel-bpf.bpf.o lavd_dispatch 260628 237930 -22698 (-8.71%)
third-party-scx-v1.0.18-__scx_lavd_bpf_skel_genskel-bpf.bpf.o lavd_enqueue 13437 12225 -1212 (-9.02%)
third-party-scx-v1.0.18-__scx_lavd_bpf_skel_genskel-bpf.bpf.o lavd_select_cpu 17744 14730 -3014 (-16.99%)
third-party-scx-v1.0.19-10-6b1958477-__scx_lavd_bpf_skel_genskel-bpf.bpf.o lavd_cpu_offline 19676 18418 -1258 (-6.39%)
third-party-scx-v1.0.19-10-6b1958477-__scx_lavd_bpf_skel_genskel-bpf.bpf.o lavd_cpu_online 19674 18416 -1258 (-6.39%)
...
Total progs: 1540
Old success: 1492
New success: 1493
total_insns diff min: -75.83%
total_insns diff max: 73.60%
0 -> value: 0
value -> 0: 0
total_insns abs max old: 434,763
total_insns abs max new: 666,036
-80 .. -70 %: 2
-55 .. -50 %: 7
-50 .. -45 %: 10
-45 .. -35 %: 4
-35 .. -25 %: 4
-25 .. -20 %: 8
-20 .. -15 %: 15
-15 .. -10 %: 11
-10 .. -5 %: 45
-5 .. 0 %: 112
0 .. 5 %: 1316
5 .. 15 %: 2
15 .. 25 %: 1
25 .. 35 %: 1
55 .. 65 %: 1
70 .. 75 %: 1
========= cilium: master vs patch-set =========
File Program Insns (A) Insns (B) Insns (DIFF)
--------------- --------------------------------- --------- --------- ----------------
bpf_host.o cil_host_policy 45801 32027 -13774 (-30.07%)
bpf_host.o cil_to_netdev 100287 69042 -31245 (-31.16%)
bpf_host.o tail_handle_ipv4_cont_from_host 60911 20962 -39949 (-65.59%)
bpf_host.o tail_handle_ipv4_from_netdev 59735 33155 -26580 (-44.50%)
bpf_host.o tail_handle_ipv6_cont_from_host 23529 17036 -6493 (-27.60%)
bpf_host.o tail_handle_ipv6_from_host 11906 10303 -1603 (-13.46%)
bpf_host.o tail_handle_ipv6_from_netdev 29778 23743 -6035 (-20.27%)
bpf_host.o tail_handle_snat_fwd_ipv4 61616 67463 +5847 (+9.49%)
bpf_host.o tail_handle_snat_fwd_ipv6 30802 22806 -7996 (-25.96%)
bpf_host.o tail_ipv4_host_policy_ingress 20017 10528 -9489 (-47.40%)
bpf_host.o tail_ipv6_host_policy_ingress 20693 17301 -3392 (-16.39%)
bpf_host.o tail_nodeport_nat_egress_ipv4 16455 13684 -2771 (-16.84%)
bpf_host.o tail_nodeport_nat_ingress_ipv4 36174 20080 -16094 (-44.49%)
bpf_host.o tail_nodeport_nat_ingress_ipv6 48039 25779 -22260 (-46.34%)
bpf_lxc.o tail_handle_ipv4 13765 10001 -3764 (-27.34%)
bpf_lxc.o tail_handle_ipv4_cont 96891 68725 -28166 (-29.07%)
bpf_lxc.o tail_handle_ipv6_cont 21809 17697 -4112 (-18.85%)
bpf_lxc.o tail_ipv4_ct_egress 15949 17746 +1797 (+11.27%)
bpf_lxc.o tail_nodeport_nat_egress_ipv4 16183 13432 -2751 (-17.00%)
bpf_lxc.o tail_nodeport_nat_ingress_ipv4 18532 10697 -7835 (-42.28%)
bpf_overlay.o tail_handle_inter_cluster_revsnat 15708 11099 -4609 (-29.34%)
bpf_overlay.o tail_handle_ipv4 105672 76108 -29564 (-27.98%)
bpf_overlay.o tail_handle_ipv6 15733 19944 +4211 (+26.77%)
bpf_overlay.o tail_handle_snat_fwd_ipv4 19327 26468 +7141 (+36.95%)
bpf_overlay.o tail_handle_snat_fwd_ipv6 20817 12556 -8261 (-39.68%)
bpf_overlay.o tail_nodeport_nat_egress_ipv4 16175 12184 -3991 (-24.67%)
bpf_overlay.o tail_nodeport_nat_ingress_ipv4 20760 11951 -8809 (-42.43%)
bpf_wireguard.o tail_handle_ipv4 27466 28909 +1443 (+5.25%)
bpf_wireguard.o tail_nodeport_nat_egress_ipv4 15937 12094 -3843 (-24.11%)
bpf_wireguard.o tail_nodeport_nat_ingress_ipv4 20624 11993 -8631 (-41.85%)
bpf_xdp.o tail_lb_ipv4 42673 60855 +18182 (+42.61%)
bpf_xdp.o tail_lb_ipv6 87903 108585 +20682 (+23.53%)
bpf_xdp.o tail_nodeport_nat_ingress_ipv4 28787 20991 -7796 (-27.08%)
bpf_xdp.o tail_nodeport_nat_ingress_ipv6 207593 152012 -55581 (-26.77%)
Total progs: 134
Old success: 134
New success: 134
total_insns diff min: -65.59%
total_insns diff max: 42.61%
0 -> value: 0
value -> 0: 0
total_insns abs max old: 207,593
total_insns abs max new: 152,012
-70 .. -60 %: 1
-50 .. -40 %: 7
-40 .. -30 %: 9
-30 .. -25 %: 9
-25 .. -20 %: 12
-20 .. -15 %: 7
-15 .. -10 %: 14
-10 .. -5 %: 6
-5 .. 0 %: 16
0 .. 5 %: 42
5 .. 15 %: 5
15 .. 25 %: 2
25 .. 35 %: 2
35 .. 45 %: 2
====================
Link: https://patch.msgid.link/20260410-patch-set-v4-0-5d4eecb343db@gmail.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
This commit is contained in:
Alexei Starovoitov
commit
e2e6a6ea24
|
|
@ -220,10 +220,61 @@ enum bpf_stack_slot_type {
|
|||
STACK_DYNPTR,
|
||||
STACK_ITER,
|
||||
STACK_IRQ_FLAG,
|
||||
STACK_POISON,
|
||||
};
|
||||
|
||||
#define BPF_REG_SIZE 8 /* size of eBPF register in bytes */
|
||||
|
||||
/* 4-byte stack slot granularity for liveness analysis */
|
||||
#define BPF_HALF_REG_SIZE 4
|
||||
#define STACK_SLOT_SZ 4
|
||||
#define STACK_SLOTS (MAX_BPF_STACK / BPF_HALF_REG_SIZE) /* 128 */
|
||||
|
||||
typedef struct {
|
||||
u64 v[2];
|
||||
} spis_t;
|
||||
|
||||
#define SPIS_ZERO ((spis_t){})
|
||||
#define SPIS_ALL ((spis_t){{ U64_MAX, U64_MAX }})
|
||||
|
||||
static inline bool spis_is_zero(spis_t s)
|
||||
{
|
||||
return s.v[0] == 0 && s.v[1] == 0;
|
||||
}
|
||||
|
||||
static inline bool spis_equal(spis_t a, spis_t b)
|
||||
{
|
||||
return a.v[0] == b.v[0] && a.v[1] == b.v[1];
|
||||
}
|
||||
|
||||
static inline spis_t spis_or(spis_t a, spis_t b)
|
||||
{
|
||||
return (spis_t){{ a.v[0] | b.v[0], a.v[1] | b.v[1] }};
|
||||
}
|
||||
|
||||
static inline spis_t spis_and(spis_t a, spis_t b)
|
||||
{
|
||||
return (spis_t){{ a.v[0] & b.v[0], a.v[1] & b.v[1] }};
|
||||
}
|
||||
|
||||
static inline spis_t spis_not(spis_t s)
|
||||
{
|
||||
return (spis_t){{ ~s.v[0], ~s.v[1] }};
|
||||
}
|
||||
|
||||
static inline bool spis_test_bit(spis_t s, u32 slot)
|
||||
{
|
||||
return s.v[slot / 64] & BIT_ULL(slot % 64);
|
||||
}
|
||||
|
||||
static inline void spis_or_range(spis_t *mask, u32 lo, u32 hi)
|
||||
{
|
||||
u32 w;
|
||||
|
||||
for (w = lo; w <= hi && w < STACK_SLOTS; w++)
|
||||
mask->v[w / 64] |= BIT_ULL(w % 64);
|
||||
}
|
||||
|
||||
#define BPF_REGMASK_ARGS ((1 << BPF_REG_1) | (1 << BPF_REG_2) | \
|
||||
(1 << BPF_REG_3) | (1 << BPF_REG_4) | \
|
||||
(1 << BPF_REG_5))
|
||||
|
|
@ -424,7 +475,6 @@ struct bpf_verifier_state {
|
|||
|
||||
bool speculative;
|
||||
bool in_sleepable;
|
||||
bool cleaned;
|
||||
|
||||
/* first and last insn idx of this verifier state */
|
||||
u32 first_insn_idx;
|
||||
|
|
@ -664,7 +714,7 @@ enum priv_stack_mode {
|
|||
};
|
||||
|
||||
struct bpf_subprog_info {
|
||||
/* 'start' has to be the first field otherwise find_subprog() won't work */
|
||||
const char *name; /* name extracted from BTF */
|
||||
u32 start; /* insn idx of function entry point */
|
||||
u32 linfo_idx; /* The idx to the main_prog->aux->linfo */
|
||||
u32 postorder_start; /* The idx to the env->cfg.insn_postorder */
|
||||
|
|
@ -819,6 +869,8 @@ struct bpf_verifier_env {
|
|||
} cfg;
|
||||
struct backtrack_state bt;
|
||||
struct bpf_jmp_history_entry *cur_hist_ent;
|
||||
/* Per-callsite copy of parent's converged at_stack_in for cross-frame fills. */
|
||||
struct arg_track **callsite_at_stack;
|
||||
u32 pass_cnt; /* number of times do_check() was called */
|
||||
u32 subprog_cnt;
|
||||
/* number of instructions analyzed by the verifier */
|
||||
|
|
@ -1121,12 +1173,14 @@ void print_verifier_state(struct bpf_verifier_env *env, const struct bpf_verifie
|
|||
u32 frameno, bool print_all);
|
||||
void print_insn_state(struct bpf_verifier_env *env, const struct bpf_verifier_state *vstate,
|
||||
u32 frameno);
|
||||
u32 bpf_vlog_alignment(u32 pos);
|
||||
|
||||
struct bpf_subprog_info *bpf_find_containing_subprog(struct bpf_verifier_env *env, int off);
|
||||
int bpf_jmp_offset(struct bpf_insn *insn);
|
||||
struct bpf_iarray *bpf_insn_successors(struct bpf_verifier_env *env, u32 idx);
|
||||
void bpf_fmt_stack_mask(char *buf, ssize_t buf_sz, u64 stack_mask);
|
||||
bool bpf_calls_callback(struct bpf_verifier_env *env, int insn_idx);
|
||||
bool bpf_subprog_is_global(const struct bpf_verifier_env *env, int subprog);
|
||||
|
||||
int bpf_find_subprog(struct bpf_verifier_env *env, int off);
|
||||
int bpf_compute_const_regs(struct bpf_verifier_env *env);
|
||||
|
|
@ -1144,16 +1198,11 @@ s64 bpf_helper_stack_access_bytes(struct bpf_verifier_env *env,
|
|||
s64 bpf_kfunc_stack_access_bytes(struct bpf_verifier_env *env,
|
||||
struct bpf_insn *insn, int arg,
|
||||
int insn_idx);
|
||||
int bpf_compute_subprog_arg_access(struct bpf_verifier_env *env);
|
||||
|
||||
int bpf_stack_liveness_init(struct bpf_verifier_env *env);
|
||||
void bpf_stack_liveness_free(struct bpf_verifier_env *env);
|
||||
int bpf_update_live_stack(struct bpf_verifier_env *env);
|
||||
int bpf_mark_stack_read(struct bpf_verifier_env *env, u32 frameno, u32 insn_idx, u64 mask);
|
||||
void bpf_mark_stack_write(struct bpf_verifier_env *env, u32 frameno, u64 mask);
|
||||
int bpf_reset_stack_write_marks(struct bpf_verifier_env *env, u32 insn_idx);
|
||||
int bpf_commit_stack_write_marks(struct bpf_verifier_env *env);
|
||||
int bpf_live_stack_query_init(struct bpf_verifier_env *env, struct bpf_verifier_state *st);
|
||||
bool bpf_stack_slot_alive(struct bpf_verifier_env *env, u32 frameno, u32 spi);
|
||||
void bpf_reset_live_stack_callchain(struct bpf_verifier_env *env);
|
||||
|
||||
#endif /* _LINUX_BPF_VERIFIER_H */
|
||||
|
|
|
|||
File diff suppressed because it is too large
Load Diff
|
|
@ -501,7 +501,8 @@ static char slot_type_char[] = {
|
|||
[STACK_ZERO] = '0',
|
||||
[STACK_DYNPTR] = 'd',
|
||||
[STACK_ITER] = 'i',
|
||||
[STACK_IRQ_FLAG] = 'f'
|
||||
[STACK_IRQ_FLAG] = 'f',
|
||||
[STACK_POISON] = 'p',
|
||||
};
|
||||
|
||||
#define UNUM_MAX_DECIMAL U16_MAX
|
||||
|
|
@ -738,7 +739,7 @@ void print_verifier_state(struct bpf_verifier_env *env, const struct bpf_verifie
|
|||
|
||||
for (j = 0; j < BPF_REG_SIZE; j++) {
|
||||
slot_type = state->stack[i].slot_type[j];
|
||||
if (slot_type != STACK_INVALID)
|
||||
if (slot_type != STACK_INVALID && slot_type != STACK_POISON)
|
||||
valid = true;
|
||||
types_buf[j] = slot_type_char[slot_type];
|
||||
}
|
||||
|
|
@ -806,7 +807,7 @@ void print_verifier_state(struct bpf_verifier_env *env, const struct bpf_verifie
|
|||
mark_verifier_state_clean(env);
|
||||
}
|
||||
|
||||
static inline u32 vlog_alignment(u32 pos)
|
||||
u32 bpf_vlog_alignment(u32 pos)
|
||||
{
|
||||
return round_up(max(pos + BPF_LOG_MIN_ALIGNMENT / 2, BPF_LOG_ALIGNMENT),
|
||||
BPF_LOG_MIN_ALIGNMENT) - pos - 1;
|
||||
|
|
@ -818,7 +819,7 @@ void print_insn_state(struct bpf_verifier_env *env, const struct bpf_verifier_st
|
|||
if (env->prev_log_pos && env->prev_log_pos == env->log.end_pos) {
|
||||
/* remove new line character */
|
||||
bpf_vlog_reset(&env->log, env->prev_log_pos - 1);
|
||||
verbose(env, "%*c;", vlog_alignment(env->prev_insn_print_pos), ' ');
|
||||
verbose(env, "%*c;", bpf_vlog_alignment(env->prev_insn_print_pos), ' ');
|
||||
} else {
|
||||
verbose(env, "%d:", env->insn_idx);
|
||||
}
|
||||
|
|
|
|||
|
|
@ -423,7 +423,7 @@ static struct btf_record *reg_btf_record(const struct bpf_reg_state *reg)
|
|||
return rec;
|
||||
}
|
||||
|
||||
static bool subprog_is_global(const struct bpf_verifier_env *env, int subprog)
|
||||
bool bpf_subprog_is_global(const struct bpf_verifier_env *env, int subprog)
|
||||
{
|
||||
struct bpf_func_info_aux *aux = env->prog->aux->func_info_aux;
|
||||
|
||||
|
|
@ -830,8 +830,6 @@ static int mark_stack_slots_dynptr(struct bpf_verifier_env *env, struct bpf_reg_
|
|||
state->stack[spi - 1].spilled_ptr.ref_obj_id = id;
|
||||
}
|
||||
|
||||
bpf_mark_stack_write(env, state->frameno, BIT(spi - 1) | BIT(spi));
|
||||
|
||||
return 0;
|
||||
}
|
||||
|
||||
|
|
@ -846,8 +844,6 @@ static void invalidate_dynptr(struct bpf_verifier_env *env, struct bpf_func_stat
|
|||
|
||||
__mark_reg_not_init(env, &state->stack[spi].spilled_ptr);
|
||||
__mark_reg_not_init(env, &state->stack[spi - 1].spilled_ptr);
|
||||
|
||||
bpf_mark_stack_write(env, state->frameno, BIT(spi - 1) | BIT(spi));
|
||||
}
|
||||
|
||||
static int unmark_stack_slots_dynptr(struct bpf_verifier_env *env, struct bpf_reg_state *reg)
|
||||
|
|
@ -984,8 +980,6 @@ static int destroy_if_dynptr_stack_slot(struct bpf_verifier_env *env,
|
|||
__mark_reg_not_init(env, &state->stack[spi].spilled_ptr);
|
||||
__mark_reg_not_init(env, &state->stack[spi - 1].spilled_ptr);
|
||||
|
||||
bpf_mark_stack_write(env, state->frameno, BIT(spi - 1) | BIT(spi));
|
||||
|
||||
return 0;
|
||||
}
|
||||
|
||||
|
|
@ -1111,7 +1105,6 @@ static int mark_stack_slots_iter(struct bpf_verifier_env *env,
|
|||
for (j = 0; j < BPF_REG_SIZE; j++)
|
||||
slot->slot_type[j] = STACK_ITER;
|
||||
|
||||
bpf_mark_stack_write(env, state->frameno, BIT(spi - i));
|
||||
mark_stack_slot_scratched(env, spi - i);
|
||||
}
|
||||
|
||||
|
|
@ -1140,7 +1133,6 @@ static int unmark_stack_slots_iter(struct bpf_verifier_env *env,
|
|||
for (j = 0; j < BPF_REG_SIZE; j++)
|
||||
slot->slot_type[j] = STACK_INVALID;
|
||||
|
||||
bpf_mark_stack_write(env, state->frameno, BIT(spi - i));
|
||||
mark_stack_slot_scratched(env, spi - i);
|
||||
}
|
||||
|
||||
|
|
@ -1230,7 +1222,6 @@ static int mark_stack_slot_irq_flag(struct bpf_verifier_env *env,
|
|||
slot = &state->stack[spi];
|
||||
st = &slot->spilled_ptr;
|
||||
|
||||
bpf_mark_stack_write(env, reg->frameno, BIT(spi));
|
||||
__mark_reg_known_zero(st);
|
||||
st->type = PTR_TO_STACK; /* we don't have dedicated reg type */
|
||||
st->ref_obj_id = id;
|
||||
|
|
@ -1286,8 +1277,6 @@ static int unmark_stack_slot_irq_flag(struct bpf_verifier_env *env, struct bpf_r
|
|||
|
||||
__mark_reg_not_init(env, st);
|
||||
|
||||
bpf_mark_stack_write(env, reg->frameno, BIT(spi));
|
||||
|
||||
for (i = 0; i < BPF_REG_SIZE; i++)
|
||||
slot->slot_type[i] = STACK_INVALID;
|
||||
|
||||
|
|
@ -1359,6 +1348,7 @@ static bool is_stack_slot_special(const struct bpf_stack_state *stack)
|
|||
case STACK_IRQ_FLAG:
|
||||
return true;
|
||||
case STACK_INVALID:
|
||||
case STACK_POISON:
|
||||
case STACK_MISC:
|
||||
case STACK_ZERO:
|
||||
return false;
|
||||
|
|
@ -1388,9 +1378,11 @@ static bool is_spilled_scalar_after(const struct bpf_stack_state *stack, int im)
|
|||
stack->spilled_ptr.type == SCALAR_VALUE;
|
||||
}
|
||||
|
||||
/* Mark stack slot as STACK_MISC, unless it is already STACK_INVALID, in which
|
||||
* case they are equivalent, or it's STACK_ZERO, in which case we preserve
|
||||
* more precise STACK_ZERO.
|
||||
/*
|
||||
* Mark stack slot as STACK_MISC, unless it is already:
|
||||
* - STACK_INVALID, in which case they are equivalent.
|
||||
* - STACK_ZERO, in which case we preserve more precise STACK_ZERO.
|
||||
* - STACK_POISON, which truly forbids access to the slot.
|
||||
* Regardless of allow_ptr_leaks setting (i.e., privileged or unprivileged
|
||||
* mode), we won't promote STACK_INVALID to STACK_MISC. In privileged case it is
|
||||
* unnecessary as both are considered equivalent when loading data and pruning,
|
||||
|
|
@ -1401,14 +1393,14 @@ static void mark_stack_slot_misc(struct bpf_verifier_env *env, u8 *stype)
|
|||
{
|
||||
if (*stype == STACK_ZERO)
|
||||
return;
|
||||
if (*stype == STACK_INVALID)
|
||||
if (*stype == STACK_INVALID || *stype == STACK_POISON)
|
||||
return;
|
||||
*stype = STACK_MISC;
|
||||
}
|
||||
|
||||
static void scrub_spilled_slot(u8 *stype)
|
||||
{
|
||||
if (*stype != STACK_INVALID)
|
||||
if (*stype != STACK_INVALID && *stype != STACK_POISON)
|
||||
*stype = STACK_MISC;
|
||||
}
|
||||
|
||||
|
|
@ -1801,7 +1793,6 @@ static int copy_verifier_state(struct bpf_verifier_state *dst_state,
|
|||
return err;
|
||||
dst_state->speculative = src->speculative;
|
||||
dst_state->in_sleepable = src->in_sleepable;
|
||||
dst_state->cleaned = src->cleaned;
|
||||
dst_state->curframe = src->curframe;
|
||||
dst_state->branches = src->branches;
|
||||
dst_state->parent = src->parent;
|
||||
|
|
@ -3864,14 +3855,10 @@ static int sort_subprogs_topo(struct bpf_verifier_env *env)
|
|||
static int mark_stack_slot_obj_read(struct bpf_verifier_env *env, struct bpf_reg_state *reg,
|
||||
int spi, int nr_slots)
|
||||
{
|
||||
int err, i;
|
||||
int i;
|
||||
|
||||
for (i = 0; i < nr_slots; i++) {
|
||||
err = bpf_mark_stack_read(env, reg->frameno, env->insn_idx, BIT(spi - i));
|
||||
if (err)
|
||||
return err;
|
||||
for (i = 0; i < nr_slots; i++)
|
||||
mark_stack_slot_scratched(env, spi - i);
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
|
||||
|
|
@ -4631,7 +4618,7 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,
|
|||
if (subprog < 0)
|
||||
return -EFAULT;
|
||||
|
||||
if (subprog_is_global(env, subprog)) {
|
||||
if (bpf_subprog_is_global(env, subprog)) {
|
||||
/* check that jump history doesn't have any
|
||||
* extra instructions from subprog; the next
|
||||
* instruction after call to global subprog
|
||||
|
|
@ -5422,18 +5409,6 @@ static int check_stack_write_fixed_off(struct bpf_verifier_env *env,
|
|||
if (err)
|
||||
return err;
|
||||
|
||||
if (!(off % BPF_REG_SIZE) && size == BPF_REG_SIZE) {
|
||||
/* only mark the slot as written if all 8 bytes were written
|
||||
* otherwise read propagation may incorrectly stop too soon
|
||||
* when stack slots are partially written.
|
||||
* This heuristic means that read propagation will be
|
||||
* conservative, since it will add reg_live_read marks
|
||||
* to stack slots all the way to first state when programs
|
||||
* writes+reads less than 8 bytes
|
||||
*/
|
||||
bpf_mark_stack_write(env, state->frameno, BIT(spi));
|
||||
}
|
||||
|
||||
check_fastcall_stack_contract(env, state, insn_idx, off);
|
||||
mark_stack_slot_scratched(env, spi);
|
||||
if (reg && !(off % BPF_REG_SIZE) && reg->type == SCALAR_VALUE && env->bpf_capable) {
|
||||
|
|
@ -5614,8 +5589,10 @@ static int check_stack_write_var_off(struct bpf_verifier_env *env,
|
|||
* For privileged programs, we will accept such reads to slots
|
||||
* that may or may not be written because, if we're reject
|
||||
* them, the error would be too confusing.
|
||||
* Conservatively, treat STACK_POISON in a similar way.
|
||||
*/
|
||||
if (*stype == STACK_INVALID && !env->allow_uninit_stack) {
|
||||
if ((*stype == STACK_INVALID || *stype == STACK_POISON) &&
|
||||
!env->allow_uninit_stack) {
|
||||
verbose(env, "uninit stack in range of var-offset write prohibited for !root; insn %d, off: %d",
|
||||
insn_idx, i);
|
||||
return -EINVAL;
|
||||
|
|
@ -5690,16 +5667,12 @@ static int check_stack_read_fixed_off(struct bpf_verifier_env *env,
|
|||
struct bpf_reg_state *reg;
|
||||
u8 *stype, type;
|
||||
int insn_flags = insn_stack_access_flags(reg_state->frameno, spi);
|
||||
int err;
|
||||
|
||||
stype = reg_state->stack[spi].slot_type;
|
||||
reg = ®_state->stack[spi].spilled_ptr;
|
||||
|
||||
mark_stack_slot_scratched(env, spi);
|
||||
check_fastcall_stack_contract(env, state, env->insn_idx, off);
|
||||
err = bpf_mark_stack_read(env, reg_state->frameno, env->insn_idx, BIT(spi));
|
||||
if (err)
|
||||
return err;
|
||||
|
||||
if (is_spilled_reg(®_state->stack[spi])) {
|
||||
u8 spill_size = 1;
|
||||
|
|
@ -5755,8 +5728,13 @@ static int check_stack_read_fixed_off(struct bpf_verifier_env *env,
|
|||
}
|
||||
if (type == STACK_INVALID && env->allow_uninit_stack)
|
||||
continue;
|
||||
verbose(env, "invalid read from stack off %d+%d size %d\n",
|
||||
off, i, size);
|
||||
if (type == STACK_POISON) {
|
||||
verbose(env, "reading from stack off %d+%d size %d, slot poisoned by dead code elimination\n",
|
||||
off, i, size);
|
||||
} else {
|
||||
verbose(env, "invalid read from stack off %d+%d size %d\n",
|
||||
off, i, size);
|
||||
}
|
||||
return -EACCES;
|
||||
}
|
||||
|
||||
|
|
@ -5805,8 +5783,13 @@ static int check_stack_read_fixed_off(struct bpf_verifier_env *env,
|
|||
continue;
|
||||
if (type == STACK_INVALID && env->allow_uninit_stack)
|
||||
continue;
|
||||
verbose(env, "invalid read from stack off %d+%d size %d\n",
|
||||
off, i, size);
|
||||
if (type == STACK_POISON) {
|
||||
verbose(env, "reading from stack off %d+%d size %d, slot poisoned by dead code elimination\n",
|
||||
off, i, size);
|
||||
} else {
|
||||
verbose(env, "invalid read from stack off %d+%d size %d\n",
|
||||
off, i, size);
|
||||
}
|
||||
return -EACCES;
|
||||
}
|
||||
if (dst_regno >= 0)
|
||||
|
|
@ -7032,7 +7015,7 @@ static int check_max_stack_depth_subprog(struct bpf_verifier_env *env, int idx,
|
|||
if (subprog[idx].has_tail_call)
|
||||
tail_call_reachable = true;
|
||||
|
||||
frame = subprog_is_global(env, idx) ? 0 : frame + 1;
|
||||
frame = bpf_subprog_is_global(env, idx) ? 0 : frame + 1;
|
||||
if (frame >= MAX_CALL_FRAMES) {
|
||||
verbose(env, "the call stack of %d frames is too deep !\n",
|
||||
frame);
|
||||
|
|
@ -8409,16 +8392,22 @@ static int check_stack_range_initialized(
|
|||
/* Some accesses can write anything into the stack, others are
|
||||
* read-only.
|
||||
*/
|
||||
bool clobber = false;
|
||||
bool clobber = type == BPF_WRITE;
|
||||
/*
|
||||
* Negative access_size signals global subprog/kfunc arg check where
|
||||
* STACK_POISON slots are acceptable. static stack liveness
|
||||
* might have determined that subprog doesn't read them,
|
||||
* but BTF based global subprog validation isn't accurate enough.
|
||||
*/
|
||||
bool allow_poison = access_size < 0 || clobber;
|
||||
|
||||
access_size = abs(access_size);
|
||||
|
||||
if (access_size == 0 && !zero_size_allowed) {
|
||||
verbose(env, "invalid zero-sized read\n");
|
||||
return -EACCES;
|
||||
}
|
||||
|
||||
if (type == BPF_WRITE)
|
||||
clobber = true;
|
||||
|
||||
err = check_stack_access_within_bounds(env, regno, off, access_size, type);
|
||||
if (err)
|
||||
return err;
|
||||
|
|
@ -8517,7 +8506,12 @@ static int check_stack_range_initialized(
|
|||
goto mark;
|
||||
}
|
||||
|
||||
if (tnum_is_const(reg->var_off)) {
|
||||
if (*stype == STACK_POISON) {
|
||||
if (allow_poison)
|
||||
goto mark;
|
||||
verbose(env, "reading from stack R%d off %d+%d size %d, slot poisoned by dead code elimination\n",
|
||||
regno, min_off, i - min_off, access_size);
|
||||
} else if (tnum_is_const(reg->var_off)) {
|
||||
verbose(env, "invalid read from stack R%d off %d+%d size %d\n",
|
||||
regno, min_off, i - min_off, access_size);
|
||||
} else {
|
||||
|
|
@ -8529,17 +8523,7 @@ static int check_stack_range_initialized(
|
|||
}
|
||||
return -EACCES;
|
||||
mark:
|
||||
/* reading any byte out of 8-byte 'spill_slot' will cause
|
||||
* the whole slot to be marked as 'read'
|
||||
*/
|
||||
err = bpf_mark_stack_read(env, reg->frameno, env->insn_idx, BIT(spi));
|
||||
if (err)
|
||||
return err;
|
||||
/* We do not call bpf_mark_stack_write(), as we can not
|
||||
* be sure that whether stack slot is written to or not. Hence,
|
||||
* we must still conservatively propagate reads upwards even if
|
||||
* helper may write to the entire memory range.
|
||||
*/
|
||||
;
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
|
|
@ -8704,8 +8688,10 @@ static int check_mem_reg(struct bpf_verifier_env *env, struct bpf_reg_state *reg
|
|||
mark_ptr_not_null_reg(reg);
|
||||
}
|
||||
|
||||
err = check_helper_mem_access(env, regno, mem_size, BPF_READ, true, NULL);
|
||||
err = err ?: check_helper_mem_access(env, regno, mem_size, BPF_WRITE, true, NULL);
|
||||
int size = base_type(reg->type) == PTR_TO_STACK ? -(int)mem_size : mem_size;
|
||||
|
||||
err = check_helper_mem_access(env, regno, size, BPF_READ, true, NULL);
|
||||
err = err ?: check_helper_mem_access(env, regno, size, BPF_WRITE, true, NULL);
|
||||
|
||||
if (may_be_null)
|
||||
*reg = saved_reg;
|
||||
|
|
@ -11107,7 +11093,7 @@ static int check_func_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
|
|||
err = btf_check_subprog_call(env, subprog, caller->regs);
|
||||
if (err == -EFAULT)
|
||||
return err;
|
||||
if (subprog_is_global(env, subprog)) {
|
||||
if (bpf_subprog_is_global(env, subprog)) {
|
||||
const char *sub_name = subprog_name(env, subprog);
|
||||
|
||||
if (env->cur_state->active_locks) {
|
||||
|
|
@ -11159,8 +11145,6 @@ static int check_func_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
|
|||
/* and go analyze first insn of the callee */
|
||||
*insn_idx = env->subprog_info[subprog].start - 1;
|
||||
|
||||
bpf_reset_live_stack_callchain(env);
|
||||
|
||||
if (env->log.level & BPF_LOG_LEVEL) {
|
||||
verbose(env, "caller:\n");
|
||||
print_verifier_state(env, state, caller->frameno, true);
|
||||
|
|
@ -11445,10 +11429,6 @@ static int prepare_func_exit(struct bpf_verifier_env *env, int *insn_idx)
|
|||
bool in_callback_fn;
|
||||
int err;
|
||||
|
||||
err = bpf_update_live_stack(env);
|
||||
if (err)
|
||||
return err;
|
||||
|
||||
callee = state->frame[state->curframe];
|
||||
r0 = &callee->regs[BPF_REG_0];
|
||||
if (r0->type == PTR_TO_STACK) {
|
||||
|
|
@ -19797,6 +19777,7 @@ static int check_btf_func(struct bpf_verifier_env *env,
|
|||
goto err_free;
|
||||
}
|
||||
|
||||
env->subprog_info[i].name = btf_name_by_offset(btf, type->name_off);
|
||||
bpfptr_add(&urecord, urec_size);
|
||||
}
|
||||
|
||||
|
|
@ -20160,11 +20141,10 @@ static bool check_scalar_ids(u32 old_id, u32 cur_id, struct bpf_idmap *idmap)
|
|||
return check_ids(old_id, cur_id, idmap);
|
||||
}
|
||||
|
||||
static void clean_func_state(struct bpf_verifier_env *env,
|
||||
struct bpf_func_state *st,
|
||||
u32 ip)
|
||||
static void __clean_func_state(struct bpf_verifier_env *env,
|
||||
struct bpf_func_state *st,
|
||||
u16 live_regs, int frame)
|
||||
{
|
||||
u16 live_regs = env->insn_aux_data[ip].live_regs_before;
|
||||
int i, j;
|
||||
|
||||
for (i = 0; i < BPF_REG_FP; i++) {
|
||||
|
|
@ -20176,58 +20156,82 @@ static void clean_func_state(struct bpf_verifier_env *env,
|
|||
__mark_reg_not_init(env, &st->regs[i]);
|
||||
}
|
||||
|
||||
/*
|
||||
* Clean dead 4-byte halves within each SPI independently.
|
||||
* half_spi 2*i → lower half: slot_type[0..3] (closer to FP)
|
||||
* half_spi 2*i+1 → upper half: slot_type[4..7] (farther from FP)
|
||||
*/
|
||||
for (i = 0; i < st->allocated_stack / BPF_REG_SIZE; i++) {
|
||||
if (!bpf_stack_slot_alive(env, st->frameno, i)) {
|
||||
__mark_reg_not_init(env, &st->stack[i].spilled_ptr);
|
||||
for (j = 0; j < BPF_REG_SIZE; j++)
|
||||
st->stack[i].slot_type[j] = STACK_INVALID;
|
||||
bool lo_live = bpf_stack_slot_alive(env, frame, i * 2);
|
||||
bool hi_live = bpf_stack_slot_alive(env, frame, i * 2 + 1);
|
||||
|
||||
if (!hi_live || !lo_live) {
|
||||
int start = !lo_live ? 0 : BPF_REG_SIZE / 2;
|
||||
int end = !hi_live ? BPF_REG_SIZE : BPF_REG_SIZE / 2;
|
||||
u8 stype = st->stack[i].slot_type[7];
|
||||
|
||||
/*
|
||||
* Don't clear special slots.
|
||||
* destroy_if_dynptr_stack_slot() needs STACK_DYNPTR to
|
||||
* detect overwrites and invalidate associated data slices.
|
||||
* is_iter_reg_valid_uninit() and is_irq_flag_reg_valid_uninit()
|
||||
* check for their respective slot types to detect double-create.
|
||||
*/
|
||||
if (stype == STACK_DYNPTR || stype == STACK_ITER ||
|
||||
stype == STACK_IRQ_FLAG)
|
||||
continue;
|
||||
|
||||
/*
|
||||
* Only destroy spilled_ptr when hi half is dead.
|
||||
* If hi half is still live with STACK_SPILL, the
|
||||
* spilled_ptr metadata is needed for correct state
|
||||
* comparison in stacksafe().
|
||||
* is_spilled_reg() is using slot_type[7], but
|
||||
* is_spilled_scalar_after() check either slot_type[0] or [4]
|
||||
*/
|
||||
if (!hi_live) {
|
||||
struct bpf_reg_state *spill = &st->stack[i].spilled_ptr;
|
||||
|
||||
if (lo_live && stype == STACK_SPILL) {
|
||||
u8 val = STACK_MISC;
|
||||
|
||||
/*
|
||||
* 8 byte spill of scalar 0 where half slot is dead
|
||||
* should become STACK_ZERO in lo 4 bytes.
|
||||
*/
|
||||
if (register_is_null(spill))
|
||||
val = STACK_ZERO;
|
||||
for (j = 0; j < 4; j++) {
|
||||
u8 *t = &st->stack[i].slot_type[j];
|
||||
|
||||
if (*t == STACK_SPILL)
|
||||
*t = val;
|
||||
}
|
||||
}
|
||||
__mark_reg_not_init(env, spill);
|
||||
}
|
||||
for (j = start; j < end; j++)
|
||||
st->stack[i].slot_type[j] = STACK_POISON;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
static void clean_verifier_state(struct bpf_verifier_env *env,
|
||||
static int clean_verifier_state(struct bpf_verifier_env *env,
|
||||
struct bpf_verifier_state *st)
|
||||
{
|
||||
int i, ip;
|
||||
int i, err;
|
||||
|
||||
bpf_live_stack_query_init(env, st);
|
||||
st->cleaned = true;
|
||||
err = bpf_live_stack_query_init(env, st);
|
||||
if (err)
|
||||
return err;
|
||||
for (i = 0; i <= st->curframe; i++) {
|
||||
ip = frame_insn_idx(st, i);
|
||||
clean_func_state(env, st->frame[i], ip);
|
||||
}
|
||||
}
|
||||
u32 ip = frame_insn_idx(st, i);
|
||||
u16 live_regs = env->insn_aux_data[ip].live_regs_before;
|
||||
|
||||
/* the parentage chains form a tree.
|
||||
* the verifier states are added to state lists at given insn and
|
||||
* pushed into state stack for future exploration.
|
||||
* when the verifier reaches bpf_exit insn some of the verifier states
|
||||
* stored in the state lists have their final liveness state already,
|
||||
* but a lot of states will get revised from liveness point of view when
|
||||
* the verifier explores other branches.
|
||||
* Example:
|
||||
* 1: *(u64)(r10 - 8) = 1
|
||||
* 2: if r1 == 100 goto pc+1
|
||||
* 3: *(u64)(r10 - 8) = 2
|
||||
* 4: r0 = *(u64)(r10 - 8)
|
||||
* 5: exit
|
||||
* when the verifier reaches exit insn the stack slot -8 in the state list of
|
||||
* insn 2 is not yet marked alive. Then the verifier pops the other_branch
|
||||
* of insn 2 and goes exploring further. After the insn 4 read, liveness
|
||||
* analysis would propagate read mark for -8 at insn 2.
|
||||
*
|
||||
* Since the verifier pushes the branch states as it sees them while exploring
|
||||
* the program the condition of walking the branch instruction for the second
|
||||
* time means that all states below this branch were already explored and
|
||||
* their final liveness marks are already propagated.
|
||||
* Hence when the verifier completes the search of state list in is_state_visited()
|
||||
* we can call this clean_live_states() function to clear dead the registers and stack
|
||||
* slots to simplify state merging.
|
||||
*
|
||||
* Important note here that walking the same branch instruction in the callee
|
||||
* doesn't meant that the states are DONE. The verifier has to compare
|
||||
* the callsites
|
||||
*/
|
||||
__clean_func_state(env, st->frame[i], live_regs, i);
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
|
||||
/* Find id in idset and increment its count, or add new entry */
|
||||
static void idset_cnt_inc(struct bpf_idset *idset, u32 id)
|
||||
|
|
@ -20292,29 +20296,6 @@ static void clear_singular_ids(struct bpf_verifier_env *env,
|
|||
}));
|
||||
}
|
||||
|
||||
static void clean_live_states(struct bpf_verifier_env *env, int insn,
|
||||
struct bpf_verifier_state *cur)
|
||||
{
|
||||
struct bpf_verifier_state_list *sl;
|
||||
struct list_head *pos, *head;
|
||||
|
||||
head = explored_state(env, insn);
|
||||
list_for_each(pos, head) {
|
||||
sl = container_of(pos, struct bpf_verifier_state_list, node);
|
||||
if (sl->state.branches)
|
||||
continue;
|
||||
if (sl->state.insn_idx != insn ||
|
||||
!same_callsites(&sl->state, cur))
|
||||
continue;
|
||||
if (sl->state.cleaned)
|
||||
/* all regs in this state in all frames were already marked */
|
||||
continue;
|
||||
if (incomplete_read_marks(env, &sl->state))
|
||||
continue;
|
||||
clean_verifier_state(env, &sl->state);
|
||||
}
|
||||
}
|
||||
|
||||
static bool regs_exact(const struct bpf_reg_state *rold,
|
||||
const struct bpf_reg_state *rcur,
|
||||
struct bpf_idmap *idmap)
|
||||
|
|
@ -20499,7 +20480,8 @@ static bool is_stack_misc_after(struct bpf_verifier_env *env,
|
|||
|
||||
for (i = im; i < ARRAY_SIZE(stack->slot_type); ++i) {
|
||||
if ((stack->slot_type[i] == STACK_MISC) ||
|
||||
(stack->slot_type[i] == STACK_INVALID && env->allow_uninit_stack))
|
||||
((stack->slot_type[i] == STACK_INVALID || stack->slot_type[i] == STACK_POISON) &&
|
||||
env->allow_uninit_stack))
|
||||
continue;
|
||||
return false;
|
||||
}
|
||||
|
|
@ -20535,13 +20517,22 @@ static bool stacksafe(struct bpf_verifier_env *env, struct bpf_func_state *old,
|
|||
|
||||
spi = i / BPF_REG_SIZE;
|
||||
|
||||
if (exact == EXACT &&
|
||||
(i >= cur->allocated_stack ||
|
||||
old->stack[spi].slot_type[i % BPF_REG_SIZE] !=
|
||||
cur->stack[spi].slot_type[i % BPF_REG_SIZE]))
|
||||
return false;
|
||||
if (exact == EXACT) {
|
||||
u8 old_type = old->stack[spi].slot_type[i % BPF_REG_SIZE];
|
||||
u8 cur_type = i < cur->allocated_stack ?
|
||||
cur->stack[spi].slot_type[i % BPF_REG_SIZE] : STACK_INVALID;
|
||||
|
||||
if (old->stack[spi].slot_type[i % BPF_REG_SIZE] == STACK_INVALID)
|
||||
/* STACK_INVALID and STACK_POISON are equivalent for pruning */
|
||||
if (old_type == STACK_POISON)
|
||||
old_type = STACK_INVALID;
|
||||
if (cur_type == STACK_POISON)
|
||||
cur_type = STACK_INVALID;
|
||||
if (i >= cur->allocated_stack || old_type != cur_type)
|
||||
return false;
|
||||
}
|
||||
|
||||
if (old->stack[spi].slot_type[i % BPF_REG_SIZE] == STACK_INVALID ||
|
||||
old->stack[spi].slot_type[i % BPF_REG_SIZE] == STACK_POISON)
|
||||
continue;
|
||||
|
||||
if (env->allow_uninit_stack &&
|
||||
|
|
@ -20639,6 +20630,7 @@ static bool stacksafe(struct bpf_verifier_env *env, struct bpf_func_state *old,
|
|||
case STACK_MISC:
|
||||
case STACK_ZERO:
|
||||
case STACK_INVALID:
|
||||
case STACK_POISON:
|
||||
continue;
|
||||
/* Ensure that new unhandled slot types return false by default */
|
||||
default:
|
||||
|
|
@ -21015,7 +21007,10 @@ static int is_state_visited(struct bpf_verifier_env *env, int insn_idx)
|
|||
env->insn_processed - env->prev_insn_processed >= 8)
|
||||
add_new_state = true;
|
||||
|
||||
clean_live_states(env, insn_idx, cur);
|
||||
/* keep cleaning the current state as registers/stack become dead */
|
||||
err = clean_verifier_state(env, cur);
|
||||
if (err)
|
||||
return err;
|
||||
|
||||
loop = false;
|
||||
head = explored_state(env, insn_idx);
|
||||
|
|
@ -21783,7 +21778,7 @@ static int do_check(struct bpf_verifier_env *env)
|
|||
for (;;) {
|
||||
struct bpf_insn *insn;
|
||||
struct bpf_insn_aux_data *insn_aux;
|
||||
int err, marks_err;
|
||||
int err;
|
||||
|
||||
/* reset current history entry on each new instruction */
|
||||
env->cur_hist_ent = NULL;
|
||||
|
|
@ -21897,15 +21892,7 @@ static int do_check(struct bpf_verifier_env *env)
|
|||
if (state->speculative && insn_aux->nospec)
|
||||
goto process_bpf_exit;
|
||||
|
||||
err = bpf_reset_stack_write_marks(env, env->insn_idx);
|
||||
if (err)
|
||||
return err;
|
||||
err = do_check_insn(env, &do_print_state);
|
||||
if (err >= 0 || error_recoverable_with_nospec(err)) {
|
||||
marks_err = bpf_commit_stack_write_marks(env);
|
||||
if (marks_err)
|
||||
return marks_err;
|
||||
}
|
||||
if (error_recoverable_with_nospec(err) && state->speculative) {
|
||||
/* Prevent this speculative path from ever reaching the
|
||||
* insn that would have been unsafe to execute.
|
||||
|
|
@ -21946,9 +21933,6 @@ static int do_check(struct bpf_verifier_env *env)
|
|||
process_bpf_exit:
|
||||
mark_verifier_state_scratched(env);
|
||||
err = update_branch_counts(env, env->cur_state);
|
||||
if (err)
|
||||
return err;
|
||||
err = bpf_update_live_stack(env);
|
||||
if (err)
|
||||
return err;
|
||||
err = pop_stack(env, &prev_insn_idx, &env->insn_idx,
|
||||
|
|
@ -25299,7 +25283,7 @@ static int do_check_subprogs(struct bpf_verifier_env *env)
|
|||
again:
|
||||
new_cnt = 0;
|
||||
for (i = 1; i < env->subprog_cnt; i++) {
|
||||
if (!subprog_is_global(env, i))
|
||||
if (!bpf_subprog_is_global(env, i))
|
||||
continue;
|
||||
|
||||
sub_aux = subprog_aux(env, i);
|
||||
|
|
@ -26338,6 +26322,11 @@ static int compute_live_registers(struct bpf_verifier_env *env)
|
|||
for (i = 0; i < insn_cnt; ++i)
|
||||
compute_insn_live_regs(env, &insns[i], &state[i]);
|
||||
|
||||
/* Forward pass: resolve stack access through FP-derived pointers */
|
||||
err = bpf_compute_subprog_arg_access(env);
|
||||
if (err)
|
||||
goto out;
|
||||
|
||||
changed = true;
|
||||
while (changed) {
|
||||
changed = false;
|
||||
|
|
|
|||
|
|
@ -54,6 +54,7 @@
|
|||
#include "verifier_leak_ptr.skel.h"
|
||||
#include "verifier_linked_scalars.skel.h"
|
||||
#include "verifier_live_stack.skel.h"
|
||||
#include "verifier_liveness_exp.skel.h"
|
||||
#include "verifier_load_acquire.skel.h"
|
||||
#include "verifier_loops1.skel.h"
|
||||
#include "verifier_lwt.skel.h"
|
||||
|
|
@ -202,6 +203,7 @@ void test_verifier_ldsx(void) { RUN(verifier_ldsx); }
|
|||
void test_verifier_leak_ptr(void) { RUN(verifier_leak_ptr); }
|
||||
void test_verifier_linked_scalars(void) { RUN(verifier_linked_scalars); }
|
||||
void test_verifier_live_stack(void) { RUN(verifier_live_stack); }
|
||||
void test_verifier_liveness_exp(void) { RUN(verifier_liveness_exp); }
|
||||
void test_verifier_loops1(void) { RUN(verifier_loops1); }
|
||||
void test_verifier_lwt(void) { RUN(verifier_lwt); }
|
||||
void test_verifier_map_in_map(void) { RUN(verifier_map_in_map); }
|
||||
|
|
|
|||
|
|
@ -25,10 +25,10 @@ static bool check_prog_load(int prog_fd, bool expect_err, const char *tag)
|
|||
|
||||
static struct {
|
||||
/* strategically placed before others to avoid accidental modification by kernel */
|
||||
char filler[1024];
|
||||
char buf[1024];
|
||||
char filler[16384];
|
||||
char buf[16384];
|
||||
/* strategically placed after buf[] to catch more accidental corruptions */
|
||||
char reference[1024];
|
||||
char reference[16384];
|
||||
} logs;
|
||||
static const struct bpf_insn *insns;
|
||||
static size_t insn_cnt;
|
||||
|
|
|
|||
|
|
@ -59,7 +59,7 @@ check_assert(s64, >=, ge_neg, INT_MIN);
|
|||
|
||||
SEC("?tc")
|
||||
__log_level(2) __failure
|
||||
__msg(": R0=0 R1=ctx() R2=scalar(smin=0xffffffff80000002,smax=smax32=0x7ffffffd,smin32=0x80000002) R10=fp0")
|
||||
__msg(": R1=ctx() R2=scalar(smin=0xffffffff80000002,smax=smax32=0x7ffffffd,smin32=0x80000002) R10=fp0")
|
||||
int check_assert_range_s64(struct __sk_buff *ctx)
|
||||
{
|
||||
struct bpf_sock *sk = ctx->sk;
|
||||
|
|
@ -86,7 +86,7 @@ int check_assert_range_u64(struct __sk_buff *ctx)
|
|||
|
||||
SEC("?tc")
|
||||
__log_level(2) __failure
|
||||
__msg(": R0=0 R1=ctx() R2=4096 R10=fp0")
|
||||
__msg(": R1=ctx() R2=4096 R10=fp0")
|
||||
int check_assert_single_range_s64(struct __sk_buff *ctx)
|
||||
{
|
||||
struct bpf_sock *sk = ctx->sk;
|
||||
|
|
@ -114,7 +114,7 @@ int check_assert_single_range_u64(struct __sk_buff *ctx)
|
|||
|
||||
SEC("?tc")
|
||||
__log_level(2) __failure
|
||||
__msg(": R1=pkt(r=64,imm=64) R2=pkt_end() R6=pkt(r=64) R10=fp0")
|
||||
__msg(": R6=pkt(r=64) R10=fp0")
|
||||
int check_assert_generic(struct __sk_buff *ctx)
|
||||
{
|
||||
u8 *data_end = (void *)(long)ctx->data_end;
|
||||
|
|
|
|||
|
|
@ -76,6 +76,7 @@ __naked int helper_uninit_to_misc(void *ctx)
|
|||
* thus showing the stack state, matched by __msg(). \
|
||||
*/ \
|
||||
call %[dummy]; \
|
||||
r1 = *(u64*)(r10 - 104); \
|
||||
r0 = 0; \
|
||||
exit; \
|
||||
"
|
||||
|
|
|
|||
|
|
@ -131,7 +131,7 @@ LBL ":" \
|
|||
SEC("tc")
|
||||
__success __log_level(2)
|
||||
__flag(BPF_F_ANY_ALIGNMENT)
|
||||
__msg("6: R0=pkt(r=8,imm=8)")
|
||||
__msg("6: {{.*}} R2=pkt(r=8)")
|
||||
__msg("6: {{.*}} R3={{[^)]*}}var_off=(0x0; 0xff)")
|
||||
__msg("7: {{.*}} R3={{[^)]*}}var_off=(0x0; 0x1fe)")
|
||||
__msg("8: {{.*}} R3={{[^)]*}}var_off=(0x0; 0x3fc)")
|
||||
|
|
@ -205,7 +205,7 @@ __success __log_level(2)
|
|||
__msg("2: {{.*}} R5=pkt(r=0)")
|
||||
__msg("4: {{.*}} R5=pkt(r=0,imm=14)")
|
||||
__msg("5: {{.*}} R4=pkt(r=0,imm=14)")
|
||||
__msg("9: {{.*}} R2=pkt(r=18)")
|
||||
__msg("9: {{.*}} R5=pkt(r=18,imm=14)")
|
||||
__msg("10: {{.*}} R4={{[^)]*}}var_off=(0x0; 0xff){{.*}} R5=pkt(r=18,imm=14)")
|
||||
__msg("13: {{.*}} R4={{[^)]*}}var_off=(0x0; 0xffff)")
|
||||
__msg("14: {{.*}} R4={{[^)]*}}var_off=(0x0; 0xffff)")
|
||||
|
|
@ -254,7 +254,7 @@ __msg("11: {{.*}} R5=pkt(id=1,{{[^)]*}},var_off=(0x2; 0x7fc)")
|
|||
* offset is considered using reg->aux_off_align which
|
||||
* is 4 and meets the load's requirements.
|
||||
*/
|
||||
__msg("15: {{.*}} R4={{[^)]*}}var_off=(0x2; 0x7fc){{.*}} R5={{[^)]*}}var_off=(0x2; 0x7fc)")
|
||||
__msg("15: {{.*}} R5={{[^)]*}}var_off=(0x2; 0x7fc)")
|
||||
/* Variable offset is added to R5 packet pointer,
|
||||
* resulting in auxiliary alignment of 4. To avoid BPF
|
||||
* verifier's precision backtracking logging
|
||||
|
|
@ -273,7 +273,7 @@ __msg("19: {{.*}} R5=pkt(id=2,{{[^)]*}}var_off=(0x2; 0x7fc)")
|
|||
* aligned, so the total offset is 4-byte aligned and
|
||||
* meets the load's requirements.
|
||||
*/
|
||||
__msg("24: {{.*}} R4={{[^)]*}}var_off=(0x2; 0x7fc){{.*}} R5={{[^)]*}}var_off=(0x2; 0x7fc)")
|
||||
__msg("24: {{.*}} R5={{[^)]*}}var_off=(0x2; 0x7fc)")
|
||||
/* Constant offset is added to R5 packet pointer,
|
||||
* resulting in reg->off value of 14.
|
||||
*/
|
||||
|
|
@ -296,7 +296,7 @@ __msg("31: {{.*}} R4={{[^)]*}}var_off=(0x2; 0xffc){{.*}} R5={{[^)]*}}var_off=(0x
|
|||
* the total offset is 4-byte aligned and meets the
|
||||
* load's requirements.
|
||||
*/
|
||||
__msg("35: {{.*}} R4={{[^)]*}}var_off=(0x2; 0xffc){{.*}} R5={{[^)]*}}var_off=(0x2; 0xffc)")
|
||||
__msg("35: {{.*}} R5={{[^)]*}}var_off=(0x2; 0xffc)")
|
||||
__naked void packet_variable_offset(void)
|
||||
{
|
||||
asm volatile (" \
|
||||
|
|
|
|||
|
|
@ -36,7 +36,7 @@ l0_%=: r0 = *(u64 *)(r1 + 0); \
|
|||
SEC("socket")
|
||||
__description("UDIV32, zero divisor")
|
||||
__success __retval(0) __log_level(2)
|
||||
__msg("w1 /= w2 {{.*}}; R1=0 R2=0")
|
||||
__msg("w1 /= w2 {{.*}}; R1=0")
|
||||
__naked void udiv32_zero_divisor(void)
|
||||
{
|
||||
asm volatile (" \
|
||||
|
|
@ -81,7 +81,7 @@ l0_%=: r0 = *(u64 *)(r1 + 0); \
|
|||
SEC("socket")
|
||||
__description("UDIV64, zero divisor")
|
||||
__success __retval(0) __log_level(2)
|
||||
__msg("r1 /= r2 {{.*}}; R1=0 R2=0")
|
||||
__msg("r1 /= r2 {{.*}}; R1=0")
|
||||
__naked void udiv64_zero_divisor(void)
|
||||
{
|
||||
asm volatile (" \
|
||||
|
|
@ -242,7 +242,7 @@ l1_%=: r0 = *(u64 *)(r1 + 0); \
|
|||
SEC("socket")
|
||||
__description("SDIV32, zero divisor")
|
||||
__success __retval(0) __log_level(2)
|
||||
__msg("w1 s/= w2 {{.*}}; R1=0 R2=0")
|
||||
__msg("w1 s/= w2 {{.*}}; R1=0")
|
||||
__naked void sdiv32_zero_divisor(void)
|
||||
{
|
||||
asm volatile (" \
|
||||
|
|
@ -275,6 +275,7 @@ __naked void sdiv32_overflow_1(void)
|
|||
w2 += 10; \
|
||||
if w1 s> w2 goto l0_%=; \
|
||||
w1 s/= -1; \
|
||||
r2 = r1; \
|
||||
l0_%=: r0 = 0; \
|
||||
exit; \
|
||||
" :
|
||||
|
|
@ -443,7 +444,7 @@ l1_%=: r0 = *(u64 *)(r1 + 0); \
|
|||
SEC("socket")
|
||||
__description("SDIV64, zero divisor")
|
||||
__success __retval(0) __log_level(2)
|
||||
__msg("r1 s/= r2 {{.*}}; R1=0 R2=0")
|
||||
__msg("r1 s/= r2 {{.*}}; R1=0")
|
||||
__naked void sdiv64_zero_divisor(void)
|
||||
{
|
||||
asm volatile (" \
|
||||
|
|
@ -476,6 +477,7 @@ __naked void sdiv64_overflow_1(void)
|
|||
r2 += 10; \
|
||||
if r1 s> r2 goto l0_%=; \
|
||||
r1 s/= -1; \
|
||||
r2 = r1; \
|
||||
l0_%=: r0 = 0; \
|
||||
exit; \
|
||||
" :
|
||||
|
|
@ -553,7 +555,7 @@ l0_%=: r0 = *(u64 *)(r1 + 0); \
|
|||
SEC("socket")
|
||||
__description("UMOD32, zero divisor")
|
||||
__success __retval(0) __log_level(2)
|
||||
__msg("w1 %= w2 {{.*}}; R1=scalar(smin=umin=smin32=umin32=1,smax=umax=smax32=umax32=9,var_off=(0x1; 0x8)) R2=0")
|
||||
__msg("w1 %= w2 {{.*}}; R1=scalar(smin=umin=smin32=umin32=1,smax=umax=smax32=umax32=9,var_off=(0x1; 0x8))")
|
||||
__naked void umod32_zero_divisor(void)
|
||||
{
|
||||
asm volatile (" \
|
||||
|
|
@ -624,7 +626,7 @@ l0_%=: r0 = *(u64 *)(r1 + 0); \
|
|||
SEC("socket")
|
||||
__description("UMOD64, zero divisor")
|
||||
__success __retval(0) __log_level(2)
|
||||
__msg("r1 %= r2 {{.*}}; R1=scalar(smin=umin=smin32=umin32=1,smax=umax=smax32=umax32=9,var_off=(0x1; 0x8)) R2=0")
|
||||
__msg("r1 %= r2 {{.*}}; R1=scalar(smin=umin=smin32=umin32=1,smax=umax=smax32=umax32=9,var_off=(0x1; 0x8))")
|
||||
__naked void umod64_zero_divisor(void)
|
||||
{
|
||||
asm volatile (" \
|
||||
|
|
@ -833,7 +835,7 @@ l1_%=: r0 = *(u64 *)(r1 + 0); \
|
|||
SEC("socket")
|
||||
__description("SMOD32, zero divisor")
|
||||
__success __retval(0) __log_level(2)
|
||||
__msg("w1 s%= w2 {{.*}}; R1=scalar(smin=0,smax=umax=0xffffffff,smin32=-8,smax32=10,var_off=(0x0; 0xffffffff)) R2=0")
|
||||
__msg("w1 s%= w2 {{.*}}; R1=scalar(smin=0,smax=umax=0xffffffff,smin32=-8,smax32=10,var_off=(0x0; 0xffffffff))")
|
||||
__naked void smod32_zero_divisor(void)
|
||||
{
|
||||
asm volatile (" \
|
||||
|
|
@ -1084,7 +1086,7 @@ l1_%=: r0 = *(u64 *)(r1 + 0); \
|
|||
SEC("socket")
|
||||
__description("SMOD64, zero divisor")
|
||||
__success __retval(0) __log_level(2)
|
||||
__msg("r1 s%= r2 {{.*}}; R1=scalar(smin=smin32=-8,smax=smax32=10) R2=0")
|
||||
__msg("r1 s%= r2 {{.*}}; R1=scalar(smin=smin32=-8,smax=smax32=10)")
|
||||
__naked void smod64_zero_divisor(void)
|
||||
{
|
||||
asm volatile (" \
|
||||
|
|
|
|||
File diff suppressed because it is too large
Load Diff
139
tools/testing/selftests/bpf/progs/verifier_liveness_exp.c
Normal file
139
tools/testing/selftests/bpf/progs/verifier_liveness_exp.c
Normal file
|
|
@ -0,0 +1,139 @@
|
|||
// SPDX-License-Identifier: GPL-2.0
|
||||
/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
|
||||
|
||||
#include <linux/bpf.h>
|
||||
#include <bpf/bpf_helpers.h>
|
||||
#include "bpf_misc.h"
|
||||
|
||||
/*
|
||||
* Exponential complexity in analyze_subprog() liveness analysis.
|
||||
*
|
||||
* analyze_subprog() recurses into each call site that passes FP-derived
|
||||
* arguments, creating a unique func_instance per (callsite, depth).
|
||||
* There is no memoization for callees reached with equivalent entry args.
|
||||
* Even if memoization were added, it can be defeated by passing a distinct
|
||||
* FP offset at each call site. arg_track keys on (frame, off[]), so
|
||||
* r1=fp-8, r1=fp-16, ... r1=fp-400 produce 50 unique cache keys per level.
|
||||
*
|
||||
* This test chains 8 subprograms (the MAX_CALL_FRAMES limit). Each
|
||||
* intermediate function calls the next one 50 times, each time with a
|
||||
* different FP-relative offset in r1.
|
||||
*
|
||||
* Without complexity limits in analyze_subprog() the resulting 50^7 ~ 7.8 * 10^11
|
||||
* recursive analyze_subprog() calls will cause a CPU soft lockup or OOM.
|
||||
*
|
||||
* The BPF program itself is ~1200 instructions and perfectly valid.
|
||||
*/
|
||||
|
||||
char _license[] SEC("license") = "GPL";
|
||||
|
||||
/* Call fn with r1 = r10 + off (a unique FP-derived arg per call site) */
|
||||
#define C(fn, off) "r1 = r10;" \
|
||||
"r1 += -" #off ";" \
|
||||
"call " #fn ";"
|
||||
|
||||
/* 50 calls, each with a distinct FP offset: -8, -16, ... -400 */
|
||||
#define CALLS_50(fn) \
|
||||
C(fn, 8) C(fn, 16) C(fn, 24) C(fn, 32) C(fn, 40) \
|
||||
C(fn, 48) C(fn, 56) C(fn, 64) C(fn, 72) C(fn, 80) \
|
||||
C(fn, 88) C(fn, 96) C(fn, 104) C(fn, 112) C(fn, 120) \
|
||||
C(fn, 128) C(fn, 136) C(fn, 144) C(fn, 152) C(fn, 160) \
|
||||
C(fn, 168) C(fn, 176) C(fn, 184) C(fn, 192) C(fn, 200) \
|
||||
C(fn, 208) C(fn, 216) C(fn, 224) C(fn, 232) C(fn, 240) \
|
||||
C(fn, 248) C(fn, 256) C(fn, 264) C(fn, 272) C(fn, 280) \
|
||||
C(fn, 288) C(fn, 296) C(fn, 304) C(fn, 312) C(fn, 320) \
|
||||
C(fn, 328) C(fn, 336) C(fn, 344) C(fn, 352) C(fn, 360) \
|
||||
C(fn, 368) C(fn, 376) C(fn, 384) C(fn, 392) C(fn, 400)
|
||||
|
||||
/* Leaf: depth 7, no further calls */
|
||||
__naked __noinline __used
|
||||
static unsigned long exp_sub7(void)
|
||||
{
|
||||
asm volatile (
|
||||
"r0 = 0;"
|
||||
"exit;"
|
||||
::: __clobber_all);
|
||||
}
|
||||
|
||||
/* depth 6 -> calls exp_sub7 x50 with distinct offsets */
|
||||
__naked __noinline __used
|
||||
static unsigned long exp_sub6(void)
|
||||
{
|
||||
asm volatile (
|
||||
CALLS_50(exp_sub7)
|
||||
"r0 = 0;"
|
||||
"exit;"
|
||||
::: __clobber_all);
|
||||
}
|
||||
|
||||
/* depth 5 -> calls exp_sub6 x50 */
|
||||
__naked __noinline __used
|
||||
static unsigned long exp_sub5(void)
|
||||
{
|
||||
asm volatile (
|
||||
CALLS_50(exp_sub6)
|
||||
"r0 = 0;"
|
||||
"exit;"
|
||||
::: __clobber_all);
|
||||
}
|
||||
|
||||
/* depth 4 -> calls exp_sub5 x50 */
|
||||
__naked __noinline __used
|
||||
static unsigned long exp_sub4(void)
|
||||
{
|
||||
asm volatile (
|
||||
CALLS_50(exp_sub5)
|
||||
"r0 = 0;"
|
||||
"exit;"
|
||||
::: __clobber_all);
|
||||
}
|
||||
|
||||
/* depth 3 -> calls exp_sub4 x50 */
|
||||
__naked __noinline __used
|
||||
static unsigned long exp_sub3(void)
|
||||
{
|
||||
asm volatile (
|
||||
CALLS_50(exp_sub4)
|
||||
"r0 = 0;"
|
||||
"exit;"
|
||||
::: __clobber_all);
|
||||
}
|
||||
|
||||
/* depth 2 -> calls exp_sub3 x50 */
|
||||
__naked __noinline __used
|
||||
static unsigned long exp_sub2(void)
|
||||
{
|
||||
asm volatile (
|
||||
CALLS_50(exp_sub3)
|
||||
"r0 = 0;"
|
||||
"exit;"
|
||||
::: __clobber_all);
|
||||
}
|
||||
|
||||
/* depth 1 -> calls exp_sub2 x50 */
|
||||
__naked __noinline __used
|
||||
static unsigned long exp_sub1(void)
|
||||
{
|
||||
asm volatile (
|
||||
CALLS_50(exp_sub2)
|
||||
"r0 = 0;"
|
||||
"exit;"
|
||||
::: __clobber_all);
|
||||
}
|
||||
|
||||
/*
|
||||
* Entry: depth 0. Calls exp_sub1 50 times, each with a distinct
|
||||
* FP offset in r1. Every call site produces a unique arg_track,
|
||||
* defeating any memoization keyed on entry args.
|
||||
*/
|
||||
SEC("?raw_tp")
|
||||
__failure __log_level(2)
|
||||
__msg("liveness analysis exceeded complexity limit")
|
||||
__naked int liveness_exponential_complexity(void)
|
||||
{
|
||||
asm volatile (
|
||||
CALLS_50(exp_sub1)
|
||||
"r0 = 0;"
|
||||
"exit;"
|
||||
::: __clobber_all);
|
||||
}
|
||||
|
|
@ -264,13 +264,13 @@ void precision_many_frames__bar(void)
|
|||
*/
|
||||
SEC("socket")
|
||||
__success __log_level(2)
|
||||
__msg("11: (0f) r2 += r1")
|
||||
__msg("12: (0f) r2 += r1")
|
||||
/* foo frame */
|
||||
__msg("frame1: regs=r1 stack= before 10: (bf) r2 = r10")
|
||||
__msg("frame1: regs=r1 stack= before 9: (25) if r1 > 0x7 goto pc+0")
|
||||
__msg("frame1: regs=r1 stack=-8,-16 before 8: (7b) *(u64 *)(r10 -16) = r1")
|
||||
__msg("frame1: regs=r1 stack=-8 before 7: (7b) *(u64 *)(r10 -8) = r1")
|
||||
__msg("frame1: regs=r1 stack= before 4: (85) call pc+2")
|
||||
__msg("frame1: regs=r1 stack= before 11: (bf) r2 = r10")
|
||||
__msg("frame1: regs=r1 stack= before 10: (25) if r1 > 0x7 goto pc+0")
|
||||
__msg("frame1: regs=r1 stack=-8,-16 before 9: (7b) *(u64 *)(r10 -16) = r1")
|
||||
__msg("frame1: regs=r1 stack=-8 before 8: (7b) *(u64 *)(r10 -8) = r1")
|
||||
__msg("frame1: regs=r1 stack= before 4: (85) call pc+3")
|
||||
/* main frame */
|
||||
__msg("frame0: regs=r1 stack=-8 before 3: (7b) *(u64 *)(r10 -8) = r1")
|
||||
__msg("frame0: regs=r1 stack= before 2: (bf) r1 = r0")
|
||||
|
|
@ -286,6 +286,7 @@ __naked void precision_stack(void)
|
|||
"r1 = r0;"
|
||||
"*(u64*)(r10 - 8) = r1;"
|
||||
"call precision_stack__foo;"
|
||||
"r0 = *(u64*)(r10 - 8);"
|
||||
"r0 = 0;"
|
||||
"exit;"
|
||||
:
|
||||
|
|
@ -309,6 +310,8 @@ void precision_stack__foo(void)
|
|||
*/
|
||||
"r2 = r10;"
|
||||
"r2 += r1;"
|
||||
"r0 = *(u64*)(r10 - 8);"
|
||||
"r0 = *(u64*)(r10 - 16);"
|
||||
"exit"
|
||||
::: __clobber_all);
|
||||
}
|
||||
|
|
@ -802,9 +805,9 @@ __success __log_level(2)
|
|||
/* The exit instruction should be reachable from two states,
|
||||
* use two matches and "processed .. insns" to ensure this.
|
||||
*/
|
||||
__msg("15: (95) exit")
|
||||
__msg("15: (95) exit")
|
||||
__msg("processed 20 insns")
|
||||
__msg("16: (95) exit")
|
||||
__msg("16: (95) exit")
|
||||
__msg("processed 22 insns")
|
||||
__flag(BPF_F_TEST_STATE_FREQ)
|
||||
__naked void two_old_ids_one_cur_id(void)
|
||||
{
|
||||
|
|
@ -835,6 +838,11 @@ __naked void two_old_ids_one_cur_id(void)
|
|||
"r2 = r10;"
|
||||
"r2 += r6;"
|
||||
"r2 += r7;"
|
||||
/*
|
||||
* keep r8 and r9 live, otherwise r6->id and r7->id
|
||||
* will become singular and reset to zero before if r6 > r7
|
||||
*/
|
||||
"r9 += r8;"
|
||||
"exit;"
|
||||
:
|
||||
: __imm(bpf_ktime_get_ns)
|
||||
|
|
|
|||
|
|
@ -650,7 +650,7 @@ __msg("mark_precise: frame0: last_idx 9 first_idx 7 subseq_idx -1")
|
|||
__msg("mark_precise: frame0: regs=r2 stack= before 8: (79) r2 = *(u64 *)(r10 -8)")
|
||||
__msg("mark_precise: frame0: regs= stack=-8 before 7: (bf) r1 = r6")
|
||||
/* note, fp-8 is precise, fp-16 is not yet precise, we'll get there */
|
||||
__msg("mark_precise: frame0: parent state regs= stack=-8: R0=1 R1=ctx() R6=map_value(map=.data.two_byte_,ks=4,vs=2) R10=fp0 fp-8=P1 fp-16=1")
|
||||
__msg("mark_precise: frame0: parent state regs= stack=-8: R6=map_value(map=.data.two_byte_,ks=4,vs=2) R10=fp0 fp-8=P1 fp-16=1")
|
||||
__msg("mark_precise: frame0: last_idx 6 first_idx 3 subseq_idx 7")
|
||||
__msg("mark_precise: frame0: regs= stack=-8 before 6: (05) goto pc+0")
|
||||
__msg("mark_precise: frame0: regs= stack=-8 before 5: (7b) *(u64 *)(r10 -16) = r0")
|
||||
|
|
@ -668,7 +668,7 @@ __msg("mark_precise: frame0: regs= stack=-16 before 9: (0f) r1 += r2")
|
|||
__msg("mark_precise: frame0: regs= stack=-16 before 8: (79) r2 = *(u64 *)(r10 -8)")
|
||||
__msg("mark_precise: frame0: regs= stack=-16 before 7: (bf) r1 = r6")
|
||||
/* now both fp-8 and fp-16 are precise, very good */
|
||||
__msg("mark_precise: frame0: parent state regs= stack=-16: R0=1 R1=ctx() R6=map_value(map=.data.two_byte_,ks=4,vs=2) R10=fp0 fp-8=P1 fp-16=P1")
|
||||
__msg("mark_precise: frame0: parent state regs= stack=-16: R6=map_value(map=.data.two_byte_,ks=4,vs=2) R10=fp0 fp-8=P1 fp-16=P1")
|
||||
__msg("mark_precise: frame0: last_idx 6 first_idx 3 subseq_idx 7")
|
||||
__msg("mark_precise: frame0: regs= stack=-16 before 6: (05) goto pc+0")
|
||||
__msg("mark_precise: frame0: regs= stack=-16 before 5: (7b) *(u64 *)(r10 -16) = r0")
|
||||
|
|
@ -726,7 +726,7 @@ __msg("9: (0f) r1 += r2")
|
|||
__msg("mark_precise: frame0: last_idx 9 first_idx 7 subseq_idx -1")
|
||||
__msg("mark_precise: frame0: regs=r2 stack= before 8: (61) r2 = *(u32 *)(r10 -8)")
|
||||
__msg("mark_precise: frame0: regs= stack=-8 before 7: (bf) r1 = r6")
|
||||
__msg("mark_precise: frame0: parent state regs= stack=-8: R0=1 R1=ctx() R6=map_value(map=.data.two_byte_,ks=4,vs=2) R10=fp0 fp-8=????P1 fp-16=????1")
|
||||
__msg("mark_precise: frame0: parent state regs= stack=-8: R6=map_value(map=.data.two_byte_,ks=4,vs=2) R10=fp0 fp-8=????P1 fp-16=????1")
|
||||
__msg("mark_precise: frame0: last_idx 6 first_idx 3 subseq_idx 7")
|
||||
__msg("mark_precise: frame0: regs= stack=-8 before 6: (05) goto pc+0")
|
||||
__msg("mark_precise: frame0: regs= stack=-8 before 5: (63) *(u32 *)(r10 -16) = r0")
|
||||
|
|
@ -743,7 +743,7 @@ __msg("mark_precise: frame0: regs= stack=-16 before 10: (73) *(u8 *)(r1 +0) = r2
|
|||
__msg("mark_precise: frame0: regs= stack=-16 before 9: (0f) r1 += r2")
|
||||
__msg("mark_precise: frame0: regs= stack=-16 before 8: (61) r2 = *(u32 *)(r10 -8)")
|
||||
__msg("mark_precise: frame0: regs= stack=-16 before 7: (bf) r1 = r6")
|
||||
__msg("mark_precise: frame0: parent state regs= stack=-16: R0=1 R1=ctx() R6=map_value(map=.data.two_byte_,ks=4,vs=2) R10=fp0 fp-8=????P1 fp-16=????P1")
|
||||
__msg("mark_precise: frame0: parent state regs= stack=-16: R6=map_value(map=.data.two_byte_,ks=4,vs=2) R10=fp0 fp-8=????P1 fp-16=????P1")
|
||||
__msg("mark_precise: frame0: last_idx 6 first_idx 3 subseq_idx 7")
|
||||
__msg("mark_precise: frame0: regs= stack=-16 before 6: (05) goto pc+0")
|
||||
__msg("mark_precise: frame0: regs= stack=-16 before 5: (63) *(u32 *)(r10 -16) = r0")
|
||||
|
|
@ -780,6 +780,8 @@ __naked void stack_load_preserves_const_precision_subreg(void)
|
|||
"r1 += r2;"
|
||||
"*(u8 *)(r1 + 0) = r2;" /* this should be fine */
|
||||
|
||||
"r2 = *(u64 *)(r10 -8);" /* keep slots alive */
|
||||
"r2 = *(u64 *)(r10 -16);"
|
||||
"r0 = 0;"
|
||||
"exit;"
|
||||
:
|
||||
|
|
|
|||
|
|
@ -282,7 +282,7 @@ __msg("mark_precise: frame0: regs=r0,r6 stack= before 10: (bf) r6 = r0")
|
|||
__msg("mark_precise: frame0: regs=r0 stack= before 9: (85) call bpf_loop")
|
||||
/* State entering callback body popped from states stack */
|
||||
__msg("from 9 to 17: frame1:")
|
||||
__msg("17: frame1: R1=scalar() R2=0 R10=fp0 cb")
|
||||
__msg("17: frame1: R10=fp0 cb")
|
||||
__msg("17: (b7) r0 = 0")
|
||||
__msg("18: (95) exit")
|
||||
__msg("returning from callee:")
|
||||
|
|
@ -411,7 +411,7 @@ __msg("mark_precise: frame0: regs=r6 stack= before 5: (b7) r1 = 1")
|
|||
__msg("mark_precise: frame0: regs=r6 stack= before 4: (b7) r6 = 3")
|
||||
/* State entering callback body popped from states stack */
|
||||
__msg("from 9 to 15: frame1:")
|
||||
__msg("15: frame1: R1=scalar() R2=0 R10=fp0 cb")
|
||||
__msg("15: frame1: R10=fp0 cb")
|
||||
__msg("15: (b7) r0 = 0")
|
||||
__msg("16: (95) exit")
|
||||
__msg("returning from callee:")
|
||||
|
|
@ -567,7 +567,7 @@ __msg("mark_precise: frame0: regs= stack=-8 before 5: (7b) *(u64 *)(r10 -8) = r6
|
|||
__msg("mark_precise: frame0: regs=r6 stack= before 4: (b7) r6 = 3")
|
||||
/* State entering callback body popped from states stack */
|
||||
__msg("from 10 to 17: frame1:")
|
||||
__msg("17: frame1: R1=scalar() R2=0 R10=fp0 cb")
|
||||
__msg("17: frame1: R10=fp0 cb")
|
||||
__msg("17: (b7) r0 = 0")
|
||||
__msg("18: (95) exit")
|
||||
__msg("returning from callee:")
|
||||
|
|
@ -681,7 +681,7 @@ __msg("mark_precise: frame0: last_idx 10 first_idx 7 subseq_idx -1")
|
|||
__msg("mark_precise: frame0: regs=r7 stack= before 9: (bf) r1 = r8")
|
||||
__msg("mark_precise: frame0: regs=r7 stack= before 8: (27) r7 *= 4")
|
||||
__msg("mark_precise: frame0: regs=r7 stack= before 7: (79) r7 = *(u64 *)(r10 -8)")
|
||||
__msg("mark_precise: frame0: parent state regs= stack=-8: R0=2 R6=1 R8=map_value(map=.data.vals,ks=4,vs=16) R10=fp0 fp-8=P1")
|
||||
__msg("mark_precise: frame0: parent state regs= stack=-8: R8=map_value(map=.data.vals,ks=4,vs=16) R10=fp0 fp-8=P1")
|
||||
__msg("mark_precise: frame0: last_idx 18 first_idx 0 subseq_idx 7")
|
||||
__msg("mark_precise: frame0: regs= stack=-8 before 18: (95) exit")
|
||||
__msg("mark_precise: frame1: regs= stack= before 17: (0f) r0 += r2")
|
||||
|
|
|
|||
Loading…
Reference in New Issue
Block a user