github-mirror/linux
1
0
Fork 0
mirror of https://github.com/torvalds/linux.git synced 2026-07-27 09:36:22 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity Graph

Merge branch 'bpf-tracing_multi-link'

Browse Source 
Jiri Olsa says:

====================
bpf: tracing_multi link

Add tracing_multi link support that allows fast attachment
of tracing program to many functions.

RFC: https://lore.kernel.org/bpf/20260203093819.2105105-1-jolsa@kernel.org/
v1: https://lore.kernel.org/bpf/20260220100649.628307-1-jolsa@kernel.org/
v2: https://lore.kernel.org/bpf/20260304222141.497203-1-jolsa@kernel.org/
v3: https://lore.kernel.org/bpf/20260316075138.465430-1-jolsa@kernel.org/
v4: https://lore.kernel.org/bpf/20260324081846.2334094-1-jolsa@kernel.org/
v5: https://lore.kernel.org/bpf/20260417192502.194548-1-jolsa@kernel.org/
v6: https://lore.kernel.org/bpf/20260527113951.46265-1-jolsa@kernel.org/
v7: https://lore.kernel.org/bpf/20260603110554.29590-1-jolsa@kernel.org/

v8 changes:
- add back the btf_is_union check to btf_get_type_size [sashiko]

v7 changes:
- added ftrace_hash_count stub for !CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS cade [sashiko]
- selftests fixes [sashiko]
- use hash_ptr in select_trampoline_lock [sashiko]
- changed the check duplicate logic in check_dup_ids [sashiko]
- use sort_r_nonatomic in check_dup_ids [sashiko]
- added BPF_TRACE_FSESSION_MULTI to can_be_sleepable,
  plus added testcase for sleepable fsession
- make bpf_tracing_multi_opts pointer fields as const
- add ___migrate_enable to trace_blacklist

v6 changes:
- move ftrace_hash_count declaration under CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS [sashiko]
- fix ftrace_hash_remove check/deref [sashiko]
- disable context access for multi programs by using stub function with no arguments
  for verification [sashiko]
- add __used for bpf_multi_func, and removed arguments, we do not allow direct access [sashiko]
- rebased on latest loongarch changes, fix ppc build
- guard update_ftrace_direct_del with ftrace_hash_count on rollback [sashiko]
- fix noreturn attachment condition in bpf_check_attach_btf_id_multi [sashiko]
- fail early on multiple same IDs provided by user [sashiko]
- fix selftests error paths [sashiko]
- add MAX_RESOLVE_DEPTH check to btf_get_type_size [sashiko]
- use btf__pointer_size [sashiko]
- fixed compilation on powerpc [sashiko]
- added verifier fails selftest
- after discussing with Song, it was determined that cleaning up FTRACE_OPS_CMD_DISABLE_SHARE_IPMODIFY_PEER
  is not strictly necessary — keeping the trampoline in the ipmodify_enabled state is acceptable.
  The race condition this introduces remains unlikely, so the concern raised in [1] will not be
  addressed at this time.
  [1] https://lore.kernel.org/bpf/aec7bAbGlnEo3R1g@krava/

v5 changes:
- add dedicated hashes used for detach, so there's no need to allocate
  them on detach [sashiko]
- safely release old trampoline images [sashiko]
- add cond_resched() to couple of loops [sashiko]
- validate attr->link_create.target_fd [sashiko]
- allow only bpf_get_func_ret() for return value retrieval [sashiko]
- do not allow attachment of fexit/fsession_multi for noreturn functions [sashiko]
- fixed double free/close in libbpf btf cleanup, in separate patch [sashiko]
- make btf_type_is_traceable_func closer to btf_distill_func_proto [sashiko]
- add prog->attach_btf_obj_fd check to collect_func_ids_by_glob,
  to check we don't load module programs for kernel [sashiko]
- make sure program is loaded in bpf_program__attach_tracing_multi [sashiko]
- several selftests fixes [sashiko]
- add attach_type to fdinfo output [Leon Hwang]
- selftests cleanup fixes [Leon Hwang]

v4 changes:
- unlink rollback fix (added ftrace_hash_count) [bot]
- use const for some bpf_link_create_opts tracing_multi members [bot]
- adding missing comment for lockdep keys [bot]
- selftest error path fixes (leaks) and other assorted test fixes [Leon Hwang]
- several compile fixes wrt CONFIG_BPF_SYSCALL and CONFIG_BPF_JIT [kernel test robot]
- make ftrace_hash_clear global, because it's needed in rollback

v3 changes:
- fix module parsing [Leon Hwang]
- use function traceable check from libbpf [Leon Hwang]
- use ptr_to_u64 and fix/updated few comments [ci]
- display cookies as decimal numbers [ci]
- added link_create.flags check [ci]
- fix error path in bpf_trampoline_multi_detach [ci]
- make fentry/fexit.multi not extendable [ci]
- add missing OPTS_VALID to bpf_program__attach_tracing_multi [ci]

v2 changes:
- allocate data.unreg in bpf_trampoline_multi_attach for rollback path [ci]
  and fixed link count setup in rollback path [ci]
- several small assorted fixes [ci]
- added loongarch and powerpc changes for struct bpf_tramp_node change
- added support to attach functions from modules
- added tests for sleepable programs
- added rollback tests

v1 changes:
- added ftrace_hash_count as wrapper for hash_count [Steven]
- added trampoline mutex pool [Andrii]
- reworked 'struct bpf_tramp_node' separatoin [Andrii]
  - the 'struct bpf_tramp_node' now holds pointer to bpf_link,
    which is similar to what we do for uprobe_multi;
    I understand it's not a fundamental change compared to previous
    version which used bpf_prog pointer instead, but I don't see better
    way of doing this.. I'm happy to discuss this further if there's
    better idea
- reworked 'struct bpf_fsession_link' based on bpf_tramp_node
- made btf__find_by_glob_kind function internal helper [Andrii]
- many small assorted fixes [Andrii,CI]
- added session support [Leon Hwang]
- added cookies support
- added more tests

Note I plan to send linkinfo support separately, the patchset is big enough.

thanks,
jirka
====================

Link: https://patch.msgid.link/20260606123955.345967-1-jolsa@kernel.org
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
This commit is contained in:
Alexei Starovoitov 2026-06-07 10:03:03 -07:00
commit c49f336dbc
42 changed files with 3135 additions and 430 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

58
arch/arm64/net/bpf_jit_comp.c
View File

@ -2335,24 +2335,24 @@ bool bpf_jit_supports_subprog_tailcalls(void)
return true;
}
static void invoke_bpf_prog(struct jit_ctx *ctx, struct bpf_tramp_link *l,
static void invoke_bpf_prog(struct jit_ctx *ctx, struct bpf_tramp_node *node,
int bargs_off, int retval_off, int run_ctx_off,
bool save_ret)
{
__le32 *branch;
u64 enter_prog;
u64 exit_prog;
struct bpf_prog *p = l->link.prog;
struct bpf_prog *p = node->link->prog;
int cookie_off = offsetof(struct bpf_tramp_run_ctx, bpf_cookie);
enter_prog = (u64)bpf_trampoline_enter(p);
exit_prog = (u64)bpf_trampoline_exit(p);
if (l->cookie == 0) {
if (node->cookie == 0) {
/* if cookie is zero, one instruction is enough to store it */
emit(A64_STR64I(A64_ZR, A64_SP, run_ctx_off + cookie_off), ctx);
} else {
emit_a64_mov_i64(A64_R(10), l->cookie, ctx);
emit_a64_mov_i64(A64_R(10), node->cookie, ctx);
emit(A64_STR64I(A64_R(10), A64_SP, run_ctx_off + cookie_off),
ctx);
}
@ -2402,7 +2402,7 @@ static void invoke_bpf_prog(struct jit_ctx *ctx, struct bpf_tramp_link *l,
emit_call(exit_prog, ctx);
}
static void invoke_bpf_mod_ret(struct jit_ctx *ctx, struct bpf_tramp_links *tl,
static void invoke_bpf_mod_ret(struct jit_ctx *ctx, struct bpf_tramp_nodes *tn,
int bargs_off, int retval_off, int run_ctx_off,
__le32 **branches)
{
@ -2412,8 +2412,8 @@ static void invoke_bpf_mod_ret(struct jit_ctx *ctx, struct bpf_tramp_links *tl,
* Set this to 0 to avoid confusing the program.
*/
emit(A64_STR64I(A64_ZR, A64_SP, retval_off), ctx);
for (i = 0; i < tl->nr_links; i++) {
invoke_bpf_prog(ctx, tl->links[i], bargs_off, retval_off,
for (i = 0; i < tn->nr_nodes; i++) {
invoke_bpf_prog(ctx, tn->nodes[i], bargs_off, retval_off,
run_ctx_off, true);
/* if (*(u64 *)(sp + retval_off) != 0)
* goto do_fexit;
@ -2544,10 +2544,10 @@ static void restore_args(struct jit_ctx *ctx, int bargs_off, int nregs)
}
}
static bool is_struct_ops_tramp(const struct bpf_tramp_links *fentry_links)
static bool is_struct_ops_tramp(const struct bpf_tramp_nodes *fentry_nodes)
{
return fentry_links->nr_links == 1 &&
fentry_links->links[0]->link.type == BPF_LINK_TYPE_STRUCT_OPS;
return fentry_nodes->nr_nodes == 1 &&
fentry_nodes->nodes[0]->link->type == BPF_LINK_TYPE_STRUCT_OPS;
}
static void store_func_meta(struct jit_ctx *ctx, u64 func_meta, int func_meta_off)
@ -2568,7 +2568,7 @@ static void store_func_meta(struct jit_ctx *ctx, u64 func_meta, int func_meta_of
*
*/
static int prepare_trampoline(struct jit_ctx *ctx, struct bpf_tramp_image *im,
struct bpf_tramp_links *tlinks, void *func_addr,
struct bpf_tramp_nodes *tnodes, void *func_addr,
const struct btf_func_model *m,
const struct arg_aux *a,
u32 flags)
@ -2584,14 +2584,14 @@ static int prepare_trampoline(struct jit_ctx *ctx, struct bpf_tramp_image *im,
int run_ctx_off;
int oargs_off;
int nfuncargs;
struct bpf_tramp_links *fentry = &tlinks[BPF_TRAMP_FENTRY];
struct bpf_tramp_links *fexit = &tlinks[BPF_TRAMP_FEXIT];
struct bpf_tramp_links *fmod_ret = &tlinks[BPF_TRAMP_MODIFY_RETURN];
struct bpf_tramp_nodes *fentry = &tnodes[BPF_TRAMP_FENTRY];
struct bpf_tramp_nodes *fexit = &tnodes[BPF_TRAMP_FEXIT];
struct bpf_tramp_nodes *fmod_ret = &tnodes[BPF_TRAMP_MODIFY_RETURN];
bool save_ret;
__le32 **branches = NULL;
bool is_struct_ops = is_struct_ops_tramp(fentry);
int cookie_off, cookie_cnt, cookie_bargs_off;
int fsession_cnt = bpf_fsession_cnt(tlinks);
int fsession_cnt = bpf_fsession_cnt(tnodes);
u64 func_meta;
/* trampoline stack layout:
@ -2637,7 +2637,7 @@ static int prepare_trampoline(struct jit_ctx *ctx, struct bpf_tramp_image *im,
cookie_off = stack_size;
/* room for session cookies */
cookie_cnt = bpf_fsession_cookie_cnt(tlinks);
cookie_cnt = bpf_fsession_cookie_cnt(tnodes);
stack_size += cookie_cnt * 8;
ip_off = stack_size;
@ -2734,20 +2734,20 @@ static int prepare_trampoline(struct jit_ctx *ctx, struct bpf_tramp_image *im,
}
cookie_bargs_off = (bargs_off - cookie_off) / 8;
for (i = 0; i < fentry->nr_links; i++) {
if (bpf_prog_calls_session_cookie(fentry->links[i])) {
for (i = 0; i < fentry->nr_nodes; i++) {
if (bpf_prog_calls_session_cookie(fentry->nodes[i])) {
u64 meta = func_meta | (cookie_bargs_off << BPF_TRAMP_COOKIE_INDEX_SHIFT);
store_func_meta(ctx, meta, func_meta_off);
cookie_bargs_off--;
}
invoke_bpf_prog(ctx, fentry->links[i], bargs_off,
invoke_bpf_prog(ctx, fentry->nodes[i], bargs_off,
retval_off, run_ctx_off,
flags & BPF_TRAMP_F_RET_FENTRY_RET);
}
if (fmod_ret->nr_links) {
branches = kcalloc(fmod_ret->nr_links, sizeof(__le32 *),
if (fmod_ret->nr_nodes) {
branches = kcalloc(fmod_ret->nr_nodes, sizeof(__le32 *),
GFP_KERNEL);
if (!branches)
return -ENOMEM;
@ -2771,7 +2771,7 @@ static int prepare_trampoline(struct jit_ctx *ctx, struct bpf_tramp_image *im,
}
/* update the branches saved in invoke_bpf_mod_ret with cbnz */
for (i = 0; i < fmod_ret->nr_links && ctx->image != NULL; i++) {
for (i = 0; i < fmod_ret->nr_nodes && ctx->image != NULL; i++) {
int offset = &ctx->image[ctx->idx] - branches[i];
*branches[i] = cpu_to_le32(A64_CBNZ(1, A64_R(10), offset));
}
@ -2782,14 +2782,14 @@ static int prepare_trampoline(struct jit_ctx *ctx, struct bpf_tramp_image *im,
store_func_meta(ctx, func_meta, func_meta_off);
cookie_bargs_off = (bargs_off - cookie_off) / 8;
for (i = 0; i < fexit->nr_links; i++) {
if (bpf_prog_calls_session_cookie(fexit->links[i])) {
for (i = 0; i < fexit->nr_nodes; i++) {
if (bpf_prog_calls_session_cookie(fexit->nodes[i])) {
u64 meta = func_meta | (cookie_bargs_off << BPF_TRAMP_COOKIE_INDEX_SHIFT);
store_func_meta(ctx, meta, func_meta_off);
cookie_bargs_off--;
}
invoke_bpf_prog(ctx, fexit->links[i], bargs_off, retval_off,
invoke_bpf_prog(ctx, fexit->nodes[i], bargs_off, retval_off,
run_ctx_off, false);
}
@ -2847,7 +2847,7 @@ bool bpf_jit_supports_fsession(void)
}
int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags,
struct bpf_tramp_links *tlinks, void *func_addr)
struct bpf_tramp_nodes *tnodes, void *func_addr)
{
struct jit_ctx ctx = {
.image = NULL,
@ -2861,7 +2861,7 @@ int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags,
if (ret < 0)
return ret;
ret = prepare_trampoline(&ctx, &im, tlinks, func_addr, m, &aaux, flags);
ret = prepare_trampoline(&ctx, &im, tnodes, func_addr, m, &aaux, flags);
if (ret < 0)
return ret;
@ -2885,7 +2885,7 @@ int arch_protect_bpf_trampoline(void *image, unsigned int size)
int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *ro_image,
void *ro_image_end, const struct btf_func_model *m,
u32 flags, struct bpf_tramp_links *tlinks,
u32 flags, struct bpf_tramp_nodes *tnodes,
void *func_addr)
{
u32 size = ro_image_end - ro_image;
@ -2912,7 +2912,7 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *ro_image,
ret = calc_arg_aux(m, &aaux);
if (ret)
goto out;
ret = prepare_trampoline(&ctx, im, tlinks, func_addr, m, &aaux, flags);
ret = prepare_trampoline(&ctx, im, tnodes, func_addr, m, &aaux, flags);
if (ret > 0 && validate_code(&ctx) < 0) {
ret = -EINVAL;

52
arch/loongarch/net/bpf_jit.c
View File

@ -1674,17 +1674,17 @@ static void restore_stk_args(struct jit_ctx *ctx, int nr_stk_args, int args_off,
}
}
static int invoke_bpf_prog(struct jit_ctx *ctx, struct bpf_tramp_link *l,
static int invoke_bpf_prog(struct jit_ctx *ctx, struct bpf_tramp_node *n,
int args_off, int retval_off, int run_ctx_off, bool save_ret)
{
int ret;
u32 *branch;
struct bpf_prog *p = l->link.prog;
struct bpf_prog *p = n->link->prog;
int cookie_off = offsetof(struct bpf_tramp_run_ctx, bpf_cookie);
if (l->cookie)
if (n->cookie)
emit_store_stack_imm64(ctx, LOONGARCH_GPR_T1,
-run_ctx_off + cookie_off, l->cookie);
-run_ctx_off + cookie_off, n->cookie);
else
emit_insn(ctx, std, LOONGARCH_GPR_ZERO, LOONGARCH_GPR_FP, -run_ctx_off + cookie_off);
@ -1737,22 +1737,22 @@ static int invoke_bpf_prog(struct jit_ctx *ctx, struct bpf_tramp_link *l,
return ret;
}
static int invoke_bpf(struct jit_ctx *ctx, struct bpf_tramp_links *tl,
static int invoke_bpf(struct jit_ctx *ctx, struct bpf_tramp_nodes *tn,
int args_off, int retval_off, int run_ctx_off,
int func_meta_off, bool save_ret, u64 func_meta, int cookie_off)
{
int i, cur_cookie = (cookie_off - args_off) / 8;
for (i = 0; i < tl->nr_links; i++) {
for (i = 0; i < tn->nr_nodes; i++) {
int err;
if (bpf_prog_calls_session_cookie(tl->links[i])) {
if (bpf_prog_calls_session_cookie(tn->nodes[i])) {
u64 meta = func_meta | ((u64)cur_cookie << BPF_TRAMP_COOKIE_INDEX_SHIFT);
emit_store_stack_imm64(ctx, LOONGARCH_GPR_T1, -func_meta_off, meta);
cur_cookie--;
}
err = invoke_bpf_prog(ctx, tl->links[i], args_off, retval_off, run_ctx_off, save_ret);
err = invoke_bpf_prog(ctx, tn->nodes[i], args_off, retval_off, run_ctx_off, save_ret);
if (err)
return err;
}
@ -1807,7 +1807,7 @@ static void sign_extend(struct jit_ctx *ctx, int rd, int rj, u8 size, bool sign)
}
static int __arch_prepare_bpf_trampoline(struct jit_ctx *ctx, struct bpf_tramp_image *im,
const struct btf_func_model *m, struct bpf_tramp_links *tlinks,
const struct btf_func_model *m, struct bpf_tramp_nodes *tnodes,
void *func_addr, u32 flags)
{
int i, ret, save_ret;
@ -1817,9 +1817,9 @@ static int __arch_prepare_bpf_trampoline(struct jit_ctx *ctx, struct bpf_tramp_i
unsigned long long func_meta;
bool is_struct_ops = flags & BPF_TRAMP_F_INDIRECT;
void *orig_call = func_addr;
struct bpf_tramp_links *fentry = &tlinks[BPF_TRAMP_FENTRY];
struct bpf_tramp_links *fexit = &tlinks[BPF_TRAMP_FEXIT];
struct bpf_tramp_links *fmod_ret = &tlinks[BPF_TRAMP_MODIFY_RETURN];
struct bpf_tramp_nodes *fentry = &tnodes[BPF_TRAMP_FENTRY];
struct bpf_tramp_nodes *fexit = &tnodes[BPF_TRAMP_FEXIT];
struct bpf_tramp_nodes *fmod_ret = &tnodes[BPF_TRAMP_MODIFY_RETURN];
u32 **branches = NULL;
/*
@ -1898,7 +1898,7 @@ static int __arch_prepare_bpf_trampoline(struct jit_ctx *ctx, struct bpf_tramp_i
ip_off = stack_size;
}
cookie_cnt = bpf_fsession_cookie_cnt(tlinks);
cookie_cnt = bpf_fsession_cookie_cnt(tnodes);
/* Room for session cookies */
stack_size += cookie_cnt * 8;
@ -1969,7 +1969,7 @@ static int __arch_prepare_bpf_trampoline(struct jit_ctx *ctx, struct bpf_tramp_i
store_args(ctx, nr_arg_slots, args_off);
if (bpf_fsession_cnt(tlinks)) {
if (bpf_fsession_cnt(tnodes)) {
/* clear all session cookies' value */
for (i = 0; i < cookie_cnt; i++)
emit_insn(ctx, std, LOONGARCH_GPR_ZERO, LOONGARCH_GPR_FP, -cookie_off + 8 * i);
@ -1994,20 +1994,20 @@ static int __arch_prepare_bpf_trampoline(struct jit_ctx *ctx, struct bpf_tramp_i
return ret;
}
if (fentry->nr_links) {
if (fentry->nr_nodes) {
ret = invoke_bpf(ctx, fentry, args_off, retval_off, run_ctx_off, func_meta_off,
flags & BPF_TRAMP_F_RET_FENTRY_RET, func_meta, cookie_off);
if (ret)
return ret;
}
if (fmod_ret->nr_links) {
branches = kcalloc(fmod_ret->nr_links, sizeof(u32 *), GFP_KERNEL);
if (fmod_ret->nr_nodes) {
branches = kcalloc(fmod_ret->nr_nodes, sizeof(u32 *), GFP_KERNEL);
if (!branches)
return -ENOMEM;
emit_insn(ctx, std, LOONGARCH_GPR_ZERO, LOONGARCH_GPR_FP, -retval_off);
for (i = 0; i < fmod_ret->nr_links; i++) {
ret = invoke_bpf_prog(ctx, fmod_ret->links[i],
for (i = 0; i < fmod_ret->nr_nodes; i++) {
ret = invoke_bpf_prog(ctx, fmod_ret->nodes[i],
args_off, retval_off, run_ctx_off, true);
if (ret)
goto out;
@ -2035,17 +2035,17 @@ static int __arch_prepare_bpf_trampoline(struct jit_ctx *ctx, struct bpf_tramp_i
emit_insn(ctx, nop);
}
for (i = 0; ctx->image && i < fmod_ret->nr_links; i++) {
for (i = 0; ctx->image && i < fmod_ret->nr_nodes; i++) {
int offset = (void *)(&ctx->image[ctx->idx]) - (void *)branches[i];
*branches[i] = larch_insn_gen_bne(LOONGARCH_GPR_T1, LOONGARCH_GPR_ZERO, offset);
}
/* Set "is_return" flag for fsession */
func_meta |= (1ULL << BPF_TRAMP_IS_RETURN_SHIFT);
if (bpf_fsession_cnt(tlinks))
if (bpf_fsession_cnt(tnodes))
emit_store_stack_imm64(ctx, LOONGARCH_GPR_T1, -func_meta_off, func_meta);
if (fexit->nr_links) {
if (fexit->nr_nodes) {
ret = invoke_bpf(ctx, fexit, args_off, retval_off, run_ctx_off,
func_meta_off, false, func_meta, cookie_off);
if (ret)
@ -2115,7 +2115,7 @@ static int __arch_prepare_bpf_trampoline(struct jit_ctx *ctx, struct bpf_tramp_i
int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *ro_image,
void *ro_image_end, const struct btf_func_model *m,
u32 flags, struct bpf_tramp_links *tlinks, void *func_addr)
u32 flags, struct bpf_tramp_nodes *tnodes, void *func_addr)
{
int ret, size;
void *image, *tmp;
@ -2131,7 +2131,7 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *ro_image,
ctx.idx = 0;
jit_fill_hole(image, (unsigned int)(ro_image_end - ro_image));
ret = __arch_prepare_bpf_trampoline(&ctx, im, m, tlinks, func_addr, flags);
ret = __arch_prepare_bpf_trampoline(&ctx, im, m, tnodes, func_addr, flags);
if (ret < 0)
goto out;
@ -2152,7 +2152,7 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *ro_image,
}
int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags,
struct bpf_tramp_links *tlinks, void *func_addr)
struct bpf_tramp_nodes *tnodes, void *func_addr)
{
int ret;
struct jit_ctx ctx;
@ -2161,7 +2161,7 @@ int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags,
ctx.image = NULL;
ctx.idx = 0;
ret = __arch_prepare_bpf_trampoline(&ctx, &im, m, tlinks, func_addr, flags);
ret = __arch_prepare_bpf_trampoline(&ctx, &im, m, tnodes, func_addr, flags);
return ret < 0 ? ret : ret * LOONGARCH_INSN_SIZE;
}

54
arch/powerpc/net/bpf_jit_comp.c
View File

@ -597,22 +597,22 @@ int arch_protect_bpf_trampoline(void *image, unsigned int size)
}
static int invoke_bpf_prog(u32 *image, u32 *ro_image, struct codegen_context *ctx,
struct bpf_tramp_link *l, int regs_off, int retval_off,
struct bpf_tramp_node *n, int regs_off, int retval_off,
int run_ctx_off, bool save_ret)
{
struct bpf_prog *p = l->link.prog;
struct bpf_prog *p = n->link->prog;
ppc_inst_t branch_insn;
u32 jmp_idx;
int ret = 0;
/* Save cookie */
if (IS_ENABLED(CONFIG_PPC64)) {
PPC_LI64(_R3, l->cookie);
PPC_LI64(_R3, n->cookie);
EMIT(PPC_RAW_STD(_R3, _R1, run_ctx_off + offsetof(struct bpf_tramp_run_ctx,
bpf_cookie)));
} else {
PPC_LI32(_R3, l->cookie >> 32);
PPC_LI32(_R4, l->cookie);
PPC_LI32(_R3, n->cookie >> 32);
PPC_LI32(_R4, n->cookie);
EMIT(PPC_RAW_STW(_R3, _R1,
run_ctx_off + offsetof(struct bpf_tramp_run_ctx, bpf_cookie)));
EMIT(PPC_RAW_STW(_R4, _R1,
@ -679,7 +679,7 @@ static int invoke_bpf_prog(u32 *image, u32 *ro_image, struct codegen_context *ct
}
static int invoke_bpf_mod_ret(u32 *image, u32 *ro_image, struct codegen_context *ctx,
struct bpf_tramp_links *tl, int regs_off, int retval_off,
struct bpf_tramp_nodes *tn, int regs_off, int retval_off,
int run_ctx_off, u32 *branches)
{
int i;
@ -690,8 +690,8 @@ static int invoke_bpf_mod_ret(u32 *image, u32 *ro_image, struct codegen_context
*/
EMIT(PPC_RAW_LI(_R3, 0));
EMIT(PPC_RAW_STL(_R3, _R1, retval_off));
for (i = 0; i < tl->nr_links; i++) {
if (invoke_bpf_prog(image, ro_image, ctx, tl->links[i], regs_off, retval_off,
for (i = 0; i < tn->nr_nodes; i++) {
if (invoke_bpf_prog(image, ro_image, ctx, tn->nodes[i], regs_off, retval_off,
run_ctx_off, true))
return -EINVAL;
@ -807,18 +807,18 @@ static void bpf_trampoline_restore_args_stack(u32 *image, struct codegen_context
static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_image,
void *rw_image_end, void *ro_image,
const struct btf_func_model *m, u32 flags,
struct bpf_tramp_links *tlinks,
struct bpf_tramp_nodes *tnodes,
void *func_addr)
{
int regs_off, func_meta_off, ip_off, run_ctx_off, retval_off;
int nvr_off, alt_lr_off, r4_off = 0;
struct bpf_tramp_links *fmod_ret = &tlinks[BPF_TRAMP_MODIFY_RETURN];
struct bpf_tramp_links *fentry = &tlinks[BPF_TRAMP_FENTRY];
struct bpf_tramp_links *fexit = &tlinks[BPF_TRAMP_FEXIT];
struct bpf_tramp_nodes *fmod_ret = &tnodes[BPF_TRAMP_MODIFY_RETURN];
struct bpf_tramp_nodes *fentry = &tnodes[BPF_TRAMP_FENTRY];
struct bpf_tramp_nodes *fexit = &tnodes[BPF_TRAMP_FEXIT];
int i, ret, nr_regs, retaddr_off, bpf_frame_size = 0;
struct codegen_context codegen_ctx, *ctx;
int cookie_off, cookie_cnt, cookie_ctx_off;
int fsession_cnt = bpf_fsession_cnt(tlinks);
int fsession_cnt = bpf_fsession_cnt(tnodes);
u64 func_meta;
u32 *image = (u32 *)rw_image;
ppc_inst_t branch_insn;
@ -893,7 +893,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im
/* room for session cookies */
cookie_off = bpf_frame_size;
cookie_cnt = bpf_fsession_cookie_cnt(tlinks);
cookie_cnt = bpf_fsession_cookie_cnt(tnodes);
bpf_frame_size += cookie_cnt * 8;
/* Room for IP address argument */
@ -1030,21 +1030,21 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im
cookie_ctx_off = (regs_off - cookie_off) / 8;
for (i = 0; i < fentry->nr_links; i++) {
if (bpf_prog_calls_session_cookie(fentry->links[i])) {
for (i = 0; i < fentry->nr_nodes; i++) {
if (bpf_prog_calls_session_cookie(fentry->nodes[i])) {
u64 meta = func_meta | (cookie_ctx_off << BPF_TRAMP_COOKIE_INDEX_SHIFT);
store_func_meta(image, ctx, meta, func_meta_off);
cookie_ctx_off--;
}
if (invoke_bpf_prog(image, ro_image, ctx, fentry->links[i], regs_off, retval_off,
if (invoke_bpf_prog(image, ro_image, ctx, fentry->nodes[i], regs_off, retval_off,
run_ctx_off, flags & BPF_TRAMP_F_RET_FENTRY_RET))
return -EINVAL;
}
if (fmod_ret->nr_links) {
branches = kcalloc(fmod_ret->nr_links, sizeof(u32), GFP_KERNEL);
if (fmod_ret->nr_nodes) {
branches = kcalloc(fmod_ret->nr_nodes, sizeof(u32), GFP_KERNEL);
if (!branches)
return -ENOMEM;
@ -1093,7 +1093,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im
}
/* Update branches saved in invoke_bpf_mod_ret with address of do_fexit */
for (i = 0; i < fmod_ret->nr_links && image; i++) {
for (i = 0; i < fmod_ret->nr_nodes && image; i++) {
if (create_cond_branch(&branch_insn, &image[branches[i]],
(unsigned long)&image[ctx->idx], COND_NE << 16)) {
ret = -EINVAL;
@ -1110,15 +1110,15 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im
cookie_ctx_off = (regs_off - cookie_off) / 8;
for (i = 0; i < fexit->nr_links; i++) {
if (bpf_prog_calls_session_cookie(fexit->links[i])) {
for (i = 0; i < fexit->nr_nodes; i++) {
if (bpf_prog_calls_session_cookie(fexit->nodes[i])) {
u64 meta = func_meta | (cookie_ctx_off << BPF_TRAMP_COOKIE_INDEX_SHIFT);
store_func_meta(image, ctx, meta, func_meta_off);
cookie_ctx_off--;
}
if (invoke_bpf_prog(image, ro_image, ctx, fexit->links[i], regs_off, retval_off,
if (invoke_bpf_prog(image, ro_image, ctx, fexit->nodes[i], regs_off, retval_off,
run_ctx_off, false)) {
ret = -EINVAL;
goto cleanup;
@ -1185,18 +1185,18 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im
}
int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags,
struct bpf_tramp_links *tlinks, void *func_addr)
struct bpf_tramp_nodes *tnodes, void *func_addr)
{
struct bpf_tramp_image im;
int ret;
ret = __arch_prepare_bpf_trampoline(&im, NULL, NULL, NULL, m, flags, tlinks, func_addr);
ret = __arch_prepare_bpf_trampoline(&im, NULL, NULL, NULL, m, flags, tnodes, func_addr);
return ret;
}
int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, void *image_end,
const struct btf_func_model *m, u32 flags,
struct bpf_tramp_links *tlinks,
struct bpf_tramp_nodes *tnodes,
void *func_addr)
{
u32 size = image_end - image;
@ -1212,7 +1212,7 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, void *i
return -ENOMEM;
ret = __arch_prepare_bpf_trampoline(im, rw_image, rw_image + size, image, m,
flags, tlinks, func_addr);
flags, tnodes, func_addr);
if (ret < 0)
goto out;

52
arch/riscv/net/bpf_jit_comp64.c
View File

@ -934,15 +934,15 @@ static void emit_store_stack_imm64(u8 reg, int stack_off, u64 imm64,
emit_sd(RV_REG_FP, stack_off, reg, ctx);
}
static int invoke_bpf_prog(struct bpf_tramp_link *l, int args_off, int retval_off,
static int invoke_bpf_prog(struct bpf_tramp_node *node, int args_off, int retval_off,
int run_ctx_off, bool save_ret, struct rv_jit_context *ctx)
{
int ret, branch_off;
struct bpf_prog *p = l->link.prog;
struct bpf_prog *p = node->link->prog;
int cookie_off = offsetof(struct bpf_tramp_run_ctx, bpf_cookie);
if (l->cookie)
emit_store_stack_imm64(RV_REG_T1, -run_ctx_off + cookie_off, l->cookie, ctx);
if (node->cookie)
emit_store_stack_imm64(RV_REG_T1, -run_ctx_off + cookie_off, node->cookie, ctx);
else
emit_sd(RV_REG_FP, -run_ctx_off + cookie_off, RV_REG_ZERO, ctx);
@ -996,22 +996,22 @@ static int invoke_bpf_prog(struct bpf_tramp_link *l, int args_off, int retval_of
return ret;
}
static int invoke_bpf(struct bpf_tramp_links *tl, int args_off, int retval_off,
static int invoke_bpf(struct bpf_tramp_nodes *tn, int args_off, int retval_off,
int run_ctx_off, int func_meta_off, bool save_ret, u64 func_meta,
int cookie_off, struct rv_jit_context *ctx)
{
int i, cur_cookie = (cookie_off - args_off) / 8;
for (i = 0; i < tl->nr_links; i++) {
for (i = 0; i < tn->nr_nodes; i++) {
int err;
if (bpf_prog_calls_session_cookie(tl->links[i])) {
if (bpf_prog_calls_session_cookie(tn->nodes[i])) {
u64 meta = func_meta | ((u64)cur_cookie << BPF_TRAMP_COOKIE_INDEX_SHIFT);
emit_store_stack_imm64(RV_REG_T1, -func_meta_off, meta, ctx);
cur_cookie--;
}
err = invoke_bpf_prog(tl->links[i], args_off, retval_off, run_ctx_off,
err = invoke_bpf_prog(tn->nodes[i], args_off, retval_off, run_ctx_off,
save_ret, ctx);
if (err)
return err;
@ -1021,7 +1021,7 @@ static int invoke_bpf(struct bpf_tramp_links *tl, int args_off, int retval_off,
static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im,
const struct btf_func_model *m,
struct bpf_tramp_links *tlinks,
struct bpf_tramp_nodes *tnodes,
void *func_addr, u32 flags,
struct rv_jit_context *ctx)
{
@ -1030,9 +1030,9 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im,
int stack_size = 0, nr_arg_slots = 0;
int retval_off, args_off, func_meta_off, ip_off, run_ctx_off, sreg_off, stk_arg_off;
int cookie_off, cookie_cnt;
struct bpf_tramp_links *fentry = &tlinks[BPF_TRAMP_FENTRY];
struct bpf_tramp_links *fexit = &tlinks[BPF_TRAMP_FEXIT];
struct bpf_tramp_links *fmod_ret = &tlinks[BPF_TRAMP_MODIFY_RETURN];
struct bpf_tramp_nodes *fentry = &tnodes[BPF_TRAMP_FENTRY];
struct bpf_tramp_nodes *fexit = &tnodes[BPF_TRAMP_FEXIT];
struct bpf_tramp_nodes *fmod_ret = &tnodes[BPF_TRAMP_MODIFY_RETURN];
bool is_struct_ops = flags & BPF_TRAMP_F_INDIRECT;
void *orig_call = func_addr;
bool save_ret;
@ -1115,7 +1115,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im,
ip_off = stack_size;
}
cookie_cnt = bpf_fsession_cookie_cnt(tlinks);
cookie_cnt = bpf_fsession_cookie_cnt(tnodes);
/* room for session cookies */
stack_size += cookie_cnt * 8;
cookie_off = stack_size;
@ -1172,7 +1172,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im,
store_args(nr_arg_slots, args_off, ctx);
if (bpf_fsession_cnt(tlinks)) {
if (bpf_fsession_cnt(tnodes)) {
/* clear all session cookies' value */
for (i = 0; i < cookie_cnt; i++)
emit_sd(RV_REG_FP, -cookie_off + 8 * i, RV_REG_ZERO, ctx);
@ -1187,22 +1187,22 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im,
return ret;
}
if (fentry->nr_links) {
if (fentry->nr_nodes) {
ret = invoke_bpf(fentry, args_off, retval_off, run_ctx_off, func_meta_off,
flags & BPF_TRAMP_F_RET_FENTRY_RET, func_meta, cookie_off, ctx);
if (ret)
return ret;
}
if (fmod_ret->nr_links) {
branches_off = kzalloc_objs(int, fmod_ret->nr_links);
if (fmod_ret->nr_nodes) {
branches_off = kzalloc_objs(int, fmod_ret->nr_nodes);
if (!branches_off)
return -ENOMEM;
/* cleanup to avoid garbage return value confusion */
emit_sd(RV_REG_FP, -retval_off, RV_REG_ZERO, ctx);
for (i = 0; i < fmod_ret->nr_links; i++) {
ret = invoke_bpf_prog(fmod_ret->links[i], args_off, retval_off,
for (i = 0; i < fmod_ret->nr_nodes; i++) {
ret = invoke_bpf_prog(fmod_ret->nodes[i], args_off, retval_off,
run_ctx_off, true, ctx);
if (ret)
goto out;
@ -1230,7 +1230,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im,
}
/* update branches saved in invoke_bpf_mod_ret with bnez */
for (i = 0; ctx->insns && i < fmod_ret->nr_links; i++) {
for (i = 0; ctx->insns && i < fmod_ret->nr_nodes; i++) {
offset = ninsns_rvoff(ctx->ninsns - branches_off[i]);
insn = rv_bne(RV_REG_T1, RV_REG_ZERO, offset >> 1);
*(u32 *)(ctx->insns + branches_off[i]) = insn;
@ -1238,10 +1238,10 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im,
/* set "is_return" flag for fsession */
func_meta |= (1ULL << BPF_TRAMP_IS_RETURN_SHIFT);
if (bpf_fsession_cnt(tlinks))
if (bpf_fsession_cnt(tnodes))
emit_store_stack_imm64(RV_REG_T1, -func_meta_off, func_meta, ctx);
if (fexit->nr_links) {
if (fexit->nr_nodes) {
ret = invoke_bpf(fexit, args_off, retval_off, run_ctx_off, func_meta_off,
false, func_meta, cookie_off, ctx);
if (ret)
@ -1305,7 +1305,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im,
}
int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags,
struct bpf_tramp_links *tlinks, void *func_addr)
struct bpf_tramp_nodes *tnodes, void *func_addr)
{
struct bpf_tramp_image im;
struct rv_jit_context ctx;
@ -1314,7 +1314,7 @@ int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags,
ctx.ninsns = 0;
ctx.insns = NULL;
ctx.ro_insns = NULL;
ret = __arch_prepare_bpf_trampoline(&im, m, tlinks, func_addr, flags, &ctx);
ret = __arch_prepare_bpf_trampoline(&im, m, tnodes, func_addr, flags, &ctx);
return ret < 0 ? ret : ninsns_rvoff(ctx.ninsns);
}
@ -1331,7 +1331,7 @@ void arch_free_bpf_trampoline(void *image, unsigned int size)
int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *ro_image,
void *ro_image_end, const struct btf_func_model *m,
u32 flags, struct bpf_tramp_links *tlinks,
u32 flags, struct bpf_tramp_nodes *tnodes,
void *func_addr)
{
int ret;
@ -1346,7 +1346,7 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *ro_image,
ctx.ninsns = 0;
ctx.insns = image;
ctx.ro_insns = ro_image;
ret = __arch_prepare_bpf_trampoline(im, m, tlinks, func_addr, flags, &ctx);
ret = __arch_prepare_bpf_trampoline(im, m, tnodes, func_addr, flags, &ctx);
if (ret < 0)
goto out;

44
arch/s390/net/bpf_jit_comp.c
View File

@ -2537,19 +2537,19 @@ static void emit_store_stack_imm64(struct bpf_jit *jit, int tmp_reg, int stack_o
static int invoke_bpf_prog(struct bpf_tramp_jit *tjit,
const struct btf_func_model *m,
struct bpf_tramp_link *tlink, bool save_ret)
struct bpf_tramp_node *node, bool save_ret)
{
struct bpf_jit *jit = &tjit->common;
int cookie_off = tjit->run_ctx_off +
offsetof(struct bpf_tramp_run_ctx, bpf_cookie);
struct bpf_prog *p = tlink->link.prog;
struct bpf_prog *p = node->link->prog;
int patch;
/*
* run_ctx.cookie = tlink->cookie;
* run_ctx.cookie = node->cookie;
*/
emit_store_stack_imm64(jit, REG_W0, cookie_off, tlink->cookie);
emit_store_stack_imm64(jit, REG_W0, cookie_off, node->cookie);
/*
* if ((start = __bpf_prog_enter(p, &run_ctx)) == 0)
@ -2609,20 +2609,20 @@ static int invoke_bpf_prog(struct bpf_tramp_jit *tjit,
static int invoke_bpf(struct bpf_tramp_jit *tjit,
const struct btf_func_model *m,
struct bpf_tramp_links *tl, bool save_ret,
struct bpf_tramp_nodes *tn, bool save_ret,
u64 func_meta, int cookie_off)
{
int i, cur_cookie = (tjit->bpf_args_off - cookie_off) / sizeof(u64);
struct bpf_jit *jit = &tjit->common;
for (i = 0; i < tl->nr_links; i++) {
if (bpf_prog_calls_session_cookie(tl->links[i])) {
for (i = 0; i < tn->nr_nodes; i++) {
if (bpf_prog_calls_session_cookie(tn->nodes[i])) {
u64 meta = func_meta | ((u64)cur_cookie << BPF_TRAMP_COOKIE_INDEX_SHIFT);
emit_store_stack_imm64(jit, REG_0, tjit->func_meta_off, meta);
cur_cookie--;
}
if (invoke_bpf_prog(tjit, m, tl->links[i], save_ret))
if (invoke_bpf_prog(tjit, m, tn->nodes[i], save_ret))
return -EINVAL;
}
@ -2651,12 +2651,12 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im,
struct bpf_tramp_jit *tjit,
const struct btf_func_model *m,
u32 flags,
struct bpf_tramp_links *tlinks,
struct bpf_tramp_nodes *tnodes,
void *func_addr)
{
struct bpf_tramp_links *fmod_ret = &tlinks[BPF_TRAMP_MODIFY_RETURN];
struct bpf_tramp_links *fentry = &tlinks[BPF_TRAMP_FENTRY];
struct bpf_tramp_links *fexit = &tlinks[BPF_TRAMP_FEXIT];
struct bpf_tramp_nodes *fmod_ret = &tnodes[BPF_TRAMP_MODIFY_RETURN];
struct bpf_tramp_nodes *fentry = &tnodes[BPF_TRAMP_FENTRY];
struct bpf_tramp_nodes *fexit = &tnodes[BPF_TRAMP_FEXIT];
int nr_bpf_args, nr_reg_args, nr_stack_args;
int cookie_cnt, cookie_off, fsession_cnt;
struct bpf_jit *jit = &tjit->common;
@ -2693,8 +2693,8 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im,
return -ENOTSUPP;
}
cookie_cnt = bpf_fsession_cookie_cnt(tlinks);
fsession_cnt = bpf_fsession_cnt(tlinks);
cookie_cnt = bpf_fsession_cookie_cnt(tnodes);
fsession_cnt = bpf_fsession_cnt(tnodes);
/*
* Calculate the stack layout.
@ -2829,7 +2829,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im,
func_meta, cookie_off))
return -EINVAL;
if (fmod_ret->nr_links) {
if (fmod_ret->nr_nodes) {
/*
* retval = 0;
*/
@ -2838,8 +2838,8 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im,
_EMIT6(0xd707f000 | tjit->retval_off,
0xf000 | tjit->retval_off);
for (i = 0; i < fmod_ret->nr_links; i++) {
if (invoke_bpf_prog(tjit, m, fmod_ret->links[i], true))
for (i = 0; i < fmod_ret->nr_nodes; i++) {
if (invoke_bpf_prog(tjit, m, fmod_ret->nodes[i], true))
return -EINVAL;
/*
@ -2964,7 +2964,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im,
}
int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags,
struct bpf_tramp_links *tlinks, void *orig_call)
struct bpf_tramp_nodes *tnodes, void *orig_call)
{
struct bpf_tramp_image im;
struct bpf_tramp_jit tjit;
@ -2973,14 +2973,14 @@ int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags,
memset(&tjit, 0, sizeof(tjit));
ret = __arch_prepare_bpf_trampoline(&im, &tjit, m, flags,
tlinks, orig_call);
tnodes, orig_call);
return ret < 0 ? ret : tjit.common.prg;
}
int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image,
void *image_end, const struct btf_func_model *m,
u32 flags, struct bpf_tramp_links *tlinks,
u32 flags, struct bpf_tramp_nodes *tnodes,
void *func_addr)
{
struct bpf_tramp_jit tjit;
@ -2989,7 +2989,7 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image,
/* Compute offsets, check whether the code fits. */
memset(&tjit, 0, sizeof(tjit));
ret = __arch_prepare_bpf_trampoline(im, &tjit, m, flags,
tlinks, func_addr);
tnodes, func_addr);
if (ret < 0)
return ret;
@ -3003,7 +3003,7 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image,
tjit.common.prg = 0;
tjit.common.prg_buf = image;
ret = __arch_prepare_bpf_trampoline(im, &tjit, m, flags,
tlinks, func_addr);
tnodes, func_addr);
return ret < 0 ? ret : tjit.common.prg;
}

54
arch/x86/net/bpf_jit_comp.c
View File

@ -3104,15 +3104,15 @@ static void restore_regs(const struct btf_func_model *m, u8 **prog,
}
static int invoke_bpf_prog(const struct btf_func_model *m, u8 **pprog,
struct bpf_tramp_link *l, int stack_size,
struct bpf_tramp_node *node, int stack_size,
int run_ctx_off, bool save_ret,
void *image, void *rw_image)
{
u8 *prog = *pprog;
u8 *jmp_insn;
int ctx_cookie_off = offsetof(struct bpf_tramp_run_ctx, bpf_cookie);
struct bpf_prog *p = l->link.prog;
u64 cookie = l->cookie;
struct bpf_prog *p = node->link->prog;
u64 cookie = node->cookie;
/* mov rdi, cookie */
emit_mov_imm64(&prog, BPF_REG_1, (long) cookie >> 32, (u32) (long) cookie);
@ -3219,7 +3219,7 @@ static int emit_cond_near_jump(u8 **pprog, void *func, void *ip, u8 jmp_cond)
}
static int invoke_bpf(const struct btf_func_model *m, u8 **pprog,
struct bpf_tramp_links *tl, int stack_size,
struct bpf_tramp_nodes *tl, int stack_size,
int run_ctx_off, int func_meta_off, bool save_ret,
void *image, void *rw_image, u64 func_meta,
int cookie_off)
@ -3227,13 +3227,13 @@ static int invoke_bpf(const struct btf_func_model *m, u8 **pprog,
int i, cur_cookie = (cookie_off - stack_size) / 8;
u8 *prog = *pprog;
for (i = 0; i < tl->nr_links; i++) {
if (tl->links[i]->link.prog->call_session_cookie) {
for (i = 0; i < tl->nr_nodes; i++) {
if (tl->nodes[i]->link->prog->call_session_cookie) {
emit_store_stack_imm64(&prog, BPF_REG_0, -func_meta_off,
func_meta | (cur_cookie << BPF_TRAMP_COOKIE_INDEX_SHIFT));
cur_cookie--;
}
if (invoke_bpf_prog(m, &prog, tl->links[i], stack_size,
if (invoke_bpf_prog(m, &prog, tl->nodes[i], stack_size,
run_ctx_off, save_ret, image, rw_image))
return -EINVAL;
}
@ -3242,7 +3242,7 @@ static int invoke_bpf(const struct btf_func_model *m, u8 **pprog,
}
static int invoke_bpf_mod_ret(const struct btf_func_model *m, u8 **pprog,
struct bpf_tramp_links *tl, int stack_size,
struct bpf_tramp_nodes *tl, int stack_size,
int run_ctx_off, u8 **branches,
void *image, void *rw_image)
{
@ -3254,8 +3254,8 @@ static int invoke_bpf_mod_ret(const struct btf_func_model *m, u8 **pprog,
*/
emit_mov_imm32(&prog, false, BPF_REG_0, 0);
emit_stx(&prog, BPF_DW, BPF_REG_FP, BPF_REG_0, -8);
for (i = 0; i < tl->nr_links; i++) {
if (invoke_bpf_prog(m, &prog, tl->links[i], stack_size, run_ctx_off, true,
for (i = 0; i < tl->nr_nodes; i++) {
if (invoke_bpf_prog(m, &prog, tl->nodes[i], stack_size, run_ctx_off, true,
image, rw_image))
return -EINVAL;
@ -3346,14 +3346,14 @@ static int invoke_bpf_mod_ret(const struct btf_func_model *m, u8 **pprog,
static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_image,
void *rw_image_end, void *image,
const struct btf_func_model *m, u32 flags,
struct bpf_tramp_links *tlinks,
struct bpf_tramp_nodes *tnodes,
void *func_addr)
{
int i, ret, nr_regs = m->nr_args, stack_size = 0;
int regs_off, func_meta_off, ip_off, run_ctx_off, arg_stack_off, rbx_off;
struct bpf_tramp_links *fentry = &tlinks[BPF_TRAMP_FENTRY];
struct bpf_tramp_links *fexit = &tlinks[BPF_TRAMP_FEXIT];
struct bpf_tramp_links *fmod_ret = &tlinks[BPF_TRAMP_MODIFY_RETURN];
struct bpf_tramp_nodes *fentry = &tnodes[BPF_TRAMP_FENTRY];
struct bpf_tramp_nodes *fexit = &tnodes[BPF_TRAMP_FEXIT];
struct bpf_tramp_nodes *fmod_ret = &tnodes[BPF_TRAMP_MODIFY_RETURN];
void *orig_call = func_addr;
int cookie_off, cookie_cnt;
u8 **branches = NULL;
@ -3425,7 +3425,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im
ip_off = stack_size;
cookie_cnt = bpf_fsession_cookie_cnt(tlinks);
cookie_cnt = bpf_fsession_cookie_cnt(tnodes);
/* room for session cookies */
stack_size += cookie_cnt * 8;
cookie_off = stack_size;
@ -3518,7 +3518,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im
}
}
if (bpf_fsession_cnt(tlinks)) {
if (bpf_fsession_cnt(tnodes)) {
/* clear all the session cookies' value */
for (int i = 0; i < cookie_cnt; i++)
emit_store_stack_imm64(&prog, BPF_REG_0, -cookie_off + 8 * i, 0);
@ -3526,15 +3526,15 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im
emit_store_stack_imm64(&prog, BPF_REG_0, -8, 0);
}
if (fentry->nr_links) {
if (fentry->nr_nodes) {
if (invoke_bpf(m, &prog, fentry, regs_off, run_ctx_off, func_meta_off,
flags & BPF_TRAMP_F_RET_FENTRY_RET, image, rw_image,
func_meta, cookie_off))
return -EINVAL;
}
if (fmod_ret->nr_links) {
branches = kcalloc(fmod_ret->nr_links, sizeof(u8 *),
if (fmod_ret->nr_nodes) {
branches = kcalloc(fmod_ret->nr_nodes, sizeof(u8 *),
GFP_KERNEL);
if (!branches)
return -ENOMEM;
@ -3573,7 +3573,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im
emit_nops(&prog, X86_PATCH_SIZE);
}
if (fmod_ret->nr_links) {
if (fmod_ret->nr_nodes) {
/* From Intel 64 and IA-32 Architectures Optimization
* Reference Manual, 3.4.1.4 Code Alignment, Assembly/Compiler
* Coding Rule 11: All branch targets should be 16-byte
@ -3583,7 +3583,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im
/* Update the branches saved in invoke_bpf_mod_ret with the
* aligned address of do_fexit.
*/
for (i = 0; i < fmod_ret->nr_links; i++) {
for (i = 0; i < fmod_ret->nr_nodes; i++) {
emit_cond_near_jump(&branches[i], image + (prog - (u8 *)rw_image),
image + (branches[i] - (u8 *)rw_image), X86_JNE);
}
@ -3591,10 +3591,10 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im
/* set the "is_return" flag for fsession */
func_meta |= (1ULL << BPF_TRAMP_IS_RETURN_SHIFT);
if (bpf_fsession_cnt(tlinks))
if (bpf_fsession_cnt(tnodes))
emit_store_stack_imm64(&prog, BPF_REG_0, -func_meta_off, func_meta);
if (fexit->nr_links) {
if (fexit->nr_nodes) {
if (invoke_bpf(m, &prog, fexit, regs_off, run_ctx_off, func_meta_off,
false, image, rw_image, func_meta, cookie_off)) {
ret = -EINVAL;
@ -3668,7 +3668,7 @@ int arch_protect_bpf_trampoline(void *image, unsigned int size)
int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, void *image_end,
const struct btf_func_model *m, u32 flags,
struct bpf_tramp_links *tlinks,
struct bpf_tramp_nodes *tnodes,
void *func_addr)
{
void *rw_image, *tmp;
@ -3683,7 +3683,7 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, void *i
return -ENOMEM;
ret = __arch_prepare_bpf_trampoline(im, rw_image, rw_image + size, image, m,
flags, tlinks, func_addr);
flags, tnodes, func_addr);
if (ret < 0)
goto out;
@ -3696,7 +3696,7 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, void *i
}
int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags,
struct bpf_tramp_links *tlinks, void *func_addr)
struct bpf_tramp_nodes *tnodes, void *func_addr)
{
struct bpf_tramp_image im;
void *image;
@ -3714,7 +3714,7 @@ int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags,
return -ENOMEM;
ret = __arch_prepare_bpf_trampoline(&im, image, image + PAGE_SIZE, image,
m, flags, tlinks, func_addr);
m, flags, tnodes, func_addr);
bpf_jit_free_exec(image);
return ret;
}

117
include/linux/bpf.h
View File

@ -33,6 +33,7 @@
#include <linux/memcontrol.h>
#include <linux/cfi.h>
#include <linux/key.h>
#include <linux/ftrace.h>
#include <asm/rqspinlock.h>
struct bpf_verifier_env;
@ -1251,9 +1252,9 @@ enum {
#define BPF_TRAMP_COOKIE_INDEX_SHIFT 8
#define BPF_TRAMP_IS_RETURN_SHIFT 63
struct bpf_tramp_links {
struct bpf_tramp_link *links[BPF_MAX_TRAMP_LINKS];
int nr_links;
struct bpf_tramp_nodes {
struct bpf_tramp_node *nodes[BPF_MAX_TRAMP_LINKS];
int nr_nodes;
};
struct bpf_tramp_run_ctx;
@ -1281,13 +1282,13 @@ struct bpf_tramp_run_ctx;
struct bpf_tramp_image;
int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, void *image_end,
const struct btf_func_model *m, u32 flags,
struct bpf_tramp_links *tlinks,
struct bpf_tramp_nodes *tnodes,
void *func_addr);
void *arch_alloc_bpf_trampoline(unsigned int size);
void arch_free_bpf_trampoline(void *image, unsigned int size);
int __must_check arch_protect_bpf_trampoline(void *image, unsigned int size);
int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags,
struct bpf_tramp_links *tlinks, void *func_addr);
struct bpf_tramp_nodes *tnodes, void *func_addr);
u64 notrace __bpf_prog_enter_sleepable_recur(struct bpf_prog *prog,
struct bpf_tramp_run_ctx *run_ctx);
@ -1353,8 +1354,6 @@ struct bpf_trampoline {
/* hlist for trampoline_ip_table */
struct hlist_node hlist_ip;
struct ftrace_ops *fops;
/* serializes access to fields of this trampoline */
struct mutex mutex;
refcount_t refcnt;
u32 flags;
u64 key;
@ -1375,6 +1374,11 @@ struct bpf_trampoline {
int progs_cnt[BPF_TRAMP_MAX];
/* Executable image of trampoline */
struct bpf_tramp_image *cur_image;
/* Used as temporary old image storage for multi_attach */
struct {
struct bpf_tramp_image *old_image;
u32 old_flags;
} multi_attach;
};
struct bpf_attach_target_info {
@ -1472,11 +1476,13 @@ static inline int bpf_dynptr_check_off_len(const struct bpf_dynptr_kern *ptr, u6
return 0;
}
struct bpf_tracing_multi_link;
#ifdef CONFIG_BPF_JIT
int bpf_trampoline_link_prog(struct bpf_tramp_link *link,
int bpf_trampoline_link_prog(struct bpf_tramp_node *node,
struct bpf_trampoline *tr,
struct bpf_prog *tgt_prog);
int bpf_trampoline_unlink_prog(struct bpf_tramp_link *link,
int bpf_trampoline_unlink_prog(struct bpf_tramp_node *node,
struct bpf_trampoline *tr,
struct bpf_prog *tgt_prog);
struct bpf_trampoline *bpf_trampoline_get(u64 key,
@ -1484,6 +1490,11 @@ struct bpf_trampoline *bpf_trampoline_get(u64 key,
void bpf_trampoline_put(struct bpf_trampoline *tr);
int arch_prepare_bpf_dispatcher(void *image, void *buf, s64 *funcs, int num_funcs);
int bpf_trampoline_multi_attach(struct bpf_prog *prog, u32 *ids,
struct bpf_tracing_multi_link *link);
int bpf_trampoline_multi_detach(struct bpf_prog *prog,
struct bpf_tracing_multi_link *link);
/*
* When the architecture supports STATIC_CALL replace the bpf_dispatcher_fn
* indirection with a direct call to the bpf program. If the architecture does
@ -1563,13 +1574,13 @@ bool bpf_insn_is_indirect_target(const struct bpf_verifier_env *env, const struc
int insn_idx);
u16 bpf_out_stack_arg_cnt(const struct bpf_verifier_env *env, const struct bpf_prog *prog);
#else
static inline int bpf_trampoline_link_prog(struct bpf_tramp_link *link,
static inline int bpf_trampoline_link_prog(struct bpf_tramp_node *node,
struct bpf_trampoline *tr,
struct bpf_prog *tgt_prog)
{
return -ENOTSUPP;
}
static inline int bpf_trampoline_unlink_prog(struct bpf_tramp_link *link,
static inline int bpf_trampoline_unlink_prog(struct bpf_tramp_node *node,
struct bpf_trampoline *tr,
struct bpf_prog *tgt_prog)
{
@ -1596,6 +1607,16 @@ static inline bool bpf_prog_has_trampoline(const struct bpf_prog *prog)
{
return false;
}
static inline int bpf_trampoline_multi_attach(struct bpf_prog *prog, u32 *ids,
struct bpf_tracing_multi_link *link)
{
return -ENOTSUPP;
}
static inline int bpf_trampoline_multi_detach(struct bpf_prog *prog,
struct bpf_tracing_multi_link *link)
{
return -ENOTSUPP;
}
#endif
struct bpf_func_info_aux {
@ -1911,12 +1932,17 @@ struct bpf_link_ops {
__poll_t (*poll)(struct file *file, struct poll_table_struct *pts);
};
struct bpf_tramp_link {
struct bpf_link link;
struct bpf_tramp_node {
struct bpf_link *link;
struct hlist_node tramp_hlist;
u64 cookie;
};
struct bpf_tramp_link {
struct bpf_link link;
struct bpf_tramp_node node;
};
struct bpf_shim_tramp_link {
struct bpf_tramp_link link;
struct bpf_trampoline *trampoline;
@ -1924,13 +1950,31 @@ struct bpf_shim_tramp_link {
struct bpf_tracing_link {
struct bpf_tramp_link link;
struct bpf_tramp_node fexit;
struct bpf_trampoline *trampoline;
struct bpf_prog *tgt_prog;
};
struct bpf_fsession_link {
struct bpf_tracing_link link;
struct bpf_tramp_link fexit;
struct bpf_tracing_multi_node {
struct bpf_tramp_node node;
struct bpf_trampoline *trampoline;
struct ftrace_func_entry entry;
};
struct bpf_tracing_multi_data {
struct ftrace_hash *unreg;
struct ftrace_hash *modify;
struct ftrace_hash *reg;
struct ftrace_func_entry *entry;
};
struct bpf_tracing_multi_link {
struct bpf_link link;
struct bpf_tracing_multi_data data;
u64 *cookies;
struct bpf_tramp_node *fexits;
int nodes_cnt;
struct bpf_tracing_multi_node nodes[] __counted_by(nodes_cnt);
};
struct bpf_raw_tp_link {
@ -2114,6 +2158,12 @@ static inline void bpf_prog_put_recursion_context(struct bpf_prog *prog)
#endif
}
static inline bool is_tracing_multi(enum bpf_attach_type type)
{
return type == BPF_TRACE_FENTRY_MULTI || type == BPF_TRACE_FEXIT_MULTI ||
type == BPF_TRACE_FSESSION_MULTI;
}
#if defined(CONFIG_BPF_JIT) && defined(CONFIG_BPF_SYSCALL)
/* This macro helps developer to register a struct_ops type and generate
* type information correctly. Developers should use this macro to register
@ -2134,8 +2184,8 @@ void bpf_struct_ops_put(const void *kdata);
int bpf_struct_ops_supported(const struct bpf_struct_ops *st_ops, u32 moff);
int bpf_struct_ops_map_sys_lookup_elem(struct bpf_map *map, void *key,
void *value);
int bpf_struct_ops_prepare_trampoline(struct bpf_tramp_links *tlinks,
struct bpf_tramp_link *link,
int bpf_struct_ops_prepare_trampoline(struct bpf_tramp_nodes *tnodes,
struct bpf_tramp_node *node,
const struct btf_func_model *model,
void *stub_func,
void **image, u32 *image_off,
@ -2230,31 +2280,33 @@ static inline void bpf_struct_ops_desc_release(struct bpf_struct_ops_desc *st_op
#endif
static inline int bpf_fsession_cnt(struct bpf_tramp_links *links)
static inline int bpf_fsession_cnt(struct bpf_tramp_nodes *nodes)
{
struct bpf_tramp_links fentries = links[BPF_TRAMP_FENTRY];
struct bpf_tramp_nodes fentries = nodes[BPF_TRAMP_FENTRY];
int cnt = 0;
for (int i = 0; i < links[BPF_TRAMP_FENTRY].nr_links; i++) {
if (fentries.links[i]->link.prog->expected_attach_type == BPF_TRACE_FSESSION)
for (int i = 0; i < nodes[BPF_TRAMP_FENTRY].nr_nodes; i++) {
if (fentries.nodes[i]->link->prog->expected_attach_type == BPF_TRACE_FSESSION)
cnt++;
if (fentries.nodes[i]->link->prog->expected_attach_type == BPF_TRACE_FSESSION_MULTI)
cnt++;
}
return cnt;
}
static inline bool bpf_prog_calls_session_cookie(struct bpf_tramp_link *link)
static inline bool bpf_prog_calls_session_cookie(struct bpf_tramp_node *node)
{
return link->link.prog->call_session_cookie;
return node->link->prog->call_session_cookie;
}
static inline int bpf_fsession_cookie_cnt(struct bpf_tramp_links *links)
static inline int bpf_fsession_cookie_cnt(struct bpf_tramp_nodes *nodes)
{
struct bpf_tramp_links fentries = links[BPF_TRAMP_FENTRY];
struct bpf_tramp_nodes fentries = nodes[BPF_TRAMP_FENTRY];
int cnt = 0;
for (int i = 0; i < links[BPF_TRAMP_FENTRY].nr_links; i++) {
if (bpf_prog_calls_session_cookie(fentries.links[i]))
for (int i = 0; i < nodes[BPF_TRAMP_FENTRY].nr_nodes; i++) {
if (bpf_prog_calls_session_cookie(fentries.nodes[i]))
cnt++;
}
@ -2802,6 +2854,9 @@ void bpf_link_init(struct bpf_link *link, enum bpf_link_type type,
void bpf_link_init_sleepable(struct bpf_link *link, enum bpf_link_type type,
const struct bpf_link_ops *ops, struct bpf_prog *prog,
enum bpf_attach_type attach_type, bool sleepable);
void bpf_tramp_link_init(struct bpf_tramp_link *link, enum bpf_link_type type,
const struct bpf_link_ops *ops, struct bpf_prog *prog,
enum bpf_attach_type attach_type, u64 cookie);
int bpf_link_prime(struct bpf_link *link, struct bpf_link_primer *primer);
int bpf_link_settle(struct bpf_link_primer *primer);
void bpf_link_cleanup(struct bpf_link_primer *primer);
@ -3225,6 +3280,12 @@ static inline void bpf_link_init_sleepable(struct bpf_link *link, enum bpf_link_
{
}
static inline void bpf_tramp_link_init(struct bpf_tramp_link *link, enum bpf_link_type type,
const struct bpf_link_ops *ops, struct bpf_prog *prog,
enum bpf_attach_type attach_type, u64 cookie)
{
}
static inline int bpf_link_prime(struct bpf_link *link,
struct bpf_link_primer *primer)
{

1
include/linux/bpf_types.h
View File

@ -156,3 +156,4 @@ BPF_LINK_TYPE(BPF_LINK_TYPE_PERF_EVENT, perf)
BPF_LINK_TYPE(BPF_LINK_TYPE_KPROBE_MULTI, kprobe_multi)
BPF_LINK_TYPE(BPF_LINK_TYPE_STRUCT_OPS, struct_ops)
BPF_LINK_TYPE(BPF_LINK_TYPE_UPROBE_MULTI, uprobe_multi)
BPF_LINK_TYPE(BPF_LINK_TYPE_TRACING_MULTI, tracing_multi)

4
include/linux/bpf_verifier.h
View File

@ -1591,6 +1591,10 @@ int bpf_add_kfunc_call(struct bpf_verifier_env *env, u32 func_id, u16 offset);
int bpf_fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
struct bpf_insn *insn_buf, int insn_idx, int *cnt);
/* Functions exported from verifier.c, used by trampoline.c */
int bpf_check_attach_btf_id_multi(struct btf *btf, struct bpf_prog *prog, u32 btf_id,
struct bpf_attach_target_info *tgt_info);
/* Functions in fixups.c, called from bpf_check() */
int bpf_remove_fastcall_spills_fills(struct bpf_verifier_env *env);
int bpf_optimize_bpf_loop(struct bpf_verifier_env *env);

1
include/linux/btf_ids.h
View File

@ -284,5 +284,6 @@ extern u32 bpf_cgroup_btf_id[];
extern u32 bpf_local_storage_map_btf_id[];
extern u32 btf_bpf_map_id[];
extern u32 bpf_kmem_cache_btf_id[];
extern u32 bpf_multi_func_btf_id[];
#endif

9
include/linux/ftrace.h
View File

@ -415,6 +415,8 @@ struct ftrace_hash *alloc_ftrace_hash(int size_bits);
void free_ftrace_hash(struct ftrace_hash *hash);
struct ftrace_func_entry *add_ftrace_hash_entry_direct(struct ftrace_hash *hash,
unsigned long ip, unsigned long direct);
void add_ftrace_hash_entry(struct ftrace_hash *hash, struct ftrace_func_entry *entry);
void ftrace_hash_remove(struct ftrace_hash *hash);
/* The hash used to know what functions callbacks trace */
struct ftrace_ops_hash {
@ -551,6 +553,8 @@ int update_ftrace_direct_mod(struct ftrace_ops *ops, struct ftrace_hash *hash, b
void ftrace_stub_direct_tramp(void);
unsigned long ftrace_hash_count(struct ftrace_hash *hash);
#else
struct ftrace_ops;
static inline unsigned long ftrace_find_rec_direct(unsigned long ip)
@ -590,6 +594,11 @@ static inline int update_ftrace_direct_mod(struct ftrace_ops *ops, struct ftrace
return -ENODEV;
}
static inline unsigned long ftrace_hash_count(struct ftrace_hash *hash)
{
return 0;
}
/*
* This must be implemented by the architecture.
* It is the way the ftrace direct_ops helper, when called

6
include/linux/trace_events.h
View File

@ -787,6 +787,7 @@ int bpf_get_perf_event_info(const struct perf_event *event, u32 *prog_id,
unsigned long *missed);
int bpf_kprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *prog);
int bpf_uprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *prog);
int bpf_tracing_multi_attach(struct bpf_prog *prog, const union bpf_attr *attr);
#else
static inline unsigned int trace_call_bpf(struct trace_event_call *call, void *ctx)
{
@ -844,6 +845,11 @@ bpf_uprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *prog)
{
return -EOPNOTSUPP;
}
static inline int
bpf_tracing_multi_attach(struct bpf_prog *prog, const union bpf_attr *attr)
{
return -EOPNOTSUPP;
}
#endif
enum {

9
include/uapi/linux/bpf.h
View File

@ -1156,6 +1156,9 @@ enum bpf_attach_type {
BPF_TRACE_KPROBE_SESSION,
BPF_TRACE_UPROBE_SESSION,
BPF_TRACE_FSESSION,
BPF_TRACE_FENTRY_MULTI,
BPF_TRACE_FEXIT_MULTI,
BPF_TRACE_FSESSION_MULTI,
__MAX_BPF_ATTACH_TYPE
};
@ -1180,6 +1183,7 @@ enum bpf_link_type {
BPF_LINK_TYPE_UPROBE_MULTI = 12,
BPF_LINK_TYPE_NETKIT = 13,
BPF_LINK_TYPE_SOCKMAP = 14,
BPF_LINK_TYPE_TRACING_MULTI = 15,
__MAX_BPF_LINK_TYPE,
};
@ -1875,6 +1879,11 @@ union bpf_attr {
};
__u64 expected_revision;
} cgroup;
struct {
__aligned_u64 ids;
__aligned_u64 cookies;
__u32 cnt;
} tracing_multi;
};
} link_create;

27
kernel/bpf/bpf_struct_ops.c
View File

@ -594,8 +594,8 @@ const struct bpf_link_ops bpf_struct_ops_link_lops = {
.dealloc = bpf_struct_ops_link_dealloc,
};
int bpf_struct_ops_prepare_trampoline(struct bpf_tramp_links *tlinks,
struct bpf_tramp_link *link,
int bpf_struct_ops_prepare_trampoline(struct bpf_tramp_nodes *tnodes,
struct bpf_tramp_node *node,
const struct btf_func_model *model,
void *stub_func,
void **_image, u32 *_image_off,
@ -605,13 +605,13 @@ int bpf_struct_ops_prepare_trampoline(struct bpf_tramp_links *tlinks,
void *image = *_image;
int size;
tlinks[BPF_TRAMP_FENTRY].links[0] = link;
tlinks[BPF_TRAMP_FENTRY].nr_links = 1;
tnodes[BPF_TRAMP_FENTRY].nodes[0] = node;
tnodes[BPF_TRAMP_FENTRY].nr_nodes = 1;
if (model->ret_size > 0)
flags |= BPF_TRAMP_F_RET_FENTRY_RET;
size = arch_bpf_trampoline_size(model, flags, tlinks, stub_func);
size = arch_bpf_trampoline_size(model, flags, tnodes, stub_func);
if (size <= 0)
return size ? : -EFAULT;
@ -628,7 +628,7 @@ int bpf_struct_ops_prepare_trampoline(struct bpf_tramp_links *tlinks,
size = arch_prepare_bpf_trampoline(NULL, image + image_off,
image + image_off + size,
model, flags, tlinks, stub_func);
model, flags, tnodes, stub_func);
if (size <= 0) {
if (image != *_image)
bpf_struct_ops_image_free(image);
@ -693,7 +693,7 @@ static long bpf_struct_ops_map_update_elem(struct bpf_map *map, void *key,
const struct btf_type *module_type;
const struct btf_member *member;
const struct btf_type *t = st_ops_desc->type;
struct bpf_tramp_links *tlinks;
struct bpf_tramp_nodes *tnodes;
void *udata, *kdata;
int prog_fd, err;
u32 i, trampoline_start, image_off = 0;
@ -720,8 +720,8 @@ static long bpf_struct_ops_map_update_elem(struct bpf_map *map, void *key,
if (uvalue->common.state || refcount_read(&uvalue->common.refcnt))
return -EINVAL;
tlinks = kzalloc_objs(*tlinks, BPF_TRAMP_MAX);
if (!tlinks)
tnodes = kzalloc_objs(*tnodes, BPF_TRAMP_MAX);
if (!tnodes)
return -ENOMEM;
uvalue = (struct bpf_struct_ops_value *)st_map->uvalue;
@ -817,8 +817,9 @@ static long bpf_struct_ops_map_update_elem(struct bpf_map *map, void *key,
err = -ENOMEM;
goto reset_unlock;
}
bpf_link_init(&link->link, BPF_LINK_TYPE_STRUCT_OPS,
&bpf_struct_ops_link_lops, prog, prog->expected_attach_type);
bpf_tramp_link_init(link, BPF_LINK_TYPE_STRUCT_OPS,
&bpf_struct_ops_link_lops, prog, prog->expected_attach_type, 0);
*plink++ = &link->link;
/* Poison pointer on error instead of return for backward compatibility */
@ -832,7 +833,7 @@ static long bpf_struct_ops_map_update_elem(struct bpf_map *map, void *key,
*pksym++ = ksym;
trampoline_start = image_off;
err = bpf_struct_ops_prepare_trampoline(tlinks, link,
err = bpf_struct_ops_prepare_trampoline(tnodes, &link->node,
&st_ops->func_models[i],
*(void **)(st_ops->cfi_stubs + moff),
&image, &image_off,
@ -911,7 +912,7 @@ static long bpf_struct_ops_map_update_elem(struct bpf_map *map, void *key,
memset(uvalue, 0, map->value_size);
memset(kvalue, 0, map->value_size);
unlock:
kfree(tlinks);
kfree(tnodes);
mutex_unlock(&st_map->lock);
if (!err)
bpf_struct_ops_map_add_ksyms(st_map);

2
kernel/bpf/fixups.c
View File

@ -2186,6 +2186,8 @@ int bpf_do_misc_fixups(struct bpf_verifier_env *env)
insn->imm == BPF_FUNC_get_func_ret) {
if (eatype == BPF_TRACE_FEXIT ||
eatype == BPF_TRACE_FSESSION ||
eatype == BPF_TRACE_FEXIT_MULTI ||
eatype == BPF_TRACE_FSESSION_MULTI ||
eatype == BPF_MODIFY_RETURN) {
/* Load nr_args from ctx - 8 */
insn_buf[0] = BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_1, -8);

83
kernel/bpf/syscall.c
View File

@ -41,6 +41,7 @@
#include <linux/overflow.h>
#include <linux/cookie.h>
#include <linux/verification.h>
#include <linux/btf_ids.h>
#include <net/netfilter/nf_bpf_link.h>
#include <net/netkit.h>
@ -2719,7 +2720,8 @@ static int
bpf_prog_load_check_attach(enum bpf_prog_type prog_type,
enum bpf_attach_type expected_attach_type,
struct btf *attach_btf, u32 btf_id,
struct bpf_prog *dst_prog)
struct bpf_prog *dst_prog,
bool multi_func)
{
if (btf_id) {
if (btf_id > BTF_MAX_TYPE)
@ -2739,6 +2741,14 @@ bpf_prog_load_check_attach(enum bpf_prog_type prog_type,
}
}
if (multi_func) {
if (prog_type != BPF_PROG_TYPE_TRACING)
return -EINVAL;
if (!attach_btf || btf_id)
return -EINVAL;
return 0;
}
if (attach_btf && (!btf_id || dst_prog))
return -EINVAL;
@ -2946,6 +2956,11 @@ static int bpf_prog_mark_insn_arrays_ready(struct bpf_prog *prog)
return 0;
}
extern int bpf_multi_func(void);
int __init __used bpf_multi_func(void) { return 0; }
BTF_ID_LIST_GLOBAL_SINGLE(bpf_multi_func_btf_id, func, bpf_multi_func)
/* last field in 'union bpf_attr' used by this command */
#define BPF_PROG_LOAD_LAST_FIELD keyring_id
@ -2958,6 +2973,7 @@ static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr, struct bpf_log_at
bool bpf_cap;
int err;
char license[128];
bool multi_func;
if (CHECK_ATTR(BPF_PROG_LOAD))
return -EINVAL;
@ -3024,6 +3040,8 @@ static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr, struct bpf_log_at
if (is_perfmon_prog_type(type) && !bpf_token_capable(token, CAP_PERFMON))
goto put_token;
multi_func = is_tracing_multi(attr->expected_attach_type);
/* attach_prog_fd/attach_btf_obj_fd can specify fd of either bpf_prog
* or btf, we need to check which one it is
*/
@ -3045,7 +3063,7 @@ static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr, struct bpf_log_at
goto put_token;
}
}
} else if (attr->attach_btf_id) {
} else if (attr->attach_btf_id || multi_func) {
/* fall back to vmlinux BTF, if BTF type ID is specified */
attach_btf = bpf_get_btf_vmlinux();
if (IS_ERR(attach_btf)) {
@ -3061,7 +3079,7 @@ static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr, struct bpf_log_at
if (bpf_prog_load_check_attach(type, attr->expected_attach_type,
attach_btf, attr->attach_btf_id,
dst_prog)) {
dst_prog, multi_func)) {
if (dst_prog)
bpf_prog_put(dst_prog);
if (attach_btf)
@ -3084,7 +3102,7 @@ static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr, struct bpf_log_at
prog->expected_attach_type = attr->expected_attach_type;
prog->sleepable = !!(attr->prog_flags & BPF_F_SLEEPABLE);
prog->aux->attach_btf = attach_btf;
prog->aux->attach_btf_id = attr->attach_btf_id;
prog->aux->attach_btf_id = multi_func ? bpf_multi_func_btf_id[0] : attr->attach_btf_id;
prog->aux->dst_prog = dst_prog;
prog->aux->dev_bound = !!attr->prog_ifindex;
prog->aux->xdp_has_frags = attr->prog_flags & BPF_F_XDP_HAS_FRAGS;
@ -3288,6 +3306,15 @@ void bpf_link_init(struct bpf_link *link, enum bpf_link_type type,
bpf_link_init_sleepable(link, type, ops, prog, attach_type, false);
}
void bpf_tramp_link_init(struct bpf_tramp_link *link, enum bpf_link_type type,
const struct bpf_link_ops *ops, struct bpf_prog *prog,
enum bpf_attach_type attach_type, u64 cookie)
{
bpf_link_init(&link->link, type, ops, prog, attach_type);
link->node.link = &link->link;
link->node.cookie = cookie;
}
static void bpf_link_free_id(int id)
{
if (!id)
@ -3595,7 +3622,7 @@ static void bpf_tracing_link_release(struct bpf_link *link)
struct bpf_tracing_link *tr_link =
container_of(link, struct bpf_tracing_link, link.link);
WARN_ON_ONCE(bpf_trampoline_unlink_prog(&tr_link->link,
WARN_ON_ONCE(bpf_trampoline_unlink_prog(&tr_link->link.node,
tr_link->trampoline,
tr_link->tgt_prog));
@ -3608,8 +3635,7 @@ static void bpf_tracing_link_release(struct bpf_link *link)
static void bpf_tracing_link_dealloc(struct bpf_link *link)
{
struct bpf_tracing_link *tr_link =
container_of(link, struct bpf_tracing_link, link.link);
struct bpf_tracing_link *tr_link = container_of(link, struct bpf_tracing_link, link.link);
kfree(tr_link);
}
@ -3617,8 +3643,8 @@ static void bpf_tracing_link_dealloc(struct bpf_link *link)
static void bpf_tracing_link_show_fdinfo(const struct bpf_link *link,
struct seq_file *seq)
{
struct bpf_tracing_link *tr_link =
container_of(link, struct bpf_tracing_link, link.link);
struct bpf_tracing_link *tr_link = container_of(link, struct bpf_tracing_link, link.link);
u32 target_btf_id, target_obj_id;
bpf_trampoline_unpack_key(tr_link->trampoline->key,
@ -3631,17 +3657,16 @@ static void bpf_tracing_link_show_fdinfo(const struct bpf_link *link,
link->attach_type,
target_obj_id,
target_btf_id,
tr_link->link.cookie);
tr_link->link.node.cookie);
}
static int bpf_tracing_link_fill_link_info(const struct bpf_link *link,
struct bpf_link_info *info)
{
struct bpf_tracing_link *tr_link =
container_of(link, struct bpf_tracing_link, link.link);
struct bpf_tracing_link *tr_link = container_of(link, struct bpf_tracing_link, link.link);
info->tracing.attach_type = link->attach_type;
info->tracing.cookie = tr_link->link.cookie;
info->tracing.cookie = tr_link->link.node.cookie;
bpf_trampoline_unpack_key(tr_link->trampoline->key,
&info->tracing.target_obj_id,
&info->tracing.target_btf_id);
@ -3723,29 +3748,18 @@ static int bpf_tracing_prog_attach(struct bpf_prog *prog,
key = bpf_trampoline_compute_key(tgt_prog, NULL, btf_id);
}
if (prog->expected_attach_type == BPF_TRACE_FSESSION) {
struct bpf_fsession_link *fslink;
fslink = kzalloc_obj(*fslink, GFP_USER);
if (fslink) {
bpf_link_init(&fslink->fexit.link, BPF_LINK_TYPE_TRACING,
&bpf_tracing_link_lops, prog, attach_type);
fslink->fexit.cookie = bpf_cookie;
link = &fslink->link;
} else {
link = NULL;
}
} else {
link = kzalloc_obj(*link, GFP_USER);
}
link = kzalloc_obj(*link, GFP_USER);
if (!link) {
err = -ENOMEM;
goto out_put_prog;
}
bpf_link_init(&link->link.link, BPF_LINK_TYPE_TRACING,
&bpf_tracing_link_lops, prog, attach_type);
bpf_tramp_link_init(&link->link, BPF_LINK_TYPE_TRACING,
&bpf_tracing_link_lops, prog, attach_type, bpf_cookie);
link->link.cookie = bpf_cookie;
if (prog->expected_attach_type == BPF_TRACE_FSESSION) {
link->fexit.link = &link->link.link;
link->fexit.cookie = bpf_cookie;
}
mutex_lock(&prog->aux->dst_mutex);
@ -3848,7 +3862,7 @@ static int bpf_tracing_prog_attach(struct bpf_prog *prog,
if (err)
goto out_unlock;
err = bpf_trampoline_link_prog(&link->link, tr, tgt_prog);
err = bpf_trampoline_link_prog(&link->link.node, tr, tgt_prog);
if (err) {
bpf_link_cleanup(&link_primer);
link = NULL;
@ -4484,6 +4498,9 @@ attach_type_to_prog_type(enum bpf_attach_type attach_type)
case BPF_TRACE_FENTRY:
case BPF_TRACE_FEXIT:
case BPF_TRACE_FSESSION:
case BPF_TRACE_FSESSION_MULTI:
case BPF_TRACE_FENTRY_MULTI:
case BPF_TRACE_FEXIT_MULTI:
case BPF_MODIFY_RETURN:
return BPF_PROG_TYPE_TRACING;
case BPF_LSM_MAC:
@ -5869,6 +5886,8 @@ static int link_create(union bpf_attr *attr, bpfptr_t uattr)
ret = bpf_iter_link_attach(attr, uattr, prog);
else if (prog->expected_attach_type == BPF_LSM_CGROUP)
ret = cgroup_bpf_link_attach(attr, prog);
else if (is_tracing_multi(prog->expected_attach_type))
ret = bpf_tracing_multi_attach(prog, attr);
else
ret = bpf_tracing_prog_attach(prog,
attr->link_create.target_fd,

672
kernel/bpf/trampoline.c
View File

@ -30,8 +30,46 @@ static struct hlist_head trampoline_ip_table[TRAMPOLINE_TABLE_SIZE];
/* serializes access to trampoline tables */
static DEFINE_MUTEX(trampoline_mutex);
/*
* Keep 32 trampoline locks (5 bits) in the pool so trampoline_lock_all()
* stays below MAX_LOCK_DEPTH. Each pool slot has a distinct lockdep
* class because trampoline_lock_all() takes all pool mutexes at once;
* otherwise lockdep would report recursive locking on same-class mutexes.
*/
#define TRAMPOLINE_LOCKS_BITS 5
#define TRAMPOLINE_LOCKS_TABLE_SIZE (1 << TRAMPOLINE_LOCKS_BITS)
static struct {
struct mutex mutex;
struct lock_class_key key;
} trampoline_locks[TRAMPOLINE_LOCKS_TABLE_SIZE];
static struct mutex *select_trampoline_lock(struct bpf_trampoline *tr)
{
return &trampoline_locks[hash_ptr(tr, TRAMPOLINE_LOCKS_BITS)].mutex;
}
static void trampoline_lock(struct bpf_trampoline *tr)
{
mutex_lock(select_trampoline_lock(tr));
}
static void trampoline_unlock(struct bpf_trampoline *tr)
{
mutex_unlock(select_trampoline_lock(tr));
}
struct bpf_trampoline_ops {
int (*register_fentry)(struct bpf_trampoline *tr, struct bpf_tramp_image *im, void *data);
int (*unregister_fentry)(struct bpf_trampoline *tr, u32 orig_flags, void *data);
int (*modify_fentry)(struct bpf_trampoline *tr, u32 orig_flags, struct bpf_tramp_image *im,
bool lock_direct_mutex, void *data);
};
#ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS
static int bpf_trampoline_update(struct bpf_trampoline *tr, bool lock_direct_mutex);
static int bpf_trampoline_update(struct bpf_trampoline *tr, bool lock_direct_mutex,
const struct bpf_trampoline_ops *ops, void *data);
static const struct bpf_trampoline_ops trampoline_ops;
#ifdef CONFIG_HAVE_SINGLE_FTRACE_DIRECT_OPS
static struct bpf_trampoline *direct_ops_ip_lookup(struct ftrace_ops *ops, unsigned long ip)
@ -69,9 +107,9 @@ static int bpf_tramp_ftrace_ops_func(struct ftrace_ops *ops, unsigned long ip,
if (cmd == FTRACE_OPS_CMD_ENABLE_SHARE_IPMODIFY_SELF) {
/* This is called inside register_ftrace_direct_multi(), so
* tr->mutex is already locked.
* trampoline's mutex is already locked.
*/
lockdep_assert_held_once(&tr->mutex);
lockdep_assert_held_once(select_trampoline_lock(tr));
/* Instead of updating the trampoline here, we propagate
* -EAGAIN to register_ftrace_direct(). Then we can
@ -91,7 +129,7 @@ static int bpf_tramp_ftrace_ops_func(struct ftrace_ops *ops, unsigned long ip,
}
/* The normal locking order is
* tr->mutex => direct_mutex (ftrace.c) => ftrace_lock (ftrace.c)
* select_trampoline_lock(tr) => direct_mutex (ftrace.c) => ftrace_lock (ftrace.c)
*
* The following two commands are called from
*
@ -99,12 +137,12 @@ static int bpf_tramp_ftrace_ops_func(struct ftrace_ops *ops, unsigned long ip,
* cleanup_direct_functions_after_ipmodify
*
* In both cases, direct_mutex is already locked. Use
* mutex_trylock(&tr->mutex) to avoid deadlock in race condition
* (something else is making changes to this same trampoline).
* mutex_trylock(select_trampoline_lock(tr)) to avoid deadlock in race condition
* (something else holds the same pool lock).
*/
if (!mutex_trylock(&tr->mutex)) {
/* sleep 1 ms to make sure whatever holding tr->mutex makes
* some progress.
if (!mutex_trylock(select_trampoline_lock(tr))) {
/* sleep 1 ms to make sure whatever holding select_trampoline_lock(tr)
* makes some progress.
*/
msleep(1);
return -EAGAIN;
@ -116,20 +154,22 @@ static int bpf_tramp_ftrace_ops_func(struct ftrace_ops *ops, unsigned long ip,
if ((tr->flags & BPF_TRAMP_F_CALL_ORIG) &&
!(tr->flags & BPF_TRAMP_F_ORIG_STACK))
ret = bpf_trampoline_update(tr, false /* lock_direct_mutex */);
ret = bpf_trampoline_update(tr, false /* lock_direct_mutex */,
&trampoline_ops, NULL);
break;
case FTRACE_OPS_CMD_DISABLE_SHARE_IPMODIFY_PEER:
tr->flags &= ~BPF_TRAMP_F_SHARE_IPMODIFY;
if (tr->flags & BPF_TRAMP_F_ORIG_STACK)
ret = bpf_trampoline_update(tr, false /* lock_direct_mutex */);
ret = bpf_trampoline_update(tr, false /* lock_direct_mutex */,
&trampoline_ops, NULL);
break;
default:
ret = -EINVAL;
break;
}
mutex_unlock(&tr->mutex);
trampoline_unlock(tr);
return ret;
}
#endif
@ -142,7 +182,9 @@ bool bpf_prog_has_trampoline(const struct bpf_prog *prog)
switch (ptype) {
case BPF_PROG_TYPE_TRACING:
if (eatype == BPF_TRACE_FENTRY || eatype == BPF_TRACE_FEXIT ||
eatype == BPF_MODIFY_RETURN || eatype == BPF_TRACE_FSESSION)
eatype == BPF_MODIFY_RETURN || eatype == BPF_TRACE_FSESSION ||
eatype == BPF_TRACE_FENTRY_MULTI || eatype == BPF_TRACE_FEXIT_MULTI ||
eatype == BPF_TRACE_FSESSION_MULTI)
return true;
return false;
case BPF_PROG_TYPE_LSM:
@ -359,7 +401,6 @@ static struct bpf_trampoline *bpf_trampoline_lookup(u64 key, unsigned long ip)
head = &trampoline_ip_table[hash_64(tr->ip, TRAMPOLINE_HASH_BITS)];
hlist_add_head(&tr->hlist_ip, head);
refcount_set(&tr->refcnt, 1);
mutex_init(&tr->mutex);
for (i = 0; i < BPF_TRAMP_MAX; i++)
INIT_HLIST_HEAD(&tr->progs_hlist[i]);
out:
@ -386,9 +427,11 @@ static int bpf_trampoline_update_fentry(struct bpf_trampoline *tr, u32 orig_flag
return bpf_arch_text_poke(ip, old_t, new_t, old_addr, new_addr);
}
static int unregister_fentry(struct bpf_trampoline *tr, u32 orig_flags,
void *old_addr)
static void bpf_tramp_image_put(struct bpf_tramp_image *im);
static int unregister_fentry(struct bpf_trampoline *tr, u32 orig_flags, void *data __maybe_unused)
{
void *old_addr = tr->cur_image->image;
int ret;
if (tr->func.ftrace_managed)
@ -396,13 +439,19 @@ static int unregister_fentry(struct bpf_trampoline *tr, u32 orig_flags,
else
ret = bpf_trampoline_update_fentry(tr, orig_flags, old_addr, NULL);
return ret;
if (ret)
return ret;
bpf_tramp_image_put(tr->cur_image);
tr->cur_image = NULL;
return 0;
}
static int modify_fentry(struct bpf_trampoline *tr, u32 orig_flags,
void *old_addr, void *new_addr,
bool lock_direct_mutex)
static int modify_fentry(struct bpf_trampoline *tr, u32 orig_flags, struct bpf_tramp_image *im,
bool lock_direct_mutex, void *data __maybe_unused)
{
void *old_addr = tr->cur_image->image;
void *new_addr = im->image;
int ret;
if (tr->func.ftrace_managed) {
@ -411,12 +460,20 @@ static int modify_fentry(struct bpf_trampoline *tr, u32 orig_flags,
ret = bpf_trampoline_update_fentry(tr, orig_flags, old_addr,
new_addr);
}
return ret;
if (ret)
return ret;
bpf_tramp_image_put(tr->cur_image);
tr->cur_image = im;
return 0;
}
/* first time registering */
static int register_fentry(struct bpf_trampoline *tr, void *new_addr)
static int register_fentry(struct bpf_trampoline *tr, struct bpf_tramp_image *im,
void *data __maybe_unused)
{
void *new_addr = im->image;
void *ip = tr->func.addr;
unsigned long faddr;
int ret;
@ -434,33 +491,42 @@ static int register_fentry(struct bpf_trampoline *tr, void *new_addr)
ret = bpf_trampoline_update_fentry(tr, 0, NULL, new_addr);
}
return ret;
if (ret)
return ret;
tr->cur_image = im;
return 0;
}
static struct bpf_tramp_links *
static const struct bpf_trampoline_ops trampoline_ops = {
.register_fentry = register_fentry,
.unregister_fentry = unregister_fentry,
.modify_fentry = modify_fentry,
};
static struct bpf_tramp_nodes *
bpf_trampoline_get_progs(const struct bpf_trampoline *tr, int *total, bool *ip_arg)
{
struct bpf_tramp_link *link;
struct bpf_tramp_links *tlinks;
struct bpf_tramp_link **links;
struct bpf_tramp_node *node, **nodes;
struct bpf_tramp_nodes *tnodes;
int kind;
*total = 0;
tlinks = kzalloc_objs(*tlinks, BPF_TRAMP_MAX);
if (!tlinks)
tnodes = kzalloc_objs(*tnodes, BPF_TRAMP_MAX);
if (!tnodes)
return ERR_PTR(-ENOMEM);
for (kind = 0; kind < BPF_TRAMP_MAX; kind++) {
tlinks[kind].nr_links = tr->progs_cnt[kind];
tnodes[kind].nr_nodes = tr->progs_cnt[kind];
*total += tr->progs_cnt[kind];
links = tlinks[kind].links;
nodes = tnodes[kind].nodes;
hlist_for_each_entry(link, &tr->progs_hlist[kind], tramp_hlist) {
*ip_arg |= link->link.prog->call_get_func_ip;
*links++ = link;
hlist_for_each_entry(node, &tr->progs_hlist[kind], tramp_hlist) {
*ip_arg |= node->link->prog->call_get_func_ip;
*nodes++ = node;
}
}
return tlinks;
return tnodes;
}
static void bpf_tramp_image_free(struct bpf_tramp_image *im)
@ -604,30 +670,29 @@ static struct bpf_tramp_image *bpf_tramp_image_alloc(u64 key, int size)
return ERR_PTR(err);
}
static int bpf_trampoline_update(struct bpf_trampoline *tr, bool lock_direct_mutex)
static int bpf_trampoline_update(struct bpf_trampoline *tr, bool lock_direct_mutex,
const struct bpf_trampoline_ops *ops, void *data)
{
struct bpf_tramp_image *im;
struct bpf_tramp_links *tlinks;
struct bpf_tramp_nodes *tnodes;
u32 orig_flags = tr->flags;
bool ip_arg = false;
int err, total, size;
tlinks = bpf_trampoline_get_progs(tr, &total, &ip_arg);
if (IS_ERR(tlinks))
return PTR_ERR(tlinks);
tnodes = bpf_trampoline_get_progs(tr, &total, &ip_arg);
if (IS_ERR(tnodes))
return PTR_ERR(tnodes);
if (total == 0) {
err = unregister_fentry(tr, orig_flags, tr->cur_image->image);
bpf_tramp_image_put(tr->cur_image);
tr->cur_image = NULL;
err = ops->unregister_fentry(tr, orig_flags, data);
goto out;
}
/* clear all bits except SHARE_IPMODIFY and TAIL_CALL_CTX */
tr->flags &= (BPF_TRAMP_F_SHARE_IPMODIFY | BPF_TRAMP_F_TAIL_CALL_CTX);
if (tlinks[BPF_TRAMP_FEXIT].nr_links ||
tlinks[BPF_TRAMP_MODIFY_RETURN].nr_links) {
if (tnodes[BPF_TRAMP_FEXIT].nr_nodes ||
tnodes[BPF_TRAMP_MODIFY_RETURN].nr_nodes) {
/* NOTE: BPF_TRAMP_F_RESTORE_REGS and BPF_TRAMP_F_SKIP_FRAME
* should not be set together.
*/
@ -658,7 +723,7 @@ static int bpf_trampoline_update(struct bpf_trampoline *tr, bool lock_direct_mut
#endif
size = arch_bpf_trampoline_size(&tr->func.model, tr->flags,
tlinks, tr->func.addr);
tnodes, tr->func.addr);
if (size < 0) {
err = size;
goto out;
@ -676,7 +741,7 @@ static int bpf_trampoline_update(struct bpf_trampoline *tr, bool lock_direct_mut
}
err = arch_prepare_bpf_trampoline(im, im->image, im->image + size,
&tr->func.model, tr->flags, tlinks,
&tr->func.model, tr->flags, tnodes,
tr->func.addr);
if (err < 0)
goto out_free;
@ -687,11 +752,10 @@ static int bpf_trampoline_update(struct bpf_trampoline *tr, bool lock_direct_mut
if (tr->cur_image)
/* progs already running at this address */
err = modify_fentry(tr, orig_flags, tr->cur_image->image,
im->image, lock_direct_mutex);
err = ops->modify_fentry(tr, orig_flags, im, lock_direct_mutex, data);
else
/* first time registering */
err = register_fentry(tr, im->image);
err = ops->register_fentry(tr, im, data);
#ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS
if (err == -EAGAIN) {
@ -703,34 +767,31 @@ static int bpf_trampoline_update(struct bpf_trampoline *tr, bool lock_direct_mut
goto again;
}
#endif
if (err)
goto out_free;
if (tr->cur_image)
bpf_tramp_image_put(tr->cur_image);
tr->cur_image = im;
out_free:
if (err)
bpf_tramp_image_free(im);
out:
/* If any error happens, restore previous flags */
if (err)
tr->flags = orig_flags;
kfree(tlinks);
kfree(tnodes);
return err;
out_free:
bpf_tramp_image_free(im);
goto out;
}
static enum bpf_tramp_prog_type bpf_attach_type_to_tramp(struct bpf_prog *prog)
{
switch (prog->expected_attach_type) {
case BPF_TRACE_FENTRY:
case BPF_TRACE_FENTRY_MULTI:
return BPF_TRAMP_FENTRY;
case BPF_MODIFY_RETURN:
return BPF_TRAMP_MODIFY_RETURN;
case BPF_TRACE_FEXIT:
case BPF_TRACE_FEXIT_MULTI:
return BPF_TRAMP_FEXIT;
case BPF_TRACE_FSESSION:
case BPF_TRACE_FSESSION_MULTI:
return BPF_TRAMP_FSESSION;
case BPF_LSM_MAC:
if (!prog->aux->attach_func_proto->type)
@ -763,18 +824,95 @@ static int bpf_freplace_check_tgt_prog(struct bpf_prog *tgt_prog)
return 0;
}
static int __bpf_trampoline_link_prog(struct bpf_tramp_link *link,
struct bpf_trampoline *tr,
struct bpf_prog *tgt_prog)
static struct bpf_tramp_node *fsession_exit(struct bpf_tramp_node *node)
{
if (node->link->type == BPF_LINK_TYPE_TRACING) {
struct bpf_tracing_link *link;
link = container_of(node->link, struct bpf_tracing_link, link.link);
return &link->fexit;
} else if (node->link->type == BPF_LINK_TYPE_TRACING_MULTI) {
struct bpf_tracing_multi_link *link;
struct bpf_tracing_multi_node *mnode;
link = container_of(node->link, struct bpf_tracing_multi_link, link);
mnode = container_of(node, struct bpf_tracing_multi_node, node);
return &link->fexits[mnode - link->nodes];
}
return NULL;
}
static int bpf_trampoline_add_prog(struct bpf_trampoline *tr,
struct bpf_tramp_node *node,
int cnt)
{
struct bpf_fsession_link *fslink = NULL;
enum bpf_tramp_prog_type kind;
struct bpf_tramp_link *link_exiting;
struct bpf_tramp_node *node_existing, *fexit;
struct hlist_head *prog_list;
kind = bpf_attach_type_to_tramp(node->link->prog);
if (kind == BPF_TRAMP_FSESSION) {
prog_list = &tr->progs_hlist[BPF_TRAMP_FENTRY];
cnt++;
} else {
prog_list = &tr->progs_hlist[kind];
}
if (cnt >= BPF_MAX_TRAMP_LINKS)
return -E2BIG;
if (!hlist_unhashed(&node->tramp_hlist))
/* prog already linked */
return -EBUSY;
hlist_for_each_entry(node_existing, prog_list, tramp_hlist) {
if (node_existing->link->prog != node->link->prog)
continue;
/* prog already linked */
return -EBUSY;
}
hlist_add_head(&node->tramp_hlist, prog_list);
if (kind == BPF_TRAMP_FSESSION) {
tr->progs_cnt[BPF_TRAMP_FENTRY]++;
fexit = fsession_exit(node);
if (WARN_ON_ONCE(!fexit))
return -EINVAL;
hlist_add_head(&fexit->tramp_hlist, &tr->progs_hlist[BPF_TRAMP_FEXIT]);
tr->progs_cnt[BPF_TRAMP_FEXIT]++;
} else {
tr->progs_cnt[kind]++;
}
return 0;
}
static void bpf_trampoline_remove_prog(struct bpf_trampoline *tr,
struct bpf_tramp_node *node)
{
enum bpf_tramp_prog_type kind;
struct bpf_tramp_node *fexit;
kind = bpf_attach_type_to_tramp(node->link->prog);
if (kind == BPF_TRAMP_FSESSION) {
fexit = fsession_exit(node);
if (WARN_ON_ONCE(!fexit))
return;
hlist_del_init(&fexit->tramp_hlist);
tr->progs_cnt[BPF_TRAMP_FEXIT]--;
kind = BPF_TRAMP_FENTRY;
}
hlist_del_init(&node->tramp_hlist);
tr->progs_cnt[kind]--;
}
static int __bpf_trampoline_link_prog(struct bpf_tramp_node *node,
struct bpf_trampoline *tr,
struct bpf_prog *tgt_prog,
const struct bpf_trampoline_ops *ops,
void *data)
{
enum bpf_tramp_prog_type kind;
int err = 0;
int cnt = 0, i;
kind = bpf_attach_type_to_tramp(link->link.prog);
kind = bpf_attach_type_to_tramp(node->link->prog);
if (tr->extension_prog)
/* cannot attach fentry/fexit if extension prog is attached.
* cannot overwrite extension prog either.
@ -791,72 +929,42 @@ static int __bpf_trampoline_link_prog(struct bpf_tramp_link *link,
err = bpf_freplace_check_tgt_prog(tgt_prog);
if (err)
return err;
tr->extension_prog = link->link.prog;
tr->extension_prog = node->link->prog;
return bpf_arch_text_poke(tr->func.addr, BPF_MOD_NOP,
BPF_MOD_JUMP, NULL,
link->link.prog->bpf_func);
}
if (kind == BPF_TRAMP_FSESSION) {
prog_list = &tr->progs_hlist[BPF_TRAMP_FENTRY];
cnt++;
} else {
prog_list = &tr->progs_hlist[kind];
}
if (cnt >= BPF_MAX_TRAMP_LINKS)
return -E2BIG;
if (!hlist_unhashed(&link->tramp_hlist))
/* prog already linked */
return -EBUSY;
hlist_for_each_entry(link_exiting, prog_list, tramp_hlist) {
if (link_exiting->link.prog != link->link.prog)
continue;
/* prog already linked */
return -EBUSY;
}
hlist_add_head(&link->tramp_hlist, prog_list);
if (kind == BPF_TRAMP_FSESSION) {
tr->progs_cnt[BPF_TRAMP_FENTRY]++;
fslink = container_of(link, struct bpf_fsession_link, link.link);
hlist_add_head(&fslink->fexit.tramp_hlist, &tr->progs_hlist[BPF_TRAMP_FEXIT]);
tr->progs_cnt[BPF_TRAMP_FEXIT]++;
} else {
tr->progs_cnt[kind]++;
}
err = bpf_trampoline_update(tr, true /* lock_direct_mutex */);
if (err) {
hlist_del_init(&link->tramp_hlist);
if (kind == BPF_TRAMP_FSESSION) {
tr->progs_cnt[BPF_TRAMP_FENTRY]--;
hlist_del_init(&fslink->fexit.tramp_hlist);
tr->progs_cnt[BPF_TRAMP_FEXIT]--;
} else {
tr->progs_cnt[kind]--;
}
node->link->prog->bpf_func);
}
err = bpf_trampoline_add_prog(tr, node, cnt);
if (err)
return err;
err = bpf_trampoline_update(tr, true /* lock_direct_mutex */, ops, data);
if (err)
bpf_trampoline_remove_prog(tr, node);
return err;
}
int bpf_trampoline_link_prog(struct bpf_tramp_link *link,
int bpf_trampoline_link_prog(struct bpf_tramp_node *node,
struct bpf_trampoline *tr,
struct bpf_prog *tgt_prog)
{
int err;
mutex_lock(&tr->mutex);
err = __bpf_trampoline_link_prog(link, tr, tgt_prog);
mutex_unlock(&tr->mutex);
trampoline_lock(tr);
err = __bpf_trampoline_link_prog(node, tr, tgt_prog, &trampoline_ops, NULL);
trampoline_unlock(tr);
return err;
}
static int __bpf_trampoline_unlink_prog(struct bpf_tramp_link *link,
static int __bpf_trampoline_unlink_prog(struct bpf_tramp_node *node,
struct bpf_trampoline *tr,
struct bpf_prog *tgt_prog)
struct bpf_prog *tgt_prog,
const struct bpf_trampoline_ops *ops,
void *data)
{
enum bpf_tramp_prog_type kind;
int err;
kind = bpf_attach_type_to_tramp(link->link.prog);
kind = bpf_attach_type_to_tramp(node->link->prog);
if (kind == BPF_TRAMP_REPLACE) {
WARN_ON_ONCE(!tr->extension_prog);
err = bpf_arch_text_poke(tr->func.addr, BPF_MOD_JUMP,
@ -866,29 +974,21 @@ static int __bpf_trampoline_unlink_prog(struct bpf_tramp_link *link,
guard(mutex)(&tgt_prog->aux->ext_mutex);
tgt_prog->aux->is_extended = false;
return err;
} else if (kind == BPF_TRAMP_FSESSION) {
struct bpf_fsession_link *fslink =
container_of(link, struct bpf_fsession_link, link.link);
hlist_del_init(&fslink->fexit.tramp_hlist);
tr->progs_cnt[BPF_TRAMP_FEXIT]--;
kind = BPF_TRAMP_FENTRY;
}
hlist_del_init(&link->tramp_hlist);
tr->progs_cnt[kind]--;
return bpf_trampoline_update(tr, true /* lock_direct_mutex */);
bpf_trampoline_remove_prog(tr, node);
return bpf_trampoline_update(tr, true /* lock_direct_mutex */, ops, data);
}
/* bpf_trampoline_unlink_prog() should never fail. */
int bpf_trampoline_unlink_prog(struct bpf_tramp_link *link,
int bpf_trampoline_unlink_prog(struct bpf_tramp_node *node,
struct bpf_trampoline *tr,
struct bpf_prog *tgt_prog)
{
int err;
mutex_lock(&tr->mutex);
err = __bpf_trampoline_unlink_prog(link, tr, tgt_prog);
mutex_unlock(&tr->mutex);
trampoline_lock(tr);
err = __bpf_trampoline_unlink_prog(node, tr, tgt_prog, &trampoline_ops, NULL);
trampoline_unlock(tr);
return err;
}
@ -902,7 +1002,7 @@ static void bpf_shim_tramp_link_release(struct bpf_link *link)
if (!shim_link->trampoline)
return;
WARN_ON_ONCE(bpf_trampoline_unlink_prog(&shim_link->link, shim_link->trampoline, NULL));
WARN_ON_ONCE(bpf_trampoline_unlink_prog(&shim_link->link.node, shim_link->trampoline, NULL));
bpf_trampoline_put(shim_link->trampoline);
}
@ -948,8 +1048,8 @@ static struct bpf_shim_tramp_link *cgroup_shim_alloc(const struct bpf_prog *prog
p->type = BPF_PROG_TYPE_LSM;
p->expected_attach_type = BPF_LSM_MAC;
bpf_prog_inc(p);
bpf_link_init(&shim_link->link.link, BPF_LINK_TYPE_UNSPEC,
&bpf_shim_tramp_link_lops, p, attach_type);
bpf_tramp_link_init(&shim_link->link, BPF_LINK_TYPE_UNSPEC,
&bpf_shim_tramp_link_lops, p, attach_type, 0);
bpf_cgroup_atype_get(p->aux->attach_btf_id, cgroup_atype);
return shim_link;
@ -958,15 +1058,15 @@ static struct bpf_shim_tramp_link *cgroup_shim_alloc(const struct bpf_prog *prog
static struct bpf_shim_tramp_link *cgroup_shim_find(struct bpf_trampoline *tr,
bpf_func_t bpf_func)
{
struct bpf_tramp_link *link;
struct bpf_tramp_node *node;
int kind;
for (kind = 0; kind < BPF_TRAMP_MAX; kind++) {
hlist_for_each_entry(link, &tr->progs_hlist[kind], tramp_hlist) {
struct bpf_prog *p = link->link.prog;
hlist_for_each_entry(node, &tr->progs_hlist[kind], tramp_hlist) {
struct bpf_prog *p = node->link->prog;
if (p->bpf_func == bpf_func)
return container_of(link, struct bpf_shim_tramp_link, link);
return container_of(node, struct bpf_shim_tramp_link, link.node);
}
}
@ -998,12 +1098,12 @@ int bpf_trampoline_link_cgroup_shim(struct bpf_prog *prog,
if (!tr)
return -ENOMEM;
mutex_lock(&tr->mutex);
trampoline_lock(tr);
shim_link = cgroup_shim_find(tr, bpf_func);
if (shim_link && !IS_ERR(bpf_link_inc_not_zero(&shim_link->link.link))) {
/* Reusing existing shim attached by the other program. */
mutex_unlock(&tr->mutex);
trampoline_unlock(tr);
bpf_trampoline_put(tr); /* bpf_trampoline_get above */
return 0;
}
@ -1016,23 +1116,23 @@ int bpf_trampoline_link_cgroup_shim(struct bpf_prog *prog,
goto err;
}
err = __bpf_trampoline_link_prog(&shim_link->link, tr, NULL);
err = __bpf_trampoline_link_prog(&shim_link->link.node, tr, NULL, &trampoline_ops, NULL);
if (err)
goto err;
shim_link->trampoline = tr;
/* note, we're still holding tr refcnt from above */
mutex_unlock(&tr->mutex);
trampoline_unlock(tr);
return 0;
err:
mutex_unlock(&tr->mutex);
trampoline_unlock(tr);
if (shim_link)
bpf_link_put(&shim_link->link.link);
/* have to release tr while _not_ holding its mutex */
/* have to release tr while _not_ holding pool mutex for trampoline */
bpf_trampoline_put(tr); /* bpf_trampoline_get above */
return err;
@ -1053,9 +1153,9 @@ void bpf_trampoline_unlink_cgroup_shim(struct bpf_prog *prog)
if (WARN_ON_ONCE(!tr))
return;
mutex_lock(&tr->mutex);
trampoline_lock(tr);
shim_link = cgroup_shim_find(tr, bpf_func);
mutex_unlock(&tr->mutex);
trampoline_unlock(tr);
if (shim_link)
bpf_link_put(&shim_link->link.link);
@ -1073,14 +1173,14 @@ struct bpf_trampoline *bpf_trampoline_get(u64 key,
if (!tr)
return NULL;
mutex_lock(&tr->mutex);
trampoline_lock(tr);
if (tr->func.addr)
goto out;
memcpy(&tr->func.model, &tgt_info->fmodel, sizeof(tgt_info->fmodel));
tr->func.addr = (void *)tgt_info->tgt_addr;
out:
mutex_unlock(&tr->mutex);
trampoline_unlock(tr);
return tr;
}
@ -1093,7 +1193,6 @@ void bpf_trampoline_put(struct bpf_trampoline *tr)
mutex_lock(&trampoline_mutex);
if (!refcount_dec_and_test(&tr->refcnt))
goto out;
WARN_ON_ONCE(mutex_is_locked(&tr->mutex));
for (i = 0; i < BPF_TRAMP_MAX; i++)
if (WARN_ON_ONCE(!hlist_empty(&tr->progs_hlist[i])))
@ -1332,7 +1431,7 @@ bpf_trampoline_exit_t bpf_trampoline_exit(const struct bpf_prog *prog)
int __weak
arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, void *image_end,
const struct btf_func_model *m, u32 flags,
struct bpf_tramp_links *tlinks,
struct bpf_tramp_nodes *tnodes,
void *func_addr)
{
return -ENOTSUPP;
@ -1366,11 +1465,288 @@ int __weak arch_protect_bpf_trampoline(void *image, unsigned int size)
}
int __weak arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags,
struct bpf_tramp_links *tlinks, void *func_addr)
struct bpf_tramp_nodes *tnodes, void *func_addr)
{
return -ENOTSUPP;
}
#if defined(CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS) && \
defined(CONFIG_HAVE_SINGLE_FTRACE_DIRECT_OPS) && \
defined(CONFIG_BPF_SYSCALL)
static void trampoline_lock_all(void)
{
int i;
for (i = 0; i < TRAMPOLINE_LOCKS_TABLE_SIZE; i++)
mutex_lock(&trampoline_locks[i].mutex);
}
static void trampoline_unlock_all(void)
{
int i;
for (i = 0; i < TRAMPOLINE_LOCKS_TABLE_SIZE; i++)
mutex_unlock(&trampoline_locks[i].mutex);
}
static void remove_tracing_multi_data(struct bpf_tracing_multi_data *data)
{
ftrace_hash_remove(data->reg);
ftrace_hash_remove(data->unreg);
ftrace_hash_remove(data->modify);
}
static void clear_tracing_multi_data(struct bpf_tracing_multi_data *data)
{
remove_tracing_multi_data(data);
free_ftrace_hash(data->reg);
free_ftrace_hash(data->unreg);
free_ftrace_hash(data->modify);
}
static int init_tracing_multi_data(struct bpf_tracing_multi_data *data)
{
data->reg = alloc_ftrace_hash(FTRACE_HASH_DEFAULT_BITS);
data->unreg = alloc_ftrace_hash(FTRACE_HASH_DEFAULT_BITS);
data->modify = alloc_ftrace_hash(FTRACE_HASH_DEFAULT_BITS);
if (!data->reg || !data->unreg || !data->modify) {
clear_tracing_multi_data(data);
return -ENOMEM;
}
return 0;
}
static void ftrace_hash_add(struct ftrace_hash *hash, struct ftrace_func_entry *entry,
unsigned long ip, unsigned long direct)
{
entry->ip = ip;
entry->direct = direct;
add_ftrace_hash_entry(hash, entry);
}
static int register_fentry_multi(struct bpf_trampoline *tr, struct bpf_tramp_image *im, void *ptr)
{
unsigned long addr = (unsigned long) im->image;
unsigned long ip = ftrace_location(tr->ip);
struct bpf_tracing_multi_data *data = ptr;
if (bpf_trampoline_use_jmp(tr->flags))
addr = ftrace_jmp_set(addr);
ftrace_hash_add(data->reg, data->entry, ip, addr);
tr->cur_image = im;
return 0;
}
static int unregister_fentry_multi(struct bpf_trampoline *tr, u32 orig_flags, void *ptr)
{
unsigned long addr = (unsigned long) tr->cur_image->image;
unsigned long ip = ftrace_location(tr->ip);
struct bpf_tracing_multi_data *data = ptr;
if (bpf_trampoline_use_jmp(tr->flags))
addr = ftrace_jmp_set(addr);
ftrace_hash_add(data->unreg, data->entry, ip, addr);
tr->cur_image = NULL;
return 0;
}
static int modify_fentry_multi(struct bpf_trampoline *tr, u32 orig_flags, struct bpf_tramp_image *im,
bool lock_direct_mutex, void *ptr)
{
unsigned long addr = (unsigned long) im->image;
unsigned long ip = ftrace_location(tr->ip);
struct bpf_tracing_multi_data *data = ptr;
if (bpf_trampoline_use_jmp(tr->flags))
addr = ftrace_jmp_set(addr);
ftrace_hash_add(data->modify, data->entry, ip, addr);
tr->cur_image = im;
return 0;
}
static const struct bpf_trampoline_ops trampoline_multi_ops = {
.register_fentry = register_fentry_multi,
.unregister_fentry = unregister_fentry_multi,
.modify_fentry = modify_fentry_multi,
};
static void bpf_trampoline_multi_attach_init(struct bpf_trampoline *tr)
{
tr->multi_attach.old_image = tr->cur_image;
tr->multi_attach.old_flags = tr->flags;
}
static void bpf_trampoline_multi_attach_free(struct bpf_trampoline *tr)
{
if (tr->multi_attach.old_image)
bpf_tramp_image_put(tr->multi_attach.old_image);
tr->multi_attach.old_image = NULL;
tr->multi_attach.old_flags = 0;
}
static void bpf_trampoline_multi_attach_rollback(struct bpf_trampoline *tr)
{
if (tr->cur_image)
bpf_tramp_image_put(tr->cur_image);
tr->cur_image = tr->multi_attach.old_image;
tr->flags = tr->multi_attach.old_flags;
tr->multi_attach.old_image = NULL;
tr->multi_attach.old_flags = 0;
}
#define for_each_mnode_cnt(mnode, link, cnt) \
for (i = 0, mnode = &link->nodes[i]; i < cnt; i++, mnode = &link->nodes[i])
#define for_each_mnode(mnode, link) \
for_each_mnode_cnt(mnode, link, link->nodes_cnt)
int bpf_trampoline_multi_attach(struct bpf_prog *prog, u32 *ids,
struct bpf_tracing_multi_link *link)
{
struct bpf_tracing_multi_data *data = &link->data;
struct bpf_attach_target_info tgt_info = {};
struct btf *btf = prog->aux->attach_btf;
struct bpf_tracing_multi_node *mnode;
struct bpf_trampoline *tr;
int i, err, rollback_cnt;
u64 key;
for_each_mnode(mnode, link) {
rollback_cnt = i;
err = bpf_check_attach_btf_id_multi(btf, prog, ids[i], &tgt_info);
if (err)
goto rollback_put;
key = bpf_trampoline_compute_key(NULL, btf, ids[i]);
tr = bpf_trampoline_get(key, &tgt_info);
if (!tr) {
err = -ENOMEM;
goto rollback_put;
}
mnode->trampoline = tr;
mnode->node.link = &link->link;
mnode->node.cookie = link->cookies ? link->cookies[i] : 0;
if (prog->expected_attach_type == BPF_TRACE_FSESSION_MULTI) {
link->fexits[i].link = &link->link;
link->fexits[i].cookie = link->cookies ? link->cookies[i] : 0;
}
cond_resched();
}
err = init_tracing_multi_data(data);
if (err) {
rollback_cnt = link->nodes_cnt;
goto rollback_put;
}
trampoline_lock_all();
for_each_mnode(mnode, link) {
bpf_trampoline_multi_attach_init(mnode->trampoline);
data->entry = &mnode->entry;
err = __bpf_trampoline_link_prog(&mnode->node, mnode->trampoline, NULL,
&trampoline_multi_ops, data);
if (err) {
rollback_cnt = i;
goto rollback_unlink;
}
}
rollback_cnt = link->nodes_cnt;
if (ftrace_hash_count(data->reg)) {
err = update_ftrace_direct_add(&direct_ops, data->reg);
if (err)
goto rollback_unlink;
}
if (ftrace_hash_count(data->modify)) {
err = update_ftrace_direct_mod(&direct_ops, data->modify, true);
if (err) {
if (ftrace_hash_count(data->reg))
WARN_ON_ONCE(update_ftrace_direct_del(&direct_ops, data->reg));
goto rollback_unlink;
}
}
for_each_mnode(mnode, link)
bpf_trampoline_multi_attach_free(mnode->trampoline);
trampoline_unlock_all();
remove_tracing_multi_data(data);
return 0;
rollback_unlink:
for_each_mnode_cnt(mnode, link, rollback_cnt) {
bpf_trampoline_remove_prog(mnode->trampoline, &mnode->node);
bpf_trampoline_multi_attach_rollback(mnode->trampoline);
}
trampoline_unlock_all();
clear_tracing_multi_data(data);
rollback_cnt = link->nodes_cnt;
rollback_put:
for_each_mnode_cnt(mnode, link, rollback_cnt)
bpf_trampoline_put(mnode->trampoline);
return err;
}
int bpf_trampoline_multi_detach(struct bpf_prog *prog, struct bpf_tracing_multi_link *link)
{
struct bpf_tracing_multi_data *data = &link->data;
struct bpf_tracing_multi_node *mnode;
int i;
trampoline_lock_all();
for_each_mnode(mnode, link) {
data->entry = &mnode->entry;
bpf_trampoline_multi_attach_init(mnode->trampoline);
WARN_ON_ONCE(__bpf_trampoline_unlink_prog(&mnode->node, mnode->trampoline,
NULL, &trampoline_multi_ops, data));
}
if (ftrace_hash_count(data->unreg))
WARN_ON_ONCE(update_ftrace_direct_del(&direct_ops, data->unreg));
if (ftrace_hash_count(data->modify))
WARN_ON_ONCE(update_ftrace_direct_mod(&direct_ops, data->modify, true));
for_each_mnode(mnode, link)
bpf_trampoline_multi_attach_free(mnode->trampoline);
trampoline_unlock_all();
for_each_mnode(mnode, link)
bpf_trampoline_put(mnode->trampoline);
clear_tracing_multi_data(data);
return 0;
}
#undef for_each_mnode_cnt
#undef for_each_mnode
#endif /* CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS &&
CONFIG_HAVE_SINGLE_FTRACE_DIRECT_OPS &&
CONFIG_BPF_SYSCALL */
static int __init init_trampolines(void)
{
int i;
@ -1379,6 +1755,8 @@ static int __init init_trampolines(void)
INIT_HLIST_HEAD(&trampoline_key_table[i]);
for (i = 0; i < TRAMPOLINE_TABLE_SIZE; i++)
INIT_HLIST_HEAD(&trampoline_ip_table[i]);
for (i = 0; i < TRAMPOLINE_LOCKS_TABLE_SIZE; i++)
__mutex_init(&trampoline_locks[i].mutex, "trampoline_lock", &trampoline_locks[i].key);
return 0;
}
late_initcall(init_trampolines);

183
kernel/bpf/verifier.c
View File

@ -16382,6 +16382,9 @@ static bool return_retval_range(struct bpf_verifier_env *env, struct bpf_retval_
case BPF_TRACE_FENTRY:
case BPF_TRACE_FEXIT:
case BPF_TRACE_FSESSION:
case BPF_TRACE_FENTRY_MULTI:
case BPF_TRACE_FEXIT_MULTI:
case BPF_TRACE_FSESSION_MULTI:
*range = retval_range(0, 0);
break;
case BPF_TRACE_RAW_TP:
@ -18772,6 +18775,60 @@ static int check_attach_modify_return(unsigned long addr, const char *func_name)
#endif /* CONFIG_FUNCTION_ERROR_INJECTION */
static bool is_tracing_multi_id(const struct bpf_prog *prog, u32 btf_id)
{
return is_tracing_multi(prog->expected_attach_type) && bpf_multi_func_btf_id[0] == btf_id;
}
static int btf_id_allow_sleepable(u32 btf_id, unsigned long addr, const struct bpf_prog *prog,
const struct btf *btf)
{
const struct btf_type *t;
const char *tname;
switch (prog->type) {
case BPF_PROG_TYPE_TRACING:
t = btf_type_by_id(btf, btf_id);
if (!t)
return -EINVAL;
tname = btf_name_by_offset(btf, t->name_off);
if (!tname)
return -EINVAL;
/*
* *.multi sleepable programs will pass initial sleepable check,
* the actual attached btf ids are checked later during the link
* attachment.
*/
if (is_tracing_multi_id(prog, btf_id))
return 0;
if (!check_attach_sleepable(btf_id, addr, tname))
return 0;
/*
* fentry/fexit/fmod_ret progs can also be sleepable if they are
* in the fmodret id set with the KF_SLEEPABLE flag.
*/
else {
u32 *flags = btf_kfunc_is_modify_return(btf, btf_id, prog);
if (flags && (*flags & KF_SLEEPABLE))
return 0;
}
break;
case BPF_PROG_TYPE_LSM:
/*
* LSM progs check that they are attached to bpf_lsm_*() funcs.
* Only some of them are sleepable.
*/
if (bpf_lsm_is_sleepable_hook(btf_id))
return 0;
break;
default:
break;
}
return -EINVAL;
}
int bpf_check_attach_target(struct bpf_verifier_log *log,
const struct bpf_prog *prog,
const struct bpf_prog *tgt_prog,
@ -18894,7 +18951,10 @@ int bpf_check_attach_target(struct bpf_verifier_log *log,
prog_extension &&
(tgt_prog->expected_attach_type == BPF_TRACE_FENTRY ||
tgt_prog->expected_attach_type == BPF_TRACE_FEXIT ||
tgt_prog->expected_attach_type == BPF_TRACE_FSESSION)) {
tgt_prog->expected_attach_type == BPF_TRACE_FENTRY_MULTI ||
tgt_prog->expected_attach_type == BPF_TRACE_FEXIT_MULTI ||
tgt_prog->expected_attach_type == BPF_TRACE_FSESSION ||
tgt_prog->expected_attach_type == BPF_TRACE_FSESSION_MULTI)) {
/* Program extensions can extend all program types
* except fentry/fexit. The reason is the following.
* The fentry/fexit programs are used for performance
@ -19000,7 +19060,11 @@ int bpf_check_attach_target(struct bpf_verifier_log *log,
case BPF_TRACE_FENTRY:
case BPF_TRACE_FEXIT:
case BPF_TRACE_FSESSION:
if (prog->expected_attach_type == BPF_TRACE_FSESSION &&
case BPF_TRACE_FSESSION_MULTI:
case BPF_TRACE_FENTRY_MULTI:
case BPF_TRACE_FEXIT_MULTI:
if ((prog->expected_attach_type == BPF_TRACE_FSESSION ||
prog->expected_attach_type == BPF_TRACE_FSESSION_MULTI) &&
!bpf_jit_supports_fsession()) {
bpf_log(log, "JIT does not support fsession\n");
return -EOPNOTSUPP;
@ -19029,7 +19093,18 @@ int bpf_check_attach_target(struct bpf_verifier_log *log,
if (ret < 0)
return ret;
if (tgt_prog) {
/*
* *.multi programs don't need an address during program
* verification, we just take the module ref if needed.
*/
if (is_tracing_multi_id(prog, btf_id)) {
if (btf_is_module(btf)) {
mod = btf_try_get_module(btf);
if (!mod)
return -ENOENT;
}
addr = 0;
} else if (tgt_prog) {
if (subprog == 0)
addr = (long) tgt_prog->bpf_func;
else
@ -19054,32 +19129,7 @@ int bpf_check_attach_target(struct bpf_verifier_log *log,
}
if (prog->sleepable) {
ret = -EINVAL;
switch (prog->type) {
case BPF_PROG_TYPE_TRACING:
if (!check_attach_sleepable(btf_id, addr, tname))
ret = 0;
/* fentry/fexit/fmod_ret progs can also be sleepable if they are
* in the fmodret id set with the KF_SLEEPABLE flag.
*/
else {
u32 *flags = btf_kfunc_is_modify_return(btf, btf_id,
prog);
if (flags && (*flags & KF_SLEEPABLE))
ret = 0;
}
break;
case BPF_PROG_TYPE_LSM:
/* LSM progs check that they are attached to bpf_lsm_*() funcs.
* Only some of them are sleepable.
*/
if (bpf_lsm_is_sleepable_hook(btf_id))
ret = 0;
break;
default:
break;
}
ret = btf_id_allow_sleepable(btf_id, addr, prog, btf);
if (ret) {
module_put(mod);
bpf_log(log, "%s is not sleepable\n", tname);
@ -19167,6 +19217,9 @@ static bool can_be_sleepable(struct bpf_prog *prog)
case BPF_TRACE_ITER:
case BPF_TRACE_FSESSION:
case BPF_TRACE_RAW_TP:
case BPF_TRACE_FENTRY_MULTI:
case BPF_TRACE_FEXIT_MULTI:
case BPF_TRACE_FSESSION_MULTI:
return true;
default:
return false;
@ -19253,6 +19306,7 @@ static int check_attach_btf_id(struct bpf_verifier_env *env)
return -EINVAL;
} else if ((prog->expected_attach_type == BPF_TRACE_FEXIT ||
prog->expected_attach_type == BPF_TRACE_FSESSION ||
prog->expected_attach_type == BPF_TRACE_FSESSION_MULTI ||
prog->expected_attach_type == BPF_MODIFY_RETURN) &&
btf_id_set_contains(&noreturn_deny, btf_id)) {
verbose(env, "Attaching fexit/fsession/fmod_ret to __noreturn function '%s' is rejected.\n",
@ -19260,6 +19314,14 @@ static int check_attach_btf_id(struct bpf_verifier_env *env)
return -EINVAL;
}
/*
* We don't get trampoline for tracing_multi programs at this point,
* it's done when tracing_multi link is created.
*/
if (prog->type == BPF_PROG_TYPE_TRACING &&
is_tracing_multi(prog->expected_attach_type))
return 0;
key = bpf_trampoline_compute_key(tgt_prog, prog->aux->attach_btf, btf_id);
tr = bpf_trampoline_get(key, &tgt_info);
if (!tr)
@ -19272,6 +19334,62 @@ static int check_attach_btf_id(struct bpf_verifier_env *env)
return 0;
}
int bpf_check_attach_btf_id_multi(struct btf *btf, struct bpf_prog *prog, u32 btf_id,
struct bpf_attach_target_info *tgt_info)
{
const struct btf_type *t;
unsigned long addr;
const char *tname;
int err;
if (!btf_id || !btf)
return -EINVAL;
/* Check noreturn attachment. */
if ((prog->expected_attach_type == BPF_TRACE_FEXIT_MULTI ||
prog->expected_attach_type == BPF_TRACE_FSESSION_MULTI) &&
btf_id_set_contains(&noreturn_deny, btf_id))
return -EINVAL;
/* Check denied attachment. */
if (btf_id_set_contains(&btf_id_deny, btf_id))
return -EINVAL;
/* Check and get function target data. */
t = btf_type_by_id(btf, btf_id);
if (!t)
return -EINVAL;
tname = btf_name_by_offset(btf, t->name_off);
if (!tname)
return -EINVAL;
if (!btf_type_is_func(t))
return -EINVAL;
t = btf_type_by_id(btf, t->type);
if (!btf_type_is_func_proto(t))
return -EINVAL;
err = btf_distill_func_proto(NULL, btf, t, tname, &tgt_info->fmodel);
if (err < 0)
return err;
if (btf_is_module(btf)) {
/* The bpf program already holds reference to module. */
if (WARN_ON_ONCE(!prog->aux->mod))
return -EINVAL;
addr = find_kallsyms_symbol_value(prog->aux->mod, tname);
} else {
addr = kallsyms_lookup_name(tname);
}
if (!addr || !ftrace_location(addr))
return -ENOENT;
/* Check sleepable program attachment. */
if (prog->sleepable) {
err = btf_id_allow_sleepable(btf_id, addr, prog, btf);
if (err)
return err;
}
tgt_info->tgt_addr = addr;
return 0;
}
struct btf *bpf_get_btf_vmlinux(void)
{
if (!btf_vmlinux && IS_ENABLED(CONFIG_DEBUG_INFO_BTF)) {
@ -19512,7 +19630,9 @@ int bpf_fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
insn_buf[0] = BPF_MOV64_REG(BPF_REG_0, BPF_REG_1);
*cnt = 1;
} else if (desc->func_id == special_kfunc_list[KF_bpf_session_is_return] &&
env->prog->expected_attach_type == BPF_TRACE_FSESSION) {
(env->prog->expected_attach_type == BPF_TRACE_FSESSION ||
env->prog->expected_attach_type == BPF_TRACE_FSESSION_MULTI)) {
/*
* inline the bpf_session_is_return() for fsession:
* bool bpf_session_is_return(void *ctx)
@ -19525,7 +19645,8 @@ int bpf_fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
insn_buf[2] = BPF_ALU64_IMM(BPF_AND, BPF_REG_0, 1);
*cnt = 3;
} else if (desc->func_id == special_kfunc_list[KF_bpf_session_cookie] &&
env->prog->expected_attach_type == BPF_TRACE_FSESSION) {
(env->prog->expected_attach_type == BPF_TRACE_FSESSION ||
env->prog->expected_attach_type == BPF_TRACE_FSESSION_MULTI)) {
/*
* inline bpf_session_cookie() for fsession:
* __u64 *bpf_session_cookie(void *ctx)

204
kernel/trace/bpf_trace.c
View File

@ -42,6 +42,7 @@
#define MAX_UPROBE_MULTI_CNT (1U << 20)
#define MAX_KPROBE_MULTI_CNT (1U << 20)
#define MAX_TRACING_MULTI_CNT (1U << 20)
#ifdef CONFIG_MODULES
struct bpf_trace_module {
@ -1333,7 +1334,8 @@ static inline bool is_uprobe_session(const struct bpf_prog *prog)
static inline bool is_trace_fsession(const struct bpf_prog *prog)
{
return prog->type == BPF_PROG_TYPE_TRACING &&
prog->expected_attach_type == BPF_TRACE_FSESSION;
(prog->expected_attach_type == BPF_TRACE_FSESSION ||
prog->expected_attach_type == BPF_TRACE_FSESSION_MULTI);
}
static const struct bpf_func_proto *
@ -3641,3 +3643,203 @@ __bpf_kfunc int bpf_copy_from_user_task_str_dynptr(const struct bpf_dynptr *dptr
}
__bpf_kfunc_end_defs();
#if defined(CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS) && \
defined(CONFIG_HAVE_SINGLE_FTRACE_DIRECT_OPS)
static void bpf_tracing_multi_link_release(struct bpf_link *link)
{
struct bpf_tracing_multi_link *tr_link =
container_of(link, struct bpf_tracing_multi_link, link);
WARN_ON_ONCE(bpf_trampoline_multi_detach(link->prog, tr_link));
}
static void bpf_tracing_multi_link_dealloc(struct bpf_link *link)
{
struct bpf_tracing_multi_link *tr_link =
container_of(link, struct bpf_tracing_multi_link, link);
kvfree(tr_link->fexits);
kvfree(tr_link->cookies);
kvfree(tr_link);
}
#ifdef CONFIG_PROC_FS
static void bpf_tracing_multi_show_fdinfo(const struct bpf_link *link,
struct seq_file *seq)
{
struct bpf_tracing_multi_link *tr_link =
container_of(link, struct bpf_tracing_multi_link, link);
bool has_cookies = !!tr_link->cookies;
seq_printf(seq, "attach_type:\t%u\n", tr_link->link.attach_type);
seq_printf(seq, "cnt:\t%u\n", tr_link->nodes_cnt);
seq_printf(seq, "%s\t %s\t %s\t %s\n", "obj-id", "btf-id", "cookie", "func");
for (int i = 0; i < tr_link->nodes_cnt; i++) {
struct bpf_tracing_multi_node *mnode = &tr_link->nodes[i];
u32 btf_id, obj_id;
bpf_trampoline_unpack_key(mnode->trampoline->key, &obj_id, &btf_id);
seq_printf(seq, "%u\t %u\t %llu\t %pS\n",
obj_id, btf_id,
has_cookies ? tr_link->cookies[i] : 0,
(void *) mnode->trampoline->ip);
cond_resched();
}
}
#endif
static const struct bpf_link_ops bpf_tracing_multi_link_lops = {
.release = bpf_tracing_multi_link_release,
.dealloc_deferred = bpf_tracing_multi_link_dealloc,
#ifdef CONFIG_PROC_FS
.show_fdinfo = bpf_tracing_multi_show_fdinfo,
#endif
};
static int ids_cmp_r(const void *pa, const void *pb, const void *priv __maybe_unused)
{
u32 a = *(u32 *) pa;
u32 b = *(u32 *) pb;
return (a > b) - (a < b);
}
static void ids_swap_r(void *a, void *b, int size __maybe_unused,
const void *priv __maybe_unused)
{
u64 *cookie_a, *cookie_b, *cookies;
u32 *id_a = a, *id_b = b, *ids;
void **data = (void **) priv;
ids = data[0];
cookies = data[1];
if (cookies) {
cookie_a = cookies + (id_a - ids);
cookie_b = cookies + (id_b - ids);
swap(*cookie_a, *cookie_b);
}
swap(*id_a, *id_b);
}
static int check_dup_ids(u32 *ids, u64 *cookies, u32 cnt)
{
void *data[2] = { ids, cookies };
int err = 0;
/*
* Sort ids array (together with cookies array if defined)
* and check it for duplicates. The ids and cookies arrays
* are left sorted.
*/
sort_r_nonatomic(ids, cnt, sizeof(ids[0]), ids_cmp_r, ids_swap_r, data);
for (int i = 1; i < cnt; i++) {
if (ids[i] == ids[i - 1]) {
err = -EINVAL;
break;
}
}
return err;
}
int bpf_tracing_multi_attach(struct bpf_prog *prog, const union bpf_attr *attr)
{
struct bpf_tracing_multi_link *link = NULL;
struct bpf_tramp_node *fexits = NULL;
struct bpf_link_primer link_primer;
u32 cnt, *ids = NULL;
u64 __user *ucookies;
u64 *cookies = NULL;
u32 __user *uids;
int err;
uids = u64_to_user_ptr(attr->link_create.tracing_multi.ids);
cnt = attr->link_create.tracing_multi.cnt;
if (!cnt || !uids)
return -EINVAL;
if (cnt > MAX_TRACING_MULTI_CNT)
return -E2BIG;
if (attr->link_create.flags || attr->link_create.target_fd)
return -EINVAL;
ids = kvmalloc_objs(*ids, cnt);
if (!ids)
return -ENOMEM;
if (copy_from_user(ids, uids, cnt * sizeof(*ids))) {
err = -EFAULT;
goto error;
}
ucookies = u64_to_user_ptr(attr->link_create.tracing_multi.cookies);
if (ucookies) {
cookies = kvmalloc_objs(*cookies, cnt);
if (!cookies) {
err = -ENOMEM;
goto error;
}
if (copy_from_user(cookies, ucookies, cnt * sizeof(*cookies))) {
err = -EFAULT;
goto error;
}
}
err = check_dup_ids(ids, cookies, cnt);
if (err)
goto error;
if (prog->expected_attach_type == BPF_TRACE_FSESSION_MULTI) {
fexits = kvmalloc_objs(*fexits, cnt);
if (!fexits) {
err = -ENOMEM;
goto error;
}
}
link = kvzalloc_flex(*link, nodes, cnt);
if (!link) {
err = -ENOMEM;
goto error;
}
bpf_link_init(&link->link, BPF_LINK_TYPE_TRACING_MULTI,
&bpf_tracing_multi_link_lops, prog, prog->expected_attach_type);
err = bpf_link_prime(&link->link, &link_primer);
if (err)
goto error;
link->nodes_cnt = cnt;
link->cookies = cookies;
link->fexits = fexits;
err = bpf_trampoline_multi_attach(prog, ids, link);
kvfree(ids);
if (err) {
bpf_link_cleanup(&link_primer);
return err;
}
return bpf_link_settle(&link_primer);
error:
kvfree(fexits);
kvfree(cookies);
kvfree(ids);
kvfree(link);
return err;
}
#else
int bpf_tracing_multi_attach(struct bpf_prog *prog, const union bpf_attr *attr)
{
return -EOPNOTSUPP;
}
#endif /* CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS && CONFIG_HAVE_SINGLE_FTRACE_DIRECT_OPS */

35
kernel/trace/ftrace.c
View File

@ -1198,8 +1198,7 @@ ftrace_lookup_ip(struct ftrace_hash *hash, unsigned long ip)
return __ftrace_lookup_ip(hash, ip);
}
static void __add_hash_entry(struct ftrace_hash *hash,
struct ftrace_func_entry *entry)
void add_ftrace_hash_entry(struct ftrace_hash *hash, struct ftrace_func_entry *entry)
{
struct hlist_head *hhd;
unsigned long key;
@ -1221,7 +1220,7 @@ add_ftrace_hash_entry_direct(struct ftrace_hash *hash, unsigned long ip, unsigne
entry->ip = ip;
entry->direct = direct;
__add_hash_entry(hash, entry);
add_ftrace_hash_entry(hash, entry);
return entry;
}
@ -1249,6 +1248,25 @@ remove_hash_entry(struct ftrace_hash *hash,
hash->count--;
}
void ftrace_hash_remove(struct ftrace_hash *hash)
{
struct ftrace_func_entry *entry;
struct hlist_head *hhd;
struct hlist_node *tn;
int size;
int i;
if (!hash || !hash->count)
return;
size = 1 << hash->size_bits;
for (i = 0; i < size; i++) {
hhd = &hash->buckets[i];
hlist_for_each_entry_safe(entry, tn, hhd, hlist)
remove_hash_entry(hash, entry);
}
FTRACE_WARN_ON(hash->count);
}
static void ftrace_hash_clear(struct ftrace_hash *hash)
{
struct hlist_head *hhd;
@ -1458,7 +1476,7 @@ static struct ftrace_hash *__move_hash(struct ftrace_hash *src, int size)
hhd = &src->buckets[i];
hlist_for_each_entry_safe(entry, tn, hhd, hlist) {
remove_hash_entry(src, entry);
__add_hash_entry(new_hash, entry);
add_ftrace_hash_entry(new_hash, entry);
}
}
return new_hash;
@ -5341,7 +5359,7 @@ int ftrace_func_mapper_add_ip(struct ftrace_func_mapper *mapper,
map->entry.ip = ip;
map->data = data;
__add_hash_entry(&mapper->hash, &map->entry);
add_ftrace_hash_entry(&mapper->hash, &map->entry);
return 0;
}
@ -6288,11 +6306,16 @@ int modify_ftrace_direct(struct ftrace_ops *ops, unsigned long addr)
}
EXPORT_SYMBOL_GPL(modify_ftrace_direct);
static unsigned long hash_count(struct ftrace_hash *hash)
static inline unsigned long hash_count(struct ftrace_hash *hash)
{
return hash ? hash->count : 0;
}
unsigned long ftrace_hash_count(struct ftrace_hash *hash)
{
return hash_count(hash);
}
/**
* hash_add - adds two struct ftrace_hash and returns the result
* @a: struct ftrace_hash object

14
net/bpf/bpf_dummy_struct_ops.c
View File

@ -132,7 +132,7 @@ int bpf_struct_ops_test_run(struct bpf_prog *prog, const union bpf_attr *kattr,
const struct bpf_struct_ops *st_ops = &bpf_bpf_dummy_ops;
const struct btf_type *func_proto;
struct bpf_dummy_ops_test_args *args;
struct bpf_tramp_links *tlinks = NULL;
struct bpf_tramp_nodes *tnodes = NULL;
struct bpf_tramp_link *link = NULL;
void *image = NULL;
unsigned int op_idx;
@ -158,8 +158,8 @@ int bpf_struct_ops_test_run(struct bpf_prog *prog, const union bpf_attr *kattr,
if (err)
goto out;
tlinks = kzalloc_objs(*tlinks, BPF_TRAMP_MAX);
if (!tlinks) {
tnodes = kzalloc_objs(*tnodes, BPF_TRAMP_MAX);
if (!tnodes) {
err = -ENOMEM;
goto out;
}
@ -171,11 +171,11 @@ int bpf_struct_ops_test_run(struct bpf_prog *prog, const union bpf_attr *kattr,
}
/* prog doesn't take the ownership of the reference from caller */
bpf_prog_inc(prog);
bpf_link_init(&link->link, BPF_LINK_TYPE_STRUCT_OPS, &bpf_struct_ops_link_lops, prog,
prog->expected_attach_type);
bpf_tramp_link_init(link, BPF_LINK_TYPE_STRUCT_OPS, &bpf_struct_ops_link_lops,
prog, prog->expected_attach_type, 0);
op_idx = prog->expected_attach_type;
err = bpf_struct_ops_prepare_trampoline(tlinks, link,
err = bpf_struct_ops_prepare_trampoline(tnodes, &link->node,
&st_ops->func_models[op_idx],
&dummy_ops_test_ret_function,
&image, &image_off,
@ -198,7 +198,7 @@ int bpf_struct_ops_test_run(struct bpf_prog *prog, const union bpf_attr *kattr,
bpf_struct_ops_image_free(image);
if (link)
bpf_link_put(&link->link);
kfree(tlinks);
kfree(tnodes);
return err;
}

3
net/bpf/test_run.c
View File

@ -703,6 +703,9 @@ int bpf_prog_test_run_tracing(struct bpf_prog *prog,
case BPF_TRACE_FENTRY:
case BPF_TRACE_FEXIT:
case BPF_TRACE_FSESSION:
case BPF_TRACE_FENTRY_MULTI:
case BPF_TRACE_FEXIT_MULTI:
case BPF_TRACE_FSESSION_MULTI:
if (bpf_fentry_test1(1) != 2 ||
bpf_fentry_test2(2, 3) != 5 ||
bpf_fentry_test3(4, 5, 6) != 15 ||

10
tools/include/uapi/linux/bpf.h
View File

@ -1156,6 +1156,9 @@ enum bpf_attach_type {
BPF_TRACE_KPROBE_SESSION,
BPF_TRACE_UPROBE_SESSION,
BPF_TRACE_FSESSION,
BPF_TRACE_FENTRY_MULTI,
BPF_TRACE_FEXIT_MULTI,
BPF_TRACE_FSESSION_MULTI,
__MAX_BPF_ATTACH_TYPE
};
@ -1180,6 +1183,7 @@ enum bpf_link_type {
BPF_LINK_TYPE_UPROBE_MULTI = 12,
BPF_LINK_TYPE_NETKIT = 13,
BPF_LINK_TYPE_SOCKMAP = 14,
BPF_LINK_TYPE_TRACING_MULTI = 15,
__MAX_BPF_LINK_TYPE,
};
@ -1875,6 +1879,11 @@ union bpf_attr {
};
__u64 expected_revision;
} cgroup;
struct {
__aligned_u64 ids;
__aligned_u64 cookies;
__u32 cnt;
} tracing_multi;
};
} link_create;
@ -7252,6 +7261,7 @@ enum {
TCP_BPF_SOCK_OPS_CB_FLAGS = 1008, /* Get or Set TCP sock ops flags */
SK_BPF_CB_FLAGS = 1009, /* Get or set sock ops flags in socket */
SK_BPF_BYPASS_PROT_MEM = 1010, /* Get or Set sk->sk_bypass_prot_mem */
};
enum {

9
tools/lib/bpf/bpf.c
View File

@ -845,6 +845,15 @@ int bpf_link_create(int prog_fd, int target_fd,
if (!OPTS_ZEROED(opts, uprobe_multi))
return libbpf_err(-EINVAL);
break;
case BPF_TRACE_FENTRY_MULTI:
case BPF_TRACE_FEXIT_MULTI:
case BPF_TRACE_FSESSION_MULTI:
attr.link_create.tracing_multi.ids = ptr_to_u64(OPTS_GET(opts, tracing_multi.ids, 0));
attr.link_create.tracing_multi.cookies = ptr_to_u64(OPTS_GET(opts, tracing_multi.cookies, 0));
attr.link_create.tracing_multi.cnt = OPTS_GET(opts, tracing_multi.cnt, 0);
if (!OPTS_ZEROED(opts, tracing_multi))
return libbpf_err(-EINVAL);
break;
case BPF_TRACE_RAW_TP:
case BPF_TRACE_FENTRY:
case BPF_TRACE_FEXIT:

5
tools/lib/bpf/bpf.h
View File

@ -469,6 +469,11 @@ struct bpf_link_create_opts {
__u32 relative_id;
__u64 expected_revision;
} cgroup;
struct {
const __u32 *ids;
const __u64 *cookies;
__u32 cnt;
} tracing_multi;
};
size_t :0;
};

375
tools/lib/bpf/libbpf.c
View File

@ -136,6 +136,9 @@ static const char * const attach_type_name[] = {
[BPF_NETKIT_PEER] = "netkit_peer",
[BPF_TRACE_KPROBE_SESSION] = "trace_kprobe_session",
[BPF_TRACE_UPROBE_SESSION] = "trace_uprobe_session",
[BPF_TRACE_FENTRY_MULTI] = "trace_fentry_multi",
[BPF_TRACE_FEXIT_MULTI] = "trace_fexit_multi",
[BPF_TRACE_FSESSION_MULTI] = "trace_fsession_multi",
};
static const char * const link_type_name[] = {
@ -154,6 +157,7 @@ static const char * const link_type_name[] = {
[BPF_LINK_TYPE_UPROBE_MULTI] = "uprobe_multi",
[BPF_LINK_TYPE_NETKIT] = "netkit",
[BPF_LINK_TYPE_SOCKMAP] = "sockmap",
[BPF_LINK_TYPE_TRACING_MULTI] = "tracing_multi",
};
static const char * const map_type_name[] = {
@ -7768,6 +7772,69 @@ static int bpf_object__sanitize_prog(struct bpf_object *obj, struct bpf_program
static int libbpf_find_attach_btf_id(struct bpf_program *prog, const char *attach_name,
int *btf_obj_fd, int *btf_type_id);
static inline bool is_tracing_multi(enum bpf_attach_type type)
{
return type == BPF_TRACE_FENTRY_MULTI || type == BPF_TRACE_FEXIT_MULTI ||
type == BPF_TRACE_FSESSION_MULTI;
}
static const struct module_btf *find_attach_module(struct bpf_object *obj, const char *attach)
{
const char *sep, *mod_name = NULL;
int i, mod_len, err;
/*
* We expect attach string in the form of either
* - function_pattern or
* - <module>:function_pattern
*/
sep = strchr(attach, ':');
if (sep) {
mod_name = attach;
mod_len = sep - mod_name;
}
if (!mod_name)
return NULL;
err = load_module_btfs(obj);
if (err)
return NULL;
for (i = 0; i < obj->btf_module_cnt; i++) {
const struct module_btf *mod = &obj->btf_modules[i];
if (strncmp(mod->name, mod_name, mod_len) == 0 && mod->name[mod_len] == '\0')
return mod;
}
return NULL;
}
static int tracing_multi_mod_fd(struct bpf_program *prog, int *btf_obj_fd)
{
const char *attach_name, *sep;
const struct module_btf *mod;
*btf_obj_fd = 0;
attach_name = strchr(prog->sec_name, '/');
/* Program with no details in spec, using kernel btf. */
if (!attach_name)
return 0;
/* Program with no module section, using kernel btf. */
sep = strchr(++attach_name, ':');
if (!sep)
return 0;
/* Program with module specified, get its btf fd. */
mod = find_attach_module(prog->obj, attach_name);
if (!mod)
return -EINVAL;
*btf_obj_fd = mod->fd;
return 0;
}
/* this is called as prog->sec_def->prog_prepare_load_fn for libbpf-supported sec_defs */
static int libbpf_prepare_prog_load(struct bpf_program *prog,
struct bpf_prog_load_opts *opts, long cookie)
@ -7831,6 +7898,18 @@ static int libbpf_prepare_prog_load(struct bpf_program *prog,
opts->attach_btf_obj_fd = btf_obj_fd;
opts->attach_btf_id = btf_type_id;
}
if (is_tracing_multi(prog->expected_attach_type)) {
int err, btf_obj_fd = 0;
err = tracing_multi_mod_fd(prog, &btf_obj_fd);
if (err < 0)
return err;
prog->attach_btf_obj_fd = btf_obj_fd;
opts->attach_btf_obj_fd = btf_obj_fd;
}
return 0;
}
@ -8937,13 +9016,10 @@ static void bpf_object_unpin(struct bpf_object *obj)
bpf_map__unpin(&obj->maps[i], NULL);
}
static void bpf_object_post_load_cleanup(struct bpf_object *obj)
static void bpf_object_cleanup_btf(struct bpf_object *obj)
{
int i;
/* clean up fd_array */
zfree(&obj->fd_array);
/* clean up module BTFs */
for (i = 0; i < obj->btf_module_cnt; i++) {
close(obj->btf_modules[i].fd);
@ -8951,6 +9027,8 @@ static void bpf_object_post_load_cleanup(struct bpf_object *obj)
free(obj->btf_modules[i].name);
}
obj->btf_module_cnt = 0;
obj->btf_module_cap = 0;
obj->btf_modules_loaded = false;
zfree(&obj->btf_modules);
/* clean up vmlinux BTF */
@ -8958,6 +9036,15 @@ static void bpf_object_post_load_cleanup(struct bpf_object *obj)
obj->btf_vmlinux = NULL;
}
static void bpf_object_post_load_cleanup(struct bpf_object *obj)
{
/* clean up fd_array */
zfree(&obj->fd_array);
/* clean up BTF */
bpf_object_cleanup_btf(obj);
}
static int bpf_object_prepare(struct bpf_object *obj, const char *target_btf_path)
{
int err;
@ -9984,6 +10071,7 @@ static int attach_kprobe_session(const struct bpf_program *prog, long cookie, st
static int attach_uprobe_multi(const struct bpf_program *prog, long cookie, struct bpf_link **link);
static int attach_lsm(const struct bpf_program *prog, long cookie, struct bpf_link **link);
static int attach_iter(const struct bpf_program *prog, long cookie, struct bpf_link **link);
static int attach_tracing_multi(const struct bpf_program *prog, long cookie, struct bpf_link **link);
static const struct bpf_sec_def section_defs[] = {
SEC_DEF("socket", SOCKET_FILTER, 0, SEC_NONE),
@ -10037,6 +10125,12 @@ static const struct bpf_sec_def section_defs[] = {
SEC_DEF("fexit.s+", TRACING, BPF_TRACE_FEXIT, SEC_ATTACH_BTF | SEC_SLEEPABLE, attach_trace),
SEC_DEF("fsession+", TRACING, BPF_TRACE_FSESSION, SEC_ATTACH_BTF, attach_trace),
SEC_DEF("fsession.s+", TRACING, BPF_TRACE_FSESSION, SEC_ATTACH_BTF | SEC_SLEEPABLE, attach_trace),
SEC_DEF("fsession.multi+", TRACING, BPF_TRACE_FSESSION_MULTI, 0, attach_tracing_multi),
SEC_DEF("fsession.multi.s+", TRACING, BPF_TRACE_FSESSION_MULTI, SEC_SLEEPABLE, attach_tracing_multi),
SEC_DEF("fentry.multi+", TRACING, BPF_TRACE_FENTRY_MULTI, 0, attach_tracing_multi),
SEC_DEF("fexit.multi+", TRACING, BPF_TRACE_FEXIT_MULTI, 0, attach_tracing_multi),
SEC_DEF("fentry.multi.s+", TRACING, BPF_TRACE_FENTRY_MULTI, SEC_SLEEPABLE, attach_tracing_multi),
SEC_DEF("fexit.multi.s+", TRACING, BPF_TRACE_FEXIT_MULTI, SEC_SLEEPABLE, attach_tracing_multi),
SEC_DEF("freplace+", EXT, 0, SEC_ATTACH_BTF, attach_trace),
SEC_DEF("lsm+", LSM, BPF_LSM_MAC, SEC_ATTACH_BTF, attach_lsm),
SEC_DEF("lsm.s+", LSM, BPF_LSM_MAC, SEC_ATTACH_BTF | SEC_SLEEPABLE, attach_lsm),
@ -12438,6 +12532,279 @@ static int attach_uprobe_multi(const struct bpf_program *prog, long cookie, stru
return ret;
}
#define MAX_BPF_FUNC_ARGS 12
static bool btf_type_is_modifier(const struct btf_type *t)
{
switch (BTF_INFO_KIND(t->info)) {
case BTF_KIND_TYPEDEF:
case BTF_KIND_VOLATILE:
case BTF_KIND_CONST:
case BTF_KIND_RESTRICT:
case BTF_KIND_TYPE_TAG:
return true;
default:
return false;
}
}
#define MAX_RESOLVE_DEPTH 32
static int btf_get_type_size(const struct btf *btf, __u32 type_id,
const struct btf_type **ret_type)
{
const struct btf_type *t;
int i;
*ret_type = btf__type_by_id(btf, 0);
if (!type_id)
return 0;
t = btf__type_by_id(btf, type_id);
for (i = 0; i < MAX_RESOLVE_DEPTH && t && btf_type_is_modifier(t); i++)
t = btf__type_by_id(btf, t->type);
if (!t || i == MAX_RESOLVE_DEPTH)
return -EINVAL;
*ret_type = t;
if (btf_is_ptr(t))
return btf__pointer_size(btf);
if (btf_is_int(t) || btf_is_any_enum(t) || btf_is_struct(t) || btf_is_union(t))
return t->size;
return -EINVAL;
}
bool btf_type_is_traceable_func(const struct btf *btf, const struct btf_type *t)
{
const struct btf_param *args;
const struct btf_type *proto;
__u32 i, nargs;
int ret;
if (!btf_is_func(t))
return false;
proto = btf__type_by_id(btf, t->type);
if (!proto || !btf_is_func_proto(proto))
return false;
args = (const struct btf_param *)(proto + 1);
nargs = btf_vlen(proto);
if (nargs > MAX_BPF_FUNC_ARGS)
return false;
/* No support for struct return type. */
ret = btf_get_type_size(btf, proto->type, &t);
if (ret < 0 || btf_is_struct(t) || btf_is_union(t))
return false;
for (i = 0; i < nargs; i++) {
/* No support for variable args. */
if (i == nargs - 1 && args[i].type == 0)
return false;
ret = btf_get_type_size(btf, args[i].type, &t);
/* No support of struct argument size greater than 16 bytes. */
if (ret < 0 || ret > 16)
return false;
/* No support for void argument. */
if (ret == 0)
return false;
}
return true;
}
static int
collect_btf_func_ids_by_glob(const struct btf *btf, const char *pattern, __u32 **ids)
{
__u32 type_id, nr_types = btf__type_cnt(btf);
size_t cap = 0, cnt = 0;
if (!pattern)
return -EINVAL;
for (type_id = 1; type_id < nr_types; type_id++) {
const struct btf_type *t = btf__type_by_id(btf, type_id);
const char *name;
int err;
if (btf_kind(t) != BTF_KIND_FUNC)
continue;
name = btf__name_by_offset(btf, t->name_off);
if (!name)
continue;
if (!glob_match(name, pattern))
continue;
if (!btf_type_is_traceable_func(btf, t))
continue;
err = libbpf_ensure_mem((void **) ids, &cap, sizeof(**ids), cnt + 1);
if (err) {
free(*ids);
return -ENOMEM;
}
(*ids)[cnt++] = type_id;
}
return cnt;
}
static int collect_func_ids_by_glob(const struct bpf_program *prog, const char *pattern, __u32 **ids)
{
struct bpf_object *obj = prog->obj;
const struct module_btf *mod;
struct btf *btf = NULL;
const char *sep;
int err;
err = bpf_object__load_vmlinux_btf(obj, true);
if (err)
return err;
/* In case we have module specified, we will find its btf and use that. */
sep = strchr(pattern, ':');
if (sep) {
mod = find_attach_module(obj, pattern);
if (!mod) {
err = -EINVAL;
goto cleanup;
}
btf = mod->btf;
pattern = sep + 1;
} else {
/* Program is loaded for kernel module. */
if (prog->attach_btf_obj_fd) {
err = -EINVAL;
goto cleanup;
}
btf = obj->btf_vmlinux;
}
err = collect_btf_func_ids_by_glob(btf, pattern, ids);
cleanup:
bpf_object_cleanup_btf(obj);
return err;
}
struct bpf_link *
bpf_program__attach_tracing_multi(const struct bpf_program *prog, const char *pattern,
const struct bpf_tracing_multi_opts *opts)
{
LIBBPF_OPTS(bpf_link_create_opts, lopts);
int prog_fd, link_fd, err, cnt;
__u32 *free_ids = NULL;
struct bpf_link *link;
const __u64 *cookies;
const __u32 *ids;
if (!OPTS_VALID(opts, bpf_tracing_multi_opts))
return libbpf_err_ptr(-EINVAL);
prog_fd = bpf_program__fd(prog);
if (prog_fd < 0) {
pr_warn("prog '%s': can't attach BPF program without FD (was it loaded?)\n",
prog->name);
return libbpf_err_ptr(-EINVAL);
}
cnt = OPTS_GET(opts, cnt, 0);
ids = OPTS_GET(opts, ids, NULL);
cookies = OPTS_GET(opts, cookies, NULL);
if (!!ids != !!cnt)
return libbpf_err_ptr(-EINVAL);
if (pattern && (ids || cookies))
return libbpf_err_ptr(-EINVAL);
if (!pattern && !ids)
return libbpf_err_ptr(-EINVAL);
if (pattern) {
cnt = collect_func_ids_by_glob(prog, pattern, &free_ids);
if (cnt < 0)
return libbpf_err_ptr(cnt);
if (cnt == 0)
return libbpf_err_ptr(-EINVAL);
ids = (const __u32 *) free_ids;
}
lopts.tracing_multi.ids = ids;
lopts.tracing_multi.cookies = cookies;
lopts.tracing_multi.cnt = cnt;
link = calloc(1, sizeof(*link));
if (!link) {
err = -ENOMEM;
goto error;
}
link->detach = &bpf_link__detach_fd;
link_fd = bpf_link_create(prog_fd, 0, prog->expected_attach_type, &lopts);
if (link_fd < 0) {
err = -errno;
pr_warn("prog '%s': failed to attach: %s\n", prog->name, errstr(err));
goto error;
}
link->fd = link_fd;
free(free_ids);
return link;
error:
free(link);
free(free_ids);
return libbpf_err_ptr(err);
}
static int attach_tracing_multi(const struct bpf_program *prog, long cookie, struct bpf_link **link)
{
static const char *const prefixes[] = {
"fentry.multi",
"fexit.multi",
"fsession.multi",
"fentry.multi.s",
"fexit.multi.s",
"fsession.multi.s",
};
const char *spec = NULL;
char *pattern;
size_t i;
int n;
*link = NULL;
for (i = 0; i < ARRAY_SIZE(prefixes); i++) {
size_t pfx_len;
if (!str_has_pfx(prog->sec_name, prefixes[i]))
continue;
pfx_len = strlen(prefixes[i]);
/* no auto-attach case of, e.g., SEC("fentry.multi") */
if (prog->sec_name[pfx_len] == '\0')
return 0;
if (prog->sec_name[pfx_len] != '/')
continue;
spec = prog->sec_name + pfx_len + 1;
break;
}
if (!spec) {
pr_warn("prog '%s': invalid section name '%s'\n",
prog->name, prog->sec_name);
return -EINVAL;
}
n = sscanf(spec, "%m[a-zA-Z0-9_.*?:]", &pattern);
if (n < 1) {
pr_warn("tracing multi pattern is invalid: %s\n", spec);
return -EINVAL;
}
*link = bpf_program__attach_tracing_multi(prog, pattern, NULL);
free(pattern);
return libbpf_get_error(*link);
}
static inline int add_uprobe_event_legacy(const char *probe_name, bool retprobe,
const char *binary_path, size_t offset)
{

15
tools/lib/bpf/libbpf.h
View File

@ -726,6 +726,21 @@ bpf_program__attach_ksyscall(const struct bpf_program *prog,
const char *syscall_name,
const struct bpf_ksyscall_opts *opts);
struct bpf_tracing_multi_opts {
/* size of this struct, for forward/backward compatibility */
size_t sz;
const __u32 *ids;
const __u64 *cookies;
size_t cnt;
size_t :0;
};
#define bpf_tracing_multi_opts__last_field cnt
LIBBPF_API struct bpf_link *
bpf_program__attach_tracing_multi(const struct bpf_program *prog, const char *pattern,
const struct bpf_tracing_multi_opts *opts);
struct bpf_uprobe_opts {
/* size of this struct, for forward/backward compatibility */
size_t sz;

1
tools/lib/bpf/libbpf.map
View File

@ -458,6 +458,7 @@ LIBBPF_1.7.0 {
LIBBPF_1.8.0 {
global:
bpf_program__attach_tracing_multi;
bpf_program__clone;
btf__new_empty_opts;
} LIBBPF_1.7.0;

1
tools/lib/bpf/libbpf_internal.h
View File

@ -250,6 +250,7 @@ const struct btf_type *skip_mods_and_typedefs(const struct btf *btf, __u32 id, _
const struct btf_header *btf_header(const struct btf *btf);
void btf_set_base_btf(struct btf *btf, const struct btf *base_btf);
int btf_relocate(struct btf *btf, const struct btf *base_btf, __u32 **id_map);
bool btf_type_is_traceable_func(const struct btf *btf, const struct btf_type *t);
static inline enum btf_func_linkage btf_func_linkage(const struct btf_type *t)
{

9
tools/testing/selftests/bpf/Makefile
View File

@ -512,7 +512,10 @@ SKEL_BLACKLIST := btf__% test_pinning_invalid.c test_sk_assign.c
LINKED_SKELS := test_static_linked.skel.h linked_funcs.skel.h \
linked_vars.skel.h linked_maps.skel.h \
test_subskeleton.skel.h test_subskeleton_lib.skel.h \
test_usdt.skel.h
test_usdt.skel.h tracing_multi.skel.h \
tracing_multi_module.skel.h \
tracing_multi_intersect.skel.h \
tracing_multi_session.skel.h
LSKELS := fexit_sleep.c trace_printk.c trace_vprintk.c map_ptr_kern.c \
core_kern.c core_kern_overflow.c test_ringbuf.c \
@ -538,6 +541,10 @@ test_usdt.skel.h-deps := test_usdt.bpf.o test_usdt_multispec.bpf.o
xsk_xdp_progs.skel.h-deps := xsk_xdp_progs.bpf.o
xdp_hw_metadata.skel.h-deps := xdp_hw_metadata.bpf.o
xdp_features.skel.h-deps := xdp_features.bpf.o
tracing_multi.skel.h-deps := tracing_multi_attach.bpf.o tracing_multi_check.bpf.o
tracing_multi_module.skel.h-deps := tracing_multi_attach_module.bpf.o tracing_multi_check.bpf.o
tracing_multi_intersect.skel.h-deps := tracing_multi_intersect_attach.bpf.o tracing_multi_check.bpf.o
tracing_multi_session.skel.h-deps := tracing_multi_session_attach.bpf.o tracing_multi_check.bpf.o
LINKED_BPF_OBJS := $(foreach skel,$(LINKED_SKELS),$($(skel)-deps))
LINKED_BPF_SRCS := $(patsubst %.bpf.o,%.c,$(LINKED_BPF_OBJS))

960
tools/testing/selftests/bpf/prog_tests/tracing_multi.c Normal file
View File

@ -0,0 +1,960 @@
// SPDX-License-Identifier: GPL-2.0
#include <test_progs.h>
#include <bpf/btf.h>
#include <search.h>
#include "bpf/libbpf_internal.h"
#include "tracing_multi.skel.h"
#include "tracing_multi_module.skel.h"
#include "tracing_multi_intersect.skel.h"
#include "tracing_multi_session.skel.h"
#include "tracing_multi_fail.skel.h"
#include "tracing_multi_verifier.skel.h"
#include "tracing_multi_bench.skel.h"
#include "tracing_multi_rollback.skel.h"
#include "trace_helpers.h"
static __u64 bpf_fentry_test_cookies[] = {
8, /* bpf_fentry_test1 */
9, /* bpf_fentry_test2 */
7, /* bpf_fentry_test3 */
5, /* bpf_fentry_test4 */
4, /* bpf_fentry_test5 */
2, /* bpf_fentry_test6 */
3, /* bpf_fentry_test7 */
1, /* bpf_fentry_test8 */
10, /* bpf_fentry_test9 */
6, /* bpf_fentry_test10 */
};
static const char * const bpf_fentry_test[] = {
"bpf_fentry_test1",
"bpf_fentry_test2",
"bpf_fentry_test3",
"bpf_fentry_test4",
"bpf_fentry_test5",
"bpf_fentry_test6",
"bpf_fentry_test7",
"bpf_fentry_test8",
"bpf_fentry_test9",
"bpf_fentry_test10",
};
static const char * const bpf_testmod_fentry_test[] = {
"bpf_testmod_fentry_test1",
"bpf_testmod_fentry_test2",
"bpf_testmod_fentry_test3",
"bpf_testmod_fentry_test7",
"bpf_testmod_fentry_test11",
};
#define FUNCS_CNT (ARRAY_SIZE(bpf_fentry_test))
static int get_random_funcs(const char **funcs)
{
int i, cnt = 0;
for (i = 0; i < FUNCS_CNT; i++) {
if (rand() % 2)
funcs[cnt++] = bpf_fentry_test[i];
}
/* we always need at least one.. */
if (!cnt)
funcs[cnt++] = bpf_fentry_test[rand() % FUNCS_CNT];
return cnt;
}
static int compare(const void *ppa, const void *ppb)
{
const char *pa = *(const char **) ppa;
const char *pb = *(const char **) ppb;
return strcmp(pa, pb);
}
static void tdestroy_free_nop(void *ptr)
{
}
static __u32 *get_ids(const char * const funcs[], int funcs_cnt, const char *mod)
{
struct btf *btf, *vmlinux_btf = NULL;
__u32 nr, type_id, cnt = 0;
void *root = NULL;
__u32 *ids = NULL;
int i, err = 0;
btf = btf__load_vmlinux_btf();
if (!ASSERT_OK_PTR(btf, "btf__load_vmlinux_btf"))
return NULL;
if (mod) {
vmlinux_btf = btf;
btf = btf__load_module_btf(mod, vmlinux_btf);
if (!ASSERT_OK_PTR(btf, "btf__load_module_btf")) {
btf__free(vmlinux_btf);
return NULL;
}
}
ids = calloc(funcs_cnt, sizeof(ids[0]));
if (!ids)
goto out;
/*
* We sort function names by name and search them
* below for each function.
*/
for (i = 0; i < funcs_cnt; i++) {
if (!tsearch(&funcs[i], &root, compare)) {
ASSERT_FAIL("tsearch failed");
err = -1;
goto error;
}
}
nr = btf__type_cnt(btf);
for (type_id = 1; type_id < nr && cnt < funcs_cnt; type_id++) {
const struct btf_type *type;
const char *str, ***val;
unsigned int idx;
type = btf__type_by_id(btf, type_id);
if (!type) {
err = -1;
break;
}
if (BTF_INFO_KIND(type->info) != BTF_KIND_FUNC)
continue;
str = btf__name_by_offset(btf, type->name_off);
if (!str) {
err = -1;
break;
}
val = tfind(&str, &root, compare);
if (!val)
continue;
/*
* We keep pointer for each function name so we can get the original
* array index and have the resulting ids array matching the original
* function array.
*
* Doing it this way allow us to easily test the cookies support,
* because each cookie is attached to particular function/id.
*/
idx = *val - funcs;
ids[idx] = type_id;
cnt++;
}
error:
if (err) {
free(ids);
ids = NULL;
}
out:
tdestroy(root, tdestroy_free_nop);
btf__free(vmlinux_btf);
btf__free(btf);
return ids;
}
static void tracing_multi_test_run(struct tracing_multi *skel)
{
LIBBPF_OPTS(bpf_test_run_opts, topts);
int err, prog_fd;
prog_fd = bpf_program__fd(skel->progs.test_fentry);
err = bpf_prog_test_run_opts(prog_fd, &topts);
ASSERT_OK(err, "test_run");
/* extra +1 count for sleepable programs */
ASSERT_EQ(skel->bss->test_result_fentry, FUNCS_CNT + 1, "test_result_fentry");
ASSERT_EQ(skel->bss->test_result_fexit, FUNCS_CNT + 1, "test_result_fexit");
}
static void test_skel_api(void)
{
struct tracing_multi *skel;
int err;
skel = tracing_multi__open_and_load();
if (!ASSERT_OK_PTR(skel, "tracing_multi__open_and_load"))
return;
skel->bss->pid = getpid();
err = tracing_multi__attach(skel);
if (!ASSERT_OK(err, "tracing_multi__attach"))
goto cleanup;
tracing_multi_test_run(skel);
cleanup:
tracing_multi__destroy(skel);
}
static void test_link_api_pattern(void)
{
struct tracing_multi *skel;
skel = tracing_multi__open_and_load();
if (!ASSERT_OK_PTR(skel, "tracing_multi__open_and_load"))
return;
skel->bss->pid = getpid();
skel->links.test_fentry = bpf_program__attach_tracing_multi(skel->progs.test_fentry,
"bpf_fentry_test*", NULL);
if (!ASSERT_OK_PTR(skel->links.test_fentry, "bpf_program__attach_tracing_multi"))
goto cleanup;
skel->links.test_fexit = bpf_program__attach_tracing_multi(skel->progs.test_fexit,
"bpf_fentry_test*", NULL);
if (!ASSERT_OK_PTR(skel->links.test_fexit, "bpf_program__attach_tracing_multi"))
goto cleanup;
skel->links.test_fentry_s = bpf_program__attach_tracing_multi(skel->progs.test_fentry_s,
"bpf_fentry_test1", NULL);
if (!ASSERT_OK_PTR(skel->links.test_fentry_s, "bpf_program__attach_tracing_multi"))
goto cleanup;
skel->links.test_fexit_s = bpf_program__attach_tracing_multi(skel->progs.test_fexit_s,
"bpf_fentry_test1", NULL);
if (!ASSERT_OK_PTR(skel->links.test_fexit_s, "bpf_program__attach_tracing_multi"))
goto cleanup;
tracing_multi_test_run(skel);
cleanup:
tracing_multi__destroy(skel);
}
static void test_link_api_ids(bool test_cookies)
{
LIBBPF_OPTS(bpf_tracing_multi_opts, opts);
struct tracing_multi *skel;
size_t cnt = FUNCS_CNT;
__u32 *ids;
skel = tracing_multi__open_and_load();
if (!ASSERT_OK_PTR(skel, "tracing_multi__open_and_load"))
return;
skel->bss->pid = getpid();
skel->bss->test_cookies = test_cookies;
ids = get_ids(bpf_fentry_test, cnt, NULL);
if (!ASSERT_OK_PTR(ids, "get_ids"))
goto cleanup;
opts.ids = ids;
opts.cnt = cnt;
if (test_cookies)
opts.cookies = bpf_fentry_test_cookies;
skel->links.test_fentry = bpf_program__attach_tracing_multi(skel->progs.test_fentry,
NULL, &opts);
if (!ASSERT_OK_PTR(skel->links.test_fentry, "bpf_program__attach_tracing_multi"))
goto cleanup;
skel->links.test_fexit = bpf_program__attach_tracing_multi(skel->progs.test_fexit,
NULL, &opts);
if (!ASSERT_OK_PTR(skel->links.test_fexit, "bpf_program__attach_tracing_multi"))
goto cleanup;
/* Only bpf_fentry_test1 is allowed for sleepable programs. */
opts.cnt = 1;
skel->links.test_fentry_s = bpf_program__attach_tracing_multi(skel->progs.test_fentry_s,
NULL, &opts);
if (!ASSERT_OK_PTR(skel->links.test_fentry_s, "bpf_program__attach_tracing_multi"))
goto cleanup;
skel->links.test_fexit_s = bpf_program__attach_tracing_multi(skel->progs.test_fexit_s,
NULL, &opts);
if (!ASSERT_OK_PTR(skel->links.test_fexit_s, "bpf_program__attach_tracing_multi"))
goto cleanup;
tracing_multi_test_run(skel);
cleanup:
tracing_multi__destroy(skel);
free(ids);
}
static void test_module_skel_api(void)
{
struct tracing_multi_module *skel = NULL;
int err;
skel = tracing_multi_module__open_and_load();
if (!ASSERT_OK_PTR(skel, "tracing_multi__open_and_load"))
return;
skel->bss->pid = getpid();
err = tracing_multi_module__attach(skel);
if (!ASSERT_OK(err, "tracing_multi__attach"))
goto cleanup;
ASSERT_OK(trigger_module_test_read(1), "trigger_read");
ASSERT_EQ(skel->bss->test_result_fentry, 5, "test_result_fentry");
ASSERT_EQ(skel->bss->test_result_fexit, 5, "test_result_fexit");
cleanup:
tracing_multi_module__destroy(skel);
}
static void test_module_link_api_pattern(void)
{
struct tracing_multi_module *skel = NULL;
skel = tracing_multi_module__open_and_load();
if (!ASSERT_OK_PTR(skel, "tracing_multi_module__open_and_load"))
return;
skel->bss->pid = getpid();
skel->links.test_fentry = bpf_program__attach_tracing_multi(skel->progs.test_fentry,
"bpf_testmod:bpf_testmod_fentry_test*", NULL);
if (!ASSERT_OK_PTR(skel->links.test_fentry, "bpf_program__attach_tracing_multi"))
goto cleanup;
skel->links.test_fexit = bpf_program__attach_tracing_multi(skel->progs.test_fexit,
"bpf_testmod:bpf_testmod_fentry_test*", NULL);
if (!ASSERT_OK_PTR(skel->links.test_fexit, "bpf_program__attach_tracing_multi"))
goto cleanup;
ASSERT_OK(trigger_module_test_read(1), "trigger_read");
ASSERT_EQ(skel->bss->test_result_fentry, 5, "test_result_fentry");
ASSERT_EQ(skel->bss->test_result_fexit, 5, "test_result_fexit");
cleanup:
tracing_multi_module__destroy(skel);
}
static void test_module_link_api_ids(void)
{
size_t cnt = ARRAY_SIZE(bpf_testmod_fentry_test);
LIBBPF_OPTS(bpf_tracing_multi_opts, opts);
struct tracing_multi_module *skel = NULL;
__u32 *ids;
skel = tracing_multi_module__open_and_load();
if (!ASSERT_OK_PTR(skel, "tracing_multi_module__open_and_load"))
return;
skel->bss->pid = getpid();
ids = get_ids(bpf_testmod_fentry_test, cnt, "bpf_testmod");
if (!ASSERT_OK_PTR(ids, "get_ids"))
goto cleanup;
opts.ids = ids;
opts.cnt = cnt;
skel->links.test_fentry = bpf_program__attach_tracing_multi(skel->progs.test_fentry,
NULL, &opts);
if (!ASSERT_OK_PTR(skel->links.test_fentry, "bpf_program__attach_tracing_multi"))
goto cleanup;
skel->links.test_fexit = bpf_program__attach_tracing_multi(skel->progs.test_fexit,
NULL, &opts);
if (!ASSERT_OK_PTR(skel->links.test_fexit, "bpf_program__attach_tracing_multi"))
goto cleanup;
ASSERT_OK(trigger_module_test_read(1), "trigger_read");
ASSERT_EQ(skel->bss->test_result_fentry, 5, "test_result_fentry");
ASSERT_EQ(skel->bss->test_result_fexit, 5, "test_result_fexit");
cleanup:
tracing_multi_module__destroy(skel);
free(ids);
}
static bool is_set(__u32 mask, __u32 bit)
{
return (1 << bit) & mask;
}
static void __test_intersect(__u32 mask, const struct bpf_program *progs[4], __u64 *test_results[4])
{
LIBBPF_OPTS(bpf_tracing_multi_opts, opts);
LIBBPF_OPTS(bpf_test_run_opts, topts);
struct bpf_link *links[4] = { NULL };
const char *funcs[FUNCS_CNT];
__u64 expected[4];
__u32 *ids, i;
int err, cnt;
/*
* We have 4 programs in progs and the mask bits pick which
* of them gets attached to randomly chosen functions.
*/
for (i = 0; i < 4; i++) {
if (!is_set(mask, i))
continue;
cnt = get_random_funcs(funcs);
ids = get_ids(funcs, cnt, NULL);
if (!ASSERT_OK_PTR(ids, "get_ids"))
goto cleanup;
opts.ids = ids;
opts.cnt = cnt;
links[i] = bpf_program__attach_tracing_multi(progs[i], NULL, &opts);
free(ids);
if (!ASSERT_OK_PTR(links[i], "bpf_program__attach_tracing_multi"))
goto cleanup;
expected[i] = *test_results[i] + cnt;
}
err = bpf_prog_test_run_opts(bpf_program__fd(progs[0]), &topts);
ASSERT_OK(err, "test_run");
for (i = 0; i < 4; i++) {
if (!is_set(mask, i))
continue;
ASSERT_EQ(*test_results[i], expected[i], "test_results");
}
cleanup:
for (i = 0; i < 4; i++)
bpf_link__destroy(links[i]);
}
static void test_intersect(void)
{
struct tracing_multi_intersect *skel;
const struct bpf_program *progs[4];
__u64 *test_results[4];
__u32 i;
skel = tracing_multi_intersect__open_and_load();
if (!ASSERT_OK_PTR(skel, "tracing_multi_intersect__open_and_load"))
return;
skel->bss->pid = getpid();
progs[0] = skel->progs.fentry_1;
progs[1] = skel->progs.fexit_1;
progs[2] = skel->progs.fentry_2;
progs[3] = skel->progs.fexit_2;
test_results[0] = &skel->bss->test_result_fentry_1;
test_results[1] = &skel->bss->test_result_fexit_1;
test_results[2] = &skel->bss->test_result_fentry_2;
test_results[3] = &skel->bss->test_result_fexit_2;
for (i = 1; i < 16; i++)
__test_intersect(i, progs, test_results);
tracing_multi_intersect__destroy(skel);
}
static void test_session(void)
{
LIBBPF_OPTS(bpf_test_run_opts, topts);
struct tracing_multi_session *skel;
int err, prog_fd;
skel = tracing_multi_session__open_and_load();
if (!ASSERT_OK_PTR(skel, "tracing_multi_session__open_and_load"))
return;
skel->bss->pid = getpid();
err = tracing_multi_session__attach(skel);
if (!ASSERT_OK(err, "tracing_multi_session__attach"))
goto cleanup;
/* execute kernel session */
prog_fd = bpf_program__fd(skel->progs.test_session_1);
err = bpf_prog_test_run_opts(prog_fd, &topts);
ASSERT_OK(err, "test_run");
/* 10 for test_session_1, 1 for test_fsession_s */
ASSERT_EQ(skel->bss->test_result_fentry, 11, "test_result_fentry");
/* extra count (+1 for each fexit execution) for test_result_fexit cookie check/inc */
ASSERT_EQ(skel->bss->test_result_fexit, 22, "test_result_fexit");
skel->bss->test_result_fentry = 0;
skel->bss->test_result_fexit = 0;
/* execute bpf_testmo.ko session */
ASSERT_OK(trigger_module_test_read(1), "trigger_read");
/* 5 for test_session_2 */
ASSERT_EQ(skel->bss->test_result_fentry, 5, "test_result_fentry");
/* extra count (+1 for each fexit execution) for test_result_fexit cookie */
ASSERT_EQ(skel->bss->test_result_fexit, 10, "test_result_fexit");
cleanup:
tracing_multi_session__destroy(skel);
}
static void test_attach_api_fails(void)
{
LIBBPF_OPTS(bpf_tracing_multi_opts, opts);
static const char * const func[] = {
"bpf_fentry_test2",
};
struct tracing_multi_fail *skel = NULL;
__u32 ids[2] = {}, *ids2 = NULL;
__u64 cookies[2];
skel = tracing_multi_fail__open_and_load();
if (!ASSERT_OK_PTR(skel, "tracing_multi_fail__open_and_load"))
return;
/* fail#1 (libbpf) pattern and opts NULL */
skel->links.test_fentry = bpf_program__attach_tracing_multi(skel->progs.test_fentry,
NULL, NULL);
if (!ASSERT_EQ(libbpf_get_error(skel->links.test_fentry), -EINVAL, "fail_1"))
goto cleanup;
/* fail#2 (libbpf) pattern and ids */
LIBBPF_OPTS_RESET(opts,
.ids = ids,
.cnt = 2,
);
skel->links.test_fentry = bpf_program__attach_tracing_multi(skel->progs.test_fentry,
"bpf_fentry_test*", &opts);
if (!ASSERT_EQ(libbpf_get_error(skel->links.test_fentry), -EINVAL, "fail_2"))
goto cleanup;
/* fail#3 (libbpf) pattern and cookies */
LIBBPF_OPTS_RESET(opts,
.ids = NULL,
.cnt = 2,
.cookies = cookies,
);
skel->links.test_fentry = bpf_program__attach_tracing_multi(skel->progs.test_fentry,
"bpf_fentry_test*", &opts);
if (!ASSERT_EQ(libbpf_get_error(skel->links.test_fentry), -EINVAL, "fail_3"))
goto cleanup;
/* fail#4 (libbpf) bogus pattern */
skel->links.test_fentry = bpf_program__attach_tracing_multi(skel->progs.test_fentry,
"bpf_not_really_a_function*", NULL);
if (!ASSERT_EQ(libbpf_get_error(skel->links.test_fentry), -EINVAL, "fail_4"))
goto cleanup;
/* fail#5 (kernel) abnormal cnt */
LIBBPF_OPTS_RESET(opts,
.ids = ids,
.cnt = INT_MAX,
);
skel->links.test_fentry = bpf_program__attach_tracing_multi(skel->progs.test_fentry,
NULL, &opts);
if (!ASSERT_EQ(libbpf_get_error(skel->links.test_fentry), -E2BIG, "fail_5"))
goto cleanup;
/* fail#6 (kernel) attach sleepable program to not-allowed function */
ids2 = get_ids(func, 1, NULL);
if (!ASSERT_OK_PTR(ids2, "get_ids"))
goto cleanup;
LIBBPF_OPTS_RESET(opts,
.ids = ids2,
.cnt = 1,
);
skel->links.test_fentry_s = bpf_program__attach_tracing_multi(skel->progs.test_fentry_s,
NULL, &opts);
if (!ASSERT_EQ(libbpf_get_error(skel->links.test_fentry_s), -EINVAL, "fail_6"))
goto cleanup;
/* fail#7 (kernel) attach with duplicate id */
ids[0] = ids2[0];
ids[1] = ids2[0];
LIBBPF_OPTS_RESET(opts,
.ids = ids,
.cnt = 2,
);
skel->links.test_fentry = bpf_program__attach_tracing_multi(skel->progs.test_fentry,
NULL, &opts);
ASSERT_EQ(libbpf_get_error(skel->links.test_fentry), -EINVAL, "fail_7");
cleanup:
tracing_multi_fail__destroy(skel);
free(ids2);
}
void serial_test_tracing_multi_bench_attach(void)
{
LIBBPF_OPTS(bpf_tracing_multi_opts, opts);
struct tracing_multi_bench *skel = NULL;
long attach_start_ns, attach_end_ns;
long detach_start_ns, detach_end_ns;
double attach_delta, detach_delta;
struct bpf_link *link = NULL;
size_t i, cap = 0, cnt = 0;
struct ksyms *ksyms = NULL;
void *root = NULL;
void *dups = NULL;
__u32 *ids = NULL;
__u32 nr, type_id;
struct btf *btf;
int err;
#ifndef __x86_64__
test__skip();
return;
#endif
btf = btf__load_vmlinux_btf();
if (!ASSERT_OK_PTR(btf, "btf__load_vmlinux_btf"))
return;
skel = tracing_multi_bench__open_and_load();
if (!ASSERT_OK_PTR(skel, "tracing_multi_bench__open_and_load"))
goto cleanup;
if (!ASSERT_OK(bpf_get_ksyms(&ksyms, true), "get_syms"))
goto cleanup;
/* Get all ftrace 'safe' symbols.. */
for (i = 0; i < ksyms->filtered_cnt; i++) {
if (!tsearch(&ksyms->filtered_syms[i], &root, compare)) {
ASSERT_FAIL("tsearch failed");
goto cleanup;
}
}
/*
* Collect names that are not unique in kallsyms. The kernel resolves a
* tracing-multi BTF id to an address with kallsyms_lookup_name(), which
* returns the first symbol of that name. For a duplicate name that may
* be a different (non-ftrace-able) instance than the ftrace-able one in
* available_filter_functions, so attaching to it by BTF id fails with
* -ENOENT (e.g. t_start/t_next/t_stop). ksyms->syms is sorted by name,
* so equal names are adjacent.
*/
for (i = 1; i < ksyms->sym_cnt; i++) {
if (strcmp(ksyms->syms[i].name, ksyms->syms[i - 1].name))
continue;
if (!tsearch(&ksyms->syms[i].name, &dups, compare)) {
ASSERT_FAIL("tsearch failed");
goto cleanup;
}
}
/* ..and filter them through BTF and btf_type_is_traceable_func. */
nr = btf__type_cnt(btf);
for (type_id = 1; type_id < nr; type_id++) {
const struct btf_type *type;
const char *str;
type = btf__type_by_id(btf, type_id);
if (!type)
break;
if (BTF_INFO_KIND(type->info) != BTF_KIND_FUNC)
continue;
str = btf__name_by_offset(btf, type->name_off);
if (!str)
break;
if (!tfind(&str, &root, compare))
continue;
/* Skip names that are not unique in kallsyms, see above. */
if (tfind(&str, &dups, compare))
continue;
if (!btf_type_is_traceable_func(btf, type))
continue;
err = libbpf_ensure_mem((void **) &ids, &cap, sizeof(*ids), cnt + 1);
if (err)
goto cleanup;
ids[cnt++] = type_id;
}
opts.ids = ids;
opts.cnt = cnt;
attach_start_ns = get_time_ns();
link = bpf_program__attach_tracing_multi(skel->progs.bench, NULL, &opts);
attach_end_ns = get_time_ns();
if (!ASSERT_OK_PTR(link, "bpf_program__attach_tracing_multi"))
goto cleanup;
detach_start_ns = get_time_ns();
bpf_link__destroy(link);
detach_end_ns = get_time_ns();
attach_delta = (attach_end_ns - attach_start_ns) / 1000000000.0;
detach_delta = (detach_end_ns - detach_start_ns) / 1000000000.0;
printf("%s: found %lu functions\n", __func__, cnt);
printf("%s: attached in %7.3lfs\n", __func__, attach_delta);
printf("%s: detached in %7.3lfs\n", __func__, detach_delta);
cleanup:
tracing_multi_bench__destroy(skel);
tdestroy(root, tdestroy_free_nop);
tdestroy(dups, tdestroy_free_nop);
free_kallsyms_local(ksyms);
free(ids);
btf__free(btf);
}
static void tracing_multi_rollback_run(struct tracing_multi_rollback *skel)
{
LIBBPF_OPTS(bpf_test_run_opts, topts);
int err, prog_fd;
prog_fd = bpf_program__fd(skel->progs.test_fentry);
err = bpf_prog_test_run_opts(prog_fd, &topts);
ASSERT_OK(err, "test_run");
/* make sure the rollback code did not leave any program attached */
ASSERT_EQ(skel->bss->test_result_fentry, 0, "test_result_fentry");
ASSERT_EQ(skel->bss->test_result_fexit, 0, "test_result_fexit");
}
static void test_rollback_put(void)
{
LIBBPF_OPTS(bpf_tracing_multi_opts, opts);
struct tracing_multi_rollback *skel = NULL;
size_t cnt = FUNCS_CNT;
__u32 *ids = NULL;
int err;
skel = tracing_multi_rollback__open();
if (!ASSERT_OK_PTR(skel, "tracing_multi_rollback__open"))
return;
bpf_program__set_autoload(skel->progs.test_fentry, true);
bpf_program__set_autoload(skel->progs.test_fexit, true);
err = tracing_multi_rollback__load(skel);
if (!ASSERT_OK(err, "tracing_multi_rollback__load"))
goto cleanup;
ids = get_ids(bpf_fentry_test, cnt, NULL);
if (!ASSERT_OK_PTR(ids, "get_ids"))
goto cleanup;
/*
* Mangle last id to trigger rollback, which needs to do put
* on get-ed trampolines.
*/
ids[9] = 0;
opts.ids = ids;
opts.cnt = cnt;
skel->bss->pid = getpid();
skel->links.test_fentry = bpf_program__attach_tracing_multi(skel->progs.test_fentry,
NULL, &opts);
if (!ASSERT_ERR_PTR(skel->links.test_fentry, "bpf_program__attach_tracing_multi"))
goto cleanup;
skel->links.test_fexit = bpf_program__attach_tracing_multi(skel->progs.test_fexit,
NULL, &opts);
if (!ASSERT_ERR_PTR(skel->links.test_fexit, "bpf_program__attach_tracing_multi"))
goto cleanup;
/* We don't really attach any program, but let's make sure. */
tracing_multi_rollback_run(skel);
cleanup:
tracing_multi_rollback__destroy(skel);
free(ids);
}
static void fillers_cleanup(struct tracing_multi_rollback **skels, int cnt)
{
int i;
for (i = 0; i < cnt; i++)
tracing_multi_rollback__destroy(skels[i]);
free(skels);
}
static struct tracing_multi_rollback *extra_load_and_link(void)
{
struct tracing_multi_rollback *skel;
int err;
skel = tracing_multi_rollback__open();
if (!ASSERT_OK_PTR(skel, "tracing_multi_rollback__open"))
goto cleanup;
bpf_program__set_autoload(skel->progs.extra, true);
err = tracing_multi_rollback__load(skel);
if (!ASSERT_OK(err, "tracing_multi_rollback__load"))
goto cleanup;
skel->links.extra = bpf_program__attach_trace(skel->progs.extra);
if (!ASSERT_OK_PTR(skel->links.extra, "bpf_program__attach_trace"))
goto cleanup;
return skel;
cleanup:
tracing_multi_rollback__destroy(skel);
return NULL;
}
static struct tracing_multi_rollback **fillers_load_and_link(int max)
{
struct tracing_multi_rollback **skels, *skel;
int i, err;
skels = calloc(max + 1, sizeof(*skels));
if (!ASSERT_OK_PTR(skels, "calloc"))
return NULL;
for (i = 0; i < max; i++) {
skel = skels[i] = tracing_multi_rollback__open();
if (!ASSERT_OK_PTR(skels[i], "tracing_multi_rollback__open"))
goto cleanup;
bpf_program__set_autoload(skel->progs.filler, true);
err = tracing_multi_rollback__load(skel);
if (!ASSERT_OK(err, "tracing_multi_rollback__load"))
goto cleanup;
skel->links.filler = bpf_program__attach_trace(skel->progs.filler);
if (!ASSERT_OK_PTR(skels[i]->links.filler, "bpf_program__attach_trace"))
goto cleanup;
}
return skels;
cleanup:
fillers_cleanup(skels, i + 1);
return NULL;
}
static void test_rollback_unlink(void)
{
struct tracing_multi_rollback *skel = NULL, *extra;
LIBBPF_OPTS(bpf_tracing_multi_opts, opts);
struct tracing_multi_rollback **fillers;
size_t cnt = FUNCS_CNT;
__u32 *ids = NULL;
int err, max;
max = get_bpf_max_tramp_links();
if (!ASSERT_GE(max, 1, "bpf_max_tramp_links"))
return;
/* Attach maximum allowed programs to bpf_fentry_test10 */
fillers = fillers_load_and_link(max);
if (!ASSERT_OK_PTR(fillers, "fillers_load_and_link"))
return;
extra = extra_load_and_link();
if (!ASSERT_OK_PTR(extra, "extra_load_and_link"))
goto cleanup;
skel = tracing_multi_rollback__open();
if (!ASSERT_OK_PTR(skel, "tracing_multi_rollback__open"))
goto cleanup;
bpf_program__set_autoload(skel->progs.test_fentry, true);
bpf_program__set_autoload(skel->progs.test_fexit, true);
/*
* Attach tracing_multi link on bpf_fentry_test1-10, which will
* fail on bpf_fentry_test10 function, because it already has
* maximum allowed programs attached.
*
* The rollback needs to unlink already link-ed trampolines and
* put all of them.
*/
err = tracing_multi_rollback__load(skel);
if (!ASSERT_OK(err, "tracing_multi_rollback__load"))
goto cleanup;
ids = get_ids(bpf_fentry_test, cnt, NULL);
if (!ASSERT_OK_PTR(ids, "get_ids"))
goto cleanup;
opts.ids = ids;
opts.cnt = cnt;
skel->bss->pid = getpid();
skel->links.test_fentry = bpf_program__attach_tracing_multi(skel->progs.test_fentry,
NULL, &opts);
if (!ASSERT_ERR_PTR(skel->links.test_fentry, "bpf_program__attach_tracing_multi"))
goto cleanup;
skel->links.test_fexit = bpf_program__attach_tracing_multi(skel->progs.test_fexit,
NULL, &opts);
if (!ASSERT_ERR_PTR(skel->links.test_fexit, "bpf_program__attach_tracing_multi"))
goto cleanup;
tracing_multi_rollback_run(skel);
cleanup:
fillers_cleanup(fillers, max);
tracing_multi_rollback__destroy(extra);
tracing_multi_rollback__destroy(skel);
free(ids);
}
void serial_test_tracing_multi_attach_rollback(void)
{
if (test__start_subtest("put"))
test_rollback_put();
if (test__start_subtest("unlink"))
test_rollback_unlink();
}
void test_tracing_multi_test(void)
{
#ifndef __x86_64__
test__skip();
return;
#endif
if (test__start_subtest("skel_api"))
test_skel_api();
if (test__start_subtest("link_api_pattern"))
test_link_api_pattern();
if (test__start_subtest("link_api_ids"))
test_link_api_ids(false);
if (test__start_subtest("module_skel_api"))
test_module_skel_api();
if (test__start_subtest("module_link_api_pattern"))
test_module_link_api_pattern();
if (test__start_subtest("module_link_api_ids"))
test_module_link_api_ids();
if (test__start_subtest("intersect"))
test_intersect();
if (test__start_subtest("cookies"))
test_link_api_ids(true);
if (test__start_subtest("session"))
test_session();
if (test__start_subtest("attach_api_fails"))
test_attach_api_fails();
RUN_TESTS(tracing_multi_verifier);
}

39
tools/testing/selftests/bpf/progs/tracing_multi_attach.c Normal file
View File

@ -0,0 +1,39 @@
// SPDX-License-Identifier: GPL-2.0
#include <vmlinux.h>
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_tracing.h>
char _license[] SEC("license") = "GPL";
__hidden extern int tracing_multi_arg_check(__u64 *ctx, __u64 *test_result, bool is_return);
__u64 test_result_fentry = 0;
__u64 test_result_fexit = 0;
SEC("fentry.multi/bpf_fentry_test*")
int BPF_PROG(test_fentry)
{
tracing_multi_arg_check(ctx, &test_result_fentry, false);
return 0;
}
SEC("fexit.multi/bpf_fentry_test*")
int BPF_PROG(test_fexit)
{
tracing_multi_arg_check(ctx, &test_result_fexit, true);
return 0;
}
SEC("fentry.multi.s/bpf_fentry_test1")
int BPF_PROG(test_fentry_s)
{
tracing_multi_arg_check(ctx, &test_result_fentry, false);
return 0;
}
SEC("fexit.multi.s/bpf_fentry_test1")
int BPF_PROG(test_fexit_s)
{
tracing_multi_arg_check(ctx, &test_result_fexit, true);
return 0;
}

25
tools/testing/selftests/bpf/progs/tracing_multi_attach_module.c Normal file
View File

@ -0,0 +1,25 @@
// SPDX-License-Identifier: GPL-2.0
#include <vmlinux.h>
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_tracing.h>
char _license[] SEC("license") = "GPL";
__hidden extern int tracing_multi_arg_check(__u64 *ctx, __u64 *test_result, bool is_return);
__u64 test_result_fentry = 0;
__u64 test_result_fexit = 0;
SEC("fentry.multi/bpf_testmod:bpf_testmod_fentry_test*")
int BPF_PROG(test_fentry)
{
tracing_multi_arg_check(ctx, &test_result_fentry, false);
return 0;
}
SEC("fexit.multi/bpf_testmod:bpf_testmod_fentry_test*")
int BPF_PROG(test_fexit)
{
tracing_multi_arg_check(ctx, &test_result_fexit, true);
return 0;
}

12
tools/testing/selftests/bpf/progs/tracing_multi_bench.c Normal file
View File

@ -0,0 +1,12 @@
// SPDX-License-Identifier: GPL-2.0
#include <vmlinux.h>
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_tracing.h>
char _license[] SEC("license") = "GPL";
SEC("fentry.multi")
int BPF_PROG(bench)
{
return 0;
}

214
tools/testing/selftests/bpf/progs/tracing_multi_check.c Normal file
View File

@ -0,0 +1,214 @@
// SPDX-License-Identifier: GPL-2.0
#include <vmlinux.h>
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_tracing.h>
char _license[] SEC("license") = "GPL";
int pid = 0;
bool test_cookies = false;
/* bpf_fentry_test1 is exported as kfunc via vmlinux.h */
extern const void bpf_fentry_test2 __ksym;
extern const void bpf_fentry_test3 __ksym;
extern const void bpf_fentry_test4 __ksym;
extern const void bpf_fentry_test5 __ksym;
extern const void bpf_fentry_test6 __ksym;
extern const void bpf_fentry_test7 __ksym;
extern const void bpf_fentry_test8 __ksym;
extern const void bpf_fentry_test9 __ksym;
extern const void bpf_fentry_test10 __ksym;
extern const void bpf_testmod_fentry_test1 __ksym;
extern const void bpf_testmod_fentry_test2 __ksym;
extern const void bpf_testmod_fentry_test3 __ksym;
extern const void bpf_testmod_fentry_test7 __ksym;
extern const void bpf_testmod_fentry_test11 __ksym;
int tracing_multi_arg_check(__u64 *ctx, __u64 *test_result, bool is_return)
{
void *ip = (void *) bpf_get_func_ip(ctx);
__u64 value = 0, ret = 0, cookie = 0;
long err = 0;
if (bpf_get_current_pid_tgid() >> 32 != pid)
return 1;
if (is_return)
err |= bpf_get_func_ret(ctx, &ret);
if (test_cookies)
cookie = bpf_get_attach_cookie(ctx);
if (ip == &bpf_fentry_test1) {
int a;
err |= bpf_get_func_arg(ctx, 0, &value);
a = (int) value;
err |= is_return ? ret != 2 : 0;
err |= test_cookies ? cookie != 8 : 0;
*test_result += err == 0 && a == 1;
} else if (ip == &bpf_fentry_test2) {
__u64 b;
int a;
err |= bpf_get_func_arg(ctx, 0, &value);
a = (int) value;
err |= bpf_get_func_arg(ctx, 1, &value);
b = value;
err |= is_return ? ret != 5 : 0;
err |= test_cookies ? cookie != 9 : 0;
*test_result += err == 0 && a == 2 && b == 3;
} else if (ip == &bpf_fentry_test3) {
__u64 c;
char a;
int b;
err |= bpf_get_func_arg(ctx, 0, &value);
a = (char) value;
err |= bpf_get_func_arg(ctx, 1, &value);
b = (int) value;
err |= bpf_get_func_arg(ctx, 2, &value);
c = value;
err |= is_return ? ret != 15 : 0;
err |= test_cookies ? cookie != 7 : 0;
*test_result += err == 0 && a == 4 && b == 5 && c == 6;
} else if (ip == &bpf_fentry_test4) {
void *a;
char b;
int c;
__u64 d;
err |= bpf_get_func_arg(ctx, 0, &value);
a = (void *) value;
err |= bpf_get_func_arg(ctx, 1, &value);
b = (char) value;
err |= bpf_get_func_arg(ctx, 2, &value);
c = (int) value;
err |= bpf_get_func_arg(ctx, 3, &value);
d = value;
err |= is_return ? ret != 34 : 0;
err |= test_cookies ? cookie != 5 : 0;
*test_result += err == 0 && a == (void *) 7 && b == 8 && c == 9 && d == 10;
} else if (ip == &bpf_fentry_test5) {
__u64 a;
void *b;
short c;
int d;
__u64 e;
err |= bpf_get_func_arg(ctx, 0, &value);
a = value;
err |= bpf_get_func_arg(ctx, 1, &value);
b = (void *) value;
err |= bpf_get_func_arg(ctx, 2, &value);
c = (short) value;
err |= bpf_get_func_arg(ctx, 3, &value);
d = (int) value;
err |= bpf_get_func_arg(ctx, 4, &value);
e = value;
err |= is_return ? ret != 65 : 0;
err |= test_cookies ? cookie != 4 : 0;
*test_result += err == 0 && a == 11 && b == (void *) 12 && c == 13 && d == 14 && e == 15;
} else if (ip == &bpf_fentry_test6) {
__u64 a;
void *b;
short c;
int d;
void *e;
__u64 f;
err |= bpf_get_func_arg(ctx, 0, &value);
a = value;
err |= bpf_get_func_arg(ctx, 1, &value);
b = (void *) value;
err |= bpf_get_func_arg(ctx, 2, &value);
c = (short) value;
err |= bpf_get_func_arg(ctx, 3, &value);
d = (int) value;
err |= bpf_get_func_arg(ctx, 4, &value);
e = (void *) value;
err |= bpf_get_func_arg(ctx, 5, &value);
f = value;
err |= is_return ? ret != 111 : 0;
err |= test_cookies ? cookie != 2 : 0;
*test_result += err == 0 && a == 16 && b == (void *) 17 && c == 18 && d == 19 && e == (void *) 20 && f == 21;
} else if (ip == &bpf_fentry_test7) {
err |= is_return ? ret != 0 : 0;
err |= test_cookies ? cookie != 3 : 0;
*test_result += err == 0 ? 1 : 0;
} else if (ip == &bpf_fentry_test8) {
err |= is_return ? ret != 0 : 0;
err |= test_cookies ? cookie != 1 : 0;
*test_result += err == 0 ? 1 : 0;
} else if (ip == &bpf_fentry_test9) {
err |= is_return ? ret != 0 : 0;
err |= test_cookies ? cookie != 10 : 0;
*test_result += err == 0 ? 1 : 0;
} else if (ip == &bpf_fentry_test10) {
err |= is_return ? ret != 0 : 0;
err |= test_cookies ? cookie != 6 : 0;
*test_result += err == 0 ? 1 : 0;
} else if (ip == &bpf_testmod_fentry_test1) {
int a;
err |= bpf_get_func_arg(ctx, 0, &value);
a = (int) value;
err |= is_return ? ret != 2 : 0;
*test_result += err == 0 && a == 1;
} else if (ip == &bpf_testmod_fentry_test2) {
int a;
__u64 b;
err |= bpf_get_func_arg(ctx, 0, &value);
a = (int) value;
err |= bpf_get_func_arg(ctx, 1, &value);
b = (__u64) value;
err |= is_return ? ret != 5 : 0;
*test_result += err == 0 && a == 2 && b == 3;
} else if (ip == &bpf_testmod_fentry_test3) {
char a;
int b;
__u64 c;
err |= bpf_get_func_arg(ctx, 0, &value);
a = (char) value;
err |= bpf_get_func_arg(ctx, 1, &value);
b = (int) value;
err |= bpf_get_func_arg(ctx, 2, &value);
c = (__u64) value;
err |= is_return ? ret != 15 : 0;
*test_result += err == 0 && a == 4 && b == 5 && c == 6;
} else if (ip == &bpf_testmod_fentry_test7) {
err |= is_return ? ret != 133 : 0;
*test_result += err == 0;
} else if (ip == &bpf_testmod_fentry_test11) {
err |= is_return ? ret != 231 : 0;
*test_result += err == 0;
}
return 0;
}

18
tools/testing/selftests/bpf/progs/tracing_multi_fail.c Normal file
View File

@ -0,0 +1,18 @@
// SPDX-License-Identifier: GPL-2.0
#include <vmlinux.h>
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_tracing.h>
char _license[] SEC("license") = "GPL";
SEC("fentry.multi")
int BPF_PROG(test_fentry)
{
return 0;
}
SEC("fentry.multi.s")
int BPF_PROG(test_fentry_s)
{
return 0;
}

41
tools/testing/selftests/bpf/progs/tracing_multi_intersect_attach.c Normal file
View File

@ -0,0 +1,41 @@
// SPDX-License-Identifier: GPL-2.0
#include <vmlinux.h>
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_tracing.h>
char _license[] SEC("license") = "GPL";
__hidden extern int tracing_multi_arg_check(__u64 *ctx, __u64 *test_result, bool is_return);
__u64 test_result_fentry_1 = 0;
__u64 test_result_fentry_2 = 0;
__u64 test_result_fexit_1 = 0;
__u64 test_result_fexit_2 = 0;
SEC("fentry.multi")
int BPF_PROG(fentry_1)
{
tracing_multi_arg_check(ctx, &test_result_fentry_1, false);
return 0;
}
SEC("fentry.multi")
int BPF_PROG(fentry_2)
{
tracing_multi_arg_check(ctx, &test_result_fentry_2, false);
return 0;
}
SEC("fexit.multi")
int BPF_PROG(fexit_1)
{
tracing_multi_arg_check(ctx, &test_result_fexit_1, true);
return 0;
}
SEC("fexit.multi")
int BPF_PROG(fexit_2)
{
tracing_multi_arg_check(ctx, &test_result_fexit_2, true);
return 0;
}

43
tools/testing/selftests/bpf/progs/tracing_multi_rollback.c Normal file
View File

@ -0,0 +1,43 @@
// SPDX-License-Identifier: GPL-2.0
#include <vmlinux.h>
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_tracing.h>
char _license[] SEC("license") = "GPL";
int pid = 0;
__u64 test_result_fentry = 0;
__u64 test_result_fexit = 0;
SEC("?fentry.multi")
int BPF_PROG(test_fentry)
{
if (bpf_get_current_pid_tgid() >> 32 != pid)
return 0;
test_result_fentry++;
return 0;
}
SEC("?fexit.multi")
int BPF_PROG(test_fexit)
{
if (bpf_get_current_pid_tgid() >> 32 != pid)
return 0;
test_result_fexit++;
return 0;
}
SEC("?fentry/bpf_fentry_test1")
int BPF_PROG(extra)
{
return 0;
}
SEC("?fentry/bpf_fentry_test10")
int BPF_PROG(filler)
{
return 0;
}

65
tools/testing/selftests/bpf/progs/tracing_multi_session_attach.c Normal file
View File

@ -0,0 +1,65 @@
// SPDX-License-Identifier: GPL-2.0
#include <vmlinux.h>
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_tracing.h>
char _license[] SEC("license") = "GPL";
__hidden extern int tracing_multi_arg_check(__u64 *ctx, __u64 *test_result, bool is_return);
__u64 test_result_fentry = 0;
__u64 test_result_fexit = 0;
SEC("fsession.multi/bpf_fentry_test*")
int BPF_PROG(test_session_1)
{
volatile __u64 *cookie = bpf_session_cookie(ctx);
if (bpf_session_is_return(ctx)) {
if (tracing_multi_arg_check(ctx, &test_result_fexit, true))
return 0;
/* extra count for test_result_fexit cookie */
test_result_fexit += *cookie == 0xbeafbeafbeafbeaf;
} else {
if (tracing_multi_arg_check(ctx, &test_result_fentry, false))
return 0;
*cookie = 0xbeafbeafbeafbeaf;
}
return 0;
}
SEC("fsession.multi.s/bpf_fentry_test1")
int BPF_PROG(test_fsession_s)
{
volatile __u64 *cookie = bpf_session_cookie(ctx);
if (bpf_session_is_return(ctx)) {
if (tracing_multi_arg_check(ctx, &test_result_fexit, true))
return 0;
/* extra count for test_result_fexit cookie */
test_result_fexit += *cookie == 0xbeafbeafbeafbeaf;
} else {
if (tracing_multi_arg_check(ctx, &test_result_fentry, false))
return 0;
*cookie = 0xbeafbeafbeafbeaf;
}
return 0;
}
SEC("fsession.multi/bpf_testmod:bpf_testmod_fentry_test*")
int BPF_PROG(test_session_2)
{
volatile __u64 *cookie = bpf_session_cookie(ctx);
if (bpf_session_is_return(ctx)) {
if (tracing_multi_arg_check(ctx, &test_result_fexit, true))
return 0;
/* extra count for test_result_fexit cookie */
test_result_fexit += *cookie == 0xbeafbeafbeafbeaf;
} else {
if (tracing_multi_arg_check(ctx, &test_result_fentry, false))
return 0;
*cookie = 0xbeafbeafbeafbeaf;
}
return 0;
}

31
tools/testing/selftests/bpf/progs/tracing_multi_verifier.c Normal file
View File

@ -0,0 +1,31 @@
// SPDX-License-Identifier: GPL-2.0
#include "vmlinux.h"
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_tracing.h>
#include "bpf_misc.h"
char _license[] SEC("license") = "GPL";
SEC("fentry.multi/bpf_fentry_test1")
__failure
__msg("func 'bpf_multi_func' doesn't have 1-th argument")
int BPF_PROG(fentry_direct_access, int a)
{
return a;
}
SEC("fexit.multi/bpf_fentry_test3")
__failure
__msg("invalid bpf_context access off=24 size=8")
int BPF_PROG(fexit_direct_access, char a, int b, __u64 c, int ret)
{
return ret;
}
SEC("fsession.multi/bpf_fentry_test4")
__failure
__msg("invalid bpf_context access off=16 size=8")
int BPF_PROG(fsession_direct_access, void *a, char b, int c, __u64 d, int ret)
{
return c;
}

7
tools/testing/selftests/bpf/trace_helpers.c
View File

@ -546,9 +546,10 @@ static const char * const trace_blacklist[] = {
"__rcu_read_lock",
"__rcu_read_unlock",
"bpf_get_numa_node_id",
"___migrate_enable",
};
static bool skip_entry(char *name)
bool is_unsafe_function(const char *name)
{
int i;
@ -651,7 +652,7 @@ int bpf_get_ksyms(struct ksyms **ksymsp, bool kernel)
free(name);
if (sscanf(buf, "%ms$*[^\n]\n", &name) != 1)
continue;
if (skip_entry(name))
if (is_unsafe_function(name))
continue;
ks = search_kallsyms_custom_local(ksyms, name, search_kallsyms_compare);
@ -728,7 +729,7 @@ int bpf_get_addrs(unsigned long **addrsp, size_t *cntp, bool kernel)
free(name);
if (sscanf(buf, "%p %ms$*[^\n]\n", &addr, &name) != 2)
continue;
if (skip_entry(name))
if (is_unsafe_function(name))
continue;
if (cnt == max_cnt) {

1
tools/testing/selftests/bpf/trace_helpers.h
View File

@ -63,4 +63,5 @@ int read_build_id(const char *path, char *build_id, size_t size);
int bpf_get_ksyms(struct ksyms **ksymsp, bool kernel);
int bpf_get_addrs(unsigned long **addrsp, size_t *cntp, bool kernel);
bool is_unsafe_function(const char *name);
#endif