github-mirror/linux
1
0
Fork 0
mirror of https://github.com/torvalds/linux.git synced 2026-11-07 06:17:02 +01:00
Code Issues Actions Packages Projects Releases Wiki Activity Graph

Merge bpf keyring work from of git://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next

Browse Source 
Bring in the bpf keyring from bpf-next. Andrew's series adds well-known
ids for the dm-verity and fs-verity keyrings. They go right next to
KEY_SPEC_BPF_KEYRING and do what commit 264d8fd279 ("bpf, keys: Add a
bpf keyring for program signature validation") did. That commit only
exists in bpf-next.

The keyring series was merged into bpf-next in 776c86a3e7 but that
conflicts in kernel/bpf/backtrack.c with the verifier fixes that went
into v7.3-rc2. bpf-next resolved that when it cross-merged v7.3-rc2 in
1b7415bf70. So pull up to that point and base this on v7.3-rc2 as
well.

Signed-off-by: Christian Brauner (Amutable) <brauner@kernel.org>
This commit is contained in:
Christian Brauner 2026-10-12 12:33:32 +02:00
commit 592d26f625
No known key found for this signature in database
GPG Key ID: 91C61BC06578DCA2
132 changed files with 6218 additions and 874 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

16
Documentation/admin-guide/kernel-parameters.txt
View File

@ -669,6 +669,22 @@ Kernel parameters
See Documentation/admin-guide/bootconfig.rst
bpf.keyring_unsealed=
[BPF] When set to 1, leave the bpf keyring unsealed
after initialization so that userspace can provision
keys. Once the keyring is restricted it becomes active
and can be used for BPF program signature verification.
Setting this also means that the bpf keyring is the
only non-system keyring a loader may select for the
rest of the boot: caller-supplied user/session
keyrings are refused with -EPERM, whether or not
provisioning actually completed. The system keyrings
stay selectable. Leaving it unset keeps the prior
behaviour, where a caller-supplied keyring is allowed.
See Documentation/bpf/signing.rst
bttv.card= [HW,V4L] bttv (bt848 + bt878 based grabber cards)
bttv.radio= Most important insmod options are available as
kernel args too.

74
Documentation/bpf/kfuncs.rst
View File

@ -575,6 +575,80 @@ is also covered by this recovery. A kfunc handed an arena pointer may
therefore access up to ``GUARD_SZ / 2`` past it without bounds-checking
against the arena. Larger accesses must verify the range explicitly.
2.9 kfunc Return Values
-----------------------
A kfunc may return a scalar, a pointer, or a small struct or union by
value. A scalar or pointer of up to 8 bytes is returned in R0, as usual.
A struct or union returned by value must be composed only of scalars and arena
pointers, where a scalar is an integer or an enum and an arena pointer is one
carrying the ``btf_type_tag("arena")`` attribute. Those may be nested in
structs and unions and in arrays of any number of dimensions, in any
combination, as long as what the nesting bottoms out in is a scalar or an arena
pointer. Its bytes are handed back to the program as the raw contents of R0
(and R2), so a member of any other pointer type would be laundered into a
scalar and escape the verifier's pointer provenance and reference tracking. A
struct or union with such a member is therefore rejected at load time, and so
is one with a floating-point member, which the ABI may not return in R0:R2 at
all.
An arena pointer member is handed back as a scalar too, but nothing is lost by
that. The program must ``cast_kern()`` the value before it can be used, and the
verifier allows that cast on any scalar, so a laundered arena address gives the
program no reach it did not already have. The result is confined to the
program's arena in either case.
A kfunc may also return a value larger than 8 bytes and up to 16 bytes -- a
struct or union of scalars and arena pointers, or an ``__int128``. Such a
value is returned in the register pair R0:R2, matching the convention LLVM
uses for the BPF target: the first 8 bytes in R0 and the second 8 bytes in R2.
A struct or union of 8 bytes or less is returned in R0 alone.
::
struct bpf_pair { __u64 a, b; }; /* 16 bytes */
__bpf_kfunc struct bpf_pair bpf_kfunc_get_pair(void)
{
struct bpf_pair p = { .a = 1, .b = 2 };
return p; /* p.a in R0, p.b in R2 */
}
Returning a value in the R0:R2 pair requires the JIT to place the second
half of the return value into R2, which not every architecture supports
right now. A kfunc with a return value larger than 8 bytes is therefore
rejected at load time on a JIT that does not advertise this capability (see
``bpf_jit_supports_kfunc_ret_reg_pair()``), and such a program is never run
by the interpreter. A return value larger than 16 bytes is not supported.
The same R0:R2 convention applies to a BPF subprogram, global or static, that
returns an ``__int128`` or a struct or union larger than 8 bytes. It is only
used when the program is JITed, since the interpreter propagates only R0 out of
a subprogram: without a JIT the return value stays in R0 alone, and a caller
reading R2 is rejected for reading an uninitialized register. A global
subprogram is verified in isolation, so its by-value struct or union return is
restricted to scalars and arena pointers just like a kfunc's; a static
subprogram is verified inline and has no such restriction. The main program is
not covered: its return value is the program's exit code, read out of R0
alone, so a declared upper half is never looked at.
A global subprogram must leave a scalar in *every* register of the pair, so
both halves of the returned value have to be assigned. Leaving the upper half
uninitialized is not merely untidy: the compiler is then free to leave R2
holding whatever it happened to hold, which for a subprogram taking a pointer
argument is typically that pointer. Handing the caller an unknown scalar built
from a pointer is a leak, so the verifier rejects it with::
At subprogram exit the register R2 is not a scalar value (...)
Initialize the whole return value, for example ``struct pair p = {};``, to
avoid this. A static subprogram is exempt from the scalar-only rule: it is
verified inline, so an unassigned R2 is simply passed back to the caller as
uninitialized and only a caller that reads it fails. A stack pointer left in
R2 is still rejected there, just as one in R0 is.
.. _BPF_kfunc_lifecycle_expectations:
3. kfunc lifecycle expectations

283
Documentation/bpf/signing.rst
View File

@ -254,21 +254,24 @@ returned. Only after the program has fully loaded, at the next hook
(``security_bpf_prog()``), does ``BPF_SIG_VERIFIED`` carry its full meaning:
validly signed *and* fully verified.
A more realistic admission policy than "is it signed at all": accept programs
signed by a system keyring, accept a user-keyring signature only if the
key/keyring it was verified against is on an explicit allowlist, and emit a
tamper-evident record of every decision so that even denied attempts are
auditable. (Illustrative - error checking elided.)
A more realistic admission policy than "is it signed at all": base trust in
the bpf keyring or an explicit allowlist of a caller-supplied staging key/
keyring, and emit a record of every decision including denied attempts.
(Illustrative - error checking elided.)
.. code-block:: c
/* Serials of user keys/keyrings we additionally trust. */
/*
* Serials of caller-supplied keyrings we are willing to stage. Empty
* on a system that has committed to the bpf keyring, where the kernel
* refuses them anyway.
*/
struct {
__uint(type, BPF_MAP_TYPE_HASH);
__type(key, __s32); /* keyring_serial */
__type(value, __u8);
__uint(max_entries, 64);
} trusted_user_keys SEC(".maps");
} staging_keys SEC(".maps");
/* Audit stream consumed by a userspace logger. */
struct {
@ -291,11 +294,18 @@ auditable. (Illustrative - error checking elided.)
if (kernel)
return 0; /* trust in-kernel loads */
if (verdict != BPF_SIG_VERIFIED)
if (verdict != BPF_SIG_VERIFIED) {
ret = -EPERM; /* must be validly signed */
else if (ktype == BPF_SIG_KEYRING_USER &&
!bpf_map_lookup_elem(&trusted_user_keys, &serial))
ret = -EPERM; /* key/keyring not allowlisted */
} else switch (ktype) {
case BPF_SIG_KEYRING_BPF:
break;
case BPF_SIG_KEYRING_USER:
if (!bpf_map_lookup_elem(&staging_keys, &serial))
ret = -EPERM;
break;
default:
ret = -EPERM; /* keyring not in policy */
}
d = bpf_ringbuf_reserve(&audit, sizeof(*d), 0);
if (d) {
@ -309,6 +319,11 @@ auditable. (Illustrative - error checking elided.)
return ret;
}
Such a policy is what makes a caller-supplied keyring usable at all on a system
that does not boot with ``bpf.keyring_unsealed=1``: the allowlist bounds which
staged keys count, and the LSM itself has to protect them from being tampered
with.
Observing a verified load: ``security_bpf_prog()``
--------------------------------------------------
@ -381,8 +396,9 @@ that verdict covered all of its exclusive maps, rejecting any that did not - so
a deny-by-default admission policy needs no second enforcement point. Use
``security_bpf_prog()`` to record or finally gate the verified programs once
they carry an id. The ``verdict``, ``keyring_type`` and ``keyring_serial`` fields
let a policy distinguish, for example, "verified and signed by a builtin key"
from "verified by a user key". A policy LSM such as IPE could consume the same
let a policy distinguish "verified against the operator's bpf keyring" from
"verified against a keyring the loader supplied itself", which is the
distinction that matters most. A policy LSM such as IPE could consume the same
hooks to enforce system policy without writing any BPF, though none implements
this today.
@ -390,33 +406,158 @@ Keyrings
========
``keyring_id`` selects the trusted keyring the PKCS#7 signature is verified
against. The well-known ids ``0`` (builtin), ``VERIFY_USE_SECONDARY_KEYRING``
and ``VERIFY_USE_PLATFORM_KEYRING`` select the corresponding system keyrings;
any other value is treated as the serial of a user/session key or keyring.
The keyring is looked up first, before the signature bytes are examined, so a
signature naming a non-existent keyring is rejected up front, and a failed
verification aborts the load - so a program that loads successfully with a
signature always has consistent keyring fields recorded.
against. Four values are well-known; anything else is resolved as a caller-
supplied key or keyring, named either by its serial or by one of the other
``KEY_SPEC_*`` ids:
.. list-table::
:header-rows: 1
* - ``keyring_id``
- Keyring
* - ``0``
- builtin trusted keyring
* - ``VERIFY_USE_SECONDARY_KEYRING`` (``1``)
- secondary trusted keyring
* - ``VERIFY_USE_PLATFORM_KEYRING`` (``2``)
- platform keyring
* - ``KEY_SPEC_BPF_KEYRING`` (``-9``)
- the bpf keyring
* - anything else
- a caller-supplied user/session key or keyring
The keyring is resolved first, before the signature bytes are examined, so a
signature naming a keyring that cannot be used is rejected up front, and a
failed verification aborts the load - a program that loads successfully with
a signature therefore always has consistent keyring fields recorded.
The bpf keyring
---------------
A system keyring needs a kernel rebuild or a vouched-for enrollment to rotate a
key, and grants BPF-signing trust to keys trusted for everything else in the
kernel too. A caller-supplied keyring, at the other extreme, is filled by the
very process that loads the program and so carries no trust of its own.
The bpf keyring fills that gap and is the trust anchor which a signed BPF
deployment should be built on top of: a keyring named ``.bpf``, selected with
``KEY_SPEC_BPF_KEYRING``, that an operator provisions at boot with a key
scoped to BPF program loading and nothing else in the kernel's trust hierarchy.
It is owned by the operator rather than by the loader, and rotatable across a
reboot without touching the kernel image. It is modelled after the dm-verity
keyring (see ``dm_verity.keyring_unsealed=``) and provisioned the same way: an
initrd runs the ``keyctl`` steps below before handing off to the rootfs.
Provisioning
~~~~~~~~~~~~
The keyring is created during ``late_initcall`` and is **sealed empty** by
default: it carries a reject-all restriction, so no key can ever be added and
``KEY_SPEC_BPF_KEYRING`` fails with ``-ENOKEY`` for the whole boot.
``bpf.keyring_unsealed=1`` leaves it unrestricted at init so the initrd can
provision it. The keyring is not linked into any process keyring, but the
``KEY_SPEC_BPF_KEYRING`` special key id addresses it directly, so its serial
does not have to be looked up first. Steps would be as follows::
keyctl padd asymmetric "" -9 < signing_key.der
keyctl restrict_keyring -9
Both steps are required: the keyring is consulted only once it is **non-empty
and restricted**. An unrestricted keyring is ignored even when it holds keys,
so a half-provisioned keyring is inert rather than a weaker trust anchor, and a
load naming it fails with ``-ENOKEY`` and a verifier log. Restricting cannot
be undone.
More than one key is enrolled by repeating the ``keyctl padd`` step; the
restriction is applied once, after the last of them::
for key in /etc/bpf/keys/*.der; do
keyctl padd asymmetric "" -9 < $key
done
keyctl restrict_keyring -9
keyctl show -9
The restriction bounds what can be added, never what can be taken away. A key
that is already enrolled can still be unlinked, and the keyring cleared or
revoked, by anything running as root. That does not weaken the anchor, since
a keyring left empty is no longer consulted and a load naming it fails with
``-ENOKEY``, but it does take signed loading out until the next boot. Dropping
the user permissions the keyring no longer needs would close that; as a third
step in the initrd::
keyctl setperm -9 0x08030000
What remains is ``KEY_POS_SEARCH`` for the in-kernel search during verification,
plus ``KEY_USR_VIEW`` and ``KEY_USR_READ`` so the keyring stays visible in
``/proc/keys``. ``KEY_SPEC_BPF_KEYRING`` names the keyring but conveys no rights
of its own - the keyring is not possessed by anyone, so the ``KEY_POS_*`` bits
are only ever reached by the in-kernel search, and userspace is left with what
the ``KEY_USR_*`` bits grant. Once they are dropped, addressing it by the serial
``/proc/keys`` reports gets no further than the special id does.
Provisioning has to complete before control passes to the rootfs. The keyring
takes any number of keys for as long as it is unrestricted, so it is the
restriction that bounds the enrolled set, not the first enrollment. Before
the initrd hands off control, it must therefore restrict the keyring after
the last enrollment.
Enforcement
~~~~~~~~~~~
``bpf.keyring_unsealed=1`` states that the bpf keyring is *the* trust anchor for
this boot, so it does more than unseal. From the first program load onwards a
caller-supplied user/session keyring is refused with ``-EPERM`` and a verifier
log message, whether or not provisioning ever completed. The system keyrings
stay selectable.
Enforcement is readable at ``/sys/module/bpf/parameters/keyring_unsealed``, and
read-only there: the flag is ``__ro_after_init`` behind a 0444 parameter. It is
also derived from the boot flag rather than from the keyring's runtime state,
so there is no window early in boot during which a caller-supplied keyring is
still accepted.
Caller-supplied keyrings are for staging
----------------------------------------
A ``keyring_id`` naming a user or session key or keyring is a *staging*
mechanism, not a trust anchor: it is filled by the same userspace that loads the
program, so verifying against it establishes only that the loader signed what it
loaded. Its purpose is to let software installed onto a running system - whose
signing key is not enrolled anywhere yet - run signed until that key reaches the
bpf keyring on the next boot.
A system that has committed to the bpf keyring refuses this path outright (see
`Enforcement`_). A system that has not can still allow it, but a policy must
never treat ``BPF_SIG_KEYRING_USER`` as equivalent to the bpf or system
keyrings; it should allowlist the specific serials it is willing to stage and
pair that with a BPF LSM policy protecting those keys from tampering, as in
`Enforcement via LSMs`_.
Recorded fields
---------------
Two fields are recorded in ``prog->aux->sig`` for an LSM to inspect:
``keyring_type`` (``enum bpf_sig_keyring``)
Classified purely from ``keyring_id`` whenever the program is signed:
``BPF_SIG_KEYRING_BUILTIN``, ``_SECONDARY``, ``_PLATFORM`` for the system
keyrings, or ``_USER`` for a user/session keyring. It is
``BPF_SIG_KEYRING_NONE`` for an unsigned program.
keyrings, ``_BPF`` for the bpf keyring, or ``_USER`` for a caller-supplied
user/session keyring. It is ``BPF_SIG_KEYRING_NONE`` for an unsigned
program.
``keyring_serial`` (``s32``)
Set **only** on a successful verification, to the serial of the
**user/session key or keyring** that ``keyring_id`` resolved to - the
**caller-supplied key or keyring** that ``keyring_id`` resolved to - the
object the signature was verified against, not the individual asymmetric
key inside it that matched the signer. Passing
``KEY_SPEC_SESSION_KEYRING``, for example, records the session keyring's
serial. The system keyrings are trusted as a whole and expose no serial
here, so the serial is ``0`` for builtin, secondary and platform
signatures, and ``0`` for unsigned programs. In other words, a non-zero
``keyring_serial`` is exactly "verified against the user key/keyring with
this serial".
serial. The system keyrings and the bpf keyring are trusted as a whole and
expose no serial here, so the serial is ``0`` for them, and ``0`` for
unsigned programs. A non-zero ``keyring_serial`` is therefore exactly
"verified against the caller-supplied key/keyring with this serial", which
is exactly the case a policy has to scrutinise.
.. list-table::
:header-rows: 1
@ -436,16 +577,51 @@ Two fields are recorded in ``prog->aux->sig`` for an LSM to inspect:
* - ``VERIFY_USE_PLATFORM_KEYRING``
- ``BPF_SIG_KEYRING_PLATFORM``
- ``0``
* - other (a user/session key serial)
* - ``KEY_SPEC_BPF_KEYRING``
- ``BPF_SIG_KEYRING_BPF``
- ``0``
* - other (a caller-supplied key serial)
- ``BPF_SIG_KEYRING_USER``
- serial of the resolved key/keyring
Producing a signed object
==========================
Producing and loading a signed object
=====================================
Generating a signing key
------------------------
Signing is algorithm agnostic: the algorithm comes from the X.509 certificate
and the PKCS#7 ``SignerInfo``. Anything the X.509 and PKCS#7 parsers understand
works with no BPF-side change. Both examples below read the certificate request
config from ``x509.genkey``, for which ``certs/x509.genkey`` serves as a
template (see Documentation/admin-guide/module-signing.rst). RSA::
openssl req -new -nodes -utf8 -sha256 -days 36500 -batch -x509 \
-config x509.genkey -outform PEM \
-out signing_key.pem -keyout signing_key.pem
openssl x509 -in signing_key.pem -outform der -out signing_key.der
ML-DSA-87 (FIPS-204), which needs openssl 3.5 or later, and both
``CONFIG_CRYPTO_MLDSA`` and ``CONFIG_CRYPTO_SHA512`` in the kernel - the latter
for the signedAttrs digest described below, which ``CONFIG_CRYPTO_MLDSA`` does
not select. Note the absence of a digest option: ML-DSA hashes the message
itself, so openssl ignores an explicit digest for it::
openssl req -new -nodes -utf8 -days 36500 -batch -x509 \
-newkey ML-DSA-87 -config x509.genkey -outform PEM \
-out signing_key.pem -keyout signing_key.pem
openssl x509 -in signing_key.pem -outform der -out signing_key.der
``bpftool`` handles the following internally: openssl before 4.0 cannot combine
ML-DSA with ``CMS_NOATTR``, so it falls back to signedAttrs, where only SHA-512
is permitted. This mirrors what module signing does as well.
Signing
-------
``bpftool`` generates and signs a light skeleton in one step::
bpftool gen skeleton -L -S -k <private_key.pem> -i <certificate.x509> \
bpftool gen skeleton -L -S -k signing_key.pem -i signing_key.der \
obj.bpf.o > obj.lskel.h
``-L`` selects the light-skeleton (``gen_loader``) backend and ``-S`` enables
@ -454,12 +630,36 @@ signing; ``-k`` and ``-i`` supply the signing key and its X.509 certificate.
reconstructs - and also computes ``excl_prog_hash`` as the digest of the loader
instructions so the metadata map can be bound to the loader. The signature and
hash are embedded in the generated header; the certificate is used only for
signing and is not included. Loading the skeleton performs the
create/populate/freeze/load sequence described above.
signing and is not included.
At runtime the trusted public key must be present in the chosen keyring (for
example added to the session keyring, or built into the kernel's builtin trusted
keyring) for verification to succeed.
Loading
-------
The generated skeleton exposes ``keyring_id``, which selects the keyring the
kernel verifies against. Set it between open and load; loading then performs
the create/populate/freeze/load sequence described above::
struct obj *skel = obj__open();
skel->keyring_id = KEY_SPEC_BPF_KEYRING;
err = obj__load(skel);
For the staging case the same object is loaded against a keyring the caller
populated itself, which only works on a system that has not set
``bpf.keyring_unsealed=1``::
/*
* Staging only: this keyring is under the loader's own control and
* carries no trust of its own. See "Caller-supplied keyrings are for
* staging".
*/
key_id = add_key("asymmetric", "", der, der_sz, KEY_SPEC_SESSION_KEYRING);
skel->keyring_id = KEY_SPEC_SESSION_KEYRING;
err = obj__load(skel);
Either way the trusted public key must already be in the chosen keyring for
verification to succeed. For the bpf keyring that enrollment happens once at
boot, see `Provisioning`_.
UAPI reference
==============
@ -487,6 +687,13 @@ UAPI reference
The map content is not hashed separately at all - it is covered, as bytes,
by the program signature.
Kernel command line:
``bpf.keyring_unsealed=``
Set to ``1`` to leave the bpf keyring unsealed for provisioning, and to make
it the only non-system keyring a loader may select for the rest of the boot
(see `The bpf keyring`_).
Notes and limitations
======================
@ -495,3 +702,5 @@ Notes and limitations
exceed it.
- The metadata container is a single-element array map, accessed through
``map_direct_value_addr``.
- The bpf keyring needs ``CONFIG_KEYS``; without it there is no bpf keyring
and ``KEY_SPEC_BPF_KEYRING`` never resolves.

5
arch/arm64/net/bpf_jit_comp.c
View File

@ -2388,6 +2388,11 @@ bool bpf_jit_supports_kfunc_call(void)
return true;
}
bool bpf_jit_supports_kfunc_ret_reg_pair(void)
{
return true;
}
bool bpf_jit_supports_stack_args(void)
{
return true;

4
arch/riscv/net/bpf_jit.h
View File

@ -12,6 +12,9 @@
#include <linux/bpf.h>
#include <linux/filter.h>
#define RV_MAX_REG_ARGS 8
#define RV_EXTRA_STK_ARGS (RV_MAX_REG_ARGS - MAX_BPF_FUNC_REG_ARGS)
/* verify runtime detection extension status */
#define rv_ext_enabled(ext) \
(IS_ENABLED(CONFIG_RISCV_ISA_##ext) && riscv_has_extension_likely(RISCV_ISA_EXT_##ext))
@ -82,6 +85,7 @@ struct rv_jit_context {
unsigned long flags;
int stack_size;
int tcc_offset;
int stack_arg_sz;
u64 arena_vm_start;
u64 user_vm_start;
};

109
arch/riscv/net/bpf_jit_comp64.c
View File

@ -15,7 +15,6 @@
#include <asm/percpu.h>
#include "bpf_jit.h"
#define RV_MAX_REG_ARGS 8
#define RV_FENTRY_NINSNS 2
#define RV_FENTRY_NBYTES (RV_FENTRY_NINSNS * 4)
/* imm that allows emit_imm to emit max count insns */
@ -23,11 +22,11 @@
/* fentry and TCC init insns will be skipped on tailcall */
#define RV_TAILCALL_OFFSET ((RV_FENTRY_NINSNS + 1) * 4)
#define RV_REG_TCC RV_REG_A6
#define RV_REG_TCC RV_REG_T5
#define RV_REG_ARENA RV_REG_S7 /* For storing arena_vm_start */
static const int regmap[] = {
[BPF_REG_0] = RV_REG_A5,
[BPF_REG_0] = RV_REG_T6,
[BPF_REG_1] = RV_REG_A0,
[BPF_REG_2] = RV_REG_A1,
[BPF_REG_3] = RV_REG_A2,
@ -47,13 +46,13 @@ static const int pt_regmap[] = {
[RV_REG_A2] = offsetof(struct pt_regs, a2),
[RV_REG_A3] = offsetof(struct pt_regs, a3),
[RV_REG_A4] = offsetof(struct pt_regs, a4),
[RV_REG_A5] = offsetof(struct pt_regs, a5),
[RV_REG_S1] = offsetof(struct pt_regs, s1),
[RV_REG_S2] = offsetof(struct pt_regs, s2),
[RV_REG_S3] = offsetof(struct pt_regs, s3),
[RV_REG_S4] = offsetof(struct pt_regs, s4),
[RV_REG_S5] = offsetof(struct pt_regs, s5),
[RV_REG_T0] = offsetof(struct pt_regs, t0),
[RV_REG_T6] = offsetof(struct pt_regs, t6),
};
enum {
@ -239,7 +238,7 @@ static void __build_epilogue(bool is_tail_call, struct rv_jit_context *ctx)
emit_addi(RV_REG_SP, RV_REG_SP, stack_adjust, ctx);
/* Set return value. */
if (!is_tail_call)
emit_addiw(RV_REG_A0, RV_REG_A5, 0, ctx);
emit_addiw(RV_REG_A0, regmap[BPF_REG_0], 0, ctx);
emit_jalr(RV_REG_ZERO, is_tail_call ? RV_REG_T3 : RV_REG_RA,
is_tail_call ? RV_TAILCALL_OFFSET : 0, ctx);
}
@ -498,6 +497,18 @@ static void emit_ldx(u8 rd, s16 off, u8 rs, u8 size, bool sign_ext,
ctx->ex_jmp_off = ctx->ninsns;
}
static void emit_stack_arg_ldx(u8 rd, s16 off, struct rv_jit_context *ctx)
{
int idx = off / 8 - 1;
if (idx < RV_EXTRA_STK_ARGS) {
emit_mv(rd, RV_REG_A5 + idx, ctx);
return;
}
emit_ldx_insn(rd, (idx - RV_EXTRA_STK_ARGS) * 8, RV_REG_FP, BPF_DW, false, ctx);
}
static void emit_st(u8 rd, s16 off, s32 imm, u8 size, struct rv_jit_context *ctx)
{
emit_imm(RV_REG_T1, imm, ctx);
@ -515,6 +526,19 @@ static void emit_st(u8 rd, s16 off, s32 imm, u8 size, struct rv_jit_context *ctx
ctx->ex_jmp_off = ctx->ninsns;
}
static void emit_stack_arg_st(s16 off, s32 imm, struct rv_jit_context *ctx)
{
int idx = -off / 8 - 1;
if (idx < RV_EXTRA_STK_ARGS) {
emit_imm(RV_REG_A5 + idx, imm, ctx);
return;
}
emit_imm(RV_REG_T1, imm, ctx);
emit_stx_insn(RV_REG_SP, (idx - RV_EXTRA_STK_ARGS) * 8, RV_REG_T1, BPF_DW, ctx);
}
static void emit_stx(u8 rd, s16 off, u8 rs, u8 size, struct rv_jit_context *ctx)
{
if (is_12b_int(off)) {
@ -531,6 +555,18 @@ static void emit_stx(u8 rd, s16 off, u8 rs, u8 size, struct rv_jit_context *ctx)
ctx->ex_jmp_off = ctx->ninsns;
}
static void emit_stack_arg_stx(s16 off, u8 rs, struct rv_jit_context *ctx)
{
int idx = -off / 8 - 1;
if (idx < RV_EXTRA_STK_ARGS) {
emit_mv(RV_REG_A5 + idx, rs, ctx);
return;
}
emit_stx_insn(RV_REG_SP, (idx - RV_EXTRA_STK_ARGS) * 8, rs, BPF_DW, ctx);
}
static int emit_atomic_ld_st(u8 rd, u8 rs, const struct bpf_insn *insn,
struct rv_jit_context *ctx)
{
@ -747,6 +783,8 @@ static int add_exception_handler(const struct bpf_insn *insn, int dst_reg,
if (BPF_MODE(insn->code) != BPF_PROBE_MEM &&
BPF_MODE(insn->code) != BPF_PROBE_MEMSX &&
BPF_MODE(insn->code) != BPF_PROBE_MEM32 &&
!(BPF_MODE(insn->code) == BPF_PROBE_MEM32SX &&
BPF_CLASS(insn->code) == BPF_LDX) &&
BPF_MODE(insn->code) != BPF_PROBE_ATOMIC)
return 0;
@ -1073,7 +1111,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im,
save_ret = flags & (BPF_TRAMP_F_CALL_ORIG | BPF_TRAMP_F_RET_FENTRY_RET);
if (save_ret)
stack_size += 16; /* Save both A5 (BPF R0) and A0 */
stack_size += 16; /* Save both RV_REG_T6 (BPF R0) and RV_REG_A0 */
retval_off = stack_size;
stack_size += nr_arg_slots * 8;
@ -1822,11 +1860,21 @@ int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx,
return -EINVAL;
for (idx = 0; idx < fm->nr_args; idx++) {
u8 reg = bpf_to_rv_reg(BPF_REG_1 + idx, ctx);
bool sign = fm->arg_flags[idx] & BTF_FMODEL_SIGNED_ARG;
u8 arg_sz = fm->arg_size[idx];
if (sign_extend(reg, reg, fm->arg_size[idx], sign, ctx))
return -EINVAL;
if (arg_sz == 8 || (arg_sz != 4 && !sign))
continue;
if (idx < RV_MAX_REG_ARGS) {
if (sign_extend(RV_REG_A0 + idx, RV_REG_A0 + idx, arg_sz, sign, ctx))
return -EINVAL;
} else {
emit_ld(RV_REG_T1, (idx - RV_MAX_REG_ARGS) * 8, RV_REG_SP, ctx);
if (sign_extend(RV_REG_T1, RV_REG_T1, arg_sz, sign, ctx))
return -EINVAL;
emit_sd(RV_REG_SP, (idx - RV_MAX_REG_ARGS) * 8, RV_REG_T1, ctx);
}
}
}
@ -1908,18 +1956,27 @@ int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx,
case BPF_LDX | BPF_PROBE_MEM32 | BPF_H:
case BPF_LDX | BPF_PROBE_MEM32 | BPF_W:
case BPF_LDX | BPF_PROBE_MEM32 | BPF_DW:
/* LDX | PROBE_MEM32SX: dst = *(signed size *)(src + RV_REG_ARENA + off) */
case BPF_LDX | BPF_PROBE_MEM32SX | BPF_B:
case BPF_LDX | BPF_PROBE_MEM32SX | BPF_H:
case BPF_LDX | BPF_PROBE_MEM32SX | BPF_W:
{
bool sign_ext;
sign_ext = BPF_MODE(insn->code) == BPF_MEMSX ||
BPF_MODE(insn->code) == BPF_PROBE_MEMSX;
BPF_MODE(insn->code) == BPF_PROBE_MEMSX ||
BPF_MODE(insn->code) == BPF_PROBE_MEM32SX;
if (BPF_MODE(insn->code) == BPF_PROBE_MEM32) {
if (BPF_MODE(insn->code) == BPF_PROBE_MEM32 ||
BPF_MODE(insn->code) == BPF_PROBE_MEM32SX) {
emit_add(RV_REG_T2, rs, RV_REG_ARENA, ctx);
rs = RV_REG_T2;
}
emit_ldx(rd, off, rs, BPF_SIZE(code), sign_ext, ctx);
if (is_stack_arg_ldx(insn))
emit_stack_arg_ldx(rd, off, ctx);
else
emit_ldx(rd, off, rs, BPF_SIZE(code), sign_ext, ctx);
ret = add_exception_handler(insn, rd, ctx);
if (ret)
@ -1949,7 +2006,10 @@ int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx,
rd = RV_REG_T3;
}
emit_st(rd, off, imm, BPF_SIZE(code), ctx);
if (is_stack_arg_st(insn))
emit_stack_arg_st(off, imm, ctx);
else
emit_st(rd, off, imm, BPF_SIZE(code), ctx);
ret = add_exception_handler(insn, REG_DONT_CLEAR_MARKER, ctx);
if (ret)
@ -1971,7 +2031,10 @@ int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx,
rd = RV_REG_T2;
}
emit_stx(rd, off, rs, BPF_SIZE(code), ctx);
if (is_stack_arg_stx(insn))
emit_stack_arg_stx(off, rs, ctx);
else
emit_stx(rd, off, rs, BPF_SIZE(code), ctx);
ret = add_exception_handler(insn, REG_DONT_CLEAR_MARKER, ctx);
if (ret)
@ -2021,9 +2084,9 @@ int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx,
void bpf_jit_build_prologue(struct rv_jit_context *ctx, bool is_subprog)
{
int i, stack_adjust = 0, store_offset, bpf_stack_adjust;
int i, stack_adjust = 0, store_offset, bpf_stack_adjust = ctx->stack_arg_sz;
bpf_stack_adjust = round_up(ctx->prog->aux->stack_depth, STACK_ALIGN);
bpf_stack_adjust += round_up(ctx->prog->aux->stack_depth, STACK_ALIGN);
if (bpf_stack_adjust)
mark_fp(ctx);
@ -2121,6 +2184,11 @@ bool bpf_jit_supports_kfunc_call(void)
return true;
}
bool bpf_jit_supports_kfunc_ret_reg_pair(void)
{
return true;
}
bool bpf_jit_supports_ptr_xchg(void)
{
return true;
@ -2148,10 +2216,6 @@ bool bpf_jit_supports_insn(struct bpf_insn *insn, bool in_arena)
if (insn->imm == BPF_CMPXCHG)
return rv_ext_enabled(ZACAS);
break;
case BPF_LDX | BPF_MEMSX | BPF_B:
case BPF_LDX | BPF_MEMSX | BPF_H:
case BPF_LDX | BPF_MEMSX | BPF_W:
return false;
}
}
@ -2189,3 +2253,8 @@ bool bpf_jit_supports_timed_may_goto(void)
{
return true;
}
bool bpf_jit_supports_stack_args(void)
{
return true;
}

7
arch/riscv/net/bpf_jit_core.c
View File

@ -48,6 +48,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_pr
int pass = 0, prev_ninsns = 0, i;
struct rv_jit_data *jit_data;
struct rv_jit_context *ctx;
u16 stack_arg_cnt;
if (!prog->jit_requested)
return prog;
@ -71,6 +72,12 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_pr
ctx->arena_vm_start = bpf_arena_get_kern_vm_start(prog->aux->arena);
ctx->user_vm_start = bpf_arena_get_user_vm_start(prog->aux->arena);
stack_arg_cnt = bpf_out_stack_arg_cnt(env, prog);
/* First 3 stack args in regs, rest on stack */
ctx->stack_arg_sz = stack_arg_cnt > RV_EXTRA_STK_ARGS ?
round_up((stack_arg_cnt - RV_EXTRA_STK_ARGS) * 8, STACK_ALIGN) : 0;
ctx->prog = prog;
ctx->offset = kvzalloc_objs(int, prog->len);
if (!ctx->offset)

8
arch/riscv/net/bpf_timed_may_goto.S
View File

@ -10,7 +10,7 @@
* - output: updated count in BPF_REG_AX (t0)
*
* Calls bpf_check_timed_may_goto(ptr) with the standard RISC-V ABI, where
* ptr = BPF_REG_FP (s5) + BPF_REG_AX (t0). BPF R0-R5 (a5, a0-a4) are saved
* ptr = BPF_REG_FP (s5) + BPF_REG_AX (t0). BPF R0-R5 (t6, a0-a4) are saved
* across the call; BPF_REG_FP (s5) is callee-saved and needs no saving.
*/
@ -20,8 +20,8 @@ SYM_FUNC_START(arch_bpf_timed_may_goto)
REG_S s0, 6*SZREG(sp)
addi s0, sp, 8*SZREG
/* Save BPF registers R0-R5 (a5, a0-a4) */
REG_S a5, 5*SZREG(sp)
/* Save BPF registers R0-R5 (t6, a0-a4) */
REG_S t6, 5*SZREG(sp)
REG_S a0, 4*SZREG(sp)
REG_S a1, 3*SZREG(sp)
REG_S a2, 2*SZREG(sp)
@ -38,7 +38,7 @@ SYM_FUNC_START(arch_bpf_timed_may_goto)
REG_L a2, 2*SZREG(sp)
REG_L a1, 3*SZREG(sp)
REG_L a0, 4*SZREG(sp)
REG_L a5, 5*SZREG(sp)
REG_L t6, 5*SZREG(sp)
REG_L s0, 6*SZREG(sp)
REG_L ra, 7*SZREG(sp)

1
arch/x86/Kconfig
View File

@ -236,6 +236,7 @@ config X86
select HAVE_SAMPLE_FTRACE_DIRECT if X86_64
select HAVE_SAMPLE_FTRACE_DIRECT_MULTI if X86_64
select HAVE_EBPF_JIT
select HAVE_EBPF_JIT_KASAN if X86_64
select HAVE_EFFICIENT_UNALIGNED_ACCESS
select HAVE_EISA if X86_32
select HAVE_EXIT_THREAD

295
arch/x86/net/bpf_jit_comp.c
View File

@ -22,6 +22,17 @@
#include <asm/unwind.h>
#include <asm/cfi.h>
#if IS_ENABLED(CONFIG_BPF_JIT_KASAN)
void __asan_load1(void *p);
void __asan_store1(void *p);
void __asan_load2(void *p);
void __asan_store2(void *p);
void __asan_load4(void *p);
void __asan_store4(void *p);
void __asan_load8(void *p);
void __asan_store8(void *p);
#endif
static bool all_callee_regs_used[4] = {true, true, true, true};
static u8 *emit_code(u8 *ptr, u32 bytes, unsigned int len)
@ -1111,6 +1122,93 @@ static void maybe_emit_1mod(u8 **pprog, u32 reg, bool is64)
*pprog = prog;
}
static int emit_kasan_check(struct bpf_verifier_env *env, u8 **pprog,
u32 addr_reg, s32 off, u32 bpf_size, u8 *ip,
bool is_write)
{
#ifdef CONFIG_BPF_JIT_KASAN
u8 *prog = *pprog;
void *kasan_func;
if (!env)
return 0;
if (addr_reg == BPF_REG_FP || addr_reg == BPF_REG_PARAMS)
return 0;
/* Derive KASAN check function from access type and size */
switch (bpf_size) {
case BPF_B:
kasan_func = is_write ? __asan_store1 : __asan_load1;
break;
case BPF_H:
kasan_func = is_write ? __asan_store2 : __asan_load2;
break;
case BPF_W:
kasan_func = is_write ? __asan_store4 : __asan_load4;
break;
case BPF_DW:
kasan_func = is_write ? __asan_store8 : __asan_load8;
break;
default:
return -EINVAL;
}
/* Save rax */
EMIT1(0x50);
/* Save rcx */
EMIT1(0x51);
/* Save rdx */
EMIT1(0x52);
/* Save rsi */
EMIT1(0x56);
/* Save rdi */
EMIT1(0x57);
/* Save r8 */
EMIT2(0x41, 0x50);
/* Save r9 */
EMIT2(0x41, 0x51);
/* Save r10 */
EMIT2(0x41, 0x52);
/* Save r11 */
EMIT2(0x41, 0x53);
/* mov rdi, addr_reg */
EMIT_mov(BPF_REG_1, addr_reg);
/* add rdi, off (if offset is non-zero) */
if (off) {
if (is_imm8(off)) {
/* add rdi, imm8 */
EMIT4(0x48, 0x83, 0xC7, (u8)off);
} else {
/* add rdi, imm32 */
EMIT3_off32(0x48, 0x81, 0xC7, off);
}
}
/* Adjust ip to account for the instrumentation generated so far */
ip += (prog - *pprog);
/* We emit a call, so update call depth counting */
ip += x86_call_depth_emit_accounting(&prog, kasan_func, ip);
/* call kasan_func */
if (emit_call(&prog, kasan_func, ip))
return -ERANGE;
EMIT2(0x41, 0x5B);
EMIT2(0x41, 0x5A);
EMIT2(0x41, 0x59);
EMIT2(0x41, 0x58);
EMIT1(0x5F);
EMIT1(0x5E);
EMIT1(0x5A);
EMIT1(0x59);
EMIT1(0x58);
*pprog = prog;
#endif /* CONFIG_BPF_JIT_KASAN */
return 0;
}
/* LDX: dst_reg = *(u8*)(src_reg + off) */
static void emit_ldx(u8 **pprog, u32 size, u32 dst_reg, u32 src_reg, int off)
{
@ -1316,6 +1414,46 @@ static void emit_st_index(u8 **pprog, u32 size, u32 dst_reg, u32 index_reg, int
*pprog = prog;
}
/* ST: *(u8*)(dst_reg + off) = imm */
static void emit_st(u8 **pprog, struct bpf_insn *insn, u32 dst_reg,
s32 insn_off)
{
s32 imm32 = insn->imm;
u8 *prog = *pprog;
switch (BPF_SIZE(insn->code)) {
case BPF_B:
if (is_ereg(dst_reg))
EMIT2(0x41, 0xC6);
else
EMIT1(0xC6);
break;
case BPF_H:
if (is_ereg(dst_reg))
EMIT3(0x66, 0x41, 0xC7);
else
EMIT2(0x66, 0xC7);
break;
case BPF_W:
if (is_ereg(dst_reg))
EMIT2(0x41, 0xC7);
else
EMIT1(0xC7);
break;
case BPF_DW:
EMIT2(add_1mod(0x48, dst_reg), 0xC7);
break;
}
if (is_imm8(insn_off))
EMIT2(add_1reg(0x40, dst_reg), insn_off);
else
EMIT1_off32(add_1reg(0x80, dst_reg), insn_off);
EMIT(imm32, bpf_size_to_x86_bytes(BPF_SIZE(insn->code)));
*pprog = prog;
}
static void emit_st_r12(u8 **pprog, u32 size, u32 dst_reg, int off, int imm)
{
emit_st_index(pprog, size, dst_reg, X86_REG_R12, off, imm);
@ -1424,17 +1562,35 @@ static int emit_atomic_rmw_index(u8 **pprog, u32 atomic_op, u32 size,
return 0;
}
static int emit_atomic_ld_st(u8 **pprog, u32 atomic_op, u32 dst_reg,
u32 src_reg, s16 off, u8 bpf_size)
static int emit_atomic_ld_st(struct bpf_verifier_env *env, u8 **pprog,
struct bpf_insn *insn, u8 *ip, u32 dst_reg,
u32 src_reg, bool accesses_stack_only)
{
u32 atomic_op = insn->imm;
int err;
switch (atomic_op) {
case BPF_LOAD_ACQ:
if (!accesses_stack_only) {
err = emit_kasan_check(env, pprog, src_reg, insn->off,
BPF_SIZE(insn->code), ip, false);
if (err)
return err;
}
/* dst_reg = smp_load_acquire(src_reg + off16) */
emit_ldx(pprog, bpf_size, dst_reg, src_reg, off);
emit_ldx(pprog, BPF_SIZE(insn->code), dst_reg, src_reg,
insn->off);
break;
case BPF_STORE_REL:
if (!accesses_stack_only) {
err = emit_kasan_check(env, pprog, dst_reg, insn->off,
BPF_SIZE(insn->code), ip, true);
if (err)
return err;
}
/* smp_store_release(dst_reg + off16, src_reg) */
emit_stx(pprog, bpf_size, dst_reg, src_reg, off);
emit_stx(pprog, BPF_SIZE(insn->code), dst_reg, src_reg,
insn->off);
break;
default:
pr_err("bpf_jit: unknown atomic load/store opcode %02x\n",
@ -1690,17 +1846,12 @@ static int emit_spectre_bhb_barrier(u8 **pprog, u8 *ip,
* arena NULL is offset 0. Return the number of emitted bytes.
*/
static int emit_kfunc_arena_args(struct bpf_prog *bpf_prog,
const struct bpf_insn *insn, u8 **pprog)
const struct btf_func_model *fm, u8 **pprog)
{
const struct btf_func_model *fm;
u8 *prog = *pprog;
u8 *start = prog;
int i;
fm = bpf_jit_find_kfunc_model(bpf_prog, insn);
if (!fm)
return -EINVAL;
for (i = 0; i < min_t(int, fm->nr_args, MAX_BPF_FUNC_REG_ARGS); i++) {
u8 flags = fm->arg_flags[i];
u32 reg = BPF_REG_1 + i;
@ -1860,10 +2011,12 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
const s32 imm32 = insn->imm;
u32 dst_reg = insn->dst_reg;
u32 src_reg = insn->src_reg;
bool accesses_stack_only;
u8 b2 = 0, b3 = 0;
u8 *start_of_ldx;
s64 jmp_offset;
s32 insn_off;
int insn_idx;
u8 jmp_cond;
u8 *func;
int nops;
@ -1880,6 +2033,10 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
EMIT_ENDBR();
ip = image + addrs[i - 1] + (prog - temp);
insn_idx = i - 1 + bpf_prog->aux->subprog_start;
accesses_stack_only =
env ? !env->insn_aux_data[insn_idx].non_stack_access :
false;
switch (insn->code) {
/* ALU */
@ -2256,49 +2413,37 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
EMIT_LFENCE();
break;
/* ST: *(u8*)(dst_reg + off) = imm */
case BPF_ST | BPF_MEM | BPF_B:
if (is_ereg(dst_reg))
EMIT2(0x41, 0xC6);
else
EMIT1(0xC6);
goto st;
case BPF_ST | BPF_MEM | BPF_H:
if (is_ereg(dst_reg))
EMIT3(0x66, 0x41, 0xC7);
else
EMIT2(0x66, 0xC7);
goto st;
case BPF_ST | BPF_MEM | BPF_W:
if (is_ereg(dst_reg))
EMIT2(0x41, 0xC7);
else
EMIT1(0xC7);
goto st;
case BPF_ST | BPF_MEM | BPF_DW:
if (dst_reg == BPF_REG_PARAMS && insn->off == -8) {
/* Arg 6: store immediate in r9 register */
emit_mov_imm64(&prog, X86_REG_R9, imm32 >> 31, (u32)imm32);
emit_mov_imm64(&prog, X86_REG_R9, imm32 >> 31,
imm32);
break;
}
EMIT2(add_1mod(0x48, dst_reg), 0xC7);
st: insn_off = insn->off;
insn_off = insn->off;
if (dst_reg == BPF_REG_PARAMS) {
/*
* Args 7+: reverse BPF negative offsets to
* x86 positive rsp offsets.
* BPF off=-16 → [rsp+0], off=-24 → [rsp+8], ...
*/
insn_off = outgoing_arg_base - outgoing_rsp - insn_off - 16;
insn_off = outgoing_arg_base - outgoing_rsp -
insn_off - 16;
dst_reg = BPF_REG_FP;
}
if (is_imm8(insn_off))
EMIT2(add_1reg(0x40, dst_reg), insn_off);
else
EMIT1_off32(add_1reg(0x80, dst_reg), insn_off);
if (!accesses_stack_only) {
err = emit_kasan_check(env, &prog, dst_reg,
insn_off,
BPF_SIZE(insn->code), ip,
true);
if (err)
return err;
}
EMIT(imm32, bpf_size_to_x86_bytes(BPF_SIZE(insn->code)));
emit_st(&prog, insn, dst_reg, insn_off);
break;
/* STX: *(u8*)(dst_reg + off) = src_reg */
@ -2316,6 +2461,14 @@ st: insn_off = insn->off;
insn_off = outgoing_arg_base - outgoing_rsp - insn_off - 16;
dst_reg = BPF_REG_FP;
}
if (!accesses_stack_only) {
err = emit_kasan_check(env, &prog, dst_reg,
insn_off,
BPF_SIZE(insn->code), ip,
true);
if (err)
return err;
}
emit_stx(&prog, BPF_SIZE(insn->code), dst_reg, src_reg, insn_off);
break;
@ -2497,6 +2650,13 @@ st: insn_off = insn->off;
/* populate jmp_offset for JAE above to jump to start_of_ldx */
start_of_ldx = prog;
end_of_jmp[-1] = start_of_ldx - end_of_jmp;
} else if (!accesses_stack_only) {
err = emit_kasan_check(env, &prog, src_reg,
insn_off,
BPF_SIZE(insn->code), ip,
false);
if (err)
return err;
}
if (BPF_MODE(insn->code) == BPF_PROBE_MEMSX ||
BPF_MODE(insn->code) == BPF_MEMSX)
@ -2558,28 +2718,42 @@ st: insn_off = insn->off;
}
fallthrough;
case BPF_STX | BPF_ATOMIC | BPF_W:
case BPF_STX | BPF_ATOMIC | BPF_DW:
if (insn->imm == (BPF_AND | BPF_FETCH) ||
insn->imm == (BPF_OR | BPF_FETCH) ||
insn->imm == (BPF_XOR | BPF_FETCH)) {
bool is64 = BPF_SIZE(insn->code) == BPF_DW;
u32 real_src_reg = src_reg;
u32 real_dst_reg = dst_reg;
u8 *branch_target;
case BPF_STX | BPF_ATOMIC | BPF_DW: {
bool is64 = BPF_SIZE(insn->code) == BPF_DW;
u32 real_src_reg = src_reg;
u32 real_dst_reg = dst_reg;
u8 *old_prog;
bool is_atomic_fetch =
(insn->imm == (BPF_AND | BPF_FETCH) ||
insn->imm == (BPF_OR | BPF_FETCH) ||
insn->imm == (BPF_XOR | BPF_FETCH));
if (is_atomic_fetch) {
/*
* Can't be implemented with a single x86 insn.
* Need to do a CMPXCHG loop.
*/
/* Will need RAX as a CMPXCHG operand so save R0 */
old_prog = prog;
emit_mov_reg(&prog, true, BPF_REG_AX, BPF_REG_0);
if (src_reg == BPF_REG_0)
real_src_reg = BPF_REG_AX;
if (dst_reg == BPF_REG_0)
real_dst_reg = BPF_REG_AX;
branch_target = prog;
ip += (prog - old_prog);
}
if (!bpf_atomic_is_load_store(insn)) {
if (!accesses_stack_only) {
err = emit_kasan_check(
env, &prog, real_dst_reg,
insn->off, BPF_SIZE(insn->code),
ip, true);
if (err)
return err;
}
}
if (is_atomic_fetch) {
u8 *branch_target = prog;
/* Load old value */
emit_ldx(&prog, BPF_SIZE(insn->code),
BPF_REG_0, real_dst_reg, insn->off);
@ -2611,15 +2785,16 @@ st: insn_off = insn->off;
}
if (bpf_atomic_is_load_store(insn))
err = emit_atomic_ld_st(&prog, insn->imm, dst_reg, src_reg,
insn->off, BPF_SIZE(insn->code));
err = emit_atomic_ld_st(env, &prog, insn, ip,
dst_reg, src_reg,
accesses_stack_only);
else
err = emit_atomic_rmw(&prog, insn->imm, dst_reg, src_reg,
insn->off, BPF_SIZE(insn->code));
if (err)
return err;
break;
}
case BPF_STX | BPF_PROBE_ATOMIC | BPF_B:
case BPF_STX | BPF_PROBE_ATOMIC | BPF_H:
if (!bpf_atomic_is_load_store(insn)) {
@ -2645,6 +2820,8 @@ st: insn_off = insn->off;
/* call */
case BPF_JMP | BPF_CALL: {
const struct btf_func_model *fm = NULL;
func = (u8 *) __bpf_call_base + imm32;
if (src_reg == BPF_PSEUDO_CALL && tail_call_reachable) {
LOAD_TAIL_CALL_CNT_PTR(stack_depth);
@ -2653,7 +2830,10 @@ st: insn_off = insn->off;
if (!imm32)
return -EINVAL;
if (src_reg == BPF_PSEUDO_KFUNC_CALL) {
err = emit_kfunc_arena_args(bpf_prog, insn, &prog);
fm = bpf_jit_find_kfunc_model(bpf_prog, insn);
if (!fm)
return -EINVAL;
err = emit_kfunc_arena_args(bpf_prog, fm, &prog);
if (err < 0)
return err;
ip += err;
@ -2667,6 +2847,14 @@ st: insn_off = insn->off;
return -EINVAL;
if (priv_frame_ptr)
pop_r9(&prog);
/*
* A kfunc returning more than 8 bytes hands the second
* half back in RDX (the native ABI's second return reg),
* but BPF expects it in R0:R2. BPF R0 is RAX (no move
* needed), while BPF R2 is RSI, so copy RDX into RSI.
*/
if (fm && fm->ret_size > 8)
emit_mov_reg(&prog, true, BPF_REG_2, BPF_REG_3);
break;
}
@ -4158,6 +4346,11 @@ bool bpf_jit_supports_kfunc_call(void)
return true;
}
bool bpf_jit_supports_kfunc_ret_reg_pair(void)
{
return true;
}
bool bpf_jit_supports_stack_args(void)
{
return true;

21
include/linux/bpf.h
View File

@ -572,7 +572,7 @@ static inline void bpf_obj_memcpy(struct btf_record *rec,
if (long_memcpy)
bpf_long_memcpy(dst, src, size);
else
memcpy(dst, src, size);
data_race(memcpy(dst, src, size));
return;
}
@ -580,10 +580,10 @@ static inline void bpf_obj_memcpy(struct btf_record *rec,
u32 next_off = rec->fields[i].offset;
u32 sz = next_off - curr_off;
memcpy(dst + curr_off, src + curr_off, sz);
data_race(memcpy(dst + curr_off, src + curr_off, sz));
curr_off += rec->fields[i].size + sz;
}
memcpy(dst + curr_off, src + curr_off, size - curr_off);
data_race(memcpy(dst + curr_off, src + curr_off, size - curr_off));
}
static inline void copy_map_value(struct bpf_map *map, void *dst, void *src)
@ -1736,6 +1736,7 @@ enum bpf_sig_keyring {
BPF_SIG_KEYRING_SECONDARY,
BPF_SIG_KEYRING_PLATFORM,
BPF_SIG_KEYRING_USER,
BPF_SIG_KEYRING_BPF,
};
struct bpf_prog_aux {
@ -3820,6 +3821,8 @@ struct bpf_key {
#if defined(CONFIG_KEYS) && defined(CONFIG_BPF_SYSCALL)
struct bpf_key *bpf_lookup_user_key(s32 serial, u64 flags);
struct bpf_key *bpf_lookup_system_key(u64 id);
struct bpf_key *bpf_lookup_keyring(void);
bool bpf_keyring_enforced(void);
void bpf_key_put(struct bpf_key *bkey);
int bpf_verify_pkcs7_signature(const struct bpf_dynptr *data_p,
const struct bpf_dynptr *sig_p,
@ -3840,6 +3843,16 @@ static inline struct bpf_key *bpf_lookup_system_key(u64 id)
return NULL;
}
static inline struct bpf_key *bpf_lookup_keyring(void)
{
return NULL;
}
static inline bool bpf_keyring_enforced(void)
{
return false;
}
static inline void bpf_key_put(struct bpf_key *bkey)
{
}
@ -4102,7 +4115,7 @@ void bpf_put_buffers(void);
void bpf_prog_stream_init(struct bpf_prog *prog);
void bpf_prog_stream_free(struct bpf_prog *prog);
int bpf_prog_stream_read(struct bpf_prog *prog, enum bpf_stream_id stream_id, void __user *buf, int len);
int bpf_prog_stream_read(struct bpf_prog *prog, enum bpf_stream_id stream_id, void __user *buf, u32 len);
void bpf_stream_stage_init(struct bpf_stream_stage *ss);
void bpf_stream_stage_free(struct bpf_stream_stage *ss);
__printf(2, 3)

32
include/linux/bpf_verifier.h
View File

@ -706,6 +706,7 @@ struct bpf_insn_aux_data {
*/
u32 calls_callback:1;
u32 indirect_target:1; /* if it is an indirect jump target */
u32 non_stack_access:1; /* instruction can access non-stack memory */
/* true if some jump or call instruction targets this instruction */
u32 jump_target:1;
/*
@ -821,6 +822,8 @@ struct bpf_subprog_info {
bool is_async_cb: 1;
bool is_exception_cb: 1;
bool args_cached: 1;
/* true if the return value is passed in the R0:R2 register pair */
bool ret_reg_pair: 1;
/* true if bpf_fastcall stack region is used by functions that can't be inlined */
bool keep_fastcall_stack: 1;
bool changes_pkt_data: 1;
@ -1057,6 +1060,11 @@ static inline struct bpf_subprog_info *subprog_info(struct bpf_verifier_env *env
return &env->subprog_info[subprog];
}
static inline bool bpf_ret_reg_pair(struct bpf_verifier_env *env, int subprog)
{
return subprog_info(env, subprog)->ret_reg_pair;
}
struct bpf_call_summary {
u8 num_params;
bool is_void;
@ -1496,6 +1504,15 @@ struct bpf_iarray *bpf_insn_successors(struct bpf_verifier_env *env, u32 idx);
void bpf_fmt_stack_mask(char *buf, ssize_t buf_sz, u64 stack_mask);
bool bpf_subprog_is_global(const struct bpf_verifier_env *env, int subprog);
/* Kinds of member a by-value struct or union may be composed of. */
enum btf_member_kind {
BTF_MEMBER_SCALAR = BIT(0), /* an int or an enum */
BTF_MEMBER_ARENA_PTR = BIT(1), /* a pointer carrying the "arena" type tag */
};
bool btf_struct_is_composed_of(struct bpf_verifier_env *env, const struct btf *btf,
const struct btf_type *t, u32 member_kinds);
int bpf_find_subprog(struct bpf_verifier_env *env, int off);
bool bpf_is_throw_kfunc(struct bpf_insn *insn);
int bpf_compute_const_regs(struct bpf_verifier_env *env);
@ -1669,6 +1686,21 @@ static inline u64 bpf_map_key_immediate(const struct bpf_insn_aux_data *aux)
return aux->map_key_state & ~(BPF_MAP_KEY_SEEN | BPF_MAP_KEY_POISON);
}
static inline bool bpf_is_mem_insn(struct bpf_insn *insn)
{
if (BPF_CLASS(insn->code) != BPF_ST &&
BPF_CLASS(insn->code) != BPF_STX &&
BPF_CLASS(insn->code) != BPF_LDX)
return false;
if (insn->code == (BPF_ST | BPF_NOSPEC))
return false;
return (BPF_MODE(insn->code) == BPF_MEM ||
BPF_MODE(insn->code) == BPF_MEMSX ||
BPF_MODE(insn->code) == BPF_ATOMIC);
}
#define MAX_PACKET_OFF 0xffff
#define CALLER_SAVED_REGS 6

1
include/linux/btf.h
View File

@ -235,6 +235,7 @@ struct btf_record *btf_parse_fields(const struct btf *btf, const struct btf_type
u32 field_mask, u32 value_size);
int btf_check_and_fixup_fields(const struct btf *btf, struct btf_record *rec);
bool btf_type_is_void(const struct btf_type *t);
bool btf_type_is_arena_ptr(const struct btf *btf, const struct btf_type *t);
s32 btf_find_by_name_kind(const struct btf *btf, const char *name, u8 kind);
s32 bpf_find_btf_id(const char *name, u32 kind, struct btf **btf_p);
struct btf *btf_get_module_btf(const struct module *module);

1
include/linux/filter.h
View File

@ -1237,6 +1237,7 @@ bool bpf_jit_inlines_helper_call(s32 imm);
bool bpf_jit_supports_subprog_tailcalls(void);
bool bpf_jit_supports_percpu_insn(void);
bool bpf_jit_supports_kfunc_call(void);
bool bpf_jit_supports_kfunc_ret_reg_pair(void);
bool bpf_jit_supports_stack_args(void);
bool bpf_jit_supports_arena_args(void);
bool bpf_jit_supports_far_kfunc_call(void);

2
include/linux/key.h
View File

@ -440,6 +440,8 @@ extern key_ref_t keyring_search(key_ref_t keyring,
extern int keyring_restrict(key_ref_t keyring, const char *type,
const char *restriction);
extern void key_register_bpf_keyring(struct key *keyring);
extern struct key *key_lookup(key_serial_t id);
static inline key_serial_t key_serial(const struct key *key)

1
include/uapi/linux/keyctl.h
View File

@ -24,6 +24,7 @@
#define KEY_SPEC_GROUP_KEYRING -6 /* - key ID for GID-specific keyring */
#define KEY_SPEC_REQKEY_AUTH_KEY -7 /* - key ID for assumed request_key auth key */
#define KEY_SPEC_REQUESTOR_KEYRING -8 /* - key ID for request_key() dest keyring */
#define KEY_SPEC_BPF_KEYRING -9 /* - key ID for the BPF-specific keyring */
/* request-key default keyrings */
#define KEY_REQKEY_DEFL_NO_CHANGE -1

17
kernel/bpf/Kconfig
View File

@ -17,6 +17,10 @@ config HAVE_CBPF_JIT
config HAVE_EBPF_JIT
bool
# KASAN support for JIT compiler
config HAVE_EBPF_JIT_KASAN
bool
# Used by archs to tell that they want the BPF JIT compiler enabled by
# default for kernels that were compiled with BPF JIT support.
config ARCH_WANT_DEFAULT_BPF_JIT
@ -101,4 +105,17 @@ config BPF_LSM
If you are unsure how to answer this question, answer N.
config BPF_JIT_KASAN
bool
depends on HAVE_EBPF_JIT_KASAN
depends on KASAN_GENERIC
depends on KASAN_VMALLOC
depends on BPF_JIT
default y
help
Makes JIT compiler insert generic outline KASAN checks in BPF
programs when they are inserted in the kernel. This feature is
automatically enabled if the needed set of KASAN and BPF
configuration options is enabled.
endmenu # "BPF subsystem"

3
kernel/bpf/Makefile
View File

@ -27,6 +27,9 @@ obj-$(CONFIG_BPF_SYSCALL) += offload.o
obj-$(CONFIG_BPF_SYSCALL) += net_namespace.o
obj-$(CONFIG_BPF_SYSCALL) += tcx.o
endif
ifeq ($(CONFIG_KEYS),y)
obj-$(CONFIG_BPF_SYSCALL) += keys.o
endif
ifeq ($(CONFIG_PERF_EVENTS),y)
obj-$(CONFIG_BPF_SYSCALL) += stackmap.o
endif

32
kernel/bpf/backtrack.c
View File

@ -423,6 +423,10 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,
*/
verifier_bug_if(idx + 1 != subseq_idx, env,
"extra insn from subprog");
/* global subprog always sets R0 */
bt_clear_reg(bt, BPF_REG_0);
/* and if it does not set R2, main pass would catch it */
bt_clear_reg(bt, BPF_REG_2);
/* r1-r5 are invalidated after subprog call,
* so for global func call it shouldn't be set
* anymore
@ -432,8 +436,6 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,
bt_reg_mask(bt));
return -EFAULT;
}
/* global subprog always sets R0 */
bt_clear_reg(bt, BPF_REG_0);
return 0;
} else {
/* static subprog call instruction, which
@ -506,6 +508,8 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,
return -ENOTSUPP;
/* regular helper call sets R0 */
bt_clear_reg(bt, BPF_REG_0);
/* kfunc might also set R2 */
bt_clear_reg(bt, BPF_REG_2);
if (bt_reg_mask(bt) & BPF_REGMASK_ARGS) {
/* if backtracking was looking for registers R1-R5
* they should have been found already.
@ -520,40 +524,50 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,
return -EFAULT;
}
} else if (opcode == BPF_EXIT) {
bool from_subprog_call, r0_precise;
bool from_subprog_call, r0_precise, r2_precise;
/* BPF_EXIT in subprog or callback always returns
* right after the call instruction, so by checking
* whether the instruction at subseq_idx-1 is subprog
* call or not we can distinguish actual exit from
* *subprog* from exit from *callback*. In the former
* case, we need to propagate r0 precision, if
* necessary. In the former we never do that.
* case, we need to propagate the precision of the
* return registers, if necessary. In the latter we
* never do that.
*/
from_subprog_call = subseq_idx - 1 >= 0 &&
bpf_pseudo_call(&env->prog->insnsi[subseq_idx - 1]);
r0_precise = from_subprog_call && bt_is_reg_set(bt, BPF_REG_0);
r2_precise = from_subprog_call && bt_is_reg_set(bt, BPF_REG_2);
/* Backtracking to a nested function call, 'idx' is a part of
* the inner frame 'subseq_idx' is a part of the outer frame.
* In case of a regular function call, instructions giving
* precision to registers R1-R5 should have been found already.
* In case of a regular function call, the callee defines the
* return registers R0 and R2, so clear them before checking
* that instructions giving precision to registers R1-R5 have
* been found already. R2 is also an argument register, hence
* it has to be cleared before that check.
* In case of a callback from bpf_loop(), R{1,4} in the calling
* frame would be set as precise and that is correct.
* frame would be set as precise and that is correct, and R2
* might be a precise helper argument as well, so leave it set.
*/
bt_clear_reg(bt, BPF_REG_0);
if (from_subprog_call)
bt_clear_reg(bt, BPF_REG_2);
if (from_subprog_call && (bt_reg_mask(bt) & BPF_REGMASK_ARGS)) {
verifier_bug(env, "backtracking exit unexpected regs %x",
bt_reg_mask(bt));
return -EFAULT;
}
bt_clear_reg(bt, BPF_REG_0);
if (bt_subprog_enter(bt))
return -EFAULT;
if (r0_precise)
bt_set_reg(bt, BPF_REG_0);
if (r2_precise)
bt_set_reg(bt, BPF_REG_2);
/* r6-r9 and stack slots will stay set in caller frame
* bitmasks until we return back from callee(s)
*/

109
kernel/bpf/btf.c
View File

@ -3538,6 +3538,22 @@ static int btf_type_tag_walk(const struct btf *btf,
return 0;
}
bool btf_type_is_arena_ptr(const struct btf *btf, const struct btf_type *t)
{
if (!btf_type_is_ptr(t))
return false;
for (t = btf_type_by_id(btf, t->type); btf_type_is_modifier(t);
t = btf_type_by_id(btf, t->type)) {
if (!btf_type_is_type_tag(t) || btf_type_kflag(t))
continue;
if (!strcmp(__btf_name_by_offset(btf, t->name_off), "arena"))
return true;
}
return false;
}
static int btf_find_kptr(const struct btf *btf, const struct btf_type *t,
u32 off, int sz, struct btf_field_info *info, u32 field_mask)
{
@ -7606,7 +7622,7 @@ int btf_distill_func_proto(struct bpf_verifier_log *log,
return -EINVAL;
}
ret = __get_type_size(btf, func->type, &t);
if (ret < 0 || btf_type_is_struct(t)) {
if (ret < 0 || ret > 16) {
bpf_log(log,
"The function %s return type %s is unsupported.\n",
tname, btf_type_str(t));
@ -7699,6 +7715,12 @@ static int btf_check_func_type_match(struct bpf_verifier_log *log,
btf_type_str(t2), fn2);
return -EINVAL;
}
if (btf_type_has_size(t1) && (t1->size > 8 || t2->size > 8)) {
bpf_log(log,
"Return type of %s() has size %u and of %s() size %u, and a size above 8 bytes cannot be replaced\n",
fn1, t1->size, fn2, t2->size);
return -EINVAL;
}
for (i = 0; i < nargs1; i++) {
t1 = btf_type_skip_modifiers(btf1, args1[i].type, NULL);
@ -7943,51 +7965,19 @@ static int btf_scan_decl_tags(struct bpf_verifier_env *env,
return 0;
}
static int btf_scan_type_tags(struct bpf_verifier_env *env,
const struct btf *btf, u32 type_id,
u32 *tags)
static void btf_scan_type_tags(const struct btf *btf, u32 type_id, u32 *tags)
{
static const struct btf_type_tag_match func_type_tags[] = {
{ "arena", ARG_TAG_ARENA },
};
struct btf_type_tag_walk_ctx ctx;
const struct btf_type *t;
int err;
/* Find the first pointer type in the chain. */
t = btf_type_skip_modifiers(btf, type_id, NULL);
const struct btf_type *t = btf_type_skip_modifiers(btf, type_id, NULL);
/*
* We currently reject type tags on non-pointer types,
* which neither LLVM nor GCC support anyway.
*/
if (!t || !btf_type_is_ptr(t))
return 0;
ctx.t = t;
err = btf_type_tag_walk(btf, &ctx, func_type_tags,
ARRAY_SIZE(func_type_tags));
if (err) {
bpf_log(&env->log,
"function signature member has multiple type tags\n");
return err;
}
*tags |= ctx.res;
return 0;
if (btf_type_is_arena_ptr(btf, t))
*tags |= ARG_TAG_ARENA;
}
/* Check whether the type is a valid return type. */
static int btf_validate_return_type(struct bpf_verifier_env *env, struct btf *btf,
const struct btf_type *t, int subprog)
const struct btf_type *t, int subprog, bool is_global)
{
u32 tags = 0;
int err;
err = btf_scan_type_tags(env, btf, t->type, &tags);
if (err)
return err;
t = btf_type_skip_modifiers(btf, t->type, NULL);
/*
@ -7995,13 +7985,36 @@ static int btf_validate_return_type(struct bpf_verifier_env *env, struct btf *bt
* General arena variables are not allowed, since it makes no sense to return by value
* a variable that's on the heap in the first place.
*/
if (subprog && (tags & ARG_TAG_ARENA) && btf_type_is_ptr(t))
if (subprog && btf_type_is_arena_ptr(btf, t))
return 0;
/* We always accept void or scalars. */
if (btf_type_is_void(t) || btf_type_is_int(t) || btf_is_any_enum(t))
return 0;
if (btf_type_is_struct(t) && t->size <= 16) {
/*
* A global function's caller models the return as an opaque
* scalar pair, so a pointer member would be laundered into a
* scalar and escape provenance and reference tracking. Only
* scalars and arena pointers are allowed: an arena pointer has
* no provenance to lose, since a program may already derive one
* from any scalar with addr_space_cast(), which confines the
* result to the arena. The main program is the exception: it
* returns to the kernel, which has no arena to cast the address
* back into. A local function is verified inline, so its caller
* receives the real register state and any member is fine.
*/
bool local_func = subprog && !is_global;
u32 member_kinds = BTF_MEMBER_SCALAR;
if (subprog)
member_kinds |= BTF_MEMBER_ARENA_PTR;
if (local_func || btf_struct_is_composed_of(env, btf, t, member_kinds))
return 0;
}
return -EOPNOTSUPP;
}
@ -8089,13 +8102,19 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog)
return -EINVAL;
}
err = btf_validate_return_type(env, btf, t, subprog);
err = btf_validate_return_type(env, btf, t, subprog, is_global);
if (err) {
if (is_global) {
/* Only a subprogram may return arena pointers. */
const char *supported = subprog ?
"void, scalar, arena pointer, or a struct/union of "
"scalars and arena pointers" :
"void, scalar, or a scalar-only struct/union";
bpf_log(log,
"Global function %s() return value not void or scalar. "
"Only those are supported.\n",
tname);
"Global function %s() has unsupported return type. "
"Only %s up to 16 bytes is supported.\n",
tname, supported);
}
return err;
}
@ -8109,9 +8128,7 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog)
if (err)
return err;
err = btf_scan_type_tags(env, btf, args[i].type, &tags);
if (err)
return err;
btf_scan_type_tags(btf, args[i].type, &tags);
t = btf_type_by_id(btf, args[i].type);
while (btf_type_is_modifier(t))

37
kernel/bpf/cnum_defs.h
View File

@ -33,7 +33,12 @@ struct cnum_t FN(from_srange)(st min, st max)
return (struct cnum_t){ .base = base, .size = size };
}
/* True if this cnum represents two unsigned ranges. */
/*
* True if this cnum represents two unsigned ranges.
*
* The caller should ensure !is_empty(cnum) holds when calling
* this function.
*/
static inline bool FN(urange_overflow)(struct cnum_t cnum)
{
/* Same as cnum.base + cnum.size > UT_MAX but avoids overflow */
@ -44,6 +49,9 @@ static inline bool FN(urange_overflow)(struct cnum_t cnum)
* cnum{T}_umin / cnum{T}_umax query an unsigned range represented by this cnum.
* If cnum represents a range crossing the UT_MAX/0 boundary, the unbound range
* [0..UT_MAX] is returned.
*
* The caller should ensure !is_empty(cnum) holds when calling
* cnum{T}_umin / cnum{T}_umax.
*/
ut FN(umin)(struct cnum_t cnum)
{
@ -57,7 +65,12 @@ ut FN(umax)(struct cnum_t cnum)
}
EXPORT_SYMBOL_GPL(FN(umax));
/* True if this cnum represents two signed ranges. */
/*
* True if this cnum represents two signed ranges.
*
* The caller should ensure !is_empty(cnum) holds when calling
* this function.
*/
static inline bool FN(srange_overflow)(struct cnum_t cnum)
{
return FN(contains)(cnum, (ut)ST_MAX) && FN(contains)(cnum, (ut)ST_MIN);
@ -67,19 +80,18 @@ static inline bool FN(srange_overflow)(struct cnum_t cnum)
* cnum{T}_smin / cnum{T}_smax query a signed range represented by this cnum.
* If cnum represents a range crossing the ST_MAX/ST_MIN boundary, the unbound range
* [ST_MIN..ST_MAX] is returned.
*
* The caller should ensure !is_empty(cnum) holds when calling
* cnum{T}_smin / cnum{T}_smax.
*/
st FN(smin)(struct cnum_t cnum)
{
return FN(srange_overflow)(cnum)
? ST_MIN
: min((st)cnum.base, (st)(cnum.base + cnum.size));
return FN(srange_overflow)(cnum) ? ST_MIN : (st)cnum.base;
}
st FN(smax)(struct cnum_t cnum)
{
return FN(srange_overflow)(cnum)
? ST_MAX
: max((st)cnum.base, (st)(cnum.base + cnum.size));
return FN(srange_overflow)(cnum) ? ST_MAX : (st)(cnum.base + cnum.size);
}
/*
@ -181,7 +193,7 @@ void FN(intersect_with_srange)(struct cnum_t *dst, st min, st max)
static inline struct cnum_t FN(normalize)(struct cnum_t cnum)
{
if (cnum.size == UT_MAX && cnum.base != 0 && cnum.base != (ut)ST_MAX)
if (cnum.size == UT_MAX)
cnum.base = 0;
return cnum;
}
@ -210,12 +222,7 @@ bool FN(is_empty)(struct cnum_t cnum)
bool FN(contains)(struct cnum_t cnum, ut v)
{
if (FN(is_empty)(cnum))
return false;
if (FN(urange_overflow)(cnum))
return v >= cnum.base || v <= (ut)cnum.base + cnum.size;
else
return v >= cnum.base && v <= (ut)cnum.base + cnum.size;
return !FN(is_empty)(cnum) && v - cnum.base <= cnum.size;
}
bool FN(is_const)(struct cnum_t cnum)

5
kernel/bpf/core.c
View File

@ -3282,6 +3282,11 @@ bool __weak bpf_jit_supports_kfunc_call(void)
return false;
}
bool __weak bpf_jit_supports_kfunc_ret_reg_pair(void)
{
return false;
}
bool __weak bpf_jit_supports_stack_args(void)
{
return false;

33
kernel/bpf/fixups.c
View File

@ -218,7 +218,8 @@ static int get_callee_stack_depth(struct bpf_verifier_env *env,
* [0, off) and [off, end) to new locations, so the patched range stays zero
*/
static void adjust_insn_aux_data(struct bpf_verifier_env *env,
struct bpf_prog *new_prog, u32 off, u32 cnt)
struct bpf_prog *new_prog, u32 off, u32 cnt,
struct bpf_insn *original_insn)
{
struct bpf_insn_aux_data *data = env->insn_aux_data;
struct bpf_insn *insn = new_prog->insnsi;
@ -232,8 +233,15 @@ static void adjust_insn_aux_data(struct bpf_verifier_env *env,
*/
data[off].zext_dst = bpf_insn_def32(new_prog, insn + off + cnt - 1) >= 0;
if (cnt == 1)
if (cnt == 1) {
/*
* A non-memory accessing insn could have been replaced by a
* memory accessing insn, systematically mark it for non-stack
* access
*/
data[off].non_stack_access = bpf_is_mem_insn(insn + off);
return;
}
prog_len = new_prog->len;
env->insn_aux_data_len = prog_len;
@ -244,8 +252,25 @@ static void adjust_insn_aux_data(struct bpf_verifier_env *env,
/* Expand insni[off]'s seen count to the patched range. */
data[i].seen = old_seen;
data[i].zext_dst = bpf_insn_def32(new_prog, insn + i) >= 0;
if (!memcmp(insn + i, original_insn, sizeof(struct bpf_insn))) {
data[i].non_stack_access =
data[off + cnt - 1].non_stack_access;
data[off + cnt - 1].non_stack_access = false;
} else if (bpf_is_mem_insn(insn + i)) {
data[i].non_stack_access = true;
}
}
/*
* Last slot instruction could be a newly generated
* BPF_ST/BPF_LDX/BPF_STX, systematically mark it for non-stack access
* if it is not the original instruction, otherwise keep the
* original marking
*/
if (bpf_is_mem_insn(insn + off + cnt - 1) &&
memcmp(insn + off + cnt - 1, original_insn, sizeof(struct bpf_insn)))
data[off + cnt - 1].non_stack_access = true;
/*
* The indirect_target flag of the original instruction was moved to the last of the
* new instructions by the above memmove and memset, but the indirect jump target is
@ -311,6 +336,7 @@ struct bpf_prog *bpf_patch_insn_data(struct bpf_verifier_env *env, u32 off,
{
struct bpf_prog *new_prog;
struct bpf_insn_aux_data *new_data = NULL;
struct bpf_insn original_insn;
if (len > 1) {
new_data = vrealloc(env->insn_aux_data,
@ -323,6 +349,7 @@ struct bpf_prog *bpf_patch_insn_data(struct bpf_verifier_env *env, u32 off,
env->insn_aux_data = new_data;
}
memcpy(&original_insn, env->prog->insnsi + off, sizeof(struct bpf_insn));
new_prog = bpf_patch_insn_single(env->prog, off, patch, len);
if (IS_ERR(new_prog)) {
if (PTR_ERR(new_prog) == -ERANGE)
@ -331,7 +358,7 @@ struct bpf_prog *bpf_patch_insn_data(struct bpf_verifier_env *env, u32 off,
env->insn_aux_data[off].orig_idx);
return NULL;
}
adjust_insn_aux_data(env, new_prog, off, len);
adjust_insn_aux_data(env, new_prog, off, len, &original_insn);
adjust_subprog_starts(env, off, len);
adjust_insn_arrays(env, off, len);
adjust_poke_descs(new_prog, off, len);

73
kernel/bpf/keys.c Normal file
View File

@ -0,0 +1,73 @@
// SPDX-License-Identifier: GPL-2.0-only
/* Copyright (c) 2026 Isovalent */
#include <linux/bpf.h>
#include <linux/cred.h>
#include <linux/err.h>
#include <linux/init.h>
#include <linux/key.h>
#include <linux/moduleparam.h>
#include <linux/slab.h>
#undef MODULE_PARAM_PREFIX
#define MODULE_PARAM_PREFIX "bpf."
static struct key *bpf_keyring;
static bool bpf_keyring_unsealed __ro_after_init;
module_param_named(keyring_unsealed, bpf_keyring_unsealed, bool, 0444);
MODULE_PARM_DESC(keyring_unsealed, "Leave the bpf keyring unsealed");
bool bpf_keyring_enforced(void)
{
return bpf_keyring_unsealed;
}
struct bpf_key *bpf_lookup_keyring(void)
{
struct bpf_key *bkey;
if (!bpf_keyring)
return NULL;
if (!READ_ONCE(bpf_keyring->keys.nr_leaves_on_tree) ||
!READ_ONCE(bpf_keyring->restrict_link))
return NULL;
bkey = kmalloc_obj(*bkey);
if (!bkey)
return NULL;
bkey->key = bpf_keyring;
bkey->has_ref = false;
return bkey;
}
static int __init bpf_keyring_init(void)
{
struct key *keyring;
keyring = keyring_alloc(".bpf",
GLOBAL_ROOT_UID, GLOBAL_ROOT_GID,
current_cred(), KEY_POS_SEARCH |
KEY_USR_VIEW | KEY_USR_READ |
KEY_USR_WRITE | KEY_USR_SEARCH |
KEY_USR_SETATTR, KEY_ALLOC_NOT_IN_QUOTA,
NULL, NULL);
if (IS_ERR(keyring)) {
pr_err("bpf: cannot allocate bpf keyring: %ld\n",
PTR_ERR(keyring));
return 0;
}
if (!bpf_keyring_unsealed &&
keyring_restrict(make_key_ref(keyring, true), NULL, NULL)) {
pr_err("bpf: cannot seal bpf keyring\n");
key_revoke(keyring);
key_put(keyring);
return 0;
}
bpf_keyring = keyring;
key_register_bpf_keyring(keyring);
return 0;
}
late_initcall(bpf_keyring_init);

20
kernel/bpf/liveness.c
View File

@ -2060,7 +2060,8 @@ static inline u16 mask_hi(u32 m) { return (u16)(m >> 16); }
/* Compute info->{use,def} fields for the instruction */
static void compute_insn_live_regs(struct bpf_verifier_env *env,
struct bpf_insn *insn,
struct insn_live_regs *info)
struct insn_live_regs *info,
bool ret_reg_pair)
{
struct bpf_call_summary cs;
const u8 class = BPF_CLASS(insn->code);
@ -2072,6 +2073,7 @@ static void compute_insn_live_regs(struct bpf_verifier_env *env,
const u32 src32 = mask_lo(src);
const u32 dst32 = mask_lo(dst);
const u32 r0 = reg64_mask(0);
const u32 r2 = reg64_mask(BPF_REG_2);
u32 def = 0;
u32 use = U32_MAX;
@ -2191,7 +2193,7 @@ static void compute_insn_live_regs(struct bpf_verifier_env *env,
break;
case BPF_EXIT:
def = 0;
use = r0;
use = ret_reg_pair ? (r0 | r2) : r0;
break;
case BPF_CALL:
def = ALL_CALLER_SAVED_REGS;
@ -2228,8 +2230,8 @@ int bpf_compute_live_registers(struct bpf_verifier_env *env)
struct insn_live_regs *state;
int insn_cnt = env->prog->len;
u64 pos, insn_pos;
int err = 0, i, j;
bool changed;
int err = 0, i, j, subprog, start, end;
bool changed, ret_reg_pair;
/* Use the following algorithm:
* - define the following:
@ -2256,8 +2258,14 @@ int bpf_compute_live_registers(struct bpf_verifier_env *env)
goto out;
}
for (i = 0; i < insn_cnt; ++i)
compute_insn_live_regs(env, &insns[i], &state[i]);
for (subprog = 0; subprog < env->subprog_cnt; subprog++) {
start = env->subprog_info[subprog].start;
end = env->subprog_info[subprog + 1].start;
ret_reg_pair = bpf_ret_reg_pair(env, subprog);
for (i = start; i < end; ++i)
compute_insn_live_regs(env, &insns[i], &state[i], ret_reg_pair);
}
/* Forward pass: resolve stack access through FP-derived pointers */
err = bpf_compute_subprog_arg_access(env);

3
kernel/bpf/queue_stack_maps.c
View File

@ -59,6 +59,9 @@ static int queue_stack_map_alloc_check(union bpf_attr *attr)
*/
return -E2BIG;
if (attr->max_entries >= U32_MAX / attr->value_size)
return -E2BIG;
return 0;
}

55
kernel/bpf/stream.c
View File

@ -22,11 +22,11 @@ static struct bpf_stream_elem *bpf_stream_elem_alloc(int len)
size_t alloc_size;
/*
* Length denotes the amount of data to be written as part of stream element,
* thus includes '\0' byte. We're capped by how much bpf_bprintf_buffers can
* accomodate, therefore deny allocations that won't fit into them.
* Length is the payload pushed into the stream, excluding the
* trailing NUL of the bprintf buffer. Reject anything that cannot
* fit without copying that NUL into the stream element.
*/
if (len < 0 || len > max_len)
if (len < 0 || len >= max_len)
return NULL;
alloc_size = offsetof(struct bpf_stream_elem, str[len]);
@ -68,10 +68,8 @@ static int bpf_stream_consume_capacity(struct bpf_stream *stream, int len)
return 0;
}
static void bpf_stream_release_capacity(struct bpf_stream *stream, struct bpf_stream_elem *elem)
static void bpf_stream_release_capacity(struct bpf_stream *stream, int len)
{
int len = elem->total_len;
atomic_sub(len, &stream->capacity);
}
@ -79,7 +77,14 @@ static int bpf_stream_push_str(struct bpf_stream *stream, const char *str, int l
{
int ret = bpf_stream_consume_capacity(stream, len);
return ret ?: __bpf_stream_push_str(&stream->log, str, len);
if (ret)
return ret;
ret = __bpf_stream_push_str(&stream->log, str, len);
if (ret)
bpf_stream_release_capacity(stream, len);
return ret;
}
static struct bpf_stream *bpf_stream_get(enum bpf_stream_id stream_id, struct bpf_prog_aux *aux)
@ -162,6 +167,7 @@ static int bpf_stream_read(struct bpf_stream *stream, void __user *buf, int len)
while (rem_len) {
int pos = len - rem_len;
int chunk, n;
bool cont;
node = bpf_stream_backlog_peek(stream);
@ -175,20 +181,21 @@ static int bpf_stream_read(struct bpf_stream *stream, void __user *buf, int len)
cons_len = elem->consumed_len;
cont = bpf_stream_consume_elem(elem, &rem_len) == false;
chunk = elem->consumed_len - cons_len;
ret = copy_to_user(buf + pos, elem->str + cons_len,
elem->consumed_len - cons_len);
/* Restore in case of error. */
if (ret) {
ret = -EFAULT;
elem->consumed_len = cons_len;
n = copy_to_user(buf + pos, elem->str + cons_len, chunk);
if (n) {
/* Keep any successfully copied bytes; -EFAULT only if none. */
elem->consumed_len -= n;
rem_len += n;
ret = (len == rem_len) ? -EFAULT : 0;
break;
}
if (cont)
continue;
bpf_stream_backlog_pop(stream);
bpf_stream_release_capacity(stream, elem);
bpf_stream_release_capacity(stream, elem->total_len);
bpf_stream_free_elem(elem);
}
@ -196,13 +203,15 @@ static int bpf_stream_read(struct bpf_stream *stream, void __user *buf, int len)
return ret ? ret : len - rem_len;
}
int bpf_prog_stream_read(struct bpf_prog *prog, enum bpf_stream_id stream_id, void __user *buf, int len)
int bpf_prog_stream_read(struct bpf_prog *prog, enum bpf_stream_id stream_id, void __user *buf, u32 len)
{
struct bpf_stream *stream;
stream = bpf_stream_get(stream_id, prog->aux);
if (!stream)
return -ENOENT;
if (len > INT_MAX)
return -EINVAL;
return bpf_stream_read(stream, buf, len);
}
@ -238,6 +247,11 @@ __bpf_kfunc int bpf_stream_vprintk(int stream_id, const char *fmt__str, const vo
return ret;
ret = bstr_printf(data.buf, MAX_BPRINTF_BUF, fmt__str, data.bin_args);
/* Truncation: reject before capacity charge (not -ENOMEM). */
if (ret >= MAX_BPRINTF_BUF) {
bpf_bprintf_cleanup(&data);
return -E2BIG;
}
/* Exclude NULL byte during push. */
ret = bpf_stream_push_str(stream, data.buf, ret);
bpf_bprintf_cleanup(&data);
@ -311,17 +325,18 @@ int bpf_stream_stage_printk(struct bpf_stream_stage *ss, const char *fmt, ...)
{
struct bpf_bprintf_buffers *buf;
va_list args;
int ret;
int len, ret;
if (bpf_try_get_buffers(&buf))
return -EBUSY;
va_start(args, fmt);
ret = vsnprintf(buf->buf, ARRAY_SIZE(buf->buf), fmt, args);
len = vscnprintf(buf->buf, ARRAY_SIZE(buf->buf), fmt, args);
va_end(args);
ss->len += ret;
/* Exclude NULL byte during push. */
ret = __bpf_stream_push_str(&ss->log, buf->buf, ret);
ret = __bpf_stream_push_str(&ss->log, buf->buf, len);
if (!ret)
ss->len += len;
bpf_put_buffers();
return ret;
}

442
kernel/bpf/verifier.c
View File

@ -24,6 +24,7 @@
#include <linux/bpf_lsm.h>
#include <linux/security.h>
#include <linux/verification.h>
#include <linux/keyctl.h>
#include <linux/btf_ids.h>
#include <linux/poison.h>
#include <linux/module.h>
@ -394,27 +395,69 @@ bool bpf_subprog_is_global(const struct bpf_verifier_env *env, int subprog)
return aux && aux[subprog].linkage == BTF_FUNC_GLOBAL;
}
static bool subprog_returns_void(struct bpf_verifier_env *env, int subprog)
static const struct btf_type *subprog_ret_type(struct bpf_verifier_env *env, int subprog)
{
const struct btf_type *type, *func, *func_proto;
const struct btf_type *func, *func_proto;
const struct btf *btf = env->prog->aux->btf;
u32 btf_id;
if (!btf || !env->prog->aux->func_info)
return NULL;
btf_id = env->prog->aux->func_info[subprog].type_id;
/* Both already validated by prepare_btf_func() at prog load. */
func = btf_type_by_id(btf, btf_id);
if (verifier_bug_if(!func, env, "btf_id %u not found", btf_id))
return false;
func_proto = btf_type_by_id(btf, func->type);
if (!func_proto)
return false;
type = btf_type_skip_modifiers(btf, func_proto->type, NULL);
if (!type)
return false;
return btf_type_skip_modifiers(btf, func_proto->type, NULL);
}
return btf_type_is_void(type);
static bool subprog_returns_void(struct bpf_verifier_env *env, int subprog)
{
const struct btf_type *type = subprog_ret_type(env, subprog);
return type && btf_type_is_void(type);
}
static u32 ret_regs_cnt(u32 size)
{
return size > 8 && size <= 16 ? 2 : 1;
}
/* Registers holding a function return value, in order. See ret_regs_cnt(). */
static const int ret_regs[] = { BPF_REG_0, BPF_REG_2 };
static int bpf_compute_subprog_ret_regs(struct bpf_verifier_env *env)
{
const struct btf *btf = env->prog->aux->btf;
const struct btf_type *type;
int subprog;
u32 size;
if (!env->prog->jit_requested || bpf_prog_is_offloaded(env->prog->aux))
return 0;
/*
* Skip the main program: its return value is the program's exit code,
* read out of R0, so it never uses the register pair. An extension does
* have a real prototype for subprog 0, but btf_check_func_type_match()
* refuses to replace a function returning more than 8 bytes.
*/
for (subprog = 1; subprog < env->subprog_cnt; subprog++) {
type = subprog_ret_type(env, subprog);
if (!type || btf_type_is_void(type))
continue;
if (verifier_bug_if(IS_ERR(btf_resolve_size(btf, type, &size)), env,
"cannot size return type of subprog %d", subprog))
return -EFAULT;
if (ret_regs_cnt(size) > 1) {
subprog_info(env, subprog)->ret_reg_pair = true;
env->prog->jit_required = 1;
}
}
return 0;
}
const char *bpf_subprog_name(const struct bpf_verifier_env *env, int subprog)
@ -2886,6 +2929,18 @@ int bpf_add_kfunc_call(struct bpf_verifier_env *env, u32 func_id, u16 offset)
err = btf_distill_func_proto(&env->log, kfunc.btf, kfunc.proto, kfunc.name, &func_model);
if (err)
return err;
if (func_model.ret_size > 8) {
if (kfunc.flags && (*kfunc.flags & KF_FASTCALL)) {
verbose(env, "kfunc %s with >8-byte return is not supported with KF_FASTCALL\n",
kfunc.name);
return -EOPNOTSUPP;
}
if (!bpf_jit_supports_kfunc_ret_reg_pair()) {
verbose(env, "kfunc %s with >8-byte return is not supported by JIT\n",
kfunc.name);
return -EOPNOTSUPP;
}
}
memset(&meta, 0, sizeof(meta));
meta.btf = kfunc.btf;
@ -3227,6 +3282,11 @@ static void mark_indirect_target(struct bpf_verifier_env *env, int idx)
env->insn_aux_data[idx].indirect_target = true;
}
static void mark_non_stack_access(struct bpf_verifier_env *env, int idx)
{
env->insn_aux_data[idx].non_stack_access = true;
}
#define LR_FRAMENO_BITS 4
#define LR_SPI_BITS 6
#define LR_ENTRY_BITS (LR_SPI_BITS + LR_FRAMENO_BITS + 1)
@ -5918,6 +5978,28 @@ BTF_TYPE_SAFE_TRUSTED(struct file) {
struct inode *f_inode;
};
/*
* The pointer fields in the sched_ext ops argument containers are pinned by the
* callers for the duration of the ops calls and are never NULL.
*/
BTF_TYPE_SAFE_TRUSTED(struct scx_init_task_args) {
#ifdef CONFIG_EXT_GROUP_SCHED
struct cgroup *cgroup;
#endif
};
BTF_TYPE_SAFE_TRUSTED(struct scx_cpu_release_args) {
struct task_struct *task;
};
BTF_TYPE_SAFE_TRUSTED(struct scx_sub_attach_args) {
struct sched_ext_ops *ops;
};
BTF_TYPE_SAFE_TRUSTED(struct scx_sub_detach_args) {
struct sched_ext_ops *ops;
};
BTF_TYPE_SAFE_TRUSTED_OR_NULL(struct dentry) {
struct inode *d_inode;
};
@ -5962,6 +6044,10 @@ static bool type_is_trusted(struct bpf_verifier_env *env,
BTF_TYPE_EMIT(BTF_TYPE_SAFE_TRUSTED(struct bpf_iter__task));
BTF_TYPE_EMIT(BTF_TYPE_SAFE_TRUSTED(struct linux_binprm));
BTF_TYPE_EMIT(BTF_TYPE_SAFE_TRUSTED(struct file));
BTF_TYPE_EMIT(BTF_TYPE_SAFE_TRUSTED(struct scx_init_task_args));
BTF_TYPE_EMIT(BTF_TYPE_SAFE_TRUSTED(struct scx_cpu_release_args));
BTF_TYPE_EMIT(BTF_TYPE_SAFE_TRUSTED(struct scx_sub_attach_args));
BTF_TYPE_EMIT(BTF_TYPE_SAFE_TRUSTED(struct scx_sub_detach_args));
return btf_nested_type_is_trusted(&env->log, reg, field_name, btf_id, "__safe_trusted");
}
@ -6383,6 +6469,7 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, struct b
int value_regno, bool strict_alignment_once, bool is_ldsx)
{
struct bpf_reg_state *regs = cur_regs(env);
enum bpf_reg_type ptr_type = reg->type;
int size, err = 0;
size = bpf_size_to_bytes(bpf_size);
@ -6621,6 +6708,10 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, struct b
clear_scalar_id(&regs[value_regno]);
}
}
if (!err && bpf_is_mem_insn(&env->prog->insnsi[insn_idx]) && ptr_type != PTR_TO_STACK)
mark_non_stack_access(env, insn_idx);
return err;
}
@ -9907,6 +9998,7 @@ static int check_func_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
u16 callee_incoming, stack_arg_cnt;
struct bpf_func_state *caller;
int err, subprog, target_insn;
u32 i, nregs;
target_insn = *insn_idx + insn->imm + 1;
subprog = bpf_find_subprog(env, target_insn);
@ -9962,10 +10054,20 @@ static int check_func_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
clear_caller_saved_regs(env, caller->regs);
invalidate_outgoing_stack_args(env, cur_func(env));
/* All non-void global functions return a 64-bit SCALAR_VALUE. */
/*
* A non-void global function returns a 64-bit SCALAR_VALUE in
* R0, or a >8 byte SCALAR_VALUE in the R0:R2 register pair.
*/
if (!returns_void) {
mark_reg_unknown(env, caller->regs, BPF_REG_0);
nregs = bpf_ret_reg_pair(env, subprog) ? 2 : 1;
mark_reg_unknown(env, caller->regs, ret_regs[0]);
bpf_diag_mod_end(env);
for (i = 1; i < nregs; i++) {
bpf_diag_mod_begin(env, &caller->regs[ret_regs[i]], NULL,
BPF_DIAG_MOD_WRITE);
mark_reg_unknown(env, caller->regs, ret_regs[i]);
bpf_diag_mod_end(env);
}
}
if (env->subprog_info[subprog].might_throw) {
@ -10337,11 +10439,15 @@ static int prepare_func_exit(struct bpf_verifier_env *env, int *insn_idx)
struct bpf_func_state *caller, *callee;
struct bpf_reg_state *r0;
bool in_callback_fn;
u32 i, nregs;
int err;
callee = state->frame[state->curframe];
r0 = &callee->regs[BPF_REG_0];
if (r0->type == PTR_TO_STACK) {
nregs = bpf_ret_reg_pair(env, callee->subprogno) ? 2 : 1;
for (i = 0; i < nregs; i++) {
if (callee->regs[ret_regs[i]].type != PTR_TO_STACK)
continue;
/* technically it's ok to return caller's stack pointer
* (or caller's caller's pointer) back to the caller,
* since these pointers are valid. Only current stack
@ -10376,10 +10482,18 @@ static int prepare_func_exit(struct bpf_verifier_env *env, int *insn_idx)
return -EFAULT;
}
} else {
/* return to the caller whatever r0 had in the callee */
bpf_diag_mod_begin(env, &caller->regs[BPF_REG_0], r0, BPF_DIAG_MOD_WRITE);
caller->regs[BPF_REG_0] = *r0;
bpf_diag_mod_end(env);
/*
* return to the caller whatever the callee had in the
* return register(s)
*/
for (i = 0; i < nregs; i++) {
u32 regno = ret_regs[i];
bpf_diag_mod_begin(env, &caller->regs[regno], &callee->regs[regno],
BPF_DIAG_MOD_WRITE);
caller->regs[regno] = callee->regs[regno];
bpf_diag_mod_end(env);
}
}
/* for callbacks like bpf_loop or bpf_for_each_map_elem go back to callsite,
@ -11341,6 +11455,37 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn
return 0;
}
/*
* Mark the register(s) holding a @size byte kfunc return value as unknown
* scalars and return how many registers that took. Both halves of a register
* pair are treated the same way.
*
* R0 is written in the diagnostic modification scope the caller opened before
* the caller-saved scrub, since R0 gets no scrub record of its own and that
* scope is what carries its pre-call value into the history. A scope tracks a
* single register and does not nest, so close it before recording the rest of
* the return value. A value that fits in R0 leaves the caller's scope open for
* the caller to close, so a later write to R0 still lands in it.
*/
static u32 mark_kfunc_ret_regs(struct bpf_verifier_env *env,
struct bpf_reg_state *regs, u32 size)
{
u32 i, nregs = ret_regs_cnt(size);
mark_reg_unknown(env, regs, ret_regs[0]);
if (nregs == 1)
return nregs;
bpf_diag_mod_end(env);
for (i = 1; i < nregs; i++) {
bpf_diag_mod_begin(env, &regs[ret_regs[i]], NULL, BPF_DIAG_MOD_WRITE);
mark_reg_unknown(env, regs, ret_regs[i]);
bpf_diag_mod_end(env);
}
return nregs;
}
static bool is_kfunc_acquire(struct bpf_call_arg_meta *meta)
{
return meta->kfunc_flags & KF_ACQUIRE;
@ -11607,13 +11752,39 @@ static bool is_kfunc_arg_implicit(const struct bpf_call_arg_meta *meta, u32 arg_
return argn <= arg_idx;
}
/* Returns true if struct is composed of scalars, 4 levels of nesting allowed */
static bool __btf_type_is_scalar_struct(struct bpf_verifier_env *env,
const struct btf *btf,
const struct btf_type *t, int rec)
#define BTF_MEMBER_MAX_DEPTH 4
#define BTF_MEMBER_PATH_LEN 64
struct btf_member_path {
const struct btf_member *member[BTF_MEMBER_MAX_DEPTH];
bool in_array[BTF_MEMBER_MAX_DEPTH];
const struct btf_type *bad_type;
int depth;
bool too_deep;
};
static bool btf_member_kind_allowed(const struct btf *btf, const struct btf_type *t,
u32 member_kinds)
{
if ((member_kinds & BTF_MEMBER_SCALAR) && btf_type_is_scalar(t))
return true;
if ((member_kinds & BTF_MEMBER_ARENA_PTR) && btf_type_is_arena_ptr(btf, t))
return true;
return false;
}
/*
* Returns true if every member of struct @t is of a kind listed in
* @member_kinds, 4 levels of nesting allowed. An array member counts as its
* element type.
*/
static bool btf_struct_member_walk(struct bpf_verifier_env *env, const struct btf *btf,
const struct btf_type *t, u32 member_kinds, int rec,
struct btf_member_path *path)
{
const struct btf_type *member_type;
const struct btf_member *member;
bool in_array;
u32 i;
if (!btf_type_is_struct(t))
@ -11622,29 +11793,83 @@ static bool __btf_type_is_scalar_struct(struct bpf_verifier_env *env,
for_each_member(i, t, member) {
const struct btf_array *array;
in_array = false;
member_type = btf_type_skip_modifiers(btf, member->type, NULL);
/*
* Every element of an array is laid out in the value being
* returned, so an array counts as its element type however many
* dimensions deep that is.
*/
while (btf_type_is_array(member_type)) {
array = btf_array(member_type);
if (!array->nelems) {
member_type = NULL;
goto bad_member;
}
member_type = btf_type_skip_modifiers(btf, array->type, NULL);
in_array = true;
}
if (btf_type_is_struct(member_type)) {
if (rec >= 3) {
if (rec >= BTF_MEMBER_MAX_DEPTH - 1) {
verbose(env, "max struct nesting depth exceeded\n");
if (path)
path->too_deep = true;
return false;
}
if (!__btf_type_is_scalar_struct(env, btf, member_type, rec + 1))
return false;
if (!btf_struct_member_walk(env, btf, member_type, member_kinds,
rec + 1, path))
goto bad_path;
continue;
}
if (btf_type_is_array(member_type)) {
array = btf_array(member_type);
if (!array->nelems)
return false;
member_type = btf_type_skip_modifiers(btf, array->type, NULL);
if (!btf_type_is_scalar(member_type))
return false;
continue;
}
if (!btf_type_is_scalar(member_type))
return false;
if (!btf_member_kind_allowed(btf, member_type, member_kinds))
goto bad_member;
}
return true;
bad_member:
if (path) {
path->depth = rec + 1;
path->bad_type = member_type;
}
bad_path:
if (path && path->depth) {
path->member[rec] = member;
path->in_array[rec] = in_array;
}
return false;
}
bool btf_struct_is_composed_of(struct bpf_verifier_env *env,
const struct btf *btf,
const struct btf_type *t, u32 member_kinds)
{
return btf_struct_member_walk(env, btf, t, member_kinds, 0, NULL);
}
static bool btf_type_is_scalar_struct(struct bpf_verifier_env *env,
const struct btf *btf,
const struct btf_type *t)
{
return btf_struct_is_composed_of(env, btf, t, BTF_MEMBER_SCALAR);
}
static void btf_member_path_str(const struct btf *btf, const struct btf_member_path *path,
char *buf, size_t buf_sz)
{
size_t len = 0;
int i;
buf[0] = '\0';
for (i = 0; i < path->depth; i++) {
const char *name = btf_name_by_offset(btf, path->member[i]->name_off);
/* An anonymous struct or union has no name to spell. */
if (!name || !name[0])
continue;
len += scnprintf(buf + len, buf_sz - len, "%s%s", len ? "." : "", name);
if (path->in_array[i])
len += scnprintf(buf + len, buf_sz - len, "[]");
}
}
enum kfunc_ptr_arg_type {
@ -12027,7 +12252,7 @@ get_kfunc_arg_type(struct bpf_verifier_env *env, struct bpf_call_arg_meta *meta,
(is_kfunc_arg_mem_size(meta->btf, &args[arg + 1]) ||
is_kfunc_arg_const_mem_size(meta->btf, &args[arg + 1]))) {
if (!btf_type_is_void(ref_t) && !btf_type_is_scalar(ref_t) &&
!__btf_type_is_scalar_struct(env, meta->btf, ref_t, 0)) {
!btf_type_is_scalar_struct(env, meta->btf, ref_t)) {
verbose(env, "%s pointer type %s %s must point to void, scalar, or struct with scalar\n",
reg_arg_name(env, argno), btf_type_str(ref_t), ref_tname);
return -EINVAL;
@ -12043,7 +12268,7 @@ get_kfunc_arg_type(struct bpf_verifier_env *env, struct bpf_call_arg_meta *meta,
* scalars. The access size is derived from the pointed-to BTF type.
*/
if (!btf_type_is_scalar(ref_t) &&
!__btf_type_is_scalar_struct(env, meta->btf, ref_t, 0)) {
!btf_type_is_scalar_struct(env, meta->btf, ref_t)) {
verbose(env, "%s pointer type %s %s must point to scalar, or struct with scalar\n",
reg_arg_name(env, argno), btf_type_str(ref_t), ref_tname);
return -EINVAL;
@ -13103,7 +13328,7 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_call_arg_me
break;
}
if (!__btf_type_is_scalar_struct(env, meta->btf, ref_t, 0)) {
if (!btf_type_is_scalar_struct(env, meta->btf, ref_t)) {
enum bpf_reg_type reg2btf_type = lookup_reg2btf_ids(ref_id);
const char *expected_type;
@ -13650,7 +13875,7 @@ static int check_special_kfunc(struct bpf_verifier_env *env, struct bpf_call_arg
struct_meta = btf_find_struct_meta(ret_btf, ret_btf_id);
if (is_bpf_percpu_obj_new_kfunc(meta->func_id)) {
if (!__btf_type_is_scalar_struct(env, ret_btf, ret_t, 0)) {
if (!btf_type_is_scalar_struct(env, ret_btf, ret_t)) {
verbose(env, "bpf_percpu_obj_new type ID argument must be of a struct of scalars\n");
return -EINVAL;
}
@ -13764,7 +13989,7 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
struct bpf_insn_aux_data *insn_aux;
const char *operation;
int err, insn_idx = *insn_idx_p;
u32 i, nargs, ptr_type_id;
u32 i, nargs, ptr_type_id, ret_nregs = 1;
struct bpf_kfunc_desc *desc;
struct btf *desc_btf;
int id;
@ -14018,10 +14243,63 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
}
if (btf_type_is_scalar(t)) {
mark_reg_unknown(env, regs, BPF_REG_0);
ret_nregs = mark_kfunc_ret_regs(env, regs, t->size);
if (meta.btf == btf_vmlinux && (meta.func_id == special_kfunc_list[KF_bpf_res_spin_lock] ||
meta.func_id == special_kfunc_list[KF_bpf_res_spin_lock_irqsave]))
__mark_reg_const_zero(env, &regs[BPF_REG_0]);
} else if (btf_type_is_struct(t)) {
struct btf_member_path path = {};
const char *member_note = "";
/*
* The returned struct comes back as raw register bits modeled
* as an unknown scalar, so a pointer member would be laundered
* into a scalar and escape provenance and reference tracking.
* Only scalars and arena pointers are allowed; see
* btf_validate_return_type() in btf.c for why an arena pointer
* is safe here.
*/
if (!btf_struct_member_walk(env, desc_btf, t,
BTF_MEMBER_SCALAR | BTF_MEMBER_ARENA_PTR, 0, &path)) {
verbose(env,
"kernel function %s returns %s %s that is not composed of scalars or arena pointers\n",
func_name, btf_type_str(t),
btf_name_by_offset(desc_btf, t->name_off));
if (path.too_deep) {
member_note = bpf_diag_fmt(
env, " It nests structs more than %d levels deep.",
BTF_MEMBER_MAX_DEPTH);
} else if (path.depth) {
char bad_name[BTF_MEMBER_PATH_LEN];
btf_member_path_str(desc_btf, &path, bad_name, sizeof(bad_name));
if (!path.bad_type) {
verbose(env, "member '%s' is a zero-length array\n",
bad_name);
member_note = bpf_diag_fmt(
env, " Its member '%s' is a zero-length array.",
bad_name);
} else {
verbose(env, "member '%s' has type %s\n", bad_name,
btf_type_str(path.bad_type));
member_note = bpf_diag_fmt(
env,
" Its member '%s' is %s, not a scalar or an arena pointer.",
bad_name, btf_type_str(path.bad_type));
}
}
bpf_diag_program_structure(
env, insn_idx, "unsupported kernel function return type",
"Call a kernel function that returns only scalars or arena pointers by value.",
"%s() returns %s %s by value.%s "
"Only kfuncs returning scalar values or arena pointers, or "
"structures composed of scalar values or arena pointers are "
"supported.",
func_name, btf_type_str(t),
btf_name_by_offset(desc_btf, t->name_off), member_note);
return -EINVAL;
}
ret_nregs = mark_kfunc_ret_regs(env, regs, t->size);
} else if (btf_type_is_ptr(t)) {
ptr_type = btf_type_skip_modifiers(desc_btf, t->type, &ptr_type_id);
err = check_special_kfunc(env, &meta, regs, insn_aux, ptr_type, desc_btf);
@ -14152,7 +14430,8 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
* Record R0 before process_iter_next_call() snapshots the alternate
* iterator path's diagnostic position.
*/
bpf_diag_mod_end(env);
if (ret_nregs == 1)
bpf_diag_mod_end(env);
if (bpf_is_iter_next_kfunc(&meta)) {
err = process_iter_next_call(env, insn_idx, &meta);
@ -17602,37 +17881,54 @@ static int check_return_code(struct bpf_verifier_env *env, int regno, const char
return 0;
}
static int check_global_subprog_return_code(struct bpf_verifier_env *env)
static int check_global_ret_scalar_reg(struct bpf_verifier_env *env, u32 regno)
{
struct bpf_reg_state *reg = reg_state(env, BPF_REG_0);
struct bpf_func_state *cur_frame = cur_func(env);
struct bpf_reg_state *reg;
int err;
if (subprog_returns_void(env, cur_frame->subprogno))
return 0;
err = check_reg_arg(env, BPF_REG_0, SRC_OP);
err = check_reg_arg(env, regno, SRC_OP);
if (err)
return err;
/* Pointers to arena are safe to pass between subprograms. */
if (is_arena_reg(env, BPF_REG_0))
if (is_arena_reg(env, regno))
return 0;
if (is_pointer_value(env, BPF_REG_0)) {
verbose(env, "R%d leaks addr as return value\n", BPF_REG_0);
if (is_pointer_value(env, regno)) {
verbose(env, "R%d leaks addr as return value\n", regno);
return -EACCES;
}
reg = reg_state(env, regno);
if (reg->type != SCALAR_VALUE) {
verbose(env, "At subprogram exit the register R0 is not a scalar value (%s)\n",
reg_type_str(env, reg->type));
verbose(env, "At subprogram exit the register R%d is not a scalar value (%s)\n",
regno, reg_type_str(env, reg->type));
return -EINVAL;
}
return 0;
}
static int check_global_subprog_return_code(struct bpf_verifier_env *env)
{
struct bpf_func_state *cur_frame = cur_func(env);
u32 subprog = cur_frame->subprogno;
u32 i, nregs;
int err;
if (subprog_returns_void(env, subprog))
return 0;
nregs = bpf_ret_reg_pair(env, subprog) ? 2 : 1;
for (i = 0; i < nregs; i++) {
err = check_global_ret_scalar_reg(env, ret_regs[i]);
if (err)
return err;
}
return 0;
}
/* Bitmask with 1s for all caller saved registers */
#define ALL_CALLER_SAVED_REGS ((1u << CALLER_SAVED_REGS) - 1)
@ -20989,6 +21285,14 @@ int bpf_fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
return 0;
}
/*
* Upper bound on the PKCS#7 signature blob passed with a program. Comfortably
* above the largest signature the kernel can verify, and far below anything
* that would make rejecting a load expensive. Deliberately a fixed number so
* that what the syscall accepts does not depend on PAGE_SIZE.
*/
#define BPF_PROG_MAX_SIGNATURE_SIZE (64 * 1024)
static enum bpf_sig_keyring bpf_classify_keyring(s32 keyring_id)
{
switch (keyring_id) {
@ -20998,6 +21302,8 @@ static enum bpf_sig_keyring bpf_classify_keyring(s32 keyring_id)
return BPF_SIG_KEYRING_SECONDARY;
case (s32)(unsigned long)VERIFY_USE_PLATFORM_KEYRING:
return BPF_SIG_KEYRING_PLATFORM;
case KEY_SPEC_BPF_KEYRING:
return BPF_SIG_KEYRING_BPF;
default:
return BPF_SIG_KEYRING_USER;
}
@ -21026,18 +21332,25 @@ static int bpf_prog_verify_signature(struct bpf_verifier_env *env,
u64 data_sz;
int err = 0;
/*
* Don't attempt to use kmalloc_large or vmalloc for signatures.
* Practical signature for BPF program should be below this limit.
*/
if (!attr->signature_size ||
attr->signature_size > KMALLOC_MAX_CACHE_SIZE)
attr->signature_size > BPF_PROG_MAX_SIGNATURE_SIZE)
return -EINVAL;
if (system_keyring_id_check(attr->keyring_id) == 0)
if (system_keyring_id_check(attr->keyring_id) == 0) {
key = bpf_lookup_system_key(attr->keyring_id);
else
} else if (attr->keyring_id == KEY_SPEC_BPF_KEYRING) {
key = bpf_lookup_keyring();
} else if (bpf_keyring_enforced()) {
verbose(env, "caller-supplied keyring refused, use bpf keyring\n");
return -EPERM;
} else {
key = bpf_lookup_user_key(attr->keyring_id, 0);
}
if (!key) {
if (attr->keyring_id == KEY_SPEC_BPF_KEYRING) {
verbose(env, "the bpf keyring is empty or has not been restricted\n");
return -ENOKEY;
}
verbose(env, "cannot resolve signing keyring with keyring_id %d\n",
attr->keyring_id);
return -EINVAL;
@ -21307,6 +21620,11 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
if (ret < 0)
goto skip_full_check;
/* must precede the first bpf_ret_reg_pair() user below */
ret = bpf_compute_subprog_ret_regs(env);
if (ret < 0)
goto skip_full_check;
ret = bpf_compute_live_registers(env);
if (ret < 0)
goto skip_full_check;

25
security/keys/process_keys.c
View File

@ -22,6 +22,9 @@
/* Session keyring create vs join semaphore */
static DEFINE_MUTEX(key_session_mutex);
/* BPF keyring reachable through KEY_SPEC_BPF_KEYRING */
static struct key *bpf_keyring __ro_after_init;
/* The root user's tracking struct */
struct key_user root_key_user = {
.usage = REFCOUNT_INIT(3),
@ -590,6 +593,20 @@ bool lookup_user_key_possessed(const struct key *key,
return key == match_data->raw_data;
}
/**
* key_register_bpf_keyring - Publish the BPF keyring for KEY_SPEC_BPF_KEYRING
* @keyring: The keyring to publish
*
* Make @keyring reachable by userspace through the KEY_SPEC_BPF_KEYRING
* special key ID, so that provisioning it does not require scraping its
* serial out of /proc/keys first. Called once, from an initcall, and never
* undone.
*/
void key_register_bpf_keyring(struct key *keyring)
{
bpf_keyring = keyring;
}
/*
* Look up a key ID given us by userspace with a given permissions mask to get
* the key it refers to.
@ -741,6 +758,14 @@ key_ref_t lookup_user_key(key_serial_t id, unsigned long lflags,
key_ref = make_key_ref(key, 1);
break;
case KEY_SPEC_BPF_KEYRING:
key = bpf_keyring;
if (!key)
goto error;
__key_get(key);
key_ref = make_key_ref(key, 0);
break;
default:
key_ref = ERR_PTR(-EINVAL);
if (id < 1)

6
tools/bpf/bpftool/btf.c
View File

@ -805,9 +805,13 @@ static int dump_btf_c(const struct btf *btf,
if (sort_dump)
datums = sort_btf_c(btf);
for (i = 1; i < cnt; i++) {
for (i = 0; i < cnt; i++) {
int idx = datums ? datums[i].index : i;
/* type ID 0 is void, skip it */
if (idx == 0)
continue;
err = btf_dump__dump_type(d, idx);
if (err)
goto done;

30
tools/bpf/bpftool/main.c
View File

@ -365,16 +365,20 @@ static int do_batch(int argc, char **argv)
if (json_output)
jsonw_start_array(json_wtr);
while (fgets(buf, sizeof(buf), fp)) {
cp = strchr(buf, '#');
if (cp)
*cp = '\0';
for (;;) {
errno = 0;
if (!fgets(buf, sizeof(buf), fp))
break;
if (strlen(buf) == sizeof(buf) - 1) {
errno = E2BIG;
break;
}
cp = strchr(buf, '#');
if (cp)
*cp = '\0';
/* Append continuation lines if any (coming after a line ending
* with '\' in the batch file).
*/
@ -387,15 +391,15 @@ static int do_batch(int argc, char **argv)
goto err_close;
}
cp = strchr(contline, '#');
if (cp)
*cp = '\0';
if (strlen(buf) + strlen(contline) + 1 > sizeof(buf)) {
p_err("command %u is too long", lines);
err = -1;
goto err_close;
}
cp = strchr(contline, '#');
if (cp)
*cp = '\0';
buf[strlen(buf) - 2] = '\0';
strcat(buf, contline);
}
@ -467,16 +471,6 @@ int main(int argc, char **argv)
setlinebuf(stdout);
#ifdef USE_LIBCAP
/* Libcap < 2.63 hooks before main() to compute the number of
* capabilities of the running kernel, and doing so it calls prctl()
* which may fail and set errno to non-zero.
* Let's reset errno to make sure this does not interfere with the
* batch mode.
*/
errno = 0;
#endif
last_do_help = do_help;
pretty_output = false;
json_output = false;

2
tools/bpf/bpftool/main.h
View File

@ -57,7 +57,7 @@ static inline void *u64_to_ptr(__u64 ptr)
})
#define ERR_MAX_LEN 1024
#define MAX_SIG_SIZE 4096
#define MAX_SIG_SIZE 16384
#define BPF_TAG_FMT "%02hhx%02hhx%02hhx%02hhx%02hhx%02hhx%02hhx%02hhx"

2
tools/bpf/bpftool/prog.c
View File

@ -1769,7 +1769,7 @@ static int load_with_options(int argc, char **argv, bool first_prog_only)
}
if (prog_type == BPF_PROG_TYPE_XDP && xdpmeta_ifindex) {
bpf_program__set_flags(pos, BPF_F_XDP_DEV_BOUND_ONLY);
bpf_program__set_flags(pos, bpf_program__flags(pos) | BPF_F_XDP_DEV_BOUND_ONLY);
bpf_program__set_ifindex(pos, xdpmeta_ifindex);
} else {
bpf_program__set_ifindex(pos, offload_ifindex);

24
tools/bpf/bpftool/sign.c
View File

@ -130,6 +130,12 @@ __u32 register_session_key(const char *key_der_path)
int bpftool_prog_sign(struct bpf_load_and_run_opts *opts)
{
unsigned int signer_flags = CMS_NOCERTS | CMS_BINARY | CMS_NOSMIMECAP |
#ifdef CMS_NO_SIGNING_TIME
CMS_NO_SIGNING_TIME |
#endif
CMS_USE_KEYID | CMS_NOATTR;
const EVP_MD *cms_digest = EVP_sha256();
BIO *bd_in = NULL, *bd_out = NULL;
EVP_PKEY *private_key = NULL;
CMS_ContentInfo *cms = NULL;
@ -167,6 +173,20 @@ int bpftool_prog_sign(struct bpf_load_and_run_opts *opts)
goto cleanup;
}
#if OPENSSL_VERSION_NUMBER >= 0x30000000L && OPENSSL_VERSION_NUMBER < 0x40000000L
if (EVP_PKEY_is_a(private_key, "ML-DSA-44") ||
EVP_PKEY_is_a(private_key, "ML-DSA-65") ||
EVP_PKEY_is_a(private_key, "ML-DSA-87")) {
/*
* See workaround in 0ad9a71933e7 ("modsign: Enable ML-DSA
* module signing"). Kernel only accepts sha512, see also
* 8bbdeb7a25b4 ("pkcs7, x509: Add ML-DSA support").
*/
signer_flags &= ~CMS_NOATTR;
cms_digest = EVP_sha512();
}
#endif
cms = CMS_sign(NULL, NULL, NULL, NULL,
CMS_NOCERTS | CMS_PARTIAL | CMS_BINARY | CMS_DETACHED |
CMS_STREAM);
@ -175,9 +195,7 @@ int bpftool_prog_sign(struct bpf_load_and_run_opts *opts)
goto cleanup;
}
if (!CMS_add1_signer(cms, x509, private_key, EVP_sha256(),
CMS_NOCERTS | CMS_BINARY | CMS_NOSMIMECAP |
CMS_USE_KEYID | CMS_NOATTR)) {
if (!CMS_add1_signer(cms, x509, private_key, cms_digest, signer_flags)) {
err = -EINVAL;
goto cleanup;
}

14
tools/lib/bpf/btf.c
View File

@ -3364,7 +3364,7 @@ static int btf_ext_parse_sec_info(struct btf_ext *btf_ext,
{
const struct btf_ext_info_sec *sinfo;
struct btf_ext_info *ext_info;
__u32 info_left, record_size;
__u32 data_left, info_left, record_size;
size_t sec_cnt = 0;
void *info;
@ -3377,16 +3377,20 @@ static int btf_ext_parse_sec_info(struct btf_ext *btf_ext,
return -EINVAL;
}
/* The start of the info sec (including the __u32 record_size). */
info = btf_ext->data + btf_ext->hdr->hdr_len + ext_sec->off;
info_left = ext_sec->len;
if (btf_ext->hdr->hdr_len > btf_ext->data_size)
return -EINVAL;
if (btf_ext->data + btf_ext->data_size < info + ext_sec->len) {
data_left = btf_ext->data_size - btf_ext->hdr->hdr_len;
if (ext_sec->off > data_left || ext_sec->len > data_left - ext_sec->off) {
pr_debug("%s section (off:%u len:%u) is beyond the end of the ELF section .BTF.ext\n",
ext_sec->desc, ext_sec->off, ext_sec->len);
return -EINVAL;
}
/* The start of the info sec (including the __u32 record_size). */
info = btf_ext->data + btf_ext->hdr->hdr_len + ext_sec->off;
info_left = ext_sec->len;
/* At least a record size */
if (info_left < sizeof(__u32)) {
pr_debug(".BTF.ext %s record size not found\n", ext_sec->desc);

22
tools/lib/bpf/libbpf.c
View File

@ -7879,6 +7879,19 @@ static int tracing_multi_mod_fd(struct bpf_program *prog, int *btf_obj_fd)
return 0;
}
static int libbpf_setup_prog_flags(struct bpf_program *prog, long cookie)
{
enum sec_def_flags def = cookie;
if (def & SEC_SLEEPABLE)
prog->prog_flags |= BPF_F_SLEEPABLE;
if (def & SEC_XDP_FRAGS)
prog->prog_flags |= BPF_F_XDP_HAS_FRAGS;
return 0;
}
/* this is called as prog->sec_def->prog_prepare_load_fn for libbpf-supported sec_defs */
static int libbpf_prepare_prog_load(struct bpf_program *prog,
struct bpf_prog_load_opts *opts, long cookie)
@ -7889,12 +7902,6 @@ static int libbpf_prepare_prog_load(struct bpf_program *prog,
if ((def & SEC_EXP_ATTACH_OPT) && !kernel_supports(prog->obj, FEAT_EXP_ATTACH_TYPE))
opts->expected_attach_type = 0;
if (def & SEC_SLEEPABLE)
opts->prog_flags |= BPF_F_SLEEPABLE;
if (prog->type == BPF_PROG_TYPE_XDP && (def & SEC_XDP_FRAGS))
opts->prog_flags |= BPF_F_XDP_HAS_FRAGS;
/* special check for usdt to use uprobe_multi link */
if ((def & SEC_USDT) && kernel_supports(prog->obj, FEAT_UPROBE_MULTI_LINK)) {
/* for BPF_TRACE_UPROBE_MULTI, user might want to query expected_attach_type
@ -10099,6 +10106,7 @@ int bpf_program__clone(struct bpf_program *prog, const struct bpf_prog_load_opts
.prog_type = BPF_PROG_TYPE_##ptype, \
.expected_attach_type = atype, \
.cookie = (long)(flags), \
.prog_setup_fn = libbpf_setup_prog_flags, \
.prog_prepare_load_fn = libbpf_prepare_prog_load, \
__VA_ARGS__ \
}
@ -11738,7 +11746,7 @@ static int perf_event_open_probe(bool uprobe, bool retprobe, const char *name,
errstr(bit));
return bit;
}
attr.config |= 1 << bit;
attr.config |= 1ULL << bit;
}
attr.size = attr_sz;
attr.type = type;

4
tools/lib/bpf/libbpf_internal.h
View File

@ -77,7 +77,7 @@
#define JUMPTABLES_SEC ".jumptables"
#define BTF_INFO_ENC(kind, kind_flag, vlen) \
((!!(kind_flag) << 31) | ((kind) << 24) | ((vlen) & BTF_MAX_VLEN))
(((__u32)(!!(kind_flag)) << 31) | ((kind) << 24) | ((vlen) & BTF_MAX_VLEN))
#define BTF_TYPE_ENC(name, info, size_or_type) (name), (info), (size_or_type)
#define BTF_INT_ENC(encoding, bits_offset, nr_bits) \
((encoding) << 24 | (bits_offset) << 16 | (nr_bits))
@ -259,7 +259,7 @@ static inline enum btf_func_linkage btf_func_linkage(const struct btf_type *t)
static inline __u32 btf_type_info(int kind, int vlen, int kflag)
{
return (kflag << 31) | (kind << 24) | vlen;
return ((__u32)kflag << 31) | (kind << 24) | vlen;
}
enum map_def_parts {

11
tools/lib/bpf/libbpf_probes.c
View File

@ -62,7 +62,7 @@ static __u32 get_ubuntu_kernel_version(void)
*/
static __u32 get_debian_kernel_version(struct utsname *info)
{
__u32 major, minor, patch;
__u32 major, minor, patch = 0;
char *p;
p = strstr(info->version, "Debian ");
@ -71,10 +71,13 @@ static __u32 get_debian_kernel_version(struct utsname *info)
return 0;
}
if (sscanf(p, "Debian %u.%u.%u", &major, &minor, &patch) != 3)
return 0;
if (sscanf(p, "Debian %u.%u.%u", &major, &minor, &patch) == 3)
return KERNEL_VERSION(major, minor, patch);
return KERNEL_VERSION(major, minor, patch);
if (sscanf(p, "Debian %u.%u", &major, &minor) == 2)
return KERNEL_VERSION(major, minor, patch);
return 0;
}
__u32 get_kernel_version(void)

4
tools/lib/bpf/usdt.c
View File

@ -608,8 +608,12 @@ static bool has_nop_combo(int fd, long off)
unsigned char nop_combo[11] = {
0x90, 0x66, 0x2e, 0x0f, 0x1f, 0x84, 0x00, 0x00, 0x00, 0x00, 0x00,
};
long page_sz = getpagesize();
unsigned char buf[11];
/* the kernel can't attach to a nop10 that crosses a page boundary */
if ((off + 1) % page_sz + 10 > page_sz)
return false;
if (pread(fd, buf, 11, off) != 11)
return false;
return memcmp(buf, nop_combo, 11) == 0;

1
tools/testing/selftests/bpf/.gitignore vendored
View File

@ -17,7 +17,6 @@ test_verifier_log
feature
urandom_read
test_sockmap
test_lirc_mode2_user
flow_dissector_load
test_tcpnotify_user
test_libbpf

31
tools/testing/selftests/bpf/Makefile
View File

@ -118,11 +118,12 @@ TEST_GEN_PROGS += test_progs-cpuv4
TEST_INST_SUBDIRS += cpuv4
endif
TEST_FILES = xsk_prereqs.sh $(wildcard progs/btf_dump_test_case_*.c)
TEST_FILES = xsk_prereqs.sh $(wildcard progs/btf_dump_test_case_*.c) \
bpftool_btf_dump_sorted.expected \
bpftool_btf_dump_unsorted.expected
# Order correspond to 'make run_tests' order
TEST_PROGS := test_kmod.sh \
test_lirc_mode2.sh \
test_bpftool_build.sh \
test_doc_build.sh \
test_xsk.sh \
@ -140,7 +141,6 @@ TEST_GEN_PROGS_EXTENDED = \
bench \
flow_dissector_load \
test_cpp \
test_lirc_mode2_user \
veristat \
xdp_features \
xdp_hw_metadata \
@ -337,7 +337,6 @@ $(OUTPUT)/test_sockmap: $(CGROUP_HELPERS) $(TESTING_HELPERS)
$(OUTPUT)/test_tcpnotify_user: $(CGROUP_HELPERS) $(TESTING_HELPERS) $(TRACE_HELPERS)
$(OUTPUT)/test_sock_fields: $(CGROUP_HELPERS) $(TESTING_HELPERS)
$(OUTPUT)/test_tag: $(TESTING_HELPERS)
$(OUTPUT)/test_lirc_mode2_user: $(TESTING_HELPERS)
$(OUTPUT)/flow_dissector_load: $(TESTING_HELPERS)
$(OUTPUT)/test_maps: $(TESTING_HELPERS)
$(OUTPUT)/test_verifier: $(TESTING_HELPERS) $(CAP_HELPERS) $(UNPRIV_HELPERS)
@ -662,7 +661,7 @@ $(TRUNNER_BPF_LSKELS): %.lskel.h: %.bpf.o $(BPFTOOL) | $(TRUNNER_OUTPUT)
}) && \
rm -f $$(<:.o=.llinked1.o) $$(<:.o=.llinked2.o) $$(<:.o=.llinked3.o)
$(TRUNNER_BPF_LSKELS_SIGNED): %.lskel.h: %.bpf.o $(BPFTOOL) | $(TRUNNER_OUTPUT)
$(TRUNNER_BPF_LSKELS_SIGNED): %.lskel.h: %.bpf.o $(BPFTOOL) $(VERIFICATION_CERT) | $(TRUNNER_OUTPUT)
$(Q)$(if $(PERMISSIVE),if [ ! -f $$< ]; then \
$$(RM) $$@; \
printf ' %-12s %s\n' 'SKIP-SKEL' '$$(notdir $$@)' 1>&2; \
@ -825,18 +824,22 @@ LIBARENA_MAKE_ARGS = \
BPF_TARGET_ENDIAN="$(BPF_TARGET_ENDIAN)" \
Q="$(Q)"
LIBARENA_BPF_DEPS := $(wildcard libarena/Makefile \
libarena/include/* \
libarena/include/libarena/* \
libarena/src/* \
libarena/selftests/* \
libarena/*.bpf.o)
LIBARENA_COMMON_DEPS := $(wildcard libarena/Makefile \
libarena/include/* \
libarena/include/libarena/* \
libarena/src/*)
LIBARENA_BPF_DEPS := $(LIBARENA_COMMON_DEPS) $(wildcard libarena/selftests/*)
LIBARENA_BENCH_BPF_DEPS := $(LIBARENA_COMMON_DEPS) $(wildcard libarena/benchs/*)
LIBARENA_SKEL := libarena/libarena.skel.h
LIBARENA_BENCH_SKEL := libarena/libarena_bench.skel.h
$(LIBARENA_SKEL): $(INCLUDE_DIR)/vmlinux.h $(BPFOBJ) $(LIBARENA_BPF_DEPS)
+$(MAKE) -C libarena libarena.skel.h $(LIBARENA_MAKE_ARGS)
$(LIBARENA_BENCH_SKEL): $(INCLUDE_DIR)/vmlinux.h $(BPFOBJ) $(LIBARENA_BENCH_BPF_DEPS) | $(LIBARENA_SKEL)
+$(MAKE) -C libarena benchmarks $(LIBARENA_MAKE_ARGS)
ifneq ($(CLANG_HAS_ARENA_ASAN),)
LIBARENA_ASAN_SKEL := libarena/libarena_asan.skel.h
CFLAGS += -DHAS_BPF_ARENA_ASAN
@ -882,7 +885,9 @@ TRUNNER_EXTRA_FILES := $(OUTPUT)/urandom_read \
ima_setup.sh \
$(VERIFY_SIG_SETUP) \
$(wildcard progs/btf_dump_test_case_*.c) \
$(wildcard progs/*.bpf.o)
$(wildcard progs/*.bpf.o) \
bpftool_btf_dump_sorted.expected \
bpftool_btf_dump_unsorted.expected
TRUNNER_BPF_BUILD_RULE := CLANG_BPF_BUILD_RULE
TRUNNER_BPF_CFLAGS := $(BPF_CFLAGS) $(CLANG_CFLAGS) -DENABLE_ATOMICS_TESTS
$(eval $(call DEFINE_TEST_RUNNER,test_progs))
@ -986,6 +991,7 @@ $(OUTPUT)/bench_sockmap.o: $(OUTPUT)/bench_sockmap_prog.skel.h
$(OUTPUT)/bench_lpm_trie_map.o: $(OUTPUT)/lpm_trie_bench.skel.h $(OUTPUT)/lpm_trie_map.skel.h
$(OUTPUT)/bench_bpf_nop.o: $(OUTPUT)/bpf_nop_bench.skel.h bench_bpf_timing.h
$(OUTPUT)/bench_xdp_lb.o: $(OUTPUT)/xdp_lb_bench.skel.h bench_bpf_timing.h
$(OUTPUT)/bench_libarena.o: $(LIBARENA_BENCH_SKEL)
$(OUTPUT)/bench_bpf_timing.o: bench_bpf_timing.h
$(OUTPUT)/bench.o: bench.h testing_helpers.h $(BPFOBJ)
$(OUTPUT)/bench: LDLIBS += -lm
@ -1013,6 +1019,7 @@ $(OUTPUT)/bench: $(OUTPUT)/bench.o \
$(OUTPUT)/bench_bpf_timing.o \
$(OUTPUT)/bench_bpf_nop.o \
$(OUTPUT)/bench_xdp_lb.o \
$(OUTPUT)/bench_libarena.o \
$(OUTPUT)/usdt_1.o \
$(OUTPUT)/usdt_2.o \
#

6
tools/testing/selftests/bpf/bench.c
View File

@ -288,6 +288,7 @@ extern struct argp bench_crypto_argp;
extern struct argp bench_sockmap_argp;
extern struct argp bench_lpm_trie_map_argp;
extern struct argp bench_xdp_lb_argp;
extern struct argp bench_libarena_argp;
static const struct argp_child bench_parsers[] = {
{ &bench_ringbufs_argp, 0, "Ring buffers benchmark", 0 },
@ -306,6 +307,7 @@ static const struct argp_child bench_parsers[] = {
{ &bench_sockmap_argp, 0, "bpf sockmap benchmark", 0 },
{ &bench_lpm_trie_map_argp, 0, "LPM trie map benchmark", 0 },
{ &bench_xdp_lb_argp, 0, "XDP load-balancer benchmark", 0 },
{ &bench_libarena_argp, 0, "libarena allocator benchmark", 0 },
{},
};
@ -585,6 +587,8 @@ extern const struct bench bench_lpm_trie_delete;
extern const struct bench bench_lpm_trie_free;
extern const struct bench bench_bpf_nop;
extern const struct bench bench_xdp_lb;
extern const struct bench bench_libarena_malloc;
extern const struct bench bench_libarena_calloc;
static const struct bench *benchs[] = {
&bench_count_global,
@ -669,6 +673,8 @@ static const struct bench *benchs[] = {
&bench_lpm_trie_free,
&bench_bpf_nop,
&bench_xdp_lb,
&bench_libarena_malloc,
&bench_libarena_calloc,
};
static void find_benchmark(void)

210
tools/testing/selftests/bpf/benchs/bench_libarena.c Normal file
View File

@ -0,0 +1,210 @@
// SPDX-License-Identifier: GPL-2.0
/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
#include <argp.h>
#include <limits.h>
#include <string.h>
#include "bench.h"
#include <libarena/common.h>
#include <libarena/asan.h>
#include <libarena/buddy.h>
#include <libarena/userspace.h>
#include "libarena/libarena_bench.skel.h"
static struct {
__u64 alloc_size;
__u64 nallocs;
} args = {
.alloc_size = 64,
.nallocs = 10000,
};
static struct {
struct libarena_bench *skel;
int bench_fd;
int reset_fd;
} ctx;
enum {
ARG_LIBARENA_ALLOC_SIZE = 12000,
ARG_LIBARENA_NALLOCS,
};
static const struct argp_option opts[] = {
{ "alloc_size", ARG_LIBARENA_ALLOC_SIZE, "BYTES", 0,
"Size of each arena allocation" },
{ "nallocs", ARG_LIBARENA_NALLOCS, "ITERS", 0,
"Number of allocation per measurement" },
{},
};
static error_t parse_arg(int key, char *arg, struct argp_state *state)
{
unsigned long value;
switch (key) {
case ARG_LIBARENA_ALLOC_SIZE:
value = strtoull(arg, NULL, 10);
if (!value || value >= UINT_MAX) {
fprintf(stderr, "invalid alloc_size: %ld", value);
argp_usage(state);
}
args.alloc_size = value;
break;
case ARG_LIBARENA_NALLOCS:
args.nallocs = strtoull(arg, NULL, 10);
break;
default:
return ARGP_ERR_UNKNOWN;
}
return 0;
}
const struct argp bench_libarena_argp = {
.options = opts,
.parser = parse_arg,
};
static void validate(void)
{
if (env.consumer_cnt != 0) {
fprintf(stderr, "benchmark doesn't support consumers\n");
exit(1);
}
if (env.producer_cnt != 1) {
fprintf(stderr, "benchmark supports exactly one producer\n");
exit(1);
}
}
static void setup_common(void)
{
struct arena_alloc_reserve_args reserve_args = {
.nr_pages = ARENA_RESERVE_PAGES_DFL,
};
int err;
setup_libbpf();
ctx.skel = libarena_bench__open_and_load();
if (!ctx.skel) {
fprintf(stderr, "failed to open and load skeleton\n");
exit(1);
}
err = libarena_run_prog_args(
bpf_program__fd(ctx.skel->progs.arena_alloc_reserve),
&reserve_args, sizeof(reserve_args));
if (err) {
fprintf(stderr, "failed to reserve arena pages: %d\n", err);
exit(1);
}
err = libarena_run_prog(
bpf_program__fd(ctx.skel->progs.arena_buddy_reset));
if (err) {
fprintf(stderr, "failed to initialize arena allocator: %d\n", err);
exit(1);
}
ctx.skel->bss->bench_alloc_size = args.alloc_size;
ctx.skel->bss->bench_nallocs = args.nallocs;
ctx.reset_fd = bpf_program__fd(ctx.skel->progs.arena_buddy_reset);
}
static void malloc_setup(void)
{
setup_common();
ctx.bench_fd = bpf_program__fd(ctx.skel->progs.bench_malloc);
}
static void calloc_setup(void)
{
setup_common();
ctx.bench_fd = bpf_program__fd(ctx.skel->progs.bench_calloc);
}
static void *producer(void *input)
{
int err;
while (true) {
err = libarena_run_prog(ctx.bench_fd);
if (err) {
fprintf(stderr, "libarena benchmark failed: %d\n", err);
exit(1);
}
err = libarena_run_prog(ctx.reset_fd);
if (err) {
fprintf(stderr, "libarena alloc reset failed: %d\n", err);
exit(1);
}
}
return NULL;
}
static void measure(struct bench_res *res)
{
res->duration_ns = atomic_swap(&ctx.skel->bss->bench_duration_ns, 0);
res->hits = atomic_swap(&ctx.skel->bss->bench_hits, 0);
}
static void report_progress(int iter, struct bench_res *res, long delta_ns)
{
double latency_ns = 0.0;
if (res->hits)
latency_ns = res->duration_ns / (double)res->hits;
printf("Iter %3d (%7.3lfus): latency %8.3lf ns/op (%ld allocations)\n",
iter, (delta_ns - 1000000000) / 1000.0, latency_ns, res->hits);
}
static void report_final(struct bench_res res[], int res_cnt)
{
unsigned long duration_ns = 0;
long hits = 0;
int i;
for (i = 0; i < res_cnt; i++) {
duration_ns += res[i].duration_ns;
hits += res[i].hits;
}
if (!hits || !res_cnt) {
printf("Summary: no runs measured\n");
return;
}
printf("Summary: %.3lf ns/op, %.0lf invocations for %u allocations/invocation)\n",
duration_ns / (double)hits, hits / (double)res_cnt,
ctx.skel->bss->bench_nallocs);
}
const struct bench bench_libarena_malloc = {
.name = "libarena-malloc",
.argp = &bench_libarena_argp,
.validate = validate,
.setup = malloc_setup,
.producer_thread = producer,
.measure = measure,
.report_progress = report_progress,
.report_final = report_final,
};
const struct bench bench_libarena_calloc = {
.name = "libarena-calloc",
.argp = &bench_libarena_argp,
.validate = validate,
.setup = calloc_setup,
.producer_thread = producer,
.measure = measure,
.report_progress = report_progress,
.report_final = report_final,
};

31
tools/testing/selftests/bpf/benchs/run_bench_libarena.sh Executable file
View File

@ -0,0 +1,31 @@
#!/bin/bash
# SPDX-License-Identifier: GPL-2.0
source ./benchs/run_common.sh
set -eufo pipefail
RUN_BENCH="./bench -d3 -q"
summarize_libarena()
{
local bench="$1"
local summary
summary=$(printf '%s\n' "$2" | tail -n1)
summary=${summary#Summary: }
printf "%-20s %s\n" "$bench" "$summary"
}
header "libarena sequential malloc\n"
for size in 16 64 256 1024 4096; do
subtitle "allocation size: $size"
printf "\t-------------------\n"
for nallocs in 10 50 100 500 1000 5000 10000; do
summarize_libarena "malloc:" \
"$($RUN_BENCH --alloc_size "$size" --nallocs "$nallocs" libarena-malloc)"
summarize_libarena "calloc:" \
"$($RUN_BENCH --alloc_size "$size" --nallocs "$nallocs" libarena-calloc)"
done
done

48
tools/testing/selftests/bpf/bpftool_btf_dump_sorted.expected Normal file
View File

@ -0,0 +1,48 @@
#ifndef __VMLINUX_H__
#define __VMLINUX_H__
#ifndef BPF_NO_PRESERVE_ACCESS_INDEX
#pragma clang attribute push (__attribute__((preserve_access_index)), apply_to = record)
#endif
#ifndef __ksym
#define __ksym __attribute__((section(".ksyms")))
#endif
#ifndef __weak
#define __weak __attribute__((weak))
#endif
#ifndef __bpf_fastcall
#if __has_attribute(bpf_fastcall)
#define __bpf_fastcall __attribute__((bpf_fastcall))
#else
#define __bpf_fastcall
#endif
#endif
enum {
E0 = 1,
};
struct holey {
int c;
long: 32;
long: 32;
int tail;
};
struct s {
int f;
};
/* BPF kfuncs */
#ifndef BPF_NO_KFUNC_PROTOTYPES
#endif
#ifndef BPF_NO_PRESERVE_ACCESS_INDEX
#pragma clang attribute pop
#endif
#endif /* __VMLINUX_H__ */

48
tools/testing/selftests/bpf/bpftool_btf_dump_unsorted.expected Normal file
View File

@ -0,0 +1,48 @@
#ifndef __VMLINUX_H__
#define __VMLINUX_H__
#ifndef BPF_NO_PRESERVE_ACCESS_INDEX
#pragma clang attribute push (__attribute__((preserve_access_index)), apply_to = record)
#endif
#ifndef __ksym
#define __ksym __attribute__((section(".ksyms")))
#endif
#ifndef __weak
#define __weak __attribute__((weak))
#endif
#ifndef __bpf_fastcall
#if __has_attribute(bpf_fastcall)
#define __bpf_fastcall __attribute__((bpf_fastcall))
#else
#define __bpf_fastcall
#endif
#endif
struct holey {
int c;
long: 32;
long: 32;
int tail;
};
enum {
E0 = 1,
};
struct s {
int f;
};
/* BPF kfuncs */
#ifndef BPF_NO_KFUNC_PROTOTYPES
#endif
#ifndef BPF_NO_PRESERVE_ACCESS_INDEX
#pragma clang attribute pop
#endif
#endif /* __VMLINUX_H__ */

7
tools/testing/selftests/bpf/bpftool_helpers.c
View File

@ -53,6 +53,7 @@ static int run_command(char *args, char *output_buf, size_t output_max_len)
static char bpftool_path[PATH_MAX] = {};
bool suppress_output = !(output_buf && output_max_len);
char command[BPFTOOL_FULL_CMD_MAX_LEN];
size_t n;
FILE *f;
int ret;
@ -68,8 +69,10 @@ static int run_command(char *args, char *output_buf, size_t output_max_len)
if (!f)
return 1;
if (!suppress_output)
fread(output_buf, 1, output_max_len, f);
if (!suppress_output) {
n = fread(output_buf, 1, output_max_len - 1, f);
output_buf[n] = '\0';
}
ret = pclose(f);
return ret;

2
tools/testing/selftests/bpf/config
View File

@ -12,7 +12,9 @@ CONFIG_BPF_SYSCALL=y
# CONFIG_BPF_UNPRIV_DEFAULT_OFF is not set
CONFIG_CGROUP_BPF=y
CONFIG_CRYPTO_HMAC=y
CONFIG_CRYPTO_MLDSA=y
CONFIG_CRYPTO_SHA256=y
CONFIG_CRYPTO_SHA512=y
CONFIG_CRYPTO_USER_API=y
CONFIG_CRYPTO_USER_API_HASH=y
CONFIG_CRYPTO_USER_API_SKCIPHER=y

31
tools/testing/selftests/bpf/libarena/Makefile
View File

@ -27,10 +27,17 @@ BPFDIR=$(abspath $(LIBARENA)/..)
INCLUDE_DIR ?= $(BPFDIR)/tools/include
LIBBPF_INCLUDE ?= $(INCLUDE_DIR)
# Scan src/ and selftests/ to generate the final binaries
LIBARENA_SOURCES = $(wildcard $(LIBARENA)/src/*.bpf.c) $(wildcard $(LIBARENA)/selftests/*.bpf.c)
LIBARENA_OBJECTS = $(notdir $(LIBARENA_SOURCES:.bpf.c=.bpf.o))
LIBARENA_OBJECTS_ASAN = $(notdir $(LIBARENA_SOURCES:.bpf.c=_asan.bpf.o))
# Build selftests and benchmarks into separate BPF objects and skeletons.
LIBARENA_CORE_SOURCES = $(wildcard $(LIBARENA)/src/*.bpf.c)
LIBARENA_TEST_SOURCES = $(wildcard $(LIBARENA)/selftests/*.bpf.c)
LIBARENA_BENCH_SOURCES = $(wildcard $(LIBARENA)/benchs/*.bpf.c)
LIBARENA_OBJECTS = $(notdir $(LIBARENA_CORE_SOURCES:.bpf.c=.bpf.o) \
$(LIBARENA_TEST_SOURCES:.bpf.c=.bpf.o))
LIBARENA_OBJECTS_ASAN = $(notdir $(LIBARENA_CORE_SOURCES:.bpf.c=_asan.bpf.o) \
$(LIBARENA_TEST_SOURCES:.bpf.c=_asan.bpf.o))
LIBARENA_BENCH_OBJECTS = $(notdir $(LIBARENA_CORE_SOURCES:.bpf.c=.bpf.o) \
$(LIBARENA_BENCH_SOURCES:.bpf.c=.bpf.o))
INCLUDES = -I$(LIBARENA)/include -I$(BPFDIR)
ifneq ($(INCLUDE_DIR),)
@ -53,17 +60,25 @@ override BPF_CFLAGS += -O2 -g
override BPF_CFLAGS += -Wno-incompatible-pointer-types-discards-qualifiers
# Required for suppressing harmless vmlinux.h-related warnings.
override BPF_CFLAGS += -Wno-missing-declarations
override BPF_CFLAGS += -fno-strict-aliasing
override BPF_CFLAGS += $(INCLUDES)
CFLAGS = -O2 -no-pie
CFLAGS += $(INCLUDES)
vpath %.bpf.c $(LIBARENA)/src $(LIBARENA)/selftests
vpath %.c $(LIBARENA)/src $(LIBARENA)/selftests
vpath %.bpf.c $(LIBARENA)/src $(LIBARENA)/selftests $(LIBARENA)/benchs
vpath %.c $(LIBARENA)/src $(LIBARENA)/selftests $(LIBARENA)/benchs
skeletons: libarena.skel.h libarena_asan.skel.h
.PHONY: skeletons
benchmarks: libarena_bench.skel.h
.PHONY: benchmarks
libarena_bench.skel.h: libarena_bench.bpf.o
$(call msg,GEN-SKEL,libarena,$@)
$(Q)$(BPFTOOL) gen skeleton $< name "libarena_bench" > $@
libarena_asan.skel.h: libarena_asan.bpf.o
$(call msg,GEN-SKEL,libarena,$@)
$(Q)$(BPFTOOL) gen skeleton $< name "libarena_asan" > $@
@ -80,6 +95,10 @@ libarena.bpf.o: $(LIBARENA_OBJECTS)
$(call msg,GEN-OBJ,libarena,$@)
$(Q)$(BPFTOOL) gen object $@ $^
libarena_bench.bpf.o: $(LIBARENA_BENCH_OBJECTS)
$(call msg,GEN-OBJ,libarena,$@)
$(Q)$(BPFTOOL) gen object $@ $^
%_asan.bpf.o: %.bpf.c
$(call msg,CLNG-BPF,libarena,$@)
$(Q)$(CLANG) $(BPF_CFLAGS) $(ASAN_FLAGS) -DBPF_ARENA_ASAN $(BPF_TARGET_ENDIAN) -c $< -o $@

51
tools/testing/selftests/bpf/libarena/benchs/bench_malloc.bpf.c Normal file
View File

@ -0,0 +1,51 @@
// SPDX-License-Identifier: LGPL-2.1 OR BSD-2-Clause
/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
#include <libarena/common.h>
#include <libarena/asan.h>
#include <libarena/buddy.h>
u32 bench_alloc_size;
u32 bench_nallocs;
long bench_hits;
long bench_duration_ns;
SEC("syscall")
int bench_malloc(void)
{
void __arena *mem;
u64 start_ns;
u32 i;
start_ns = bpf_ktime_get_ns();
for (i = zero; i < bench_nallocs && can_loop; i++) {
mem = arena_malloc(bench_alloc_size);
if (!mem)
return -ENOMEM;
}
__sync_add_and_fetch(&bench_duration_ns,
bpf_ktime_get_ns() - start_ns);
__sync_add_and_fetch(&bench_hits, i);
return 0;
}
SEC("syscall")
int bench_calloc(void)
{
void __arena *mem;
u64 start_ns;
u32 i;
start_ns = bpf_ktime_get_ns();
for (i = zero; i < bench_nallocs && can_loop; i++) {
mem = arena_calloc(1, bench_alloc_size);
if (!mem)
return -ENOMEM;
}
__sync_add_and_fetch(&bench_duration_ns,
bpf_ktime_get_ns() - start_ns);
__sync_add_and_fetch(&bench_hits, i);
return 0;
}

2
tools/testing/selftests/bpf/libarena/include/bpf_arena_spin_lock.h
View File

@ -1,4 +1,4 @@
// SPDX-License-Identifier: GPL-2.0
// SPDX-License-Identifier: GPL-2.0 OR BSD-2-Clause
/* Copyright (c) 2025 Meta Platforms, Inc. and affiliates. */
#ifndef BPF_ARENA_SPIN_LOCK_H
#define BPF_ARENA_SPIN_LOCK_H

2
tools/testing/selftests/bpf/libarena/include/bpf_atomic.h
View File

@ -1,4 +1,4 @@
// SPDX-License-Identifier: GPL-2.0
// SPDX-License-Identifier: GPL-2.0 OR BSD-2-Clause
/* Copyright (c) 2025 Meta Platforms, Inc. and affiliates. */
#ifndef BPF_ATOMIC_H
#define BPF_ATOMIC_H

1
tools/testing/selftests/bpf/libarena/include/bpf_may_goto.h
View File

@ -1,3 +1,4 @@
// SPDX-License-Identifier: LGPL-2.1 OR BSD-2-Clause
#pragma once
/*

30
tools/testing/selftests/bpf/libarena/include/libarena/bitmap.h
View File

@ -1,5 +1,8 @@
// SPDX-License-Identifier: LGPL-2.1 OR BSD-2-Clause
#pragma once
#include <bpf_atomic.h>
#define BITS_PER_BYTE 8
#define BYTES_TO_BITS(nb) ((nb) * BITS_PER_BYTE)
@ -15,11 +18,8 @@ struct arena_bitmap {
struct arena_bitmap __arena *bmp_alloc(size_t bits);
void bmp_free(struct arena_bitmap __arena *bmp);
void __bmp_set_bit(u32 bit, struct arena_bitmap __arena *bmp);
void __bmp_clear_bit(u32 bit, struct arena_bitmap __arena *bmp);
void bmp_set_bit(u32 bit, struct arena_bitmap __arena *bmp);
void bmp_clear_bit(u32 bit, struct arena_bitmap __arena *bmp);
bool bmp_test_bit(u32 bit, struct arena_bitmap __arena *bmp);
bool bmp_test_and_clear_bit(u32 bit, struct arena_bitmap __arena *bmp);
bool bmp_test_and_set_bit(u32 bit, struct arena_bitmap __arena *bmp);
@ -32,3 +32,27 @@ void bmp_copy(size_t bits, struct arena_bitmap __arena *dst, struct arena_bitmap
bool bmp_intersects(size_t bits, struct arena_bitmap __arena *arg1, struct arena_bitmap __arena *arg2);
bool bmp_subset(size_t bits, struct arena_bitmap __arena *big, struct arena_bitmap __arena *small);
void bmp_print(size_t bits, struct arena_bitmap __arena *bmp);
static __always_inline
void __bmp_set_bit(u32 bit, struct arena_bitmap __arena *bmp)
{
volatile u64 __arena *word = &bmp->bits[BIT_WORD(bit)];
*word |= BIT_MASK(bit);
}
static __always_inline
void __bmp_clear_bit(u32 bit, struct arena_bitmap __arena *bmp)
{
volatile u64 __arena *word = &bmp->bits[BIT_WORD(bit)];
*word &= ~BIT_MASK(bit);
}
static __always_inline
bool bmp_test_bit(u32 bit, struct arena_bitmap __arena *bmp)
{
u64 word = READ_ONCE(bmp->bits[BIT_WORD(bit)]);
return word & BIT_MASK(bit);
}

71
tools/testing/selftests/bpf/libarena/include/libarena/common.h
View File

@ -49,6 +49,7 @@ extern volatile u64 asan_violated;
int arena_fls(__u64 word);
void __arena *arena_malloc(size_t size);
void __arena *arena_calloc(size_t ncount, size_t size);
void arena_free(void __arena *ptr);
/*
@ -61,6 +62,76 @@ void arena_free(void __arena *ptr);
*/
#define arena_subprog_init() do { asm volatile ("" :: "r"(&arena)); } while (0)
/*
* BPF does not currently support the memset intrinsics. for large
* sequential copies, or assignments of large data structures,
* the frontend will generate an intrinsic that causes the BPF
* backend to exit due to a missing implementation. Provide
* implementations for the intrinsic.
*/
static inline int arena_memset(s8 __arena *dst, s8 val, size_t size)
{
size_t headalign;
size_t tailalign;
u8 uval = (u8)val;
size_t val64;
size_t i;
/*
* Calculate how many bytes to the next word-aligned one.
* We get this by truncating the 2s complement of the
* pointer to the last 3 bits. Intuitively, since
*
* The N LSBs of dst and -dst add to 1 << N, which
* is why dst + (-dst) = 0x0ULL through overflow. So the
* last N = 3 bits of the negative are the number of
* bytes to align dst on the last 3 bits.
*
*/
headalign = -(u64)dst & (sizeof(u64) - 1);
if (!headalign || size < headalign)
goto ptraligned;
for (i = zero; i < headalign && can_loop; i++)
dst[i] = uval;
dst += headalign;
size -= headalign;
ptraligned:
/*
* Make a word with all bytes equal to the byte we are setting.
* Since 1 byte -> 2 hex digits.
*
* Shifting the value by a 0 bytes is equal to multiplication by 0x01
* Shifting by 1 bytes is equal to multiplication by 0x01 << 8,
* ...
* Shifting by 7 bytes is equal to multiplication by 0x01 << 56.
*
* End operation to replicate the byte into all the bytes of a word
* is (since a | b = a + b when a & b == 0):
*
* val + val * (1UL << 8) + val * (1UL << 16) + .. + val * (1UL << 56)
* = val * (1UL << 56 + 1UL << 48 + ... + 1UL << 0)
* = val * (0x01UL << 56 | 0x01UL << 48 + ... + 1UL << 0)
* = val * 0x0101 0101 0101 0101
*/
val64 = (u8)val * 0x0101010101010101ULL;
/* Pointer is now aligned, use word-aligned assignments. */
for (i = zero; i < size / sizeof(u64) && can_loop; i++)
((u64 __arena *)dst)[i] = val64;
/* Go back to byte-aligned for the tail. */
tailalign = size % sizeof(u64);
dst += size - tailalign;
for (i = zero; i < tailalign && can_loop; i++)
dst[i] = uval;
return 0;
}
#else /* ! __BPF__ */
#include <stdint.h>

3
tools/testing/selftests/bpf/libarena/selftests/test_bitmap.bpf.c
View File

@ -1,3 +1,6 @@
// SPDX-License-Identifier: LGPL-2.1 OR BSD-2-Clause
/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
#include <libarena/common.h>
#include <libarena/asan.h>

21
tools/testing/selftests/bpf/libarena/src/asan.bpf.c
View File

@ -103,23 +103,6 @@ volatile bool asan_inited = false;
*/
volatile bool asan_report_once = false;
/*
* BPF does not currently support the memset/memcpy/memcmp intrinsics.
* For large sequential copies, or assignments of large data structures,
* the frontend will generate an intrinsic that causes the BPF backend
* to exit due to a missing implementation. Provide a simple implementation
* just for memset to use it for poisoning/unpoisoning the map.
*/
__weak int asan_memset(s8 __arena *dst, s8 val, size_t size)
{
size_t i;
for (i = zero; i < size && can_loop; i++)
dst[i] = val;
return 0;
}
/* Validate a 1-byte access, always within a single byte. */
static __always_inline bool memory_is_poisoned_1(s8 __arena *addr)
{
@ -422,7 +405,7 @@ __hidden __noasan int asan_poison(void __arena *addr, s8 val, size_t size)
shadow = mem_to_shadow(addr);
len = size >> ASAN_SHADOW_SHIFT;
asan_memset(shadow, val, len);
arena_memset(shadow, val, len);
return 0;
}
@ -463,7 +446,7 @@ __hidden __noasan int asan_unpoison(void __arena *addr, size_t size)
shadow = mem_to_shadow(addr);
len = size >> ASAN_SHADOW_SHIFT;
asan_memset(shadow, 0, len);
arena_memset(shadow, 0, len);
/*
* If we are allocating a non-granule aligned region, we need to adjust

18
tools/testing/selftests/bpf/libarena/src/bitmap.bpf.c
View File

@ -34,24 +34,6 @@ void bmp_free(struct arena_bitmap __arena *bmp)
arena_free(bmp);
}
__weak
void __bmp_set_bit(u32 bit, struct arena_bitmap __arena *bmp)
{
bmp->bits[BIT_WORD(bit)] |= BIT_MASK(bit);
}
__weak
void __bmp_clear_bit(u32 bit, struct arena_bitmap __arena *bmp)
{
bmp->bits[BIT_WORD(bit)] &= ~BIT_MASK(bit);
}
__weak
bool bmp_test_bit(u32 bit, struct arena_bitmap __arena *bmp)
{
return bmp->bits[BIT_WORD(bit)] & BIT_MASK(bit);
}
__weak
bool bmp_test_and_clear_bit(u32 bit, struct arena_bitmap __arena *bmp)
{

44
tools/testing/selftests/bpf/libarena/src/buddy.bpf.c
View File

@ -5,6 +5,8 @@
#include <libarena/asan.h>
#include <libarena/buddy.h>
#include <bpf_arena_spin_lock.h>
/*
* Buddy allocator arena-based implementation.
*
@ -45,15 +47,8 @@ enum {
BUDDY_CHUNK_PAGES = BUDDY_CHUNK_BYTES / __PAGE_SIZE
};
static inline int buddy_lock(struct buddy __arena *buddy)
{
return arena_spin_lock(&buddy->lock);
}
static inline void buddy_unlock(struct buddy __arena *buddy)
{
arena_spin_unlock(&buddy->lock);
}
#define buddy_lock(buddy, flags) (arena_spin_lock_irqsave(&(buddy)->lock, (flags)))
#define buddy_unlock(buddy, flags) (arena_spin_unlock_irqrestore(&(buddy)->lock, (flags)))
/*
* Reserve part of the arena address space for the allocator. We use
@ -385,6 +380,7 @@ static struct buddy_chunk __arena *buddy_chunk_get(struct buddy __arena *buddy)
{
u64 order, ord, min_order, max_order;
struct buddy_chunk __arena *chunk;
unsigned long flags;
size_t left;
int power2;
u64 vaddr;
@ -416,7 +412,7 @@ static struct buddy_chunk __arena *buddy_chunk_get(struct buddy __arena *buddy)
return NULL;
}
if (buddy_lock(buddy)) {
if (buddy_lock(buddy, flags)) {
/*
* We cannot reclaim the vaddr space, but that is ok - this
* operation should always succeed. The error path is to catch
@ -520,7 +516,7 @@ static struct buddy_chunk __arena *buddy_chunk_get(struct buddy __arena *buddy)
arena_stderr(
"chunk has size of 0x%lx bytes (left %lx bytes)\n",
sizeof(*chunk), left);
buddy_unlock(buddy);
buddy_unlock(buddy, flags);
return NULL;
}
@ -531,7 +527,7 @@ static struct buddy_chunk __arena *buddy_chunk_get(struct buddy __arena *buddy)
order = (power2 >= BUDDY_MIN_ALLOC_SHIFT) ? power2 - BUDDY_MIN_ALLOC_SHIFT : 0;
if (idx_set_allocated(chunk, idx, true)) {
buddy_unlock(buddy);
buddy_unlock(buddy, flags);
return NULL;
}
@ -547,7 +543,7 @@ static struct buddy_chunk __arena *buddy_chunk_get(struct buddy __arena *buddy)
*/
min_order = left ? order + 1 : order;
if (add_leftovers_to_freelist(chunk, idx, min_order, max_order)) {
buddy_unlock(buddy);
buddy_unlock(buddy, flags);
return NULL;
}
@ -556,7 +552,7 @@ static struct buddy_chunk __arena *buddy_chunk_get(struct buddy __arena *buddy)
max_order = order;
}
buddy_unlock(buddy);
buddy_unlock(buddy, flags);
return chunk;
}
@ -564,6 +560,7 @@ static struct buddy_chunk __arena *buddy_chunk_get(struct buddy __arena *buddy)
__weak int buddy_init(struct buddy __arena *buddy)
{
struct buddy_chunk __arena *chunk;
unsigned long flags;
int ret;
if (!asan_ready())
@ -579,7 +576,7 @@ __weak int buddy_init(struct buddy __arena *buddy)
chunk = buddy_chunk_get(buddy);
if (buddy_lock(buddy)) {
if (buddy_lock(buddy, flags)) {
bpf_arena_free_pages(&arena, chunk, BUDDY_CHUNK_PAGES);
return -EINVAL;
}
@ -591,7 +588,7 @@ __weak int buddy_init(struct buddy __arena *buddy)
/* Put the chunk at the beginning of the list. */
buddy->first_chunk = chunk;
buddy_unlock(buddy);
buddy_unlock(buddy, flags);
return chunk ? 0 : -ENOMEM;
}
@ -730,9 +727,10 @@ static u64 buddy_alloc_from_existing_chunks(struct buddy __arena *buddy, int ord
*/
static u64 buddy_alloc_from_new_chunk(struct buddy __arena *buddy, struct buddy_chunk __arena *chunk, int order)
{
unsigned long flags;
u64 address;
if (buddy_lock(buddy))
if (buddy_lock(buddy, flags))
return (u64)NULL;
@ -745,7 +743,7 @@ static u64 buddy_alloc_from_new_chunk(struct buddy __arena *buddy, struct buddy_
address = buddy_chunk_alloc(buddy->first_chunk, order);
buddy_unlock(buddy);
buddy_unlock(buddy, flags);
return (u64)address;
}
@ -754,6 +752,7 @@ void __arena *buddy_alloc(struct buddy __arena *buddy, size_t size)
{
void __arena *address = NULL;
struct buddy_chunk __arena *chunk;
unsigned long flags;
int order;
if (!buddy)
@ -765,11 +764,11 @@ void __arena *buddy_alloc(struct buddy __arena *buddy, size_t size)
return NULL;
}
if (buddy_lock(buddy))
if (buddy_lock(buddy, flags))
return NULL;
address = (u8 __arena *)buddy_alloc_from_existing_chunks(buddy, order);
buddy_unlock(buddy);
buddy_unlock(buddy, flags);
if (address)
goto done;
@ -880,6 +879,7 @@ static __always_inline int buddy_free_unlocked(struct buddy __arena *buddy, u64
__weak int buddy_free(struct buddy __arena *buddy, void __arena *addr)
{
unsigned long flags;
int ret;
if (!buddy)
@ -889,13 +889,13 @@ __weak int buddy_free(struct buddy __arena *buddy, void __arena *addr)
if (!addr)
return 0;
ret = buddy_lock(buddy);
ret = buddy_lock(buddy, flags);
if (ret)
return ret;
ret = buddy_free_unlocked(buddy, (u64)addr);
buddy_unlock(buddy);
buddy_unlock(buddy, flags);
return ret;
}

27
tools/testing/selftests/bpf/libarena/src/common.bpf.c
View File

@ -1,5 +1,7 @@
// SPDX-License-Identifier: LGPL-2.1 OR BSD-2-Clause
/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
#include <limits.h>
#include <libarena/common.h>
#include <libarena/asan.h>
#include <libarena/buddy.h>
@ -55,10 +57,33 @@ __weak void __arena *arena_malloc(size_t size)
return buddy_alloc(&buddy, size);
}
__weak void __arena *arena_calloc(size_t ncount, size_t size)
{
void __arena *mem;
size_t total;
/*
* Ideally we'd be using __builtin_mul_overflow here,
* but the BPF compiler backend doesn't implement __multi3.
* There are ways to optimize the division away from the
* overflow check, but any costs are dwarfed by the
* buddy_alloc() call. Keep it simple for now.
*/
if (unlikely(ncount && size >= ULLONG_MAX / ncount))
return NULL;
total = ncount * size;
mem = buddy_alloc(&buddy, total);
if (likely(mem))
arena_memset(mem, 0, total);
return mem;
}
__weak void arena_free(void __arena *ptr)
{
buddy_free(&buddy, ptr);
}
char _license[] SEC("license") = "GPL";

46
tools/testing/selftests/bpf/network_helpers.c
View File

@ -49,6 +49,8 @@
errno = __save; \
})
#define CONNECT_MIN_TIMEOUT_MS 5000
struct ipv4_packet pkt_v4 = {
.eth.h_proto = __bpf_constant_htons(ETH_P_IP),
.iph.ihl = 5,
@ -291,6 +293,37 @@ int client_socket(int family, int type,
return -1;
}
static int connect_with_timeout(int fd, const struct sockaddr_storage *addr,
socklen_t addrlen, int timeout_ms)
{
int connect_timeout_ms = MAX(timeout_ms, CONNECT_MIN_TIMEOUT_MS);
/*
* Override timeout configuration with a larger value for the
* connection
*/
if (settimeo(fd, connect_timeout_ms)) {
log_err("Failed to set connect timeout");
return -1;
}
if (connect(fd, (const struct sockaddr *)addr, addrlen)) {
log_err("Failed to connect");
return -1;
}
/*
* If the timeout configured by the test is different from the
* connect timeout, restore it
*/
if (timeout_ms != connect_timeout_ms && settimeo(fd, timeout_ms)) {
log_err("Failed to set timeout for connected socket");
return -1;
}
return 0;
}
int connect_to_addr(int type, const struct sockaddr_storage *addr, socklen_t addrlen,
const struct network_helper_opts *opts)
{
@ -305,8 +338,7 @@ int connect_to_addr(int type, const struct sockaddr_storage *addr, socklen_t add
return -1;
}
if (connect(fd, (const struct sockaddr *)addr, addrlen)) {
log_err("Failed to connect to server");
if (connect_with_timeout(fd, addr, addrlen, opts->timeout_ms)) {
save_errno_close(fd);
return -1;
}
@ -376,20 +408,12 @@ int connect_fd_to_fd(int client_fd, int server_fd, int timeout_ms)
struct sockaddr_storage addr;
socklen_t len = sizeof(addr);
if (settimeo(client_fd, timeout_ms))
return -1;
if (getsockname(server_fd, (struct sockaddr *)&addr, &len)) {
log_err("Failed to get server addr");
return -1;
}
if (connect(client_fd, (const struct sockaddr *)&addr, len)) {
log_err("Failed to connect to server");
return -1;
}
return 0;
return connect_with_timeout(client_fd, &addr, len, timeout_ms);
}
int make_sockaddr(int family, const char *addr_str, __u16 port,

53
tools/testing/selftests/bpf/prog_tests/aggregate_ret.c Normal file
View File

@ -0,0 +1,53 @@
// SPDX-License-Identifier: GPL-2.0
/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
#include <test_progs.h>
#include <bpf/btf.h>
#include "aggregate_ret_func.skel.h"
#include "aggregate_ret_kfunc.skel.h"
#include "aggregate_ret_kfunc_arena.skel.h"
static bool testmod_has_arena_tagged_member(void)
{
struct btf *vmlinux_btf, *module_btf = NULL;
const struct btf_type *t;
bool tagged = false;
__s32 id;
vmlinux_btf = btf__load_vmlinux_btf();
if (!vmlinux_btf)
return false;
module_btf = btf__load_module_btf("bpf_testmod", vmlinux_btf);
if (!module_btf)
goto out;
/* prog_test_ret_arena::a is 'void __arena_tag *': PTR -> TYPE_TAG -> void */
id = btf__find_by_name_kind(module_btf, "prog_test_ret_arena", BTF_KIND_STRUCT);
if (id <= 0)
goto out;
t = btf__type_by_id(module_btf, btf_members(btf__type_by_id(module_btf, id))[0].type);
if (!t || !btf_is_ptr(t))
goto out;
t = btf__type_by_id(module_btf, t->type);
tagged = t && btf_is_type_tag(t) &&
!strcmp(btf__name_by_offset(module_btf, t->name_off), "arena");
out:
btf__free(module_btf);
btf__free(vmlinux_btf);
return tagged;
}
void test_aggregate_ret(void)
{
RUN_TESTS(aggregate_ret_func);
RUN_TESTS(aggregate_ret_kfunc);
if (testmod_has_arena_tagged_member())
RUN_TESTS(aggregate_ret_kfunc_arena);
else
test__skip();
}

6
tools/testing/selftests/bpf/prog_tests/attach_probe.c
View File

@ -543,8 +543,10 @@ static void test_kprobe_sleepable(void)
return;
/* sleepable kprobe test case needs flags set before loading */
if (!ASSERT_OK(bpf_program__set_flags(skel->progs.handle_kprobe_sleepable,
BPF_F_SLEEPABLE), "kprobe_sleepable_flags"))
if (!ASSERT_OK(bpf_program__set_flags(
skel->progs.handle_kprobe_sleepable,
bpf_program__flags(skel->progs.handle_kprobe_sleepable) | BPF_F_SLEEPABLE),
"kprobe_sleepable_flags"))
goto cleanup;
if (!ASSERT_OK(test_attach_kprobe_sleepable__load(skel),

2
tools/testing/selftests/bpf/prog_tests/bpf_verif_scale.c
View File

@ -35,7 +35,7 @@ static int check_load(const char *file, enum bpf_prog_type type)
}
bpf_program__set_type(prog, type);
bpf_program__set_flags(prog, testing_prog_flags());
bpf_program__set_flags(prog, bpf_program__flags(prog) | testing_prog_flags());
bpf_program__set_log_level(prog, 4 | extra_prog_load_log_flags);
err = bpf_object__load(obj);

178
tools/testing/selftests/bpf/prog_tests/bpftool_btf_dump.c Normal file
View File

@ -0,0 +1,178 @@
// SPDX-License-Identifier: GPL-2.0
#include <test_progs.h>
#include <bpftool_helpers.h>
#include <bpf/btf.h>
#include <limits.h>
#include <unistd.h>
#include "btf_helpers.h"
#include "testing_helpers.h"
#define DUMP_BUF_SZ (16 * 1024)
#define EXPECTED_SORTED "bpftool_btf_dump_sorted.expected"
#define EXPECTED_UNSORTED "bpftool_btf_dump_unsorted.expected"
/*
* struct holey {
* int c;
* <64-bit hole>
* int tail;
* };
* enum { E0 = 1 };
* struct s { int f; };
*/
static struct btf *mk_btf(void)
{
struct btf *btf;
btf = btf__new_empty();
if (!ASSERT_OK_PTR(btf, "new_empty"))
return NULL;
btf__add_int(btf, "int", 4, BTF_INT_SIGNED);
btf__add_int(btf, "long int", 4, BTF_INT_SIGNED);
btf__add_struct(btf, "holey", 16);
btf__add_field(btf, "c", 1, 0, 0);
btf__add_field(btf, "tail", 1, 96, 0);
btf__add_enum(btf, NULL, 4);
btf__add_enum_value(btf, "E0", 1);
btf__add_struct(btf, "s", 4);
btf__add_field(btf, "f", 1, 0, 0);
VALIDATE_RAW_BTF(
btf,
"[1] INT 'int' size=4 bits_offset=0 nr_bits=32 encoding=SIGNED",
"[2] INT 'long int' size=4 bits_offset=0 nr_bits=32 encoding=SIGNED",
"[3] STRUCT 'holey' size=16 vlen=2\n"
"\t'c' type_id=1 bits_offset=0\n"
"\t'tail' type_id=1 bits_offset=96",
"[4] ENUM '(anon)' encoding=UNSIGNED size=4 vlen=1\n"
"\t'E0' val=1",
"[5] STRUCT 's' size=4 vlen=1\n"
"\t'f' type_id=1 bits_offset=0");
return btf;
}
static int btf_to_tmpfile(const struct btf *btf, char *path)
{
ssize_t written;
const void *raw;
__u32 sz;
int fd;
raw = btf__raw_data(btf, &sz);
if (!ASSERT_OK_PTR(raw, "raw_data"))
return -1;
snprintf(path, PATH_MAX, "/tmp/bpftool_btf_dump.XXXXXX");
fd = mkstemp(path);
if (!ASSERT_OK_FD(fd, "mkstemp_btf"))
return -1;
written = write(fd, raw, sz);
close(fd);
if (!ASSERT_EQ(written, sz, "write_btf")) {
unlink(path);
return -1;
}
return 0;
}
static char *dump_c(const char *btf_path, bool sorted)
{
char args[MAX_BPFTOOL_CMD_LEN];
char *buf;
int err;
buf = malloc(DUMP_BUF_SZ);
if (!ASSERT_OK_PTR(buf, "alloc_dump"))
return NULL;
snprintf(args, sizeof(args), "btf dump file %s format c%s",
btf_path, sorted ? "" : " unsorted");
err = get_bpftool_command_output(args, buf, DUMP_BUF_SZ);
if (!ASSERT_OK(err, "btf_dump_format_c")) {
free(buf);
return NULL;
}
return buf;
}
static char *read_expected(const char *path)
{
char *buf = NULL;
size_t cap = 0;
FILE *f;
int err;
f = fopen(path, "r");
if (!f) {
err = errno;
PRINT_FAIL("can't open expected output '%s': errno %d\n", path, err);
return NULL;
}
/* no NUL in a generated header, so this reads to the end */
if (getdelim(&buf, &cap, '\0', f) < 0) {
err = errno;
PRINT_FAIL("can't read expected output '%s': errno %d\n", path, err);
free(buf);
buf = NULL;
}
fclose(f);
return buf;
}
static void test_dump(const char *btf_path, bool sorted)
{
const char *exp_path;
char *dump, *exp;
int err;
exp_path = sorted ? EXPECTED_SORTED : EXPECTED_UNSORTED;
dump = dump_c(btf_path, sorted);
if (!dump)
return;
exp = read_expected(exp_path);
if (!exp)
goto out_dump;
err = compare_text_to_expected(dump, exp);
ASSERT_OK(err, sorted ? "cmp_sorted" : "cmp_unsorted");
free(exp);
out_dump:
free(dump);
}
void test_bpftool_btf_dump(void)
{
char path[PATH_MAX];
struct btf *btf;
btf = mk_btf();
if (!btf)
return;
if (btf_to_tmpfile(btf, path))
goto out_btf;
if (test__start_subtest("c_sorted"))
test_dump(path, true);
if (test__start_subtest("c_unsorted"))
test_dump(path, false);
unlink(path);
out_btf:
btf__free(btf);
}

127
tools/testing/selftests/bpf/prog_tests/btf_dump.c
View File

@ -1,8 +1,7 @@
// SPDX-License-Identifier: GPL-2.0
#include <test_progs.h>
#include <bpf/btf.h>
static int duration = 0;
#include "testing_helpers.h"
void btf_dump_printf(void *ctx, const char *fmt, va_list args)
{
@ -45,12 +44,75 @@ static int btf_dump_all_types(const struct btf *btf, void *ctx)
return err;
}
/*
* Expected output is embedded in the test case source, between
* START-EXPECTED-OUTPUT and END-EXPECTED-OUTPUT markers. A region is either
* plain C, where a declaration doubles as its own expectation, or C wrapped in
* a block comment, where the rendered form differs from the source. All
* regions of a file concatenate into one expectation, compared against one
* whole-file dump.
*
* Returns a malloc'd buffer for the caller to free, or NULL on failure.
*/
static char *read_expected_output(const char *path)
{
size_t out_sz = 0, line_cap = 0;
char *out = NULL, *line = NULL;
bool in_region = false;
FILE *f, *out_file;
f = fopen(path, "r");
if (!f)
return NULL;
out_file = open_memstream(&out, &out_sz);
if (!out_file) {
fclose(f);
return NULL;
}
while (getline(&line, &line_cap, f) > 0) {
const char *p;
if (strstr(line, "START-EXPECTED-OUTPUT")) {
in_region = true;
continue;
}
if (strstr(line, "END-EXPECTED-OUTPUT"))
in_region = false;
if (!in_region)
continue;
p = line + strspn(line, " \t");
/* opening or closing line of a commented out region */
if (!strncmp(p, "/*", 2) || !strncmp(p, "*/", 2))
continue;
/*
* Only a '*' directly after the indentation is a comment
* prefix. Without one the line is taken as it is, leading
* whitespace included.
*/
p = *p == '*' ? p + 1 : line;
fputs(p, out_file);
}
free(line);
fclose(f);
fclose(out_file);
return out;
}
static int test_btf_dump_case(int n, struct btf_dump_test_case *t)
{
char test_file[256], out_file[256], diff_cmd[1024];
char *dump = NULL, *expected = NULL;
struct btf *btf = NULL;
int err = 0, fd = -1;
FILE *f = NULL;
char test_file[256];
size_t dump_sz = 0;
int err = 0;
FILE *f;
snprintf(test_file, sizeof(test_file), "%s.bpf.o", t->file);
@ -69,29 +131,21 @@ static int test_btf_dump_case(int n, struct btf_dump_test_case *t)
if (!t->known_ptr_sz) {
btf__set_pointer_size(btf, 8);
} else {
CHECK(btf__pointer_size(btf) != 8, "ptr_sz", "exp %d, got %zu\n",
8, btf__pointer_size(btf));
size_t ptr_sz = btf__pointer_size(btf);
ASSERT_EQ(ptr_sz, (size_t)8, "ptr_sz");
}
snprintf(out_file, sizeof(out_file), "/tmp/%s.output.XXXXXX", t->file);
fd = mkstemp(out_file);
if (!ASSERT_GE(fd, 0, "create_tmp")) {
err = fd;
goto done;
}
f = fdopen(fd, "w");
if (CHECK(f == NULL, "open_tmp", "failed to open file: %s(%d)\n",
strerror(errno), errno)) {
close(fd);
f = open_memstream(&dump, &dump_sz);
if (!ASSERT_OK_PTR(f, "open_memstream")) {
err = -errno;
goto done;
}
err = btf_dump_all_types(btf, f);
fclose(f);
close(fd);
if (CHECK(err, "btf_dump", "failure during C dumping: %d\n", err)) {
if (!ASSERT_OK(err, "btf_dump"))
goto done;
}
snprintf(test_file, sizeof(test_file), "progs/%s.c", t->file);
if (access(test_file, R_OK) == -1)
@ -100,28 +154,23 @@ static int test_btf_dump_case(int n, struct btf_dump_test_case *t)
* without preserving the directory structure.
*/
snprintf(test_file, sizeof(test_file), "%s.c", t->file);
/*
* Diff test output and expected test output, contained between
* START-EXPECTED-OUTPUT and END-EXPECTED-OUTPUT lines in test case.
* For expected output lines, everything before '*' is stripped out.
* Also lines containing comment start and comment end markers are
* ignored.
*/
snprintf(diff_cmd, sizeof(diff_cmd),
"awk '/START-EXPECTED-OUTPUT/{out=1;next} "
"/END-EXPECTED-OUTPUT/{out=0} "
"/\\/\\*|\\*\\//{next} " /* ignore comment start/end lines */
"out {sub(/^[ \\t]*\\*/, \"\"); print}' '%s' | diff -u - '%s'",
test_file, out_file);
err = system(diff_cmd);
if (CHECK(err, "diff",
"differing test output, output=%s, err=%d, diff cmd:\n%s\n",
out_file, err, diff_cmd))
goto done;
remove(out_file);
expected = read_expected_output(test_file);
if (!ASSERT_OK_PTR(expected, "read_expected_output")) {
err = -errno;
goto done;
}
/*
* The mismatch has already been reported, so this only has to
* register the failure. ASSERT_OK() would append a stale errno to it.
*/
err = compare_text_to_expected(dump, expected);
ASSERT_EQ(err, 0, "compare_text_to_expected");
done:
free(expected);
free(dump);
btf__free(btf);
return err;
}

10
tools/testing/selftests/bpf/prog_tests/core_reloc.c
View File

@ -7,6 +7,7 @@
#include <sys/mman.h>
#include <sys/syscall.h>
#include <bpf/btf.h>
#include "bpftool_helpers.h"
static int duration = 0;
@ -985,16 +986,15 @@ static size_t roundup_page(size_t sz)
static int run_btfgen(const char *src_btf, const char *dst_btf, const char *objpath)
{
char command[4096];
char args[4096];
int n;
n = snprintf(command, sizeof(command),
"./bpftool gen min_core_btf %s %s %s",
n = snprintf(args, sizeof(args), "gen min_core_btf %s %s %s",
src_btf, dst_btf, objpath);
if (n < 0 || n >= sizeof(command))
if (n < 0 || n >= sizeof(args))
return -1;
return system(command);
return run_bpftool_command(args);
}
static void run_core_reloc_tests(bool use_btfgen)

16
tools/testing/selftests/bpf/prog_tests/fexit_bpf2bpf.c
View File

@ -441,6 +441,20 @@ static void test_func_replace_int_with_void(void)
" doesn't match type INT of global_func2()");
}
static void test_func_replace_ret_pair(void)
{
const char *msg = "Return type of new_agg_ret_target_func() has size 8 "
"and of agg_ret_target_func() size 16";
/*
* An extension cannot replace a function whose return value comes back
* in the R0:R2 pair: only R0 is checked at the extension's exit, so it
* would leave R2 stale for the target's callers.
*/
test_obj_load_failure_common("freplace_ret_pair.bpf.o",
"./aggregate_ret_target.bpf.o", msg);
}
static int find_prog_btf_id(const char *name, __u32 attach_prog_fd)
{
struct bpf_prog_info info = {};
@ -660,6 +674,8 @@ void serial_test_fexit_bpf2bpf(void)
test_func_replace_progmap();
if (test__start_subtest("freplace_int_with_void"))
test_func_replace_int_with_void();
if (test__start_subtest("freplace_ret_pair"))
test_func_replace_ret_pair();
if (test__start_subtest("freplace_void"))
test_func_replace_void();
if (test__start_subtest("sleepable_fentry_to_xdp"))

96
tools/testing/selftests/bpf/prog_tests/global_data_init.c
View File

@ -68,12 +68,9 @@ static void test_percpu_data_on_cpus(struct bpf_map *map, int map_fd, int prog_f
{
struct test_global_percpu_data__percpu *data = NULL;
int i, err, key = 0, num_online, run = 0;
__u64 args[2] = {0x1234ULL, 0x5678ULL};
size_t data_sz;
bool *online;
LIBBPF_OPTS(bpf_test_run_opts, topts,
.ctx_in = args,
.ctx_size_in = sizeof(args),
.flags = BPF_F_TEST_RUN_ON_CPU,
);
@ -81,7 +78,7 @@ static void test_percpu_data_on_cpus(struct bpf_map *map, int map_fd, int prog_f
if (!ASSERT_OK(err, "parse_cpu_mask_file"))
return;
data_sz = map ? bpf_map__value_size(map) : sizeof(*data);
data_sz = sizeof(*data);
data = calloc(1, data_sz);
if (!ASSERT_OK_PTR(data, "calloc percpu data"))
goto out;
@ -109,7 +106,7 @@ static void test_percpu_data_on_cpus(struct bpf_map *map, int map_fd, int prog_f
break;
ASSERT_EQ(*runp, ++run, "run");
ASSERT_EQ(data->cpu_id[0], i, "cpu_id");
ASSERT_EQ(data->cpu_id, i, "cpu_id");
ASSERT_EQ(data->data, 1, "data");
ASSERT_TRUE(data->set, "set");
ASSERT_EQ(data->nums[6], 0xc0de, "nums[6]");
@ -127,7 +124,6 @@ static void test_global_percpu_data_init(void)
{
struct test_global_percpu_data__percpu init_value = {};
struct test_global_percpu_data__percpu *init_data;
const __u32 desired_sz = sysconf(_SC_PAGE_SIZE);
struct test_global_percpu_data *skel = NULL;
size_t init_data_sz;
struct bpf_map *map;
@ -163,9 +159,11 @@ static void test_global_percpu_data_init(void)
if (!ASSERT_EQ(bpf_map__type(map), BPF_MAP_TYPE_PERCPU_ARRAY, "bpf_map__type"))
goto out;
init_value.set = 1;
init_value.data = 2;
init_value.nums[6] = -1;
init_value.struct_data.i = 2;
init_value.struct_data.set = 1;
init_value.struct_data.nums[6] = -1;
err = bpf_map__set_initial_value(map, &init_value, sizeof(init_value));
if (!ASSERT_OK(err, "bpf_map__set_initial_value"))
@ -177,35 +175,24 @@ static void test_global_percpu_data_init(void)
ASSERT_EQ(init_data->data, init_value.data, "init_value data");
ASSERT_EQ(init_data->set, init_value.set, "init_value set");
ASSERT_EQ(init_data->nums[6], init_value.nums[6], "init_value nums[6]");
ASSERT_EQ(init_data->struct_data.i, init_value.struct_data.i, "init_value struct_data.i");
ASSERT_EQ(init_data->struct_data.set, init_value.struct_data.set,
"init_value struct_data.set");
ASSERT_EQ(init_data->struct_data.nums[6], init_value.struct_data.nums[6],
"init_value struct_data.nums[6]");
ASSERT_EQ(init_data_sz, sizeof(init_value), "init_value size");
ASSERT_EQ((void *) init_data, (void *) skel->percpu, "skel->percpu eq init_data");
ASSERT_EQ(skel->percpu->data, init_value.data, "skel->percpu->data");
ASSERT_EQ(skel->percpu->set, init_value.set, "skel->percpu->set");
ASSERT_EQ(skel->percpu->nums[6], init_value.nums[6], "skel->percpu->nums[6]");
ASSERT_EQ(skel->percpu->struct_data.i, init_value.struct_data.i,
"skel->percpu->struct_data.i");
ASSERT_EQ(skel->percpu->struct_data.set, init_value.struct_data.set,
"skel->percpu->struct_data.set");
ASSERT_EQ(skel->percpu->struct_data.nums[6], init_value.struct_data.nums[6],
"skel->percpu->struct_data.nums[6]");
ASSERT_GT(desired_sz, sizeof(init_value), "desired_sz");
err = bpf_map__set_value_size(map, desired_sz);
if (!ASSERT_OK(err, "bpf_map__set_value_size"))
goto out;
if (!ASSERT_EQ(bpf_map__value_size(map), desired_sz, "percpu value size"))
goto out;
if (!ASSERT_NEQ(bpf_map__btf_value_type_id(map), 0, "percpu BTF value type"))
goto out;
init_data = bpf_map__initial_value(map, &init_data_sz);
if (!ASSERT_OK_PTR(init_data, "resized bpf_map__initial_value"))
goto out;
if (!ASSERT_EQ(init_data_sz, desired_sz, "resized initial value size"))
goto out;
if (!ASSERT_EQ(init_data->data, init_value.data, "resized initial value data"))
goto out;
err = test_global_percpu_data__load(skel);
if (!ASSERT_OK(err, "test_global_percpu_data__load"))
goto out;
@ -236,6 +223,67 @@ static void test_global_percpu_data_lskel(void)
test_global_percpu_data_lskel__destroy(lskel);
}
#define PCPU_MIN_UNIT_SIZE (32 << 10)
static void test_global_percpu_data_map_resize(void)
{
const __u32 page_size = sysconf(_SC_PAGE_SIZE);
const __u32 pcpu_max_value_size = ALIGN(PCPU_MIN_UNIT_SIZE, page_size);
const __u32 desired_sz = page_size * 2;
struct test_global_percpu_data *skel;
const int arr_value = -1;
struct bpf_map *map;
int err, prog_fd;
size_t data_sz;
skel = test_global_percpu_data__open();
if (!ASSERT_OK_PTR(skel, "test_global_percpu_data__open"))
return;
map = skel->maps.percpu_arr;
skel->percpu_arr->arr[0] = arr_value;
err = bpf_map__set_value_size(map, desired_sz);
if (!ASSERT_OK(err, "bpf_map__set_value_size"))
goto out;
if (!ASSERT_EQ(bpf_map__value_size(map), desired_sz, "bpf_map__value_size"))
goto out;
if (!ASSERT_NEQ(bpf_map__btf_value_type_id(map), 0, "bpf_map__btf_value_type_id"))
goto out;
skel->percpu_arr = bpf_map__initial_value(map, &data_sz);
if (!ASSERT_EQ(data_sz, desired_sz, "data_sz"))
goto out;
if (!ASSERT_OK_PTR(skel->percpu_arr, "skel->percpu_arr"))
goto out;
if (!ASSERT_EQ(skel->percpu_arr->arr[0], arr_value, "arr[0]"))
goto out;
err = test_global_percpu_data__load(skel);
if (bpf_map__value_size(map) > pcpu_max_value_size) {
if (ASSERT_EQ(err, -E2BIG, "test_global_percpu_data__load E2BIG"))
/*
* When the PAGE_SIZE is 64KiB, the new value size will
* be 128KiB. Creating a percpu_array map with the new
* value size will fail with -E2BIG due to percpu_array
* map value size limit 64KiB, which is calculated by
* PFN_ALIGN(32KiB).
* Let's skip the test for such case.
*/
test__skip();
goto out;
}
if (!ASSERT_OK(err, "test_global_percpu_data__load"))
goto out;
map = skel->maps.percpu;
prog_fd = bpf_program__fd(skel->progs.update_percpu_data);
test_percpu_data_on_cpus(map, bpf_map__fd(map), prog_fd, &skel->bss->run);
ASSERT_EQ(skel->bss->arr_sum, arr_value * skel->bss->run, "arr_sum");
out:
test_global_percpu_data__destroy(skel);
}
static int create_rdonly_percpu_array(void)
{
LIBBPF_OPTS(bpf_map_create_opts, map_opts,
@ -388,6 +436,8 @@ void test_global_percpu_data(void)
test_global_percpu_data_init();
if (test__start_subtest("lskel"))
test_global_percpu_data_lskel();
if (test__start_subtest("map_resize"))
test_global_percpu_data_map_resize();
if (test__start_subtest("rdonly_direct_read"))
test_global_percpu_data_rdonly_direct_read();
if (test__start_subtest("rdonly_direct_write"))

479
tools/testing/selftests/bpf/prog_tests/kasan.c Normal file
View File

@ -0,0 +1,479 @@
// SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause
/*
* Tests validating that KASAN reports are properly instrumented and
* generated on a wide variety of instructions. The running kernel needs
* kasan_multi_shot to run multiple kasan-generating subtests at once
*/
#include <bpf/bpf.h>
#include <errno.h>
#include <fcntl.h>
#include <linux/if_ether.h>
#include <unistd.h>
#include <test_progs.h>
#include <unpriv_helpers.h>
#include "sysctl_helpers.h"
#include "kasan.skel.h"
#include "kasan_harden.skel.h"
#define SUBTEST_NAME_MAX_LEN 128
#define PROG_NAME_MAX_LEN 128
#define MAX_LOG_SIZE (8 * 1024)
#define READ_CHUNK_SIZE 256
#define KASAN_PATTERN_SLAB_UAF "BUG: KASAN: slab-use-after-free " \
"in bpf_prog_%02x%02x%02x%02x%02x%02x%02x%02x_%s"
#define KASAN_PATTERN_SLAB_OOB "BUG: KASAN: slab-out-of-bounds " \
"in bpf_prog_%02x%02x%02x%02x%02x%02x%02x%02x_%s"
#define KASAN_PATTERN_REPORT "%s of size %d at addr"
static char klog_buffer[MAX_LOG_SIZE];
static char record[MAX_LOG_SIZE];
struct test_spec {
char *prog_type;
bool is_write;
bool only_32_or_64;
bool needs_load_acq_store_rel;
bool needs_st;
bool skip_multi_size_testing;
bool skip_on_stack_testing;
int run_size;
bool expect_no_report;
bool rnd_hi32;
bool is_oob;
};
struct kasan_write_val {
__u8 data_1;
__u16 data_2;
__u32 data_4;
__u64 data_8;
};
struct test_ctx {
__u8 prog_tag[BPF_TAG_SIZE];
struct bpf_object *obj;
int *access_size;
bool skip_load_acq_store_rel;
bool skip_st_tests;
struct bpf_program *prog;
char prog_name[SUBTEST_NAME_MAX_LEN];
int klog_fd;
};
static int open_kernel_logs(void)
{
int fd;
fd = open("/dev/kmsg", O_RDONLY | O_NONBLOCK);
return fd;
}
static void skip_kernel_logs(int fd)
{
lseek(fd, 0, SEEK_END);
}
static int read_kernel_logs(int fd, char *buf, size_t max_len)
{
size_t total = 0;
ssize_t n;
buf[0] = '\0';
while (1) {
char *msg, *eol;
size_t len;
n = read(fd, record, sizeof(record) - 1);
if (n == 0)
break;
if (n < 0) {
if (errno == EAGAIN)
break;
return n;
}
record[n] = '\0';
/*
* Each kmsg record starts with some metadata, separated
* from the actual content by a semi-colon
*/
msg = strchr(record, ';');
if (!msg)
continue;
msg++;
eol = strchr(msg, '\n');
if (eol)
*eol = '\0';
len = strlen(msg);
if (total + len + 2 > max_len)
break;
memcpy(buf + total, msg, len);
total += len;
buf[total++] = '\n';
buf[total] = '\0';
}
return total;
}
static int check_kasan_report_in_kernel_logs(char *buf, struct test_ctx *ctx,
bool is_write, int size,
bool is_oob)
{
char access_log[READ_CHUNK_SIZE];
const char *pattern;
char *kasan_report_start;
int nsize;
pattern = is_oob ? KASAN_PATTERN_SLAB_OOB : KASAN_PATTERN_SLAB_UAF;
nsize = snprintf(access_log, READ_CHUNK_SIZE, pattern,
ctx->prog_tag[0], ctx->prog_tag[1], ctx->prog_tag[2],
ctx->prog_tag[3], ctx->prog_tag[4], ctx->prog_tag[5],
ctx->prog_tag[6], ctx->prog_tag[7], ctx->prog_name);
if (!ASSERT_GE(nsize, 0, "format kasan access header line"))
return nsize;
/*
* Searched kasan report is valid if
* - it contains the expected kasan pattern
* - the description of the faulty access is found somewhere
* after the header (not necessarily on the very next line,
* because other kernel messages may interleave)
* - faulty access properties match the tested type and size
*/
kasan_report_start = strstr(buf, access_log);
if (!kasan_report_start)
return 1;
nsize = snprintf(access_log, READ_CHUNK_SIZE, KASAN_PATTERN_REPORT,
is_write ? "Write" : "Read", size);
if (!ASSERT_GE(nsize, 0, "format kasan access report line"))
return nsize;
if (!strstr(kasan_report_start, access_log))
return 1;
return 0;
}
static void exec_subtest(struct test_ctx *ctx, struct test_spec *test,
int access_size, bool on_stack)
{
LIBBPF_OPTS(bpf_test_run_opts, topts);
struct bpf_prog_info info;
uint8_t buf[ETH_HLEN] = {0};
int ret, prog_fd;
__u32 info_len;
ctx->prog = bpf_object__find_program_by_name(ctx->obj,
ctx->prog_name);
if (!ASSERT_OK_PTR(ctx->prog, "find test prog"))
return;
info_len = sizeof(info);
memset(&info, 0, info_len);
prog_fd = bpf_program__fd(ctx->prog);
if (!ASSERT_OK_FD(prog_fd, "get prog fd"))
return;
ret = bpf_prog_get_info_by_fd(prog_fd, &info, &info_len);
if (!ASSERT_OK(ret, "fetch loaded program info"))
return;
memcpy(ctx->prog_tag, info.tag, BPF_TAG_SIZE);
skip_kernel_logs(ctx->klog_fd);
topts.sz = sizeof(struct bpf_test_run_opts);
topts.data_size_in = ETH_HLEN;
topts.data_in = buf;
if (ctx->access_size)
*ctx->access_size = access_size;
ret = bpf_prog_test_run_opts(bpf_program__fd(ctx->prog),
&topts);
if (!ASSERT_OK(ret, "run prog"))
return;
ret = read_kernel_logs(ctx->klog_fd, klog_buffer, MAX_LOG_SIZE);
if (!ASSERT_GE(ret, 0, "read kernel logs"))
return;
ret = check_kasan_report_in_kernel_logs(klog_buffer, ctx,
test->is_write, access_size,
test->is_oob);
if (on_stack || test->expect_no_report)
ASSERT_NEQ(ret, 0, "no report should be generated");
else
ASSERT_OK(ret, "report should be generated");
}
static void run_subtest_with_size_and_location(struct test_ctx *ctx,
struct test_spec *test,
int access_size,
bool on_stack)
{
char subtest_name[SUBTEST_NAME_MAX_LEN];
if (test->skip_multi_size_testing) {
snprintf(subtest_name, SUBTEST_NAME_MAX_LEN, "%s%s",
test->prog_type,
test->skip_on_stack_testing ? "" :
on_stack ? "_on_stack" :
"_not_on_stack");
} else {
snprintf(subtest_name, SUBTEST_NAME_MAX_LEN, "%s_%d_%s",
test->prog_type, access_size,
on_stack ? "on_stack" : "not_on_stack");
}
snprintf(ctx->prog_name, PROG_NAME_MAX_LEN, "%s%s", test->prog_type,
test->skip_on_stack_testing ? "" :
on_stack ? "_on_stack" :
"_not_on_stack");
if (!test__start_subtest(subtest_name))
return;
if (test->needs_load_acq_store_rel && ctx->skip_load_acq_store_rel) {
test__skip();
return;
}
if (test->needs_st && ctx->skip_st_tests) {
test__skip();
return;
}
exec_subtest(ctx, test, access_size, on_stack);
}
static void run_subtest_with_size(struct test_ctx *ctx, struct test_spec *test,
int size)
{
run_subtest_with_size_and_location(ctx, test, size, false);
if (!test->skip_on_stack_testing)
run_subtest_with_size_and_location(ctx, test, size, true);
}
static void run_subtest(struct test_ctx *ctx, struct test_spec *test)
{
if (test->skip_multi_size_testing) {
run_subtest_with_size(ctx, test, test->run_size);
return;
}
if (!test->only_32_or_64) {
run_subtest_with_size(ctx, test, 1);
run_subtest_with_size(ctx, test, 2);
}
run_subtest_with_size(ctx, test, 4);
run_subtest_with_size(ctx, test, 8);
}
static void run_blinding_subtest(void)
{
struct test_spec blinding_spec = {
.prog_type = "st_blinded",
.is_write = true,
};
char bpf_jit_harden_orig[2];
struct kasan_harden *skel;
struct test_ctx *ctx;
if (!test__start_subtest("st_blinded"))
return;
ctx = calloc(1, sizeof(*ctx));
if (!ASSERT_OK_PTR(ctx, "alloc blinding ctx"))
return;
ctx->klog_fd = -1;
if (sysctl_set_or_fail("/proc/sys/net/core/bpf_jit_harden",
bpf_jit_harden_orig, "2"))
goto free_ctx;
skel = kasan_harden__open_and_load();
if (!ASSERT_OK_PTR(skel, "open and load blinded prog"))
goto restore;
if (skel->data->skip_st_tests) {
test__skip();
goto destroy;
}
ctx->klog_fd = open_kernel_logs();
if (!ASSERT_OK_FD(ctx->klog_fd, "open kernel logs"))
goto destroy;
ctx->obj = skel->obj;
strncpy(ctx->prog_name, "st_blinded", PROG_NAME_MAX_LEN);
exec_subtest(ctx, &blinding_spec, 1, false);
destroy:
close(ctx->klog_fd);
kasan_harden__destroy(skel);
restore:
sysctl_set_or_fail("/proc/sys/net/core/bpf_jit_harden", NULL,
bpf_jit_harden_orig);
free_ctx:
free(ctx);
}
static struct test_spec tests[] = {
{
.prog_type = "st",
.is_write = true,
.needs_st = true
},
{
.prog_type = "stx",
.is_write = true
},
{
.prog_type = "ldx",
.is_write = false
},
{
.prog_type = "simple_atomic",
.is_write = true,
.only_32_or_64 = true
},
{
.prog_type = "simple_atomic_fetch",
.is_write = true,
.skip_multi_size_testing = true,
.run_size = 8,
},
{
.prog_type = "load_acquire",
.is_write = false,
.needs_load_acq_store_rel = true
},
{
.prog_type = "store_release",
.is_write = true,
.needs_load_acq_store_rel = true
},
{
.prog_type = "ldx_patched",
.is_write = false,
.skip_multi_size_testing = true,
.run_size = 4,
.rnd_hi32 = true
},
{
.prog_type = "verifier_paths_stack_and_non_stack",
.is_write = true,
.skip_multi_size_testing = true,
.skip_on_stack_testing = true,
.run_size = 1
},
{
.prog_type = "ldx_oob",
.is_write = false,
.skip_on_stack_testing = true,
.is_oob = true
},
{
.prog_type = "ldx_self_alias_on_stack",
.is_write = false,
.skip_multi_size_testing = true,
.skip_on_stack_testing = true,
.run_size = 8,
.expect_no_report = true
}
};
void serial_test_kasan(void)
{
struct kasan_write_val val;
struct test_spec *test;
struct test_ctx *ctx;
struct kasan *skel;
__u32 key = 0;
int i, ret;
ctx = calloc(1, sizeof(struct test_ctx));
if (!ASSERT_OK_PTR(ctx, "alloc test ctx"))
return;
if (!is_jit_enabled() || !get_kasan_jit_enabled() ||
!get_kasan_multi_shot_enabled()) {
test__skip();
goto end;
}
skel = kasan__open();
if (!ASSERT_OK_PTR(skel, "open prog"))
goto end;
for (i = 0; i < ARRAY_SIZE(tests); i++) {
char prog_name[SUBTEST_NAME_MAX_LEN];
struct bpf_program *prog;
if (!tests[i].rnd_hi32)
continue;
snprintf(prog_name, SUBTEST_NAME_MAX_LEN, "%s_%s",
tests[i].prog_type, "on_stack");
prog = bpf_object__find_program_by_name(skel->obj, prog_name);
if (!ASSERT_OK_PTR(prog, "find rnd_hi32 on_stack prog"))
goto destroy;
bpf_program__set_flags(prog, BPF_F_TEST_RND_HI32);
snprintf(prog_name, SUBTEST_NAME_MAX_LEN, "%s_%s",
tests[i].prog_type, "not_on_stack");
prog = bpf_object__find_program_by_name(skel->obj, prog_name);
if (!ASSERT_OK_PTR(prog, "find rnd_hi32 not_on_stack prog"))
goto destroy;
bpf_program__set_flags(prog, BPF_F_TEST_RND_HI32);
}
if (!ASSERT_OK(kasan__load(skel), "load prog"))
goto destroy;
ctx->obj = skel->obj;
ctx->access_size = &skel->bss->access_size;
ctx->skip_load_acq_store_rel = skel->data->skip_load_acq_store_rel_tests;
ctx->skip_st_tests = skel->data->skip_st_tests;
ctx->klog_fd = open_kernel_logs();
if (!ASSERT_OK_FD(ctx->klog_fd, "open kernel logs"))
goto destroy;
/* Fill map with recognizable values */
ret = bpf_map__lookup_elem(skel->maps.test_map, &key, sizeof(key),
&val, sizeof(val), 0);
if (!ASSERT_OK(ret, "get map"))
goto close;
val.data_1 = 0xAA;
val.data_2 = 0xBBBB;
val.data_4 = 0xCCCCCCCC;
val.data_8 = 0xDDDDDDDDDDDDDDDD;
ret = bpf_map__update_elem(skel->maps.test_map, &key, sizeof(key),
&val, sizeof(val), 0);
if (!ASSERT_OK(ret, "set map"))
goto close;
for (i = 0; i < ARRAY_SIZE(tests); i++) {
test = &tests[i];
run_subtest(ctx, test);
}
/*
* Blinding subtest is handled differently as it needs the
* corresponding program to be loaded with bpf_jit_harden raised
*/
run_blinding_subtest();
close:
close(ctx->klog_fd);
destroy:
kasan__destroy(skel);
end:
free(ctx);
}

3
tools/testing/selftests/bpf/prog_tests/kernel_flag.c
View File

@ -16,6 +16,9 @@ void test_kernel_flag(void)
if (!ASSERT_OK_PTR(lsm_skel, "lsm_skel"))
return;
ASSERT_EQ(bpf_program__flags(lsm_skel->progs.bpf) & BPF_F_SLEEPABLE,
BPF_F_SLEEPABLE, "sleepable in program flags");
lsm_skel->bss->monitored_tid = sys_gettid();
ret = test_kernel_flag__attach(lsm_skel);

2
tools/testing/selftests/bpf/prog_tests/kprobe_multi_test.c
View File

@ -362,7 +362,7 @@ static void test_attach_api_fails(void)
sl_skel->bss->user_ptr = sl_skel;
err = bpf_program__set_flags(sl_skel->progs.handle_kprobe_multi_sleepable,
BPF_F_SLEEPABLE);
bpf_program__flags(sl_skel->progs.handle_kprobe_multi_sleepable) | BPF_F_SLEEPABLE);
if (!ASSERT_OK(err, "sleep_skel_set_flags"))
goto cleanup;

334
tools/testing/selftests/bpf/prog_tests/lirc_mode2.c Normal file
View File

@ -0,0 +1,334 @@
// SPDX-License-Identifier: GPL-2.0
// test ir decoder
//
// Copyright (C) 2018 Sean Young <sean@mess.org>
// A lirc chardev is a device representing a consumer IR (cir) device which
// can receive infrared signals from remote control and/or transmit IR.
//
// IR is sent as a series of pulses and space somewhat like morse code. The
// BPF program can decode this into scancodes so that rc-core can translate
// this into input key codes using the rc keymap.
//
// This test works by sending IR over rc-loopback, so the IR is processed by
// BPF and then decoded into scancodes. The lirc chardev must be the one
// associated with rc-loopback, see the output of ir-keytable(1).
//
// The following CONFIG options must be enabled for the test to succeed:
// CONFIG_RC_CORE=y
// CONFIG_BPF_LIRC_MODE2=y
// CONFIG_RC_LOOPBACK=y
// CONFIG_LIRC=y
#include <linux/input.h>
#include <linux/lirc.h>
#include <glob.h>
#include <limits.h>
#include <poll.h>
#include <test_progs.h>
#include "lirc_mode2.skel.h"
/*
* Read the DEVNAME= line out of the first uevent file that matches
* pattern, and turn it into a /dev/<name> path.
*/
static bool find_devname(const char *pattern, char *path, size_t path_sz)
{
glob_t gl = {};
bool found = false;
FILE *f;
if (glob(pattern, 0, NULL, &gl) || gl.gl_pathc == 0)
goto out;
f = fopen(gl.gl_pathv[0], "r");
if (!f)
goto out;
char line[256];
while (fgets(line, sizeof(line), f)) {
char *val;
if (strncmp(line, "DEVNAME=", 8))
continue;
val = line + 8;
val[strcspn(val, "\n")] = '\0';
snprintf(path, path_sz, "/dev/%s", val);
found = true;
break;
}
fclose(f);
out:
globfree(&gl);
return found;
}
/* Load rc-loopback and find the lirc and input chardevs it created. */
static bool find_loopback_devices(char *lirc_path, char *input_path,
size_t path_sz)
{
glob_t gl = {};
bool found = false;
/* Ignore failure, we check for the resulting devices below. */
SYS_NOFAIL("modprobe rc-loopback > /dev/null 2>&1");
if (glob("/sys/class/rc/rc*", 0, NULL, &gl))
goto out;
for (size_t i = 0; i < gl.gl_pathc; i++) {
const char *rcdir = gl.gl_pathv[i];
char uevent_path[PATH_MAX];
char uevent[4096];
char pattern[PATH_MAX];
FILE *f;
size_t n;
snprintf(uevent_path, sizeof(uevent_path), "%s/uevent", rcdir);
f = fopen(uevent_path, "r");
if (!f)
continue;
n = fread(uevent, 1, sizeof(uevent) - 1, f);
fclose(f);
uevent[n] = '\0';
if (!strstr(uevent, "DRV_NAME=rc-loopback"))
continue;
snprintf(pattern, sizeof(pattern), "%s/lirc*/uevent", rcdir);
if (!find_devname(pattern, lirc_path, path_sz))
continue;
snprintf(pattern, sizeof(pattern), "%s/input*/event*/uevent", rcdir);
if (!find_devname(pattern, input_path, path_sz))
continue;
found = true;
break;
}
out:
if (!found) {
fprintf(stderr, "No rc devices found\n");
fprintf(stderr, "Enable CONFIG_RC_LOOPBACK and CONFIG_BPF_LIRC_MODE2\n");
}
globfree(&gl);
return found;
}
void test_lirc_mode2(void)
{
char lirc_path[PATH_MAX], input_path[PATH_MAX];
int lircfd = -1, inputfd = -1, progfd, progfd2 = -1;
struct lirc_mode2 *skel = NULL, *skel2 = NULL;
__u32 prog_ids[10], prog_flags[10], prog_cnt;
struct bpf_prog_info info;
__u32 info_len, prog_id, prog_id2;
int testir1 = 0x8ead; /* keydown flag (0x8000) | scancode 0xead */
int testir2 = 0x4081; /* pointer_rel flag (0x4000) | rel_x=1 | rel_y=1 */
struct input_event event;
struct pollfd pfd = {};
int ret;
if (getuid() != 0) {
test__skip();
return;
}
if (!find_loopback_devices(lirc_path, input_path, sizeof(lirc_path))) {
test__skip();
return;
}
skel = lirc_mode2__open_and_load();
if (!ASSERT_OK_PTR(skel, "lirc_mode2__open_and_load"))
return;
progfd = bpf_program__fd(skel->progs.bpf_decoder);
memset(&info, 0, sizeof(info));
info_len = sizeof(info);
ret = bpf_prog_get_info_by_fd(progfd, &info, &info_len);
if (!ASSERT_OK(ret, "get first program's info"))
goto out;
prog_id = info.id;
lircfd = open(lirc_path, O_RDWR | O_NONBLOCK);
if (!ASSERT_GE(lircfd, 0, "open lirc device"))
goto out;
/* Try to detach it before it was ever attached, should fail. */
ret = bpf_prog_detach2(progfd, lircfd, BPF_LIRC_MODE2);
if (!ASSERT_EQ(ret, -ENOENT, "detach unattached program"))
goto out;
inputfd = open(input_path, O_RDONLY | O_NONBLOCK);
if (!ASSERT_GE(inputfd, 0, "open input device"))
goto out;
prog_cnt = ARRAY_SIZE(prog_ids);
ret = bpf_prog_query(lircfd, BPF_LIRC_MODE2, 0, prog_flags, prog_ids,
&prog_cnt);
if (!ASSERT_OK(ret, "query programs before attach"))
goto out;
if (!ASSERT_EQ(prog_cnt, 0, "no programs should be attached yet"))
goto out;
/* Invalid attach flags must be rejected, and must not attach. */
ret = bpf_prog_attach(progfd, lircfd, BPF_LIRC_MODE2, 1);
if (!ASSERT_EQ(ret, -EINVAL, "attach with invalid flags"))
goto out;
prog_cnt = ARRAY_SIZE(prog_ids);
ret = bpf_prog_query(lircfd, BPF_LIRC_MODE2, 0, prog_flags, prog_ids,
&prog_cnt);
if (!ASSERT_OK(ret, "query programs after rejected attach"))
goto out;
if (!ASSERT_EQ(prog_cnt, 0, "rejected attach should not attach"))
goto out;
ret = bpf_prog_attach(progfd, lircfd, BPF_LIRC_MODE2, 0);
if (!ASSERT_OK(ret, "attach program to lirc device"))
goto out;
/* Invalid query flags must be rejected too, without upsetting state. */
prog_cnt = ARRAY_SIZE(prog_ids);
ret = bpf_prog_query(lircfd, BPF_LIRC_MODE2, 1, prog_flags, prog_ids,
&prog_cnt);
ASSERT_EQ(ret, -EINVAL, "query with invalid flags");
prog_cnt = ARRAY_SIZE(prog_ids);
ret = bpf_prog_query(lircfd, BPF_LIRC_MODE2, 0, prog_flags, prog_ids,
&prog_cnt);
if (!ASSERT_OK(ret, "query programs after attach"))
goto out_detach;
if (!ASSERT_EQ(prog_cnt, 1, "one program should be attached"))
goto out_detach;
ASSERT_EQ(prog_ids[0], prog_id, "queried id should match attached program");
/* Write raw IR */
ret = write(lircfd, &testir1, sizeof(testir1));
if (!ASSERT_EQ(ret, sizeof(testir1), "send test IR message 1"))
goto out_detach;
pfd.fd = inputfd;
pfd.events = POLLIN;
for (;;) {
poll(&pfd, 1, 100);
/* Read decoded IR */
ret = read(inputfd, &event, sizeof(event));
if (!ASSERT_EQ(ret, sizeof(event), "read decoded IR 1"))
goto out_detach;
if (event.type == EV_MSC && event.code == MSC_SCAN &&
event.value == 0xead)
break;
}
/* Write raw IR */
ret = write(lircfd, &testir2, sizeof(testir2));
if (!ASSERT_EQ(ret, sizeof(testir2), "send test IR message 2"))
goto out_detach;
for (;;) {
poll(&pfd, 1, 100);
/* Read decoded IR */
ret = read(inputfd, &event, sizeof(event));
if (!ASSERT_EQ(ret, sizeof(event), "read decoded IR 2"))
goto out_detach;
if (event.type == EV_REL && event.code == REL_Y &&
event.value == 1)
break;
}
prog_cnt = ARRAY_SIZE(prog_ids);
ret = bpf_prog_query(lircfd, BPF_LIRC_MODE2, 0, prog_flags, prog_ids,
&prog_cnt);
if (!ASSERT_OK(ret, "query programs after IR was decoded"))
goto out_detach;
if (!ASSERT_EQ(prog_cnt, 1, "one program should still be attached"))
goto out_detach;
/*
* The lirc chardev can hold more than one attached program at once.
* Load a second, independent instance and check it can be attached
* alongside the first, queried, and then detached on its own
* without disturbing the first program's attachment.
*/
skel2 = lirc_mode2__open_and_load();
if (!ASSERT_OK_PTR(skel2, "lirc_mode2__open_and_load (2nd)"))
goto out_detach;
progfd2 = bpf_program__fd(skel2->progs.bpf_decoder);
memset(&info, 0, sizeof(info));
info_len = sizeof(info);
ret = bpf_prog_get_info_by_fd(progfd2, &info, &info_len);
if (!ASSERT_OK(ret, "get second program's info"))
goto out_detach;
prog_id2 = info.id;
ret = bpf_prog_attach(progfd2, lircfd, BPF_LIRC_MODE2, 0);
if (!ASSERT_OK(ret, "attach second program to lirc device"))
goto out_detach;
prog_cnt = ARRAY_SIZE(prog_ids);
ret = bpf_prog_query(lircfd, BPF_LIRC_MODE2, 0, prog_flags, prog_ids,
&prog_cnt);
if (!ASSERT_OK(ret, "query programs after second attach"))
goto out_detach2;
if (!ASSERT_EQ(prog_cnt, 2, "two programs should be attached"))
goto out_detach2;
ASSERT_TRUE((prog_ids[0] == prog_id && prog_ids[1] == prog_id2) ||
(prog_ids[0] == prog_id2 && prog_ids[1] == prog_id),
"queried ids should be the two attached programs");
/* Detach the second program; the first should remain attached. */
ret = bpf_prog_detach2(progfd2, lircfd, BPF_LIRC_MODE2);
if (!ASSERT_OK(ret, "detach second program"))
goto out_detach2;
/* Detaching an already-detached program should now fail. */
ret = bpf_prog_detach2(progfd2, lircfd, BPF_LIRC_MODE2);
ASSERT_EQ(ret, -ENOENT, "detach second program again");
prog_cnt = ARRAY_SIZE(prog_ids);
ret = bpf_prog_query(lircfd, BPF_LIRC_MODE2, 0, prog_flags, prog_ids,
&prog_cnt);
if (!ASSERT_OK(ret, "query programs after second detach"))
goto out_detach;
if (!ASSERT_EQ(prog_cnt, 1, "one program should remain attached"))
goto out_detach;
ASSERT_EQ(prog_ids[0], prog_id, "remaining program should be the first one");
out_detach:
/* Let's try detaching it now it is actually attached. */
ret = bpf_prog_detach2(progfd, lircfd, BPF_LIRC_MODE2);
ASSERT_OK(ret, "detach program from lirc device");
/* Detaching it again should now fail the same way. */
ret = bpf_prog_detach2(progfd, lircfd, BPF_LIRC_MODE2);
ASSERT_EQ(ret, -ENOENT, "detach program from lirc device again");
goto out;
out_detach2:
/* Best-effort cleanup of the second program before bailing out. */
bpf_prog_detach2(progfd2, lircfd, BPF_LIRC_MODE2);
goto out_detach;
out:
if (inputfd >= 0)
close(inputfd);
if (lircfd >= 0)
close(lircfd);
lirc_mode2__destroy(skel2);
lirc_mode2__destroy(skel);
}

23
tools/testing/selftests/bpf/prog_tests/prog_tests_framework.c
View File

@ -179,3 +179,26 @@ void test_prog_tests_framework_expected_msgs(void)
}
}
}
void test_prog_tests_framework_compare_text(void)
{
int err;
if (test__start_subtest("compare_text_match")) {
err = compare_text_to_expected("same\n", "same\n");
ASSERT_EQ(err, 0, "match_rc");
test__end_subtest();
}
if (test__start_subtest("compare_text_mismatch")) {
err = compare_text_to_expected("line two\n", "line one\n");
fflush(stdout);
ASSERT_EQ(err, -1, "mismatch_rc");
ASSERT_HAS_SUBSTR(env.subtest_state->log_buf, "-line one",
"diff_has_expected");
ASSERT_HAS_SUBSTR(env.subtest_state->log_buf, "+line two",
"diff_has_actual");
test__end_subtest();
}
}

29
tools/testing/selftests/bpf/prog_tests/queue_stack_map.c
View File

@ -2,6 +2,8 @@
#include <test_progs.h>
#include <network_helpers.h>
#define U32_MAX ((u32)UINT_MAX)
enum {
QUEUE,
STACK,
@ -101,8 +103,35 @@ static void test_queue_stack_map_by_type(int type)
bpf_object__close(obj);
}
static void test_queue_stack_map_alloc_check(void)
{
int fd;
fd = bpf_map_create(BPF_MAP_TYPE_QUEUE, NULL, 0, 1000000, 8192, NULL);
ASSERT_EQ(fd, -E2BIG, "queue_oversize");
if (fd >= 0)
close(fd);
fd = bpf_map_create(BPF_MAP_TYPE_QUEUE, NULL, 0, 1, U32_MAX, NULL);
ASSERT_EQ(fd, -E2BIG, "queue_u32max");
if (fd >= 0)
close(fd);
fd = bpf_map_create(BPF_MAP_TYPE_STACK, NULL, 0, 1000000, 8192, NULL);
ASSERT_EQ(fd, -E2BIG, "stack_oversize");
if (fd >= 0)
close(fd);
/* A normal-sized map must still be created successfully. */
fd = bpf_map_create(BPF_MAP_TYPE_QUEUE, NULL, 0, 64, 100, NULL);
ASSERT_GE(fd, 0, "queue_normal");
if (fd >= 0)
close(fd);
}
void test_queue_stack_map(void)
{
test_queue_stack_map_by_type(QUEUE);
test_queue_stack_map_by_type(STACK);
test_queue_stack_map_alloc_check();
}

600
tools/testing/selftests/bpf/prog_tests/signed_loader.c
View File

@ -32,11 +32,22 @@ enum {
BPF_SIG_KEYRING_SECONDARY,
BPF_SIG_KEYRING_PLATFORM,
BPF_SIG_KEYRING_USER,
BPF_SIG_KEYRING_BPF,
};
static int load_loader(const void *insns, __u32 insns_sz, int map_fd,
const void *sig, __u32 sig_sz, __s32 keyring_id,
__u32 fd_array_cnt)
#ifndef KEY_SPEC_BPF_KEYRING
#define KEY_SPEC_BPF_KEYRING -9
#endif
/* verify_sig_setup.sh exits with this when openssl cannot do ML-DSA. */
#define SETUP_SKIP (-77)
/* FIPS-204 ML-DSA-87 signature size, see include/crypto/mldsa.h. */
#define MLDSA87_SIGNATURE_SIZE 4627
static int load_loader_log(const void *insns, __u32 insns_sz, int map_fd,
const void *sig, __u32 sig_sz, __s32 keyring_id,
__u32 fd_array_cnt, char *log_buf, __u32 log_sz)
{
union bpf_attr attr;
int fd;
@ -48,18 +59,31 @@ static int load_loader(const void *insns, __u32 insns_sz, int map_fd,
attr.license = ptr_to_u64("Dual BSD/GPL");
attr.prog_flags = BPF_F_SLEEPABLE;
attr.fd_array = ptr_to_u64(&map_fd);
attr.fd_array_cnt = fd_array_cnt;
if (sig) {
attr.signature = ptr_to_u64(sig);
attr.signature_size = sig_sz;
attr.keyring_id = keyring_id;
}
attr.fd_array_cnt = fd_array_cnt;
if (log_buf) {
attr.log_level = 1;
attr.log_buf = ptr_to_u64(log_buf);
attr.log_size = log_sz;
}
memcpy(attr.prog_name, "__loader.prog", sizeof("__loader.prog"));
fd = syscall(__NR_bpf, BPF_PROG_LOAD, &attr,
offsetofend(union bpf_attr, keyring_id));
return fd < 0 ? -errno : fd;
}
static int load_loader(const void *insns, __u32 insns_sz, int map_fd,
const void *sig, __u32 sig_sz, __s32 keyring_id,
__u32 fd_array_cnt)
{
return load_loader_log(insns, insns_sz, map_fd, sig, sig_sz, keyring_id,
fd_array_cnt, NULL, 0);
}
static int run_gen_loader(const void *insns, __u32 insns_sz,
const void *data, __u32 data_sz,
const void *excl, __u32 excl_sz,
@ -156,12 +180,30 @@ static int run_setup(const char *cmd, const char *dir)
}
if (waitpid(pid, &status, 0) < 0)
return -errno;
return (WIFEXITED(status) &&
WEXITSTATUS(status) == 0) ? 0 : -EINVAL;
if (!WIFEXITED(status))
return -EINVAL;
return -WEXITSTATUS(status);
}
static int sign_buf(const char *dir, const void *buf, __u32 len,
void *sig, __u32 *sig_sz)
static void genkey_dir_fini(const char *dir)
{
static const char * const files[] = {
"signing_key.der", "signing_key.pem", "x509.genkey",
};
char path[PATH_MAX];
size_t i;
if (!dir)
return;
for (i = 0; i < ARRAY_SIZE(files); i++) {
snprintf(path, sizeof(path), "%s/%s", dir, files[i]);
unlink(path);
}
rmdir(dir);
}
static int sign_buf_digest(const char *dir, const void *buf, __u32 len,
void *sig, __u32 *sig_sz, const char *digest)
{
char data_tmpl[PATH_MAX], key[PATH_MAX];
char sigpath[PATH_MAX + sizeof(".p7s")];
@ -176,6 +218,7 @@ static int sign_buf(const char *dir, const void *buf, __u32 len,
fd = mkstemp(data_tmpl);
if (fd < 0)
return -errno;
snprintf(sigpath, sizeof(sigpath), "%s.p7s", data_tmpl);
if (write(fd, buf, len) != (ssize_t)len) {
close(fd);
ret = -EIO;
@ -190,7 +233,7 @@ static int sign_buf(const char *dir, const void *buf, __u32 len,
}
if (pid == 0) {
snprintf(key, sizeof(key), "%s/signing_key.pem", dir);
execlp("./sign-file", "./sign-file", "-d", "sha256",
execlp("./sign-file", "./sign-file", "-d", digest,
key, key, data_tmpl, NULL);
exit(1);
}
@ -200,34 +243,38 @@ static int sign_buf(const char *dir, const void *buf, __u32 len,
goto out;
}
snprintf(sigpath, sizeof(sigpath), "%s.p7s", data_tmpl);
if (stat(sigpath, &st) < 0) {
ret = -errno;
goto out;
}
if (st.st_size > (off_t)*sig_sz) {
ret = -E2BIG;
goto out_sig;
goto out;
}
fd = open(sigpath, O_RDONLY);
if (fd < 0) {
ret = -errno;
goto out_sig;
goto out;
}
if (read(fd, sig, st.st_size) != st.st_size) {
close(fd);
ret = -EIO;
goto out_sig;
goto out;
}
close(fd);
*sig_sz = st.st_size;
out_sig:
unlink(sigpath);
out:
unlink(sigpath);
unlink(data_tmpl);
return ret;
}
static int sign_buf(const char *dir, const void *buf, __u32 len,
void *sig, __u32 *sig_sz)
{
return sign_buf_digest(dir, buf, len, sig, sig_sz, "sha256");
}
struct gen_loader_fixture {
struct test_signed_loader *skel;
struct gen_loader_opts gopts;
@ -457,7 +504,7 @@ static void signed_btf_fd_array_rejected(void)
dir = mkdtemp(dir_tmpl);
if (!ASSERT_OK_PTR(dir, "mkdtemp"))
return;
if (!ASSERT_OK(run_setup("setup", dir), "verify_sig_setup")) {
if (!ASSERT_OK(run_setup("setup-rsa", dir), "verify_sig_setup")) {
rmdir(dir);
return;
}
@ -529,7 +576,6 @@ static void signature_failure_logs(void)
static const __u8 junk[64] = { 0x30, 0x42, 0x13, 0x37, };
char log_buf[1024] = {};
struct gen_loader_fixture f;
union bpf_attr attr;
int fd;
if (gen_loader_fixture_init(&f) == 0) {
@ -538,22 +584,9 @@ static void signature_failure_logs(void)
* failure is reported through the verifier log. A present-but-
* invalid signature is rejected and the log says why.
*/
memset(&attr, 0, sizeof(attr));
attr.prog_type = BPF_PROG_TYPE_SYSCALL;
attr.insns = ptr_to_u64(f.gopts.insns);
attr.insn_cnt = f.gopts.insns_sz / sizeof(struct bpf_insn);
attr.license = ptr_to_u64("Dual BSD/GPL");
attr.prog_flags = BPF_F_SLEEPABLE;
attr.signature = ptr_to_u64(junk);
attr.signature_size = sizeof(junk);
attr.keyring_id = KEY_SPEC_SESSION_KEYRING;
attr.log_level = 1;
attr.log_buf = ptr_to_u64(log_buf);
attr.log_size = sizeof(log_buf);
memcpy(attr.prog_name, "__loader.prog", sizeof("__loader.prog"));
fd = syscall(__NR_bpf, BPF_PROG_LOAD, &attr,
offsetofend(union bpf_attr, keyring_id));
fd = load_loader_log(f.gopts.insns, f.gopts.insns_sz, -1, junk,
sizeof(junk), KEY_SPEC_SESSION_KEYRING, 0,
log_buf, sizeof(log_buf));
ASSERT_LT(fd, 0, "invalid signature rejected at load");
if (fd >= 0)
close(fd);
@ -571,8 +604,9 @@ static void signature_too_large(void)
if (gen_loader_fixture_init(&f) == 0) {
/*
* signature_size beyond the kernel's bound (KMALLOC_MAX_CACHE_SIZE)
* is rejected before the buffer is read.
* signature_size beyond the kernel's bound
* (BPF_PROG_MAX_SIGNATURE_SIZE) is rejected before the buffer
* is read.
*/
fd = load_loader(f.gopts.insns, f.gopts.insns_sz, -1, junk,
64 << 20, KEY_SPEC_SESSION_KEYRING, 0);
@ -626,6 +660,280 @@ static void signature_bad_keyring(void)
gen_loader_fixture_fini(&f);
}
static bool keyring_unsealed_boot(void)
{
char val = 0;
int fd;
fd = open("/sys/module/bpf/parameters/keyring_unsealed", O_RDONLY);
if (fd < 0)
return false;
if (read(fd, &val, 1) != 1)
val = 0;
close(fd);
return val == 'Y' || val == '1';
}
static int bpf_keyring_lookup(int *nr_keys)
{
char line[512], type[32], desc[64];
int serial = -ENOENT;
FILE *f;
f = fopen("/proc/keys", "r");
if (!f)
return -errno;
while (fgets(line, sizeof(line), f)) {
unsigned int hex;
char *sum;
if (sscanf(line, "%x %*s %*s %*s %*s %*s %*s %31s %63s",
&hex, type, desc) != 3)
continue;
if (strcmp(type, "keyring") || strcmp(desc, ".bpf:"))
continue;
serial = (int)hex;
if (nr_keys) {
sum = strstr(line, ".bpf: ");
*nr_keys = !sum || !strncmp(sum + 6, "empty", 5) ?
0 : atoi(sum + 6);
}
break;
}
fclose(f);
return serial;
}
static long keyctl_ret(int cmd, unsigned long arg2, unsigned long arg3)
{
long ret = syscall(__NR_keyctl, cmd, arg2, arg3);
return ret < 0 ? -errno : ret;
}
/*
* What the bpf keyring still needs once it got provisioned: KEY_POS_SEARCH
* for the in-kernel search during verification, and the user view/read bits
* so it stays visible in /proc/keys, rest is dropped so the enrolled is
* therefore final.
*/
#define BPF_KEYRING_PERM_LOCKED 0x08030000
/* What bpf_keyring_init() grants at boot. */
#define BPF_KEYRING_PERM_INITIAL 0x082f0000
static void bpf_keyring_sealed(void)
{
static const __u8 junk[64] = {};
struct gen_loader_fixture f;
int serial, key, fd;
if (keyring_unsealed_boot()) {
printf("%s:SKIP:the bpf keyring was unsealed at boot\n", __func__);
test__skip();
return;
}
serial = bpf_keyring_lookup(NULL);
if (serial >= 0) {
ASSERT_EQ(keyctl_ret(KEYCTL_GET_KEYRING_ID,
KEY_SPEC_BPF_KEYRING, 0), serial,
"KEY_SPEC_BPF_KEYRING resolves to the bpf keyring");
key = syscall(__NR_add_key, "user", "sealprobe", "x", 1,
KEY_SPEC_BPF_KEYRING);
if (key >= 0)
syscall(__NR_keyctl, KEYCTL_UNLINK, key,
KEY_SPEC_BPF_KEYRING);
ASSERT_EQ(key < 0 ? -errno : 0, -EPERM,
"nothing links into a sealed keyring");
}
if (gen_loader_fixture_init(&f) == 0) {
fd = load_loader(f.gopts.insns, f.gopts.insns_sz, -1, junk,
sizeof(junk), KEY_SPEC_BPF_KEYRING, 0);
ASSERT_EQ(fd, -ENOKEY, "sealed bpf keyring rejected");
if (fd >= 0)
close(fd);
}
gen_loader_fixture_fini(&f);
}
static int try_load(const struct gen_loader_fixture *f, const void *sig,
__u32 sig_sz, __s32 keyring_id, char *log_buf, __u32 log_sz)
{
int map_fd, prog_fd;
map_fd = setup_meta_map(f);
if (!ASSERT_OK_FD(map_fd, "meta_map"))
return map_fd;
prog_fd = load_loader_log(f->gopts.insns, f->gopts.insns_sz, map_fd,
sig, sig_sz, keyring_id, 1, log_buf, log_sz);
close(map_fd);
if (prog_fd >= 0)
close(prog_fd);
return prog_fd;
}
/*
* This needs bpf.keyring_unsealed=1 on the guest kernel command line, which
* vmtest.sh can pass via KERNEL_CMDLINE_EXTRA. There is no way to unseal the
* keyring from here, so without it the test skips. It also only works once
* per boot, as restricting a keyring cannot be undone.
*/
static void bpf_keyring_provisioned(void)
{
char dir_tmpl[] = "/tmp/bpfkeyringXXXXXX";
char bad_tmpl[] = "/tmp/bpfkeyringbadXXXXXX";
__u8 *sig = NULL, *bad = NULL, *buf = NULL;
int serial, err;
int nr_keys = 0, der_fd = -1;
struct gen_loader_fixture f;
__u32 sig_sz = 8192, bad_sz;
bool have_fixture = false;
char *dir, *bad_dir = NULL;
char log_buf[1024] = {};
char path[PATH_MAX];
__u8 der[4096];
ssize_t der_sz;
serial = bpf_keyring_lookup(&nr_keys);
if (serial < 0) {
printf("%s:SKIP:no bpf keyring (needs CONFIG_KEYS)\n", __func__);
test__skip();
return;
}
if (nr_keys != 0) {
printf("%s:SKIP:the bpf keyring has already been provisioned\n",
__func__);
test__skip();
return;
}
dir = mkdtemp(dir_tmpl);
if (!ASSERT_OK_PTR(dir, "mkdtemp"))
return;
if (!ASSERT_OK(run_setup("genkey", dir), "verify_sig_setup genkey"))
goto rmdir;
snprintf(path, sizeof(path), "%s/signing_key.der", dir);
der_fd = open(path, O_RDONLY);
if (!ASSERT_OK_FD(der_fd, "open signing_key.der"))
goto rmdir;
der_sz = read(der_fd, der, sizeof(der));
close(der_fd);
if (!ASSERT_GT(der_sz, 0, "read signing_key.der"))
goto rmdir;
ASSERT_EQ(keyctl_ret(KEYCTL_GET_KEYRING_ID, KEY_SPEC_BPF_KEYRING, 0),
serial, "KEY_SPEC_BPF_KEYRING resolves to the bpf keyring");
err = syscall(__NR_add_key, "asymmetric", "", der, (size_t)der_sz,
KEY_SPEC_BPF_KEYRING);
if (err < 0 && errno == EPERM) {
printf("%s:SKIP:the bpf keyring is sealed, need bpf.keyring_unsealed=1\n",
__func__);
test__skip();
goto rmdir;
}
if (!ASSERT_GE(err, 0, "add the signing key to the bpf keyring"))
goto rmdir;
sig = malloc(sig_sz);
if (!ASSERT_OK_PTR(sig, "sig buf"))
goto out;
have_fixture = true;
if (gen_loader_fixture_init(&f) != 0)
goto out;
buf = malloc((size_t)f.gopts.insns_sz + f.data_sz);
if (!ASSERT_OK_PTR(buf, "signbuf"))
goto out;
memcpy(buf, f.gopts.insns, f.gopts.insns_sz);
memcpy(buf + f.gopts.insns_sz, f.blob, f.data_sz);
if (!ASSERT_OK(sign_buf(dir, buf, f.gopts.insns_sz + f.data_sz, sig,
&sig_sz), "sign insns||metadata"))
goto out;
ASSERT_EQ(try_load(&f, sig, sig_sz, KEY_SPEC_BPF_KEYRING, NULL, 0),
-ENOKEY, "unrestricted keyring still not consulted");
ASSERT_EQ(try_load(&f, sig, sig_sz, KEY_SPEC_SESSION_KEYRING, NULL, 0),
-EPERM, "caller-supplied keyring refused before provisioning");
if (!ASSERT_OK(syscall(__NR_keyctl, KEYCTL_RESTRICT_KEYRING,
KEY_SPEC_BPF_KEYRING, NULL, NULL),
"restrict bpf keyring"))
goto out;
if (!ASSERT_OK_FD(try_load(&f, sig, sig_sz, KEY_SPEC_BPF_KEYRING,
NULL, 0),
"load signed by a key in the .bpf keyring"))
goto out;
bad_dir = mkdtemp(bad_tmpl);
if (!ASSERT_OK_PTR(bad_dir, "mkdtemp unenrolled"))
goto out;
if (!ASSERT_OK(run_setup("genkey", bad_dir), "verify_sig_setup genkey unenrolled"))
goto out;
bad_sz = 8192;
bad = malloc(bad_sz);
if (!ASSERT_OK_PTR(bad, "bad sig buf"))
goto out;
if (!ASSERT_OK(sign_buf(bad_dir, buf, f.gopts.insns_sz + f.data_sz, bad,
&bad_sz), "sign with an unenrolled key"))
goto out;
ASSERT_EQ(try_load(&f, bad, bad_sz, KEY_SPEC_BPF_KEYRING, log_buf,
sizeof(log_buf)), -ENOKEY,
"key outside the bpf keyring refused");
ASSERT_HAS_SUBSTR(log_buf, "signature verification failed",
"the bpf keyring was consulted");
f.blob[0] ^= 0xff;
err = try_load(&f, sig, sig_sz, KEY_SPEC_BPF_KEYRING, NULL, 0);
f.blob[0] ^= 0xff;
ASSERT_EQ(err, -EKEYREJECTED, "tampered metadata refused");
ASSERT_EQ(try_load(&f, sig, sig_sz, KEY_SPEC_SESSION_KEYRING, NULL, 0),
-EPERM, "caller-supplied keyring refused once .bpf is in use");
err = keyctl_ret(KEYCTL_UNLINK, KEY_SPEC_SESSION_KEYRING, serial);
ASSERT_EQ(err, -ENOENT, "keyring writable while the user bits are there");
err = keyctl_ret(KEYCTL_SETPERM, serial, BPF_KEYRING_PERM_LOCKED);
if (!ASSERT_OK(err, "drop the user bits on the bpf keyring"))
goto out;
ASSERT_OK_FD(try_load(&f, sig, sig_sz, KEY_SPEC_BPF_KEYRING, NULL, 0),
"load still verified against the locked keyring");
ASSERT_EQ(keyctl_ret(KEYCTL_GET_KEYRING_ID, KEY_SPEC_BPF_KEYRING, 0),
-EACCES, "the special id grants no rights of its own");
err = keyctl_ret(KEYCTL_UNLINK, KEY_SPEC_SESSION_KEYRING, serial);
ASSERT_EQ(err, -EACCES, "unlink refused");
err = keyctl_ret(KEYCTL_CLEAR, serial, 0);
ASSERT_EQ(err, -EACCES, "clear refused");
err = keyctl_ret(KEYCTL_REVOKE, serial, 0);
ASSERT_EQ(err, -EACCES, "revoke refused");
err = keyctl_ret(KEYCTL_INVALIDATE, serial, 0);
ASSERT_EQ(err, -EACCES, "invalidate refused");
err = keyctl_ret(KEYCTL_SET_TIMEOUT, serial, 1);
ASSERT_EQ(err, -EACCES, "timeout refused");
err = keyctl_ret(KEYCTL_SETPERM, serial, BPF_KEYRING_PERM_INITIAL);
ASSERT_EQ(err, -EACCES, "the bits cannot be granted back");
ASSERT_EQ(bpf_keyring_lookup(&nr_keys), serial, "keyring still there");
ASSERT_EQ(nr_keys, 1, "the enrolled key survived");
out:
if (have_fixture)
gen_loader_fixture_fini(&f);
genkey_dir_fini(bad_dir);
free(buf);
free(bad);
free(sig);
rmdir:
genkey_dir_fini(dir);
}
/*
* A signed loader must ignore ctx-supplied map dimensions: the host cannot
* resize a signed program's maps via the loader ctx. Drive a one-map program
@ -831,7 +1139,7 @@ static void signature_authenticates_insns(void)
dir = mkdtemp(dir_tmpl);
if (!ASSERT_OK_PTR(dir, "mkdtemp"))
return;
if (!ASSERT_OK(run_setup("setup", dir), "verify_sig_setup")) {
if (!ASSERT_OK(run_setup("setup-rsa", dir), "verify_sig_setup")) {
rmdir(dir);
return;
}
@ -931,7 +1239,7 @@ static void signature_authenticates_metadata(void)
dir = mkdtemp(dir_tmpl);
if (!ASSERT_OK_PTR(dir, "mkdtemp"))
return;
if (!ASSERT_OK(run_setup("setup", dir), "verify_sig_setup")) {
if (!ASSERT_OK(run_setup("setup-rsa", dir), "verify_sig_setup")) {
rmdir(dir);
return;
}
@ -1267,7 +1575,7 @@ static void lsm_signature_verdict(void)
dir = mkdtemp(dir_tmpl);
if (!ASSERT_OK_PTR(dir, "mkdtemp"))
goto out;
if (!ASSERT_OK(run_setup("setup", dir), "verify_sig_setup")) {
if (!ASSERT_OK(run_setup("setup-rsa", dir), "verify_sig_setup")) {
rmdir(dir);
dir = NULL;
goto out;
@ -1450,7 +1758,7 @@ static void loadtime_verify(struct bpf_object *obj, int expect_maps)
dir = mkdtemp(dir_tmpl);
if (!ASSERT_OK_PTR(dir, "mkdtemp"))
return;
if (!ASSERT_OK(run_setup("setup", dir), "verify_sig_setup")) {
if (!ASSERT_OK(run_setup("setup-rsa", dir), "verify_sig_setup")) {
rmdir(dir);
return;
}
@ -1524,6 +1832,89 @@ static void loadtime_with_map(void)
test_signed_loader_map__destroy(skel);
}
/*
* End-to-end signed load with a post-quantum key. ML-DSA (FIPS-204) is wired
* through the X.509 and PKCS#7 parsers, and BPF reaches them via
* verify_pkcs7_signature() without knowing the algorithm, so an ML-DSA key in
* the keyring should verify an ML-DSA signed program with no BPF-side work.
*/
static void mldsa_signed_load(void)
{
char dir_tmpl[] = "/tmp/bpfmldsaXXXXXX";
int map_fd = -1, prog_fd = -1, err;
__u8 *sig = NULL, *buf = NULL;
struct gen_loader_fixture f;
bool have_fixture = false;
__u32 sig_sz = 16384;
char *dir;
syscall(__NR_request_key, "keyring", "_uid.0", NULL,
KEY_SPEC_SESSION_KEYRING);
dir = mkdtemp(dir_tmpl);
if (!ASSERT_OK_PTR(dir, "mkdtemp"))
return;
err = run_setup("setup-mldsa", dir);
if (err == SETUP_SKIP) {
printf("%s:SKIP:no working ML-DSA signing, set SELFTESTS_VERBOSE=1\n",
__func__);
test__skip();
genkey_dir_fini(dir);
return;
}
if (!ASSERT_OK(err, "verify_sig_setup setup-mldsa")) {
genkey_dir_fini(dir);
return;
}
sig = malloc(sig_sz);
if (!ASSERT_OK_PTR(sig, "sig buf"))
goto out;
have_fixture = true;
if (gen_loader_fixture_init(&f) != 0)
goto out;
buf = malloc((size_t)f.gopts.insns_sz + f.data_sz);
if (!ASSERT_OK_PTR(buf, "signbuf"))
goto out;
memcpy(buf, f.gopts.insns, f.gopts.insns_sz);
memcpy(buf + f.gopts.insns_sz, f.blob, f.data_sz);
/*
* ML-DSA hashes the message itself, but openssl before 4.0 cannot
* produce a CMS message without signedAttrs for it, and with those in
* play only SHA-512 is permitted for the messageDigest attribute.
*/
if (!ASSERT_OK(sign_buf_digest(dir, buf, f.gopts.insns_sz + f.data_sz,
sig, &sig_sz, "sha512"),
"sign insns||metadata with ML-DSA"))
goto out;
/*
* Guard against the setup silently handing back some other key type:
* an RSA or ECDSA signature is a few hundred bytes, where an ML-DSA-87
* one cannot be smaller than the raw signature it carries.
*/
ASSERT_GT(sig_sz, MLDSA87_SIGNATURE_SIZE, "ML-DSA-87 signature size");
map_fd = setup_meta_map(&f);
if (!ASSERT_OK_FD(map_fd, "meta_map"))
goto out;
prog_fd = load_loader(f.gopts.insns, f.gopts.insns_sz, map_fd, sig,
sig_sz, KEY_SPEC_SESSION_KEYRING, 1);
ASSERT_OK_FD(prog_fd, "ML-DSA signed loader load");
out:
if (prog_fd >= 0)
close(prog_fd);
if (map_fd >= 0)
close(map_fd);
if (have_fixture)
gen_loader_fixture_fini(&f);
free(buf);
free(sig);
run_setup("cleanup", dir);
}
/*
* A signed program need not bind any map. A plain BPF_PROG_TYPE_SYSCALL
* program with no fd_array is signed over its instructions alone: the kernel
@ -1548,7 +1939,7 @@ static void signed_no_fd_array(void)
dir = mkdtemp(dir_tmpl);
if (!ASSERT_OK_PTR(dir, "mkdtemp"))
return;
if (!ASSERT_OK(run_setup("setup", dir), "verify_sig_setup")) {
if (!ASSERT_OK(run_setup("setup-rsa", dir), "verify_sig_setup")) {
rmdir(dir);
return;
}
@ -1619,7 +2010,7 @@ static void signed_map_by_fd_rejected(void)
dir = mkdtemp(dir_tmpl);
if (!ASSERT_OK_PTR(dir, "mkdtemp"))
goto out_map;
if (!ASSERT_OK(run_setup("setup", dir), "verify_sig_setup")) {
if (!ASSERT_OK(run_setup("setup-rsa", dir), "verify_sig_setup")) {
rmdir(dir);
goto out_map;
}
@ -1681,7 +2072,7 @@ static void signed_sparse_fd_array_rejected(void)
dir = mkdtemp(dir_tmpl);
if (!ASSERT_OK_PTR(dir, "mkdtemp"))
goto out_map;
if (!ASSERT_OK(run_setup("setup", dir), "verify_sig_setup")) {
if (!ASSERT_OK(run_setup("setup-rsa", dir), "verify_sig_setup")) {
rmdir(dir);
goto out_map;
}
@ -1735,7 +2126,7 @@ static void signed_module_kfunc_rejected(void)
dir = mkdtemp(dir_tmpl);
if (!ASSERT_OK_PTR(dir, "mkdtemp"))
return;
if (!ASSERT_OK(run_setup("setup", dir), "verify_sig_setup")) {
if (!ASSERT_OK(run_setup("setup-rsa", dir), "verify_sig_setup")) {
rmdir(dir);
return;
}
@ -1777,64 +2168,71 @@ static void signed_module_kfunc_rejected(void)
run_setup("cleanup", dir);
}
enum subtest_boot {
BOOT_ANY,
BOOT_SEALED,
BOOT_UNSEALED,
};
static const struct {
const char *name;
void (*fn)(void);
enum subtest_boot boot;
} subtests[] = {
{ "loadtime_no_map", loadtime_no_map, BOOT_SEALED },
{ "loadtime_with_map", loadtime_with_map, BOOT_SEALED },
{ "metadata_match", metadata_match, BOOT_ANY },
{ "signature_enforced", signature_enforced, BOOT_SEALED },
{ "signed_nonexcl_fd_array_rejected", signed_nonexcl_fd_array_rejected, BOOT_SEALED },
{ "signed_unfrozen_fd_array_rejected", signed_unfrozen_fd_array_rejected, BOOT_SEALED },
{ "signed_nonarray_fd_array_rejected", signed_nonarray_fd_array_rejected, BOOT_SEALED },
{ "signed_btf_fd_array_rejected", signed_btf_fd_array_rejected, BOOT_ANY },
{ "signed_module_kfunc_rejected", signed_module_kfunc_rejected, BOOT_SEALED },
{ "signature_failure_logs", signature_failure_logs, BOOT_SEALED },
{ "signature_too_large", signature_too_large, BOOT_ANY },
{ "signature_zero_size", signature_zero_size, BOOT_ANY },
{ "signature_bad_keyring", signature_bad_keyring, BOOT_SEALED },
{ "bpf_keyring_sealed", bpf_keyring_sealed, BOOT_ANY },
{ "mldsa_signed_load", mldsa_signed_load, BOOT_SEALED },
{ "metadata_ctx_max_entries_ignored", metadata_ctx_max_entries_ignored, BOOT_ANY },
{ "metadata_ctx_initial_value_ignored", metadata_ctx_initial_value_ignored, BOOT_ANY },
{ "signature_authenticates_insns", signature_authenticates_insns, BOOT_SEALED },
{ "signature_authenticates_metadata", signature_authenticates_metadata, BOOT_SEALED },
{ "hash_requires_frozen", hash_requires_frozen, BOOT_ANY },
{ "no_update_after_freeze", no_update_after_freeze, BOOT_ANY },
{ "freeze_writable_mmap", freeze_writable_mmap, BOOT_ANY },
{ "no_writable_mmap_frozen", no_writable_mmap_frozen, BOOT_ANY },
{ "map_hash_matches_libbpf", map_hash_matches_libbpf, BOOT_ANY },
{ "map_hash_multi_element", map_hash_multi_element, BOOT_ANY },
{ "map_hash_bad_size", map_hash_bad_size, BOOT_ANY },
{ "map_hash_unsupported_type", map_hash_unsupported_type, BOOT_ANY },
{ "lsm_signature_verdict", lsm_signature_verdict, BOOT_SEALED },
{ "signed_no_fd_array", signed_no_fd_array, BOOT_SEALED },
{ "signed_map_by_fd_rejected", signed_map_by_fd_rejected, BOOT_SEALED },
{ "signed_sparse_fd_array_rejected", signed_sparse_fd_array_rejected, BOOT_SEALED },
{ "bpf_keyring_provisioned", bpf_keyring_provisioned, BOOT_UNSEALED },
};
void test_signed_loader(void)
{
if (test__start_subtest("loadtime_no_map"))
loadtime_no_map();
if (test__start_subtest("loadtime_with_map"))
loadtime_with_map();
if (test__start_subtest("metadata_match"))
metadata_match();
if (test__start_subtest("signature_enforced"))
signature_enforced();
if (test__start_subtest("signed_nonexcl_fd_array_rejected"))
signed_nonexcl_fd_array_rejected();
if (test__start_subtest("signed_unfrozen_fd_array_rejected"))
signed_unfrozen_fd_array_rejected();
if (test__start_subtest("signed_nonarray_fd_array_rejected"))
signed_nonarray_fd_array_rejected();
if (test__start_subtest("signed_btf_fd_array_rejected"))
signed_btf_fd_array_rejected();
if (test__start_subtest("signed_module_kfunc_rejected"))
signed_module_kfunc_rejected();
if (test__start_subtest("signature_failure_logs"))
signature_failure_logs();
if (test__start_subtest("signature_too_large"))
signature_too_large();
if (test__start_subtest("signature_zero_size"))
signature_zero_size();
if (test__start_subtest("signature_bad_keyring"))
signature_bad_keyring();
if (test__start_subtest("metadata_ctx_max_entries_ignored"))
metadata_ctx_max_entries_ignored();
if (test__start_subtest("metadata_ctx_initial_value_ignored"))
metadata_ctx_initial_value_ignored();
if (test__start_subtest("signature_authenticates_insns"))
signature_authenticates_insns();
if (test__start_subtest("signature_authenticates_metadata"))
signature_authenticates_metadata();
if (test__start_subtest("hash_requires_frozen"))
hash_requires_frozen();
if (test__start_subtest("no_update_after_freeze"))
no_update_after_freeze();
if (test__start_subtest("freeze_writable_mmap"))
freeze_writable_mmap();
if (test__start_subtest("no_writable_mmap_frozen"))
no_writable_mmap_frozen();
if (test__start_subtest("map_hash_matches_libbpf"))
map_hash_matches_libbpf();
if (test__start_subtest("map_hash_multi_element"))
map_hash_multi_element();
if (test__start_subtest("map_hash_bad_size"))
map_hash_bad_size();
if (test__start_subtest("map_hash_unsupported_type"))
map_hash_unsupported_type();
if (test__start_subtest("lsm_signature_verdict"))
lsm_signature_verdict();
if (test__start_subtest("signed_no_fd_array"))
signed_no_fd_array();
if (test__start_subtest("signed_map_by_fd_rejected"))
signed_map_by_fd_rejected();
if (test__start_subtest("signed_sparse_fd_array_rejected"))
signed_sparse_fd_array_rejected();
bool unsealed = keyring_unsealed_boot();
unsigned int i;
for (i = 0; i < ARRAY_SIZE(subtests); i++) {
if (!test__start_subtest(subtests[i].name))
continue;
if (subtests[i].boot == BOOT_SEALED && unsealed) {
printf("%s:SKIP:needs a boot without bpf.keyring_unsealed=1\n",
subtests[i].name);
test__skip();
continue;
}
if (subtests[i].boot == BOOT_UNSEALED && !unsealed) {
printf("%s:SKIP:needs bpf.keyring_unsealed=1\n",
subtests[i].name);
test__skip();
continue;
}
subtests[i].fn();
}
}

67
tools/testing/selftests/bpf/prog_tests/stream.c
View File

@ -46,6 +46,10 @@ void test_stream_syscall(void)
ret = -errno;
ASSERT_EQ(ret, -EFAULT, "bad stream buf");
ASSERT_LT(bpf_prog_stream_read(prog_fd, BPF_STREAM_STDOUT, buf, UINT_MAX, NULL), 0, "error");
ret = -errno;
ASSERT_EQ(ret, -EINVAL, "large stream buf");
ret = bpf_prog_stream_read(prog_fd, BPF_STREAM_STDOUT, buf, 2, NULL);
ASSERT_EQ(ret, 2, "bytes");
ret = bpf_prog_stream_read(prog_fd, BPF_STREAM_STDOUT, buf, 2, NULL);
@ -58,6 +62,69 @@ void test_stream_syscall(void)
stream__destroy(skel);
}
void test_stream_oversize(void)
{
LIBBPF_OPTS(bpf_test_run_opts, opts);
struct stream *skel;
int ret, prog_fd;
skel = stream__open_and_load();
if (!ASSERT_OK_PTR(skel, "stream__open_and_load"))
return;
prog_fd = bpf_program__fd(skel->progs.stream_oversize);
ret = bpf_prog_test_run_opts(prog_fd, &opts);
ASSERT_OK(ret, "oversize run");
ASSERT_OK(opts.retval, "oversize retval");
stream__destroy(skel);
}
void test_stream_partial_read(void)
{
LIBBPF_OPTS(bpf_test_run_opts, opts);
struct stream *skel;
int ret, prog_fd;
long page_size;
char *page, *buf;
char rest[8] = {};
skel = stream__open_and_load();
if (!ASSERT_OK_PTR(skel, "stream__open_and_load"))
return;
prog_fd = bpf_program__fd(skel->progs.stream_syscall);
ret = bpf_prog_test_run_opts(prog_fd, &opts);
ASSERT_OK(ret, "ret");
ASSERT_OK(opts.retval, "retval");
page_size = sysconf(_SC_PAGESIZE);
page = mmap(NULL, page_size * 2, PROT_READ | PROT_WRITE,
MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
if (!ASSERT_NEQ(page, MAP_FAILED, "mmap")) {
stream__destroy(skel);
return;
}
/* Leave only the first page mapped so a straddling copy faults. */
if (!ASSERT_OK(munmap(page + page_size, page_size), "munmap second page")) {
munmap(page, page_size * 2);
stream__destroy(skel);
return;
}
buf = page + page_size - 1;
ret = bpf_prog_stream_read(prog_fd, BPF_STREAM_STDOUT, buf, 3, NULL);
ASSERT_EQ(ret, 1, "partial bytes");
ASSERT_EQ(buf[0], 'f', "first byte");
ret = bpf_prog_stream_read(prog_fd, BPF_STREAM_STDOUT, rest, sizeof(rest), NULL);
ASSERT_EQ(ret, 2, "remaining bytes");
ASSERT_OK(memcmp(rest, "oo", 2), "remaining data");
munmap(page, page_size);
stream__destroy(skel);
}
static void test_address(struct bpf_program *prog, unsigned long *fault_addr_p)
{
LIBBPF_OPTS(bpf_test_run_opts, opts);

54
tools/testing/selftests/bpf/prog_tests/usdt.c
View File

@ -250,6 +250,7 @@ static void subtest_basic_usdt(bool optimized)
#ifdef __x86_64__
extern void usdt_1(void);
extern void usdt_2(void);
extern void usdt_2_cross_page(void);
extern void usdt_red_zone_trigger(void);
static unsigned char nop1[1] = { 0x90 };
@ -342,6 +343,57 @@ static void subtest_optimized_attach(void)
test_usdt__destroy(skel);
}
/*
* Test attachment to a USDT probe whose nop10 crosses a page boundary.
* The kernel can't optimize such nop10, so libbpf keeps the uprobe on
* the preceding 1-byte nop. Do not assume any particular placement
* here, though: however the probe ends up attached, the attachment
* must succeed and the probe must fire.
*/
static void subtest_optimized_attach_cross_page(void)
{
long page_sz = sysconf(_SC_PAGESIZE);
struct test_usdt *skel;
__u8 *addr = NULL;
long i;
/* combo is placed up to a page of padding after the function start */
for (i = 0; i < 2 * page_sz; i++) {
if (!memcmp((void *)usdt_2_cross_page + i, nop1_nop10_combo, 11)) {
addr = (void *)usdt_2_cross_page + i;
break;
}
}
if (!ASSERT_OK_PTR(addr, "find_nop1_nop10_combo"))
return;
/* layout sanity check: the nop10 must cross the page boundary */
if (!ASSERT_GT((unsigned long)(addr + 1) % page_sz + 10, page_sz,
"nop10_crosses_page"))
return;
skel = test_usdt__open_and_load();
if (!ASSERT_OK_PTR(skel, "test_usdt__open_and_load"))
return;
skel->bss->my_pid = getpid();
skel->links.usdt0 = bpf_program__attach_usdt(skel->progs.usdt0,
0 /*self*/, "/proc/self/exe",
"optimized_attach",
"usdt_2_cross_page", NULL);
if (!ASSERT_OK_PTR(skel->links.usdt0, "bpf_program__attach_usdt"))
goto cleanup;
usdt_2_cross_page();
usdt_2_cross_page();
ASSERT_EQ(skel->bss->usdt0_called, 2, "usdt0_called");
cleanup:
test_usdt__destroy(skel);
}
/*
* Test that USDT arguments survive nop10 optimization in a function where
* the compiler places operands in the red zone.
@ -660,6 +712,8 @@ void test_usdt(void)
subtest_basic_usdt(true);
if (test__start_subtest("optimized_attach"))
subtest_optimized_attach();
if (test__start_subtest("optimized_attach_cross_page"))
subtest_optimized_attach_cross_page();
if (test__start_subtest("optimized_red_zone"))
subtest_optimized_red_zone();
#endif

2
tools/testing/selftests/bpf/prog_tests/verifier.c
View File

@ -5,6 +5,7 @@
#include "arena_kfunc.skel.h"
#include "arena_kfunc_jit.skel.h"
#include "cap_helpers.h"
#include "verifier_aggregate_ret.skel.h"
#include "verifier_align.skel.h"
#include "verifier_and.skel.h"
#include "verifier_arena.skel.h"
@ -170,6 +171,7 @@ void test_arena_kfunc(void) { RUN_TESTS(arena_kfunc); }
void test_arena_kfunc_jit(void) { RUN_TESTS(arena_kfunc_jit); }
void test_verifier_aggregate_ret(void) { RUN_TESTS(verifier_aggregate_ret); }
void test_verifier_align(void) { RUN(verifier_align); }
void test_verifier_and(void) { RUN(verifier_and); }
void test_verifier_arena(void) { RUN(verifier_arena); }

4
tools/testing/selftests/bpf/prog_tests/verify_pkcs7_sig.c
View File

@ -257,7 +257,7 @@ static void test_verify_pkcs7_sig_from_map(void)
if (!ASSERT_OK_PTR(tmp_dir, "mkdtemp"))
return;
ret = _run_setup_process(tmp_dir, "setup");
ret = _run_setup_process(tmp_dir, "setup-rsa");
if (!ASSERT_OK(ret, "_run_setup_process"))
goto close_prog;
@ -458,7 +458,7 @@ static void test_pkcs7_sig_fsverity(void)
snprintf(data_path, PATH_MAX, "%s/data-file", tmp_dir);
snprintf(sig_path, PATH_MAX, "%s/sig-file", tmp_dir);
ret = _run_setup_process(tmp_dir, "setup");
ret = _run_setup_process(tmp_dir, "setup-rsa");
if (!ASSERT_OK(ret, "_run_setup_process"))
goto out;

3
tools/testing/selftests/bpf/prog_tests/xdp_adjust_frags.c
View File

@ -18,6 +18,9 @@ static void test_xdp_update_frags(void)
return;
prog = bpf_object__next_program(obj, NULL);
ASSERT_EQ(bpf_program__flags(prog) & BPF_F_XDP_HAS_FRAGS,
BPF_F_XDP_HAS_FRAGS, "frags in program flags");
if (bpf_object__load(obj))
return;

5
tools/testing/selftests/bpf/prog_tests/xdp_devmap_attach.c
View File

@ -146,6 +146,11 @@ static void test_xdp_with_devmap_frags_helpers(void)
if (!ASSERT_OK_PTR(skel, "test_xdp_with_devmap_helpers__open_and_load"))
return;
ASSERT_EQ(bpf_program__flags(skel->progs.xdp_dummy_dm_frags) & BPF_F_XDP_HAS_FRAGS,
BPF_F_XDP_HAS_FRAGS, "frags in program flags");
ASSERT_EQ(bpf_program__flags(skel->progs.xdp_dummy_dm) & BPF_F_XDP_HAS_FRAGS,
0, "frags not in program flags");
dm_fd_frags = bpf_program__fd(skel->progs.xdp_dummy_dm_frags);
map_fd = bpf_map__fd(skel->maps.dm_ports);
err = bpf_prog_get_info_by_fd(dm_fd_frags, &info, &len);

4
tools/testing/selftests/bpf/prog_tests/xdp_metadata.c
View File

@ -408,14 +408,14 @@ void test_xdp_metadata(void)
prog = bpf_object__find_program_by_name(bpf_obj->obj, "rx");
bpf_program__set_ifindex(prog, rx_ifindex);
bpf_program__set_flags(prog, BPF_F_XDP_DEV_BOUND_ONLY);
bpf_program__set_flags(prog, bpf_program__flags(prog) | BPF_F_XDP_DEV_BOUND_ONLY);
/* Make sure we can load a dev-bound program that performs
* XDP_REDIRECT into a devmap.
*/
new_prog = bpf_object__find_program_by_name(bpf_obj->obj, "redirect");
bpf_program__set_ifindex(new_prog, rx_ifindex);
bpf_program__set_flags(new_prog, BPF_F_XDP_DEV_BOUND_ONLY);
bpf_program__set_flags(new_prog, bpf_program__flags(new_prog) | BPF_F_XDP_DEV_BOUND_ONLY);
if (!ASSERT_OK(xdp_metadata__load(bpf_obj), "load skeleton"))
goto out;

383
tools/testing/selftests/bpf/progs/aggregate_ret_func.c Normal file
View File

@ -0,0 +1,383 @@
// SPDX-License-Identifier: GPL-2.0
/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
#include <linux/bpf.h>
#include <bpf/bpf_helpers.h>
#include <bpf_arena_common.h>
#include "bpf_misc.h"
typedef unsigned __int128 u128;
__naked u128 global_agg_good(void)
{
asm volatile (
"r0 = 0x1234;" /* low 64 bits */
"r2 = 0x5678;" /* high 64 bits */
"exit;"
);
}
__naked u128 global_agg_bad(void)
{
asm volatile (
"r0 = 0;"
"exit;"
);
}
__naked u128 global_agg_bad_ptr(void)
{
asm volatile (
"r0 = 0;"
"r2 = r10;"
"exit;"
);
}
SEC("tc")
__failure __msg("R2 !read_ok")
__naked int aggregate_ret_global_fail(void)
{
asm volatile (
"call %[global_agg_bad];"
"r0 = r2;"
"exit;"
:
: __imm(global_agg_bad)
: __clobber_all);
}
SEC("tc")
__load_if_JITed()
__failure __msg("At subprogram exit the register R2 is not a scalar value")
__naked int aggregate_ret_global_ptr_fail(void)
{
asm volatile (
"call %[global_agg_bad_ptr];"
"r0 = r2;"
"exit;"
:
: __imm(global_agg_bad_ptr)
: __clobber_all);
}
static __naked __noinline u128 static_agg_bad_ptr(void)
{
asm volatile (
"r0 = 0;"
"r2 = r10;" /* stack pointer placed in the second return register */
"exit;"
);
}
/*
* R2 is a return register once the subprogram returns a pair, so a stack
* pointer left in it is rejected at the callee's exit exactly as one in R0
* is: the callee frame is gone by the time the caller could use it.
*/
SEC("tc")
__load_if_JITed()
__failure __msg("cannot return stack pointer to the caller")
__naked int aggregate_ret_static_ptr_fail(void)
{
asm volatile (
"call %[static_agg_bad_ptr];"
"r0 = 0;"
"exit;"
:
: __imm(static_agg_bad_ptr)
: __clobber_all);
}
static __naked __noinline u128 static_agg_no_r2(void)
{
asm volatile (
"r0 = 0;"
"exit;"
);
}
SEC("tc")
__failure __msg("R2 !read_ok")
__naked int aggregate_ret_static_uninit_fail(void)
{
asm volatile (
"call %[static_agg_no_r2];"
"r0 = r2;"
"exit;"
:
: __imm(static_agg_no_r2)
: __clobber_all);
}
static __naked __noinline u128 static_agg_precise(void)
{
asm volatile (
"r0 = 0;"
"r2 = 4;" /* second half; its value is made precise below */
"exit;"
);
}
SEC("tc")
__load_if_JITed()
__success __retval(0)
__log_level(2)
__msg("mark_precise: frame0: last_idx 5 first_idx 0 subseq_idx -1")
__msg("mark_precise: frame0: regs=r6 stack= before 4: (07) r1 += -8")
__msg("mark_precise: frame0: regs=r6 stack= before 3: (bf) r1 = r10")
__msg("mark_precise: frame0: regs=r6 stack= before 2: (57) r6 &= 7")
__msg("mark_precise: frame0: regs=r6 stack= before 1: (bf) r6 = r2")
__msg("mark_precise: frame0: regs=r2 stack= before 12: (95) exit")
__msg("mark_precise: frame1: regs=r2 stack= before 11: (b7) r2 = 4")
__naked int aggregate_ret_static_precise(void)
{
asm volatile (
"call %[static_agg_precise];"
"r6 = r2;" /* derived from the aggregate's second half */
"r6 &= 7;" /* keep it in [0, 7] to index the stack */
"r1 = r10;"
"r1 += -8;"
"r1 += r6;" /* ptr += scalar marks r6 (hence R2) precise */
"r0 = 0;"
"*(u8 *)(r1 + 0) = r0;"
"r0 = 0;"
"exit;"
:
: __imm(static_agg_precise)
: __clobber_all);
}
SEC("tc")
__load_if_JITed()
__success __retval(0)
__log_level(2)
__msg("mark_precise: frame0: last_idx 5 first_idx 0 subseq_idx -1")
__msg("mark_precise: frame0: regs=r6 stack= before 4: (07) r1 += -8")
__msg("mark_precise: frame0: regs=r6 stack= before 3: (bf) r1 = r10")
__msg("mark_precise: frame0: regs=r6 stack= before 2: (57) r6 &= 7")
__msg("mark_precise: frame0: regs=r6 stack= before 1: (bf) r6 = r2")
__msg("mark_precise: frame0: regs=r2 stack= before 0: (85) call pc+9")
__naked int aggregate_ret_global_precise(void)
{
asm volatile (
"call %[global_agg_good];"
"r6 = r2;" /* derived from the aggregate's second half */
"r6 &= 7;" /* keep it in [0, 7] to index the stack */
"r1 = r10;"
"r1 += -8;"
"r1 += r6;" /* ptr += scalar marks r6 (hence R2) precise */
"r0 = 0;"
"*(u8 *)(r1 + 0) = r0;"
"r0 = 0;"
"exit;"
:
: __imm(global_agg_good)
: __clobber_all);
}
#if defined(__clang_major__) && __clang_major__ >= 23
/* A by-value struct that smuggles a pointer, which must be rejected. */
struct with_ptr {
void *p;
__u64 x;
};
/* A by-value union that smuggles a pointer, which must be rejected too. */
union upair_with_ptr {
void *p;
__u64 halves[2];
};
__naked struct with_ptr global_ret_struct_ptr(void)
{
asm volatile (
"r0 = 0;"
"r2 = 0;"
"exit;"
);
}
SEC("tc")
__failure __msg("Global function global_ret_struct_ptr() has unsupported return type")
__naked int aggregate_ret_global_struct_ptr_fail(void)
{
asm volatile (
"call %[global_ret_struct_ptr];"
"r0 = 0;"
"exit;"
:
: __imm(global_ret_struct_ptr)
: __clobber_all);
}
__naked union upair_with_ptr global_ret_union_ptr(void)
{
asm volatile (
"r0 = 0;"
"r2 = 0;"
"exit;"
);
}
SEC("tc")
__failure __msg("Global function global_ret_union_ptr() has unsupported return type")
__naked int aggregate_ret_global_union_ptr_fail(void)
{
asm volatile (
"call %[global_ret_union_ptr];"
"r0 = 0;"
"exit;"
:
: __imm(global_ret_union_ptr)
: __clobber_all);
}
#endif
/*
* gcc returns a by-value struct through a hidden pointer, and emits the
* 'r0 = r1' returning it after the __naked body's exit, leaving the
* subprogram falling through. Build these with clang only.
*/
#if defined(__clang__)
struct arena_pair {
void __arena *lo;
void __arena *hi;
};
struct arena_and_scalar {
void __arena *p;
__u64 x;
};
struct arena_array {
void __arena *p[2];
};
struct arena_single {
void __arena *p;
};
union arena_upair {
void __arena *p;
__u64 halves[2];
};
__naked struct arena_pair global_ret_arena_pair(void)
{
asm volatile (
"r0 = 0;"
"r2 = 0;"
"exit;"
);
}
SEC("tc")
__load_if_JITed()
__success __retval(0)
__naked int aggregate_ret_global_arena_pair(void)
{
asm volatile (
"call %[global_ret_arena_pair];"
"r0 = 0;"
"exit;"
:
: __imm(global_ret_arena_pair)
: __clobber_all);
}
__naked struct arena_and_scalar global_ret_arena_and_scalar(void)
{
asm volatile (
"r0 = 0;"
"r2 = 0;"
"exit;"
);
}
SEC("tc")
__load_if_JITed()
__success __retval(0)
__naked int aggregate_ret_global_arena_and_scalar(void)
{
asm volatile (
"call %[global_ret_arena_and_scalar];"
"r0 = 0;"
"exit;"
:
: __imm(global_ret_arena_and_scalar)
: __clobber_all);
}
__naked struct arena_array global_ret_arena_array(void)
{
asm volatile (
"r0 = 0;"
"r2 = 0;"
"exit;"
);
}
SEC("tc")
__load_if_JITed()
__success __retval(0)
__naked int aggregate_ret_global_arena_array(void)
{
asm volatile (
"call %[global_ret_arena_array];"
"r0 = 0;"
"exit;"
:
: __imm(global_ret_arena_array)
: __clobber_all);
}
__naked struct arena_single global_ret_arena_single(void)
{
asm volatile (
"r0 = 0;"
"exit;"
);
}
SEC("tc")
__success __retval(0)
__naked int aggregate_ret_global_arena_single(void)
{
asm volatile (
"call %[global_ret_arena_single];"
"r0 = 0;"
"exit;"
:
: __imm(global_ret_arena_single)
: __clobber_all);
}
__naked union arena_upair global_ret_arena_union(void)
{
asm volatile (
"r0 = 0;"
"r2 = 0;"
"exit;"
);
}
SEC("tc")
__load_if_JITed()
__success __retval(0)
__naked int aggregate_ret_global_arena_union(void)
{
asm volatile (
"call %[global_ret_arena_union];"
"r0 = 0;"
"exit;"
:
: __imm(global_ret_arena_union)
: __clobber_all);
}
#endif
char _license[] SEC("license") = "GPL";

203
tools/testing/selftests/bpf/progs/aggregate_ret_kfunc.c Normal file
View File

@ -0,0 +1,203 @@
// SPDX-License-Identifier: GPL-2.0
/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
#include <vmlinux.h>
#include <bpf/bpf_helpers.h>
#include "bpf_misc.h"
#include "../test_kmods/bpf_testmod_kfunc.h"
/*
* Reference kfunc addresses to force those BTF to be emitted. Taking the address
* (rather than calling) avoids any dependence on the compiler lowering an
* __int128 or struct return value, which the BPF backend only supports from
* LLVM 23 on.
*/
void __kfunc_btf_root(void)
{
asm volatile (""
:
: "r"(&bpf_kfunc_call_test_i128),
"r"(&bpf_kfunc_call_test_ret_fastcall),
"r"(&bpf_kfunc_call_test_ret_ptr),
"r"(&bpf_kfunc_call_test_ret_ii),
"r"(&bpf_kfunc_call_test_ret_nested),
"r"(&bpf_kfunc_call_test_ret_ptr_arr),
"r"(&bpf_kfunc_call_test_ret_deep),
"r"(&bpf_kfunc_call_test_ret_arr_struct),
"r"(&bpf_kfunc_call_test_ret_arr2d),
"r"(&bpf_kfunc_call_test_ret_big));
}
SEC("tc")
__arch_x86_64 __arch_arm64
__load_if_JITed()
__success __retval(0)
__log_level(2)
__msg("mark_precise: frame0: last_idx 7 first_idx 0 subseq_idx -1")
__msg("mark_precise: frame0: regs=r6 stack= before 6: (07) r1 += -8")
__msg("mark_precise: frame0: regs=r6 stack= before 5: (bf) r1 = r10")
__msg("mark_precise: frame0: regs=r6 stack= before 4: (57) r6 &= 7")
__msg("mark_precise: frame0: regs=r6 stack= before 3: (bf) r6 = r2")
__msg("mark_precise: frame0: regs=r2 stack= before 2: (85) call bpf_kfunc_call_test_i128")
__naked int aggregate_ret_kfunc_precise(void)
{
asm volatile (
"r1 = 1;"
"r2 = 2;"
"call %[bpf_kfunc_call_test_i128];"
"r6 = r2;" /* second return half */
"r6 &= 7;" /* keep it in [0, 7] to index the stack */
"r1 = r10;"
"r1 += -8;"
"r1 += r6;" /* ptr += scalar marks r6 (hence R2) precise */
"r0 = 0;"
"*(u8 *)(r1 + 0) = r0;"
"r0 = 0;"
"exit;"
:
: __imm(bpf_kfunc_call_test_i128)
: __clobber_all);
}
SEC("tc")
__arch_x86_64 __arch_arm64
__failure __msg("kfunc bpf_kfunc_call_test_ret_fastcall with >8-byte return is not supported with KF_FASTCALL")
__naked int aggregate_ret_kfunc_fastcall_fail(void)
{
asm volatile (
"r1 = 1;"
"r2 = 2;"
"call %[bpf_kfunc_call_test_ret_fastcall];"
"r0 = 0;"
"exit;"
:
: __imm(bpf_kfunc_call_test_ret_fastcall)
: __clobber_all);
}
SEC("tc")
__arch_x86_64 __arch_arm64
__failure __msg("is not composed of scalars or arena pointers")
__msg("member 'p' has type PTR")
__naked int aggregate_ret_kfunc_ptr_fail(void)
{
asm volatile (
"r1 = 0;"
"call %[bpf_kfunc_call_test_ret_ptr];"
"r0 = 0;"
"exit;"
:
: __imm(bpf_kfunc_call_test_ret_ptr)
: __clobber_all);
}
SEC("tc")
__arch_x86_64 __arch_arm64
__failure __msg("is not composed of scalars or arena pointers")
__msg("member 'in.p' has type PTR")
__naked int aggregate_ret_kfunc_nested_ptr_fail(void)
{
asm volatile (
"r1 = 0;"
"call %[bpf_kfunc_call_test_ret_nested];"
"r0 = 0;"
"exit;"
:
: __imm(bpf_kfunc_call_test_ret_nested)
: __clobber_all);
}
SEC("tc")
__arch_x86_64 __arch_arm64
__failure __msg("is not composed of scalars or arena pointers")
__msg("member 'p[]' has type PTR")
__naked int aggregate_ret_kfunc_ptr_arr_fail(void)
{
asm volatile (
"call %[bpf_kfunc_call_test_ret_ptr_arr];"
"r0 = 0;"
"exit;"
:
: __imm(bpf_kfunc_call_test_ret_ptr_arr)
: __clobber_all);
}
SEC("tc")
__arch_x86_64 __arch_arm64
__failure __msg("is not composed of scalars or arena pointers")
__msg("member 'in1[].in2.p' has type PTR")
__naked int aggregate_ret_kfunc_arr_struct_fail(void)
{
asm volatile (
"call %[bpf_kfunc_call_test_ret_arr_struct];"
"r0 = 0;"
"exit;"
:
: __imm(bpf_kfunc_call_test_ret_arr_struct)
: __clobber_all);
}
SEC("tc")
__arch_x86_64 __arch_arm64
__failure __msg("is not composed of scalars or arena pointers")
__msg("member 'a[].p' has type PTR")
__naked int aggregate_ret_kfunc_arr2d_fail(void)
{
asm volatile (
"call %[bpf_kfunc_call_test_ret_arr2d];"
"r0 = 0;"
"exit;"
:
: __imm(bpf_kfunc_call_test_ret_arr2d)
: __clobber_all);
}
SEC("tc")
__arch_x86_64 __arch_arm64
__failure __msg("max struct nesting depth exceeded")
__naked int aggregate_ret_kfunc_too_deep_fail(void)
{
asm volatile (
"r1 = 0;"
"call %[bpf_kfunc_call_test_ret_deep];"
"r0 = 0;"
"exit;"
:
: __imm(bpf_kfunc_call_test_ret_deep)
: __clobber_all);
}
SEC("tc")
__arch_x86_64 __arch_arm64
__failure __msg("R2 !read_ok")
__naked int aggregate_ret_kfunc_small_no_r2(void)
{
asm volatile (
"r1 = 0;"
"r2 = 0;"
"call %[bpf_kfunc_call_test_ret_ii];"
"r0 = r2;" /* R2 is not a return register for a <=8 byte struct */
"exit;"
:
: __imm(bpf_kfunc_call_test_ret_ii)
: __clobber_all);
}
/*
* A return value larger than 16 bytes does not fit in R0:R2 and is rejected by
* btf_distill_func_proto() before the KF_FASTCALL and JIT-capability checks.
*/
SEC("tc")
__arch_x86_64 __arch_arm64
__failure __msg("The function bpf_kfunc_call_test_ret_big return type STRUCT is unsupported")
__naked int aggregate_ret_kfunc_too_big_fail(void)
{
asm volatile (
"call %[bpf_kfunc_call_test_ret_big];"
"r0 = 0;"
"exit;"
:
: __imm(bpf_kfunc_call_test_ret_big)
: __clobber_all);
}
char _license[] SEC("license") = "GPL";

121
tools/testing/selftests/bpf/progs/aggregate_ret_kfunc_arena.c Normal file
View File

@ -0,0 +1,121 @@
// SPDX-License-Identifier: GPL-2.0
/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
#include <vmlinux.h>
#include <bpf/bpf_helpers.h>
#include <bpf_arena_common.h>
#include "bpf_misc.h"
#include "../test_kmods/bpf_testmod_kfunc.h"
#if defined(__clang_major__) && __clang_major__ >= 23
struct {
__uint(type, BPF_MAP_TYPE_ARENA);
__uint(map_flags, BPF_F_MMAPABLE);
__uint(max_entries, 2);
} arena SEC(".maps");
SEC("syscall")
__arch_x86_64 __arch_arm64
__load_if_JITed()
__success __retval(0)
int aggregate_ret_kfunc_arena(void *ctx)
{
u32 volatile __arena *page = bpf_arena_alloc_pages(&arena, NULL, 1, NUMA_NO_NODE, 0);
u32 volatile __arena *a, *b;
struct prog_test_ret_arena r;
if (!page)
return 1;
/* Both halves come back in R0:R2, pointing at page and page + 4. */
r = bpf_kfunc_call_test_ret_arena((u64)page);
if (!r.a || !r.b)
return 2;
a = (u32 __arena *)r.a;
b = (u32 __arena *)r.b;
*a = 1;
*b = 2;
if (*a != 1)
return 3;
if (*b != 2)
return 4;
page[0] = 7;
if (*a != 7)
return 5;
page[1] = 9;
if (*b != 9)
return 6;
return 0;
}
SEC("syscall")
__arch_x86_64 __arch_arm64
__load_if_JITed()
__success __retval(0)
int aggregate_ret_kfunc_arena_mixed(void *ctx)
{
u32 __arena *page = bpf_arena_alloc_pages(&arena, NULL, 1, NUMA_NO_NODE, 0);
struct prog_test_ret_arena_mixed r;
u32 volatile __arena *p;
if (!page)
return 1;
/* An arena pointer in R0 beside a scalar in R2. */
r = bpf_kfunc_call_test_ret_arena_mixed((u64)page);
if (!r.p)
return 2;
if (r.tag != 0xbeef)
return 3;
p = (u32 __arena *)r.p;
*p = 3;
if (*p != 3)
return 4;
return 0;
}
SEC("syscall")
__arch_x86_64 __arch_arm64
__failure __msg("is not composed of scalars or arena pointers")
__msg("member 'b' has type PTR")
int aggregate_ret_kfunc_arena_untagged_fail(void *ctx)
{
struct prog_test_ret_arena_untagged r;
r = bpf_kfunc_call_test_ret_arena_untagged(0);
return r.a == r.b;
}
SEC("syscall")
__arch_x86_64 __arch_arm64
__failure __msg("is not composed of scalars or arena pointers")
__msg("member 'b' has type PTR")
int aggregate_ret_kfunc_arena_union_fail(void *ctx)
{
union prog_test_ret_arena_union r;
r = bpf_kfunc_call_test_ret_arena_union(0);
return r.a == r.b;
}
#else
SEC("socket")
__description("aggregate_ret_kfunc_arena: needs LLVM 23, dummy test")
__skip("needs LLVM 23")
__success
int dummy_test(void)
{
return 0;
}
#endif
char _license[] SEC("license") = "GPL";

29
tools/testing/selftests/bpf/progs/aggregate_ret_target.c Normal file
View File

@ -0,0 +1,29 @@
// SPDX-License-Identifier: GPL-2.0
/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
#include <linux/bpf.h>
#include <bpf/bpf_helpers.h>
#include "bpf_misc.h"
/* freplace target: a global subprogram returning 16 bytes in R0:R2. */
__naked unsigned __int128 agg_ret_target_func(void)
{
asm volatile (
"r0 = 0x1234;"
"r2 = 0x5678;"
"exit;"
);
}
SEC("tc")
__naked int agg_ret_target(void)
{
asm volatile (
"call %[agg_ret_target_func];"
"r0 = 0;"
"exit;"
:
: __imm(agg_ret_target_func)
: __clobber_all);
}
char _license[] SEC("license") = "GPL";

3
tools/testing/selftests/bpf/progs/arena_kfunc.c
View File

@ -233,8 +233,9 @@ SEC("syscall")
__arch_x86_64
__arch_arm64
__description("arena_arg_stack: not supported, dummy test")
__skip("arena_arg_stack: not supported")
__success
int arena_arg_stack(void *ctx)
int dummy_test(void *ctx)
{
return 0;
}

10
tools/testing/selftests/bpf/progs/bpf_misc.h
View File

@ -106,6 +106,11 @@
* __description Text to be used for display and as an additional filter
* alias, while the original program name stays matchable.
*
* __skip Report the test as SKIP with the given reason instead of
* running it. For placeholder programs that stand in for a
* feature the toolchain or JIT cannot provide, so a run does
* not report OK for coverage it never executed.
*
* __log_level Log level to use for the program, numeric value expected.
*
* __flag Adds one flag use for the program, the following values are valid:
@ -139,6 +144,7 @@
#define __failure __test_tag("test_expect_failure")
#define __success __test_tag("test_expect_success")
#define __description(desc) __test_tag("test_description=" desc)
#define __skip(reason) __test_tag("test_skip=" reason)
#define __msg_unpriv(msg) __test_tag("test_expect_msg_unpriv=" msg)
#define __not_msg_unpriv(msg) __test_tag("test_expect_not_msg_unpriv=" msg)
#define __xlated_unpriv(msg) __test_tag("test_expect_xlated_unpriv=" msg)
@ -255,11 +261,11 @@
(defined(__TARGET_ARCH_riscv) && __riscv_xlen == 64) || \
defined(__TARGET_ARCH_arm) || defined(__TARGET_ARCH_s390) || \
defined(__TARGET_ARCH_loongarch)) && \
__clang_major__ >= 18
(__clang_major__ >= 18 || defined(__BPF_FEATURE_GOTOL))
#define CAN_USE_GOTOL
#endif
#if __clang_major__ >= 18
#if __clang_major__ >= 18 || defined(__BPF_FEATURE_ST)
#define CAN_USE_BPF_ST
#endif

3
tools/testing/selftests/bpf/progs/btf__stack_arg_precision.c
View File

@ -4,7 +4,8 @@
#include <bpf/bpf_helpers.h>
#include "../test_kmods/bpf_testmod_kfunc.h"
#if (defined(__TARGET_ARCH_x86) || defined(__TARGET_ARCH_arm64)) && \
#if (defined(__TARGET_ARCH_x86) || defined(__TARGET_ARCH_arm64) || \
(defined(__TARGET_ARCH_riscv) && __riscv_xlen == 64)) && \
defined(__BPF_FEATURE_STACK_ARGUMENT)
long subprog_call_mem_kfunc(long a, long b, long c, long d, long e, long size)

3
tools/testing/selftests/bpf/progs/btf__verifier_stack_arg_order.c
View File

@ -3,7 +3,8 @@
#include <vmlinux.h>
#include <bpf/bpf_helpers.h>
#if (defined(__TARGET_ARCH_x86) || defined(__TARGET_ARCH_arm64)) && \
#if (defined(__TARGET_ARCH_x86) || defined(__TARGET_ARCH_arm64) || \
(defined(__TARGET_ARCH_riscv) && __riscv_xlen == 64)) && \
defined(__BPF_FEATURE_STACK_ARGUMENT)
int subprog_bad_order_6args(int a, int b, int c, int d, int e, int f)

36
tools/testing/selftests/bpf/progs/compute_live_registers.c
View File

@ -299,7 +299,15 @@ __naked void gotol(void)
"r3 = 24;"
"if r1 > 0x7 goto +2;"
"r0 = r2;"
#ifdef __clang__
"gotol +1;"
#else
/*
* gas mis-parses 'gotol +1' as 'goto l+1', same encoding
* without the sign (binutils PR gas/34558).
*/
"gotol 1;"
#endif
"r0 = r3;"
"exit;"
:
@ -431,6 +439,34 @@ __naked void subprog1(void)
::: __clobber_all);
}
static __used __naked unsigned __int128 aux2(void)
{
asm volatile (
"r0 = 1;"
"r2 = 2;"
"exit;"
::: __clobber_all);
}
SEC("socket")
/* A program observing the pair needs the JIT; see bpf_compute_subprog_ret_regs(). */
__load_if_JITed()
__log_level(2)
__msg("0: .12345.... (85) call pc+2")
__msg("1: ..2....... (bf) r0 = r2")
__msg("2: 0......... (95) exit")
__msg("3: .......... (b7) r0 = 1")
__msg("4: 0......... (b7) r2 = 2")
__msg("5: 0.2....... (95) exit")
__naked void subprog_ret_reg_pair(void)
{
asm volatile (
"call aux2;"
"r0 = r2;"
"exit;"
::: __clobber_all);
}
#if defined(__TARGET_ARCH_x86) || defined(__TARGET_ARCH_arm64)
SEC("socket")

2
tools/testing/selftests/bpf/progs/exceptions_fail.c
View File

@ -60,7 +60,7 @@ __noinline int exception_cb_ok_arg_small(int a)
SEC("?tc")
__exception_cb(exception_cb_bad_ret_type1)
__failure __msg("Global function exception_cb_bad_ret_type1() return value not void or scalar.")
__failure __msg("Only void, scalar, arena pointer, or a struct/union of scalars and arena pointers up to 16 bytes is supported.")
int reject_exception_cb_type_1(struct __sk_buff *ctx)
{
bpf_throw(0);

12
tools/testing/selftests/bpf/progs/freplace_ret_pair.c Normal file
View File

@ -0,0 +1,12 @@
// SPDX-License-Identifier: GPL-2.0
/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
#include <linux/bpf.h>
#include <bpf/bpf_helpers.h>
SEC("freplace/agg_ret_target_func")
__u64 new_agg_ret_target_func(void)
{
return 0;
}
char _license[] SEC("license") = "GPL";

502
tools/testing/selftests/bpf/progs/kasan.c Normal file
View File

@ -0,0 +1,502 @@
// SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause
#include <stdbool.h>
#include <linux/bpf.h>
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_tracing.h>
#include "bpf_misc.h"
extern void bpf_kfunc_kasan_poison(void *mem, __u32 mem__sz) __ksym;
extern void bpf_kfunc_kasan_unpoison(void *mem, __u32 mem__sz) __ksym;
struct bpf_testmod_oob {
__u8 data;
union {
__u8 redzone_1;
__u16 redzone_2;
__u32 redzone_4;
__u64 redzone_8;
};
};
extern struct bpf_testmod_oob *bpf_testmod_oob_alloc(void) __ksym;
extern void bpf_testmod_oob_free(struct bpf_testmod_oob *oob) __ksym;
int access_size;
struct kasan_test_val {
__u8 data_1;
__u16 data_2;
__u32 data_4;
__u64 data_8;
};
struct {
__uint(type, BPF_MAP_TYPE_ARRAY);
__uint(max_entries, 1);
__type(key, __u32);
__type(value, struct kasan_test_val);
} test_map SEC(".maps");
/*
* ST instructions are only emitted if the BPF cpu supports it (eg cpuv4),
* they are otherwise turned into MOV + STX, so compile and exercise ST
* only if supported.
*/
#ifdef __BPF_FEATURE_ST
SEC("tcx/ingress")
int st_on_stack(struct __sk_buff *skb)
{
struct kasan_test_val val;
bpf_kfunc_kasan_poison(&val, sizeof(struct kasan_test_val));
switch (access_size) {
case 1:
val.data_1 = 0xAA;
break;
case 2:
val.data_2 = 0xAA;
break;
case 4:
val.data_4 = 0xAA;
break;
case 8:
val.data_8 = 0xAA;
break;
}
bpf_kfunc_kasan_unpoison(&val, sizeof(struct kasan_test_val));
return 0;
}
SEC("tcx/ingress")
int st_not_on_stack(struct __sk_buff *skb)
{
struct kasan_test_val *val;
__u32 key = 0;
val = bpf_map_lookup_elem(&test_map, &key);
if (!val)
return 0;
bpf_kfunc_kasan_poison(val, sizeof(struct kasan_test_val));
switch (access_size) {
case 1:
val->data_1 = 0xAA;
break;
case 2:
val->data_2 = 0xAA;
break;
case 4:
val->data_4 = 0xAA;
break;
case 8:
val->data_8 = 0xAA;
break;
}
bpf_kfunc_kasan_unpoison(val, sizeof(struct kasan_test_val));
return 0;
}
bool skip_st_tests SEC(".data") = 0;
#else
bool skip_st_tests SEC(".data") = 1;
#endif
SEC("tcx/ingress")
int stx_on_stack(struct __sk_buff *skb)
{
struct kasan_test_val val;
bpf_kfunc_kasan_poison(&val, sizeof(struct kasan_test_val));
/*
* Unlike the st() programs above, the stored value comes from a
* runtime source (skb->len), so it cannot be constant-folded and
* clang always emits a genuine BPF_STX (register store) regardless
* of the target cpu version.
*/
switch (access_size) {
case 1:
val.data_1 = (__u8)skb->len;
break;
case 2:
val.data_2 = (__u16)skb->len;
break;
case 4:
val.data_4 = (__u32)skb->len;
break;
case 8:
val.data_8 = (__u64)skb->len;
break;
}
bpf_kfunc_kasan_unpoison(&val, sizeof(struct kasan_test_val));
return 0;
}
SEC("tcx/ingress")
int stx_not_on_stack(struct __sk_buff *skb)
{
struct kasan_test_val *val;
__u32 key = 0;
val = bpf_map_lookup_elem(&test_map, &key);
if (!val)
return 0;
bpf_kfunc_kasan_poison(val, sizeof(struct kasan_test_val));
switch (access_size) {
case 1:
val->data_1 = (__u8)skb->len;
break;
case 2:
val->data_2 = (__u16)skb->len;
break;
case 4:
val->data_4 = (__u32)skb->len;
break;
case 8:
val->data_8 = (__u64)skb->len;
break;
}
bpf_kfunc_kasan_unpoison(val, sizeof(struct kasan_test_val));
return 0;
}
SEC("tcx/ingress")
int ldx_on_stack(struct __sk_buff *skb)
{
struct kasan_test_val val;
bpf_kfunc_kasan_poison(&val, sizeof(struct kasan_test_val));
switch (access_size) {
case 1:
__sink(val.data_1);
break;
case 2:
__sink(val.data_2);
break;
case 4:
__sink(val.data_4);
break;
case 8:
__sink(val.data_8);
break;
}
bpf_kfunc_kasan_unpoison(&val, sizeof(struct kasan_test_val));
return 0;
}
SEC("tcx/ingress")
int ldx_not_on_stack(struct __sk_buff *skb)
{
struct kasan_test_val *val;
__u32 key = 0;
val = bpf_map_lookup_elem(&test_map, &key);
if (!val)
return 0;
bpf_kfunc_kasan_poison(val, sizeof(struct kasan_test_val));
switch (access_size) {
case 1:
__sink(val->data_1);
break;
case 2:
__sink(val->data_2);
break;
case 4:
__sink(val->data_4);
break;
case 8:
__sink(val->data_8);
break;
}
bpf_kfunc_kasan_unpoison(val, sizeof(struct kasan_test_val));
return 0;
}
SEC("tcx/ingress")
int ldx_patched_not_on_stack(struct __sk_buff *skb)
{
struct kasan_test_val *val;
__u32 key = 0;
val = bpf_map_lookup_elem(&test_map, &key);
if (!val)
return 0;
bpf_kfunc_kasan_poison(val, sizeof(struct kasan_test_val));
__sink(val->data_4);
bpf_kfunc_kasan_unpoison(val, sizeof(struct kasan_test_val));
return 0;
}
SEC("tcx/ingress")
int ldx_patched_on_stack(struct __sk_buff *skb)
{
struct kasan_test_val val;
bpf_kfunc_kasan_poison(&val, sizeof(struct kasan_test_val));
__sink(val.data_4);
bpf_kfunc_kasan_unpoison(&val, sizeof(struct kasan_test_val));
return 0;
}
SEC("tcx/ingress")
int simple_atomic_on_stack(struct __sk_buff *skb)
{
struct kasan_test_val val;
bpf_kfunc_kasan_poison(&val, sizeof(struct kasan_test_val));
switch (access_size) {
case 4:
__sync_fetch_and_add(&val.data_4, 4);
break;
case 8:
__sync_fetch_and_add(&val.data_8, 8);
break;
}
bpf_kfunc_kasan_unpoison(&val, sizeof(struct kasan_test_val));
return 0;
}
SEC("tcx/ingress")
int simple_atomic_not_on_stack(struct __sk_buff *skb)
{
struct kasan_test_val *val;
__u32 key = 0;
val = bpf_map_lookup_elem(&test_map, &key);
if (!val)
return 0;
bpf_kfunc_kasan_poison(val, sizeof(struct kasan_test_val));
switch (access_size) {
case 4:
__sync_fetch_and_add(&val->data_4, 4);
break;
case 8:
__sync_fetch_and_add(&val->data_8, 8);
break;
}
bpf_kfunc_kasan_unpoison(val, sizeof(struct kasan_test_val));
return 0;
}
SEC("tcx/ingress")
int simple_atomic_fetch_on_stack(struct __sk_buff *skb)
{
struct kasan_test_val val;
bpf_kfunc_kasan_poison(&val, sizeof(struct kasan_test_val));
__sync_fetch_and_or(&val.data_8, 8);
bpf_kfunc_kasan_unpoison(&val, sizeof(struct kasan_test_val));
return 0;
}
SEC("tcx/ingress")
int simple_atomic_fetch_not_on_stack(struct __sk_buff *skb)
{
struct kasan_test_val *val;
__u32 key = 0;
val = bpf_map_lookup_elem(&test_map, &key);
if (!val)
return 0;
bpf_kfunc_kasan_poison(val, sizeof(struct kasan_test_val));
__sync_fetch_and_or(&val->data_8, 8);
bpf_kfunc_kasan_unpoison(val, sizeof(struct kasan_test_val));
return 0;
}
#ifdef __BPF_FEATURE_LOAD_ACQ_STORE_REL
bool skip_load_acq_store_rel_tests SEC(".data") = 0;
SEC("tcx/ingress")
int load_acquire_on_stack(struct __sk_buff *skb)
{
struct kasan_test_val val;
bpf_kfunc_kasan_poison(&val, sizeof(struct kasan_test_val));
switch (access_size) {
case 1:
__atomic_load_n(&val.data_1, __ATOMIC_ACQUIRE);
break;
case 2:
__atomic_load_n(&val.data_2, __ATOMIC_ACQUIRE);
break;
case 4:
__atomic_load_n(&val.data_4, __ATOMIC_ACQUIRE);
break;
case 8:
__atomic_load_n(&val.data_8, __ATOMIC_ACQUIRE);
break;
}
bpf_kfunc_kasan_unpoison(&val, sizeof(struct kasan_test_val));
return 0;
}
SEC("tcx/ingress")
int load_acquire_not_on_stack(struct __sk_buff *skb)
{
struct kasan_test_val *val;
__u32 key = 0;
val = bpf_map_lookup_elem(&test_map, &key);
if (!val)
return 0;
bpf_kfunc_kasan_poison(val, sizeof(struct kasan_test_val));
switch (access_size) {
case 1:
__atomic_load_n(&val->data_1, __ATOMIC_ACQUIRE);
break;
case 2:
__atomic_load_n(&val->data_2, __ATOMIC_ACQUIRE);
break;
case 4:
__atomic_load_n(&val->data_4, __ATOMIC_ACQUIRE);
break;
case 8:
__atomic_load_n(&val->data_8, __ATOMIC_ACQUIRE);
break;
}
bpf_kfunc_kasan_unpoison(val, sizeof(struct kasan_test_val));
return 0;
}
SEC("tcx/ingress")
int store_release_on_stack(struct __sk_buff *skb)
{
struct kasan_test_val val;
bpf_kfunc_kasan_poison(&val, sizeof(struct kasan_test_val));
switch (access_size) {
case 1:
__atomic_store_n(&val.data_1, 0xAA, __ATOMIC_RELEASE);
break;
case 2:
__atomic_store_n(&val.data_2, 0xBBBB, __ATOMIC_RELEASE);
break;
case 4:
__atomic_store_n(&val.data_4, 0xCCCCCCCC, __ATOMIC_RELEASE);
break;
case 8:
__atomic_store_n(&val.data_8, 0xDDDDDDDDDDDDDDDD,
__ATOMIC_RELEASE);
break;
}
bpf_kfunc_kasan_unpoison(&val, sizeof(struct kasan_test_val));
return 0;
}
SEC("tcx/ingress")
int store_release_not_on_stack(struct __sk_buff *skb)
{
struct kasan_test_val *val;
__u32 key = 0;
val = bpf_map_lookup_elem(&test_map, &key);
if (!val)
return 0;
bpf_kfunc_kasan_poison(val, sizeof(struct kasan_test_val));
switch (access_size) {
case 1:
__atomic_store_n(&val->data_1, 0xAA, __ATOMIC_RELEASE);
break;
case 2:
__atomic_store_n(&val->data_2, 0xBBBB, __ATOMIC_RELEASE);
break;
case 4:
__atomic_store_n(&val->data_4, 0xCCCCCCCC, __ATOMIC_RELEASE);
break;
case 8:
__atomic_store_n(&val->data_8, 0xDDDDDDDDDDDDDDDD,
__ATOMIC_RELEASE);
break;
}
bpf_kfunc_kasan_unpoison(val, sizeof(struct kasan_test_val));
return 0;
}
#else
bool skip_load_acq_store_rel_tests SEC(".data") = 1;
#endif
SEC("tcx/ingress")
int verifier_paths_stack_and_non_stack(struct __sk_buff *skb)
{
struct kasan_test_val stack_val = {};
struct kasan_test_val *val;
void *ptr;
__u32 key = 0;
val = bpf_map_lookup_elem(&test_map, &key);
if (!val)
return 0;
if (access_size)
ptr = val;
else
ptr = &stack_val;
bpf_kfunc_kasan_poison(val, sizeof(*val));
*(__u8 *)ptr = 0xAA;
bpf_kfunc_kasan_unpoison(val, sizeof(*val));
return 0;
}
SEC("tcx/ingress")
int ldx_oob(struct __sk_buff *skb)
{
struct bpf_testmod_oob *val;
struct kasan_test_val volatile tmp;
val = bpf_testmod_oob_alloc();
if (!val)
return 0;
switch (access_size) {
case 1:
tmp.data_1 = (__u8)val->redzone_1;
break;
case 2:
tmp.data_2 = (__u16)val->redzone_2;
break;
case 4:
tmp.data_4 = (__u32)val->redzone_4;
break;
case 8:
tmp.data_8 = (__u64)val->redzone_8;
break;
}
bpf_testmod_oob_free(val);
return tmp.data_1;
}
SEC("tcx/ingress")
int ldx_self_alias_on_stack(struct __sk_buff *skb)
{
struct kasan_test_val val;
__u64 addr;
bpf_kfunc_kasan_poison(&val, sizeof(val));
/*
* Check that a stack access with dst_reg == src_reg is correctly
* flagged as stack-only access
*/
addr = (__u64)&val;
asm volatile(
"r1 = %0\n"
"r1 = *(u64 *)(r1 + 0)\n"
:
: "r"(addr)
: "r1", "memory");
bpf_kfunc_kasan_unpoison(&val, sizeof(val));
return 0;
}
char LICENSE[] SEC("license") = "GPL";

52
tools/testing/selftests/bpf/progs/kasan_harden.c Normal file
View File

@ -0,0 +1,52 @@
// SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause
#include <stdbool.h>
#include <linux/bpf.h>
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_tracing.h>
extern void bpf_kfunc_kasan_poison(void *mem, __u32 mem__sz) __ksym;
extern void bpf_kfunc_kasan_unpoison(void *mem, __u32 mem__sz) __ksym;
struct kasan_test_val {
__u8 data_1;
__u16 data_2;
__u32 data_4;
__u64 data_8;
};
struct {
__uint(type, BPF_MAP_TYPE_ARRAY);
__uint(max_entries, 1);
__type(key, __u32);
__type(value, struct kasan_test_val);
} test_map SEC(".maps");
/*
* ST instructions are only emitted if the BPF cpu supports it (eg cpuv4),
* they are otherwise turned into MOV + STX, so compile and exercise ST
* only if supported.
*/
#ifdef __BPF_FEATURE_ST
SEC("tcx/ingress")
int st_blinded(struct __sk_buff *skb)
{
struct kasan_test_val *val;
__u32 key = 0;
val = bpf_map_lookup_elem(&test_map, &key);
if (!val)
return 0;
bpf_kfunc_kasan_poison(val, sizeof(struct kasan_test_val));
val->data_1 = 0xAA;
bpf_kfunc_kasan_unpoison(val, sizeof(struct kasan_test_val));
return 0;
}
bool skip_st_tests SEC(".data") = 0;
#else
bool skip_st_tests SEC(".data") = 1;
#endif
char LICENSE[] SEC("license") = "GPL";

32
tools/testing/selftests/bpf/progs/lirc_mode2.c Normal file
View File

@ -0,0 +1,32 @@
// SPDX-License-Identifier: GPL-2.0
// test ir decoder
//
// Copyright (C) 2018 Sean Young <sean@mess.org>
#include <linux/bpf.h>
#include <linux/lirc.h>
#include <bpf/bpf_helpers.h>
SEC("lirc_mode2")
int bpf_decoder(unsigned int *sample)
{
if (LIRC_IS_PULSE(*sample)) {
unsigned int duration = LIRC_VALUE(*sample);
/*
* Flag bits picked deliberately low: rc-loopback simulates
* a receiver overflow for any pulse over MS_TO_US(50) (see
* loop_tx_ir() in rc-loopback.c), which would silently
* swallow the sample before it ever reaches this decoder.
*/
if (duration & 0x8000)
bpf_rc_keydown(sample, 0x40, duration & 0x3fff, 0);
if (duration & 0x4000)
bpf_rc_pointer_rel(sample, (duration >> 7) & 0x7f,
duration & 0x7f);
}
return 0;
}
char _license[] SEC("license") = "GPL";

3
tools/testing/selftests/bpf/progs/stack_arg.c
View File

@ -21,7 +21,8 @@ struct {
int timer_result;
#if (defined(__TARGET_ARCH_x86) || defined(__TARGET_ARCH_arm64)) && \
#if (defined(__TARGET_ARCH_x86) || defined(__TARGET_ARCH_arm64) || \
(defined(__TARGET_ARCH_riscv) && __riscv_xlen == 64)) && \
defined(__BPF_FEATURE_STACK_ARGUMENT)
const volatile bool has_stack_arg = true;

3
tools/testing/selftests/bpf/progs/stack_arg_fail.c
View File

@ -103,8 +103,9 @@ __naked void r11_store_zero_off(void)
SEC("tc")
__description("stack_arg_fail: not supported, dummy test")
__skip("stack_arg_fail: not supported")
__success
int test_stack_arg_big(struct __sk_buff *skb)
int dummy_test(struct __sk_buff *skb)
{
return 0;
}

3
tools/testing/selftests/bpf/progs/stack_arg_kfunc.c
View File

@ -6,7 +6,8 @@
#include "bpf_kfuncs.h"
#include "../test_kmods/bpf_testmod_kfunc.h"
#if (defined(__TARGET_ARCH_x86) || defined(__TARGET_ARCH_arm64)) && \
#if (defined(__TARGET_ARCH_x86) || defined(__TARGET_ARCH_arm64) || \
(defined(__TARGET_ARCH_riscv) && __riscv_xlen == 64)) && \
defined(__BPF_FEATURE_STACK_ARGUMENT)
const volatile bool has_stack_arg = true;

Some files were not shown because too many files have changed in this diff Show More