The KHO radix tree constants are somewhat hard to understand. The tree
depth essentially comes from the max key width. The max key width comes
from the need to store a 52-bit PFN plus one more bit for the order.
All this is very obscure with the corrent code. The PFN width is defined
as KHO_ORDER_0_LOG2, which makes very little sense to a new reader not
already familiar with what the value means. Then the fact that an extra
bit is needed is hidden in the KHO_TREE_MAX_DEPTH calculation.
Simplify this by removing KHO_ORDER_0_LOG2 and replace it with
KHO_RADIX_KEY_WIDTH. Update the comment to explain why this value is
used. This moves the +1 from KHO_TREE_MAX_DEPTH to KHO_RADIX_KEY_WIDTH,
making things clearer.
Update kho_{encode,decode}_radix_key() to not use KHO_ORDER_0_LOG2.
Instead, refactor the code and comments to make it clearer how the
encoding and decoding is done.
In kho_encode_radix_key(), add a new variable for the shift for physical
address. Use that in calculating where the order bit goes and in
calculating the shifted PFN. Update comments to make this clearer.
In kho_radix_decode_key(), turn order_bit to 0-indexed to simplify the
eventual calculation for order. Touch up comments to make the
computation clearer.
Signed-off-by: Pratyush Yadav (Google) <pratyush@kernel.org>
Link: https://patch.msgid.link/20260801084833.1897543-3-pratyush@kernel.org
Signed-off-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
The KHO radix tree is a data structure that can track the presence or
absence of an arbitrary key, with nothing inherently tied to KHO memory
preservation tracking. This was one of the design goals of the radix
tree. This was done to enable it to be re-used by other users of KHO.
Despite that, the radix tree APIs are very closely tied to KHO memory
preservation tracking. Adding a key is done by kho_radix_add_page(),
which encodes it as a page tracking operation and takes in PFN and
order. kho_radix_del_page() does the same. These functions encode the
key internally that goes into the radix tree. kho_radix_walk_tree() does
the same by baking the PFN and order into the callback arguments.
Generalize the APIs by taking the key directly and doing the encoding at
the callers. Rename the functions to kho_radix_add_key() and
kho_radix_del_key(). In practice, this removes a line each from the
functions and moves the encoding function call to the callers.
Similarly, update kho_radix_tree_walk_callback_t to take the key
directly.
Now that key encoding is no longer an inherent part of the radix tree
and can be decided by the user, rename kho_radix_{encode,decode}_key()
to kho_{encode,decode}_radix_key(). This moves them out of the
"kho_radix_" name space into the "kho_" namespace. This emphasizes that
this is KHO's way of encoding the key for its radix tree.
Reviewed-by: Pasha Tatashin <pasha.tatashin@soleen.com>
Signed-off-by: Pratyush Yadav (Google) <pratyush@kernel.org>
Link: https://patch.msgid.link/20260801084833.1897543-2-pratyush@kernel.org
Signed-off-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
* fix a regression caused by allowing coexistence of KHO with deferred
initialization of the memory map.
-----BEGIN PGP SIGNATURE-----
iQEzBAABCgAdFiEEeOVYVaWZL5900a/pOQOGJssO/ZEFAmpweGsACgkQOQOGJssO
/ZHe4gf/b06J+YW9GO7FGQclCf3s30DbL8nNeEkb74lpWU8CAfk2vbo9QyrlYH8+
5wtISj9+SdBwMfY4XFdDWOur1EHD47N52xbT5zL3yp1UEPpXYbggiiKuzwCXK1MZ
lVhyc3XfPmF4eRP5V0Nr7AsN1pAujH83uRrTgg//ZJziOaOtLi7ZWJZa0Oae3UY1
z+/bRIM21DzkL2wD2fitlYxYBmrooYRS6FVr+wv/khBnLxAtcWGM+nfHGI4/i4bK
5oz/9D3+nD/rQBqOT6F+Vf6Dz3zPtJPLe7+B459x24f2yPzJplIt6cDq9nxZKbY2
L7q3npvcsts7iLXVsKSeaHeEfn2l7g==
=xzQP
-----END PGP SIGNATURE-----
Merge tag 'liveupdate-fixes-2026-08-03' of git://git.kernel.org/pub/scm/linux/kernel/git/liveupdate/linux
Pull liveupdate fix from Mike Rapoport:
- fix a regression caused by allowing coexistence of KHO with deferred
initialization of the memory map
* tag 'liveupdate-fixes-2026-08-03' of git://git.kernel.org/pub/scm/linux/kernel/git/liveupdate/linux:
kho: align kho_scratch to MAX_ORDER_NR_PAGES pages
The built-in idle masks are reset with all online CPUs marked idle
before sched_ext is enabled. Busy CPUs can therefore be incorrectly
advertised as idle until their next idle transition.
Initialize the masks empty so that the initial state is conservative.
When bypass is lifted, every CPU is rescheduled and idle-to-idle
re-picks populate the masks with CPUs that are actually idle. Later
idle transitions keep the masks up to date.
Suggested-by: Tejun Heo <tj@kernel.org>
Signed-off-by: Andrea Righi <arighi@nvidia.com>
Reviewed-by: Kuba Piecuch <jpiecuch@google.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
- More lifecycle fixes for the new sub-scheduler support: a failed enable
could tear down a never-linked sub-scheduler in a way that races the root
scheduler's disable and leads to a use-after-free, tasks that were not on
the ext class could still get the enable callback, and a policy-rejection
path silently rewrote a running task's scheduling policy instead of
aborting the scheduler.
- Scheduler enable/disable could deadlock with cgroup removal and a
concurrent cgroup weight write through kernfs. Fixed by reordering lock
acquisition.
- Sync wakeups could leave the waker CPU incorrectly marked idle in the
built-in idle-CPU tracking.
- A selftest fix for sleeping tasks whose CPU affinity changes before
wakeup.
-----BEGIN PGP SIGNATURE-----
iIQEABYKACwWIQTfIjM1kS57o3GsC/uxYfJx3gVYGQUCam/mog4cdGpAa2VybmVs
Lm9yZwAKCRCxYfJx3gVYGe0NAQC2UrrXeSm97RTCv+5HBfom/8sjPdB2tiAwGiUt
fUUQ+AD9FP30urJaoiALUiL+EGWpLFejwUCXNXlT+0kN2TiFxQM=
=EuWf
-----END PGP SIGNATURE-----
Merge tag 'sched_ext-for-7.2-rc6-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/sched_ext
Pull sched_ext fixes from Tejun Heo:
- More lifecycle fixes for the new sub-scheduler support: a failed
enable could tear down a never-linked sub-scheduler in a way that
races the root scheduler's disable and leads to a use-after-free,
tasks that were not on the ext class could still get the enable
callback, and a policy-rejection path silently rewrote a running
task's scheduling policy instead of aborting the scheduler.
- Scheduler enable/disable could deadlock with cgroup removal and a
concurrent cgroup weight write through kernfs. Fixed by reordering
lock acquisition.
- Sync wakeups could leave the waker CPU incorrectly marked idle in the
built-in idle-CPU tracking.
- A selftest fix for sleeping tasks whose CPU affinity changes before
wakeup.
* tag 'sched_ext-for-7.2-rc6-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/sched_ext:
selftests/sched_ext: Handle sleeping task affinity changes in numa test
sched_ext: Mark waker CPU busy when selected in WAKE_SYNC case
sched_ext: Don't enable non-ext tasks in the sub-sched task loops
sched_ext: Skip sub-disable teardown for never-linked sub-schedulers
sched_ext: Take cgroup_lock() first in scx_cgroup_lock()
sched_ext: Reject setting disallow from init_task outside the enable path
- A pressure trigger's poll timer could be re-armed while the last trigger
was being torn down and then fire after the cgroup was freed. Tie the
timer to the cgroup's lifetime and shut it down when the cgroup is freed.
- Writing to a pressure file forked a worker kthread while holding the
cgroup mutex, creating lock dependencies from the mutex to the whole fork
path. A pressure write racing a sched_ext scheduler enable, which blocks
forks before grabbing the mutex, deadlocked. Fork the worker with the
mutex dropped.
- Documentation fix for io.latency behavior on non-rotational devices.
-----BEGIN PGP SIGNATURE-----
iIQEABYKACwWIQTfIjM1kS57o3GsC/uxYfJx3gVYGQUCam/mlw4cdGpAa2VybmVs
Lm9yZwAKCRCxYfJx3gVYGVhRAQCyNBHCHpaY9erKTezenGDK6D+LqbfLWiWuIntB
swiwLQEA6h6Rgob2GDDRHOey7+XwF6PHh6xoh4FhSYboeEZd1wc=
=Re+7
-----END PGP SIGNATURE-----
Merge tag 'cgroup-for-7.2-rc6-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/cgroup
Pull cgroup fixes from Tejun Heo:
- A pressure trigger's poll timer could be re-armed while the last
trigger was being torn down and then fire after the cgroup was freed.
Tie the timer to the cgroup's lifetime and shut it down when the
cgroup is freed.
- Writing to a pressure file forked a worker kthread while holding the
cgroup mutex, creating lock dependencies from the mutex to the whole
fork path. A pressure write racing a sched_ext scheduler enable,
which blocks forks before grabbing the mutex, deadlocked.
Fork the worker with the mutex dropped.
- Documentation fix for io.latency behavior on non-rotational devices.
* tag 'cgroup-for-7.2-rc6-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/cgroup:
Docs/admin-guide/cgroup-v2: document io.latency rotational vs non-rotational behavior
sched/psi: Shut down rtpoll_timer in psi_cgroup_free()
sched/psi: Create the psimon kthread outside of cgroup_mutex
Commit 814cba835e ("bpf, x86: Fix trampoline stack size for 128-bit
arguments") changed the x86 trampoline to compute the number of
registers from arg_size for every argument, which removed the last user
of BTF_FMODEL_STRUCT_ARG. No other architecture or verifier code looks
at the flag, so remove the macro and the code in __get_type_fmodel_flags()
which sets it.
Keep BTF_FMODEL_SIGNED_ARG at BIT(1) rather than renumbering it to
BIT(0), so BIT(0) is available for a future flag.
No functional change.
Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Acked-by: Leon Hwang <leon.hwang@linux.dev>
Acked-by: Jiri Olsa <jolsa@kernel.org>
Link: https://lore.kernel.org/bpf/20260803052726.2821447-1-yonghong.song@linux.dev
uretprobes_srcu currently uses normal SRCU, which issues
two smp_mb() per read lock/unlock pair. This overhead is
paid on every uretprobe hit.
Switch to SRCU-fast-updown, which eliminates the per-reader
memory barriers by moving the ordering cost to the
grace-period side (synchronize_rcu() instead of smp_mb()).
This is acceptable because grace periods (uprobe
unregistration) are infrequent compared to reader-side
uretprobe hits.
The updown flavor is required because the SRCU read lock is
taken in prepare_uretprobe() when a return instance is
created and is held until that return instance is finalized.
The traced thread returns to user space in between, so the
lock is inherently released in a different context from
where it was acquired: on the normal return path via
uprobe_handle_trampoline() -> hprobe_finalize(), or from
ri_timer() (expiry) or dup_utask() (fork) via
hprobe_expire(). srcu_down_read_fast() / srcu_up_read_fast()
are designed for this acquire-here / release-elsewhere
pattern and, unlike the same-context srcu_read_lock_fast()
variant, do not carry the lockdep read-side tracking that
would warn on it.
The short, same-context SRCU sections in ri_timer() and
dup_utask() (which guard the uprobe against reuse across the
hprobe_expire() cmpxchg) instead use guard(srcu_fast_updown)
for proper lockdep coverage.
Signed-off-by: Puranjay Mohan <puranjay@kernel.org>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Oleg Nesterov <oleg@redhat.com>
Acked-by: Andrii Nakryiko <andrii@kernel.org>
Link: https://patch.msgid.link/20260706172744.3920417-3-puranjay@kernel.org
Once binfmt_misc is loaded load_misc_binary() runs for every execve()
on the system since binfmt_misc registers at the head of the formats
list. Every exec therefore performs read_lock() and read_unlock() on
the entries_lock of the relevant binfmt_misc instance, i.e., two
atomic read-modify-writes on a shared cacheline. User namespaces
without their own binfmt_misc mount fall back to an ancestor's
instance so on container-heavy systems every exec on the machine
typically ends up hammering the cacheline of init_binfmt_misc. On
PREEMPT_RT the rwlock additionally turns the handler lookup into a
sleeping lock on the exec fast path.
The lock protects very little. Entries are immutable after publication
except for the Enabled bit which is already toggled locklessly via
set_bit()/clear_bit() and entry lifetime is already handled by the
users refcount via get_binfmt_handler()/put_binfmt_handler(). The read
lock's only remaining job is to make "the entry is still linked" and
"take a reference" atomic with respect to the unlink sites.
Switch the lookup to an RCU walk:
* Lookup walks the entry list under rcu_read_lock() and acquires a
reference via refcount_inc_not_zero(). The refcount can only drop to
zero after an entry has been unlinked so a failed increment means
the walk raced with an unlink. Restarting the search is bounded
because an unlinked entry cannot be found again.
* The unlink sites use hlist_del_init_rcu() which keeps the forward
pointer intact for concurrent walkers and preserves hlist_unhashed()
as the protection against double removal.
* The final put frees the entry via kfree_rcu() as a concurrent walker
may still dereference its flags, magic, mask, and inline strings.
They all live in the entry allocation itself and thus stay valid
until a grace period has elapsed. Closing the interpreter file stays
synchronous. It is only used with a reference already held and all
final puts run in process context.
* Writers remain serialized by the inode lock of the root dentry with
one exception. bm_evict_inode() called from generic_shutdown_super()
during umount unlinks entries without holding it. Keep a spinlock
around the unlink sites instead of relying on superblock lifetime
rules to make that exclusion implicit.
Handler removal semantics are unchanged. An exec that acquired a
reference just before its handler was unregistered already completes
with the removed handler today. The read lock never protected against
that, it only made the window smaller.
With this an exec that matches no binfmt_misc entry, the common case,
no longer writes to any shared cacheline at all.
Link: https://patch.msgid.link/20260710-work-binfmt_misc-locking-v3-5-a162f7cb58d6@kernel.org
Reviewed-by: Jori Koolstra <jkoolstra@xs4all.nl>
Signed-off-by: Christian Brauner (Amutable) <brauner@kernel.org>
The upcoming conversion of the handler lookup to RCU walks cannot use
list_del_init(): reinitializing the forward pointer of a removed entry
would make a concurrent lockless walker standing on that entry loop
back onto it indefinitely. The removal paths do rely on
reinitialization though because bm_{entry,status}_write() and
bm_evict_inode() need to detect whether an entry has already been
unlinked.
hlists support exactly this pattern: hlist_del_init_rcu() keeps the
forward pointer of the removed entry intact for concurrent walkers and
only zeroes ->pprev with hlist_unhashed() serving as the linked test.
Convert the entry list to an hlist now while keeping the rwlock so the
subsequent RCU conversion is a pure locking change. hlist_add_head()
inserts at the head just as list_add() did so lookup precedence
between registered handlers is unchanged.
Link: https://patch.msgid.link/20260710-work-binfmt_misc-locking-v3-4-a162f7cb58d6@kernel.org
Reviewed-by: Jori Koolstra <jkoolstra@xs4all.nl>
Signed-off-by: Christian Brauner (Amutable) <brauner@kernel.org>
Reported by the Sashiko AI review.
arena_vm_fault() returns VM_FAULT_RETRY when it can't take
arena->spinlock, but it never took mmap_lock. The fault path assumes a
VM_FAULT_RETRY handler already dropped mmap_lock and re-takes it on the
retry, so mmap_lock gets taken twice and can deadlock:
do_user_addr_fault()
{
fault = handle_mm_fault(...); // calls arena_vm_fault()
if (fault & VM_FAULT_RETRY)
goto retry; // re-locks mmap_lock
mmap_read_unlock(mm);
}
Return VM_FAULT_SIGBUS instead, for two reasons:
1. We could keep VM_FAULT_RETRY, but then we'd have to drop the fault
lock first and cap the retry ourselves, the way __folio_lock_or_retry()
does.
2. A failed raw_res_spin_lock_irqsave() already means a possible deadlock
was detected, so retrying just hits the same lock again.
So returning VM_FAULT_RETRY here is overkill.
Fixes: b8467290ed ("bpf: arena: make arena kfuncs any context safe")
Signed-off-by: Jiayuan Chen <jiayuan.chen@linux.dev>
Reviewed-by: Emil Tsalapatis <emil@etsalapatis.com>
Link: https://lore.kernel.org/bpf/20260728060517.95183-1-jiayuan.chen@linux.dev
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Kfunc argument checking re-derives each argument's kfunc_ptr_arg_type
from BTF on every verification of a call in check_kfunc_args(). Now that
get_kfunc_arg_type() is a function of the kfunc's BTF alone, it no
longer inspects register state. The classification can be computed once
when the call is added and cached. This is a step toward describing
kfuncs with a bpf_func_proto and sharing the helper argument-checking
path.
Generate the classification at bpf_add_kfunc_call() time:
- Extend struct bpf_func_proto to be able to describe a kfunc: widen
arg_type[] and the arg_btf_id[]/arg_size[] union from 5 to
MAX_BPF_FUNC_ARGS, since a kfunc may take up to 12 arguments (5 in
registers, 7 on the stack).
- Embed a bpf_func_proto in struct bpf_kfunc_desc, populated by
gen_kfunc_arg_proto() which runs get_kfunc_arg_type() for each
argument and stores the result in proto.arg_type[]. Grow the
descriptor table's descs[] as a flexible array to not waste memory.
- check_kfunc_args() reads the cached classification from meta->fn
The KF_ARG_PTR_TO_CTX classification depends on the resolved program type,
and for BPF_PROG_TYPE_EXT that is the target program's type, which
resolve_prog_type() reads from prog->aux->saved_dst_prog_type. That field
is normally recorded later during verification in check_attach_btf_id(),
after bpf_add_kfunc_call() has run.
Record saved_dst_prog_type and saved_dst_attach_type from dst_prog at
program load time in bpf_prog_load() so the resolved type is available
at add-call time without reordering check_attach_btf_id(). This keeps
e.g. an freplace of an XDP program calling bpf_xdp_metadata_rx_hash()
classifying its struct xdp_md * argument as context.
The classification result is unchanged; it is only computed earlier and
cached.
Signed-off-by: Amery Hung <ameryhung@gmail.com>
Link: https://lore.kernel.org/bpf/20260801074633.1595644-19-ameryhung@gmail.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Add kfunc scalar argument types, classify them in get_kfunc_arg_type()
along side with pointer arguments and move scalar type verification
into the main switch in check_kfunc_args(). This keeps BTF-based
classification separate from register validation for every argument,
paving the way for generating the kfunc argument prototype at add-call
time. No functional change intended.
KF_ARG_MEM_SIZE and KF_ARG_CONST_MEM_SIZE now are reachable. Therefore,
remove the fallthrough from KF_ARG_PTR_TO_MEM case and adjust the
register indexing.
Signed-off-by: Amery Hung <ameryhung@gmail.com>
Reviewed-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://lore.kernel.org/bpf/20260801074633.1595644-18-ameryhung@gmail.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Now that get_kfunc_ptr_arg_type() classifies a kfunc pointer argument
from its BTF alone, express a nullable argument by OR-ing PTR_MAYBE_NULL
into the classified type, and resolve a NULL register after
classification instead of before it.
Previously check_kfunc_args() short-circuited a nullable argument passed
a NULL register with a continue placed before get_kfunc_ptr_arg_type(),
so the NULL never reached classification. That kept a register-state
decision (bpf_register_is_null()) ahead of the BTF-based classification.
This mirrors how helper arguments carry PTR_MAYBE_NULL in their
bpf_arg_type and is a step toward describing kfuncs with a bpf_func_proto:
the nullability now travels with the per-argument classification, so it is
captured when the prototype is generated at add-call time.
Signed-off-by: Amery Hung <ameryhung@gmail.com>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://lore.kernel.org/bpf/20260801074633.1595644-17-ameryhung@gmail.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
get_kfunc_ptr_arg_type() decided part of a kfunc pointer argument's type
from the caller's register: a PTR_TO_BTF_ID (or reg2btf_ids) register made
the argument KF_ARG_PTR_TO_BTF_ID, otherwise it fell through to a memory
buffer. Folding register state into argument classification prevents
describing a kfunc's arguments from its BTF alone, which is a prerequisite
for generating a helper-like prototype and eventually sharing the argument
checking (check_func_arg()) between helpers and kfuncs.
Classify pointer arguments from BTF only, and resolve them against the
register in check_kfunc_args():
- A pointer to a struct that is not paired with a __sz/__szk size
argument is classified KF_ARG_PTR_TO_BTF_ID and then checked against
the register. A register carrying a BTF ID (PTR_TO_BTF_ID or a
reg2btf_ids type) must be referenced or trusted and is matched against
the expected type. The only relaxation is when the struct is composed
of scalars, the register may be verified as a fixed-size memory buffer
sized from the BTF type; anything else is rejected.
- A pointer paired with a size argument is always a memory buffer and is
never classified as BTF_ID, so the __sz/__szk case no longer detours
through BTF_ID.
The new design now accepts one previously rejected case: passing
PTR_TO_BTF_ID to a pointer to scalar w/o a following __sz/__szk. The
argument will be classified as KF_ARG_PTR_TO_MEM | MEM_FIXED_SIZE. The
PTR_TO_BTF_ID register will go through check_mem_reg() ->
check_helper_mem_access() -> check_ptr_to_btf_access(). For a pointer to
scalar arg, a kernel btf id will be rejected unless explicitly granted
by btf_struct_access(); a program allocated btf id will be allowed.
The referenced-or-trusted check thus moves into the KF_ARG_PTR_TO_BTF_ID
resolution, alongside the type match.
get_kfunc_ptr_arg_type() no longer needs the register, so drop its regs
and reg parameters; it is now a pure function of the kfunc's BTF.
When a register cannot satisfy a BTF_ID argument, report the register type
passed and, when the expected struct has a reg2btf_ids mapping, the
register type that would be accepted, instead of a confusing "socket".
Update the affected selftest messages accordingly.
Signed-off-by: Amery Hung <ameryhung@gmail.com>
Link: https://lore.kernel.org/bpf/20260801074633.1595644-16-ameryhung@gmail.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
A kfunc memory-pointer argument comes in two flavors: a fixed-size buffer
whose access size is derived from the pointed-to BTF type, and a
variable-size buffer paired with a following __sz/__szk size argument.
Both were represented by separate kfunc_ptr_arg_type values
(KF_ARG_PTR_TO_MEM vs KF_ARG_PTR_TO_MEM_SIZE) with the pointer classified
as the latter when a size argument followed.
Mirror how helpers describe the same distinction: classify both as
KF_ARG_PTR_TO_MEM and OR in MEM_FIXED_SIZE for the fixed-size case, just
as helpers use ARG_PTR_TO_MEM | MEM_FIXED_SIZE. The switches now key on
base_type(kf_arg_type) so the flag rides along, and the KF_ARG_PTR_TO_MEM
handler either resolves the size from BTF (MEM_FIXED_SIZE) or falls
through to the mem/size-pair check, which validates the buffer against the
following size register and skips it. No functional change.
Currently, KF_ARG_MEM_SIZE and KF_ARG_CONST_MEM_SIZE are only reachable
from ARG_PTR_TO_MEM fallthrough. A patch later will merge scalar
checking into the same switch and remove the fallthrough.
Signed-off-by: Amery Hung <ameryhung@gmail.com>
Reviewed-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://lore.kernel.org/bpf/20260801074633.1595644-15-ameryhung@gmail.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
get_kfunc_ptr_arg_type() returned KF_ARG_PTR_TO_NULL when a nullable
pointer argument was passed a NULL register. This folded a register-state
decision (bpf_register_is_null()) into what is otherwise BTF-based
argument classification, and it short-circuited before the BTF_ID/MEM
resolution.
Drop KF_ARG_PTR_TO_NULL and handle the NULL case in check_kfunc_args()
instead: a nullable argument that is actually NULL is skipped. Note that
it is okay to skip even when it is a mem+size pair because the size
argument check has been moved to the scalar section. The skip is done
before get_kfunc_ptr_arg_type() so that a NULL passed to a nullable
non-scalar-struct argument is not newly rejected by the BTF_ID/MEM
resolution.
No functional change.
Signed-off-by: Amery Hung <ameryhung@gmail.com>
Reviewed-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://lore.kernel.org/bpf/20260801074633.1595644-14-ameryhung@gmail.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
check_kfunc_args() already makes sure a scalar value is passed to a
scalar kfunc argument. Drop the check in is_kfunc_arg_mem_size() and
is_kfunc_arg_const_mem_size() to further decouple
get_kfunc_ptr_arg_type() from register state (a prerequisite for
generating a helper-like prototype from kfunc's BTF).
Signed-off-by: Amery Hung <ameryhung@gmail.com>
Reviewed-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://lore.kernel.org/bpf/20260801074633.1595644-13-ameryhung@gmail.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
To align helper and kfunc pointer to memory argument handling, move
kfunc constant memorry size argument handling to the kfunc scalar
section. In addition, factor out constant scalar argument handling.
The constant size argument (__szk) of a kfunc memory/size pair was
recorded into meta->arg_constant by a dedicated block in the
KF_ARG_PTR_TO_MEM_SIZE case, duplicating the "only one constant
argument" and "must be a known constant" checks already in the generic
scalar argument handling. That block also did an explicit i++ to skip
the size argument.
This also fixes a precision gap: the old dedicated block did not mark
the size register precise, relying on check_mem_size_reg() for that. But
check_mem_size_reg() is skipped when the buffer is a nullable arg passed
as NULL (e.g. bpf_dynptr_slice(_rdwr) with a NULL buffer), so in that
case the __szk value was recorded and used for regs[R0].mem_size without
marking it precise. Routing the size through the scalar path marks it
precise in all cases.
Signed-off-by: Amery Hung <ameryhung@gmail.com>
Reviewed-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://lore.kernel.org/bpf/20260801074633.1595644-11-ameryhung@gmail.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
ARG_CONST_SIZE does not require a constant: check_mem_size_reg() accepts
any bounded scalar and verifies the memory access against its maximum
(reg_umax). Rename ARG_CONST_SIZE and ARG_CONST_SIZE_OR_ZERO to
ARG_MEM_SIZE and ARG_MEM_SIZE_OR_ZERO to reflect that. ARG_CONST_ALLOC_
SIZE_OR_ZERO, which does require a constant, is left unchanged.
Pure rename, no functional change.
Signed-off-by: Amery Hung <ameryhung@gmail.com>
Link: https://lore.kernel.org/bpf/20260801074633.1595644-10-ameryhung@gmail.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Fixed-size memory arguments went through two paths: helpers called
check_helper_mem_access() directly, while kfuncs and global subprogs
used check_mem_reg(). Route the helper MEM_FIXED_SIZE case through
check_mem_reg() too so all three share the same check.
This also fixes a bug in the helper path. When passing a NULL to
PTR_MAYBE_NULL | ARG_PTR_TO_FIXED_SIZE_MEM argument, the program would
be falsely rejected by check_helper_mem_access(). This is not
triggerable since there is no such kind of helper. Also, note that
check_reg_type() still make sure NULL cannot be passed to an argument
not marked with PTR_MAYBE_NULL.
It also tightens the poisoned-stack-slot check. check_mem_reg() encoded
"a STACK_POISON slot may be read" as a negative access size for any
PTR_TO_STACK argument, but that is only sound for global subprogs, where
static stack liveness proved the callee body does not read those slots
(2cb27158ad ("bpf: poison dead stack slots")). Since check_mem_reg() is
also used for kfuncs, kfuncs accidentally inherited it and could read a
poisoned (dead, possibly uninitialized) stack slot. Restrict the negative
size to global subprogs (meta == NULL) so kfuncs, like helpers, require
the whole argument initialized.
Signed-off-by: Amery Hung <ameryhung@gmail.com>
Link: https://lore.kernel.org/bpf/20260801074633.1595644-9-ameryhung@gmail.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Helper ARG_CONST_SIZE and kfunc KF_ARG_PTR_TO_MEM_SIZE memory arguments
already share check_mem_size_reg(), but the kfunc path reached it
through a thin wrapper, check_kfunc_mem_size_reg(). The wrapper existed
only to invoke check_mem_size_reg() twice. Once for BPF_READ and once for
BPF_WRITE because a kfunc mem argument may be both read and written,
whereas a helper argument carries a single access direction.
Let check_mem_size_reg() take a bitmask of access directions (widening
access_type to u32) and perform each requested access, then pass
BPF_READ | BPF_WRITE from the kfunc call site. This removes the
check_kfunc_mem_size_reg() wrapper so helper and kfunc mem+size arguments
run through exactly the same code.
Signed-off-by: Amery Hung <ameryhung@gmail.com>
Reviewed-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://lore.kernel.org/bpf/20260801074633.1595644-7-ameryhung@gmail.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
bpf_map_lookup_elem() is typed to return PTR_TO_MAP_VALUE for every
map, but for some map kinds the looked up value is actually a
different object: an inner map, a socket or an xsk socket.
Until now this reinterpretation happened once the pointer was
converted from its NULL-able form to a concrete value.
Such reinterpretation logic placement led to mark_ptr_not_null_reg()
being called for a temporary register copy in check_mem_reg() and
check_kfunc_mem_size_reg() (check_mem_size_reg() was buggy because of
not calling it). The temporary copy was necessary to pass
reinterpreted parameters as nullable helper and kfunc arguments.
Avoid this complication by refining map lookup result type right away.
The test case verifier_map_in_map/on_the_inner_map_pointer needs an
update because the verifier now prints a concrete NULL-able type for
the lookup.
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Signed-off-by: Amery Hung <ameryhung@gmail.com>
Link: https://lore.kernel.org/bpf/20260801074633.1595644-6-ameryhung@gmail.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
kfunc now shares the same bpf_call_arg_meta with helpers. Pass kfunc's
own meta to check_mem_reg() and check_kfunc_mem_size() instead of NULL
or a temporary meta on the stack.
Signed-off-by: Amery Hung <ameryhung@gmail.com>
Reviewed-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://lore.kernel.org/bpf/20260801074633.1595644-5-ameryhung@gmail.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Kfuncs used a single '__map' suffix (KF_ARG_PTR_TO_MAP) for two different
things: a verifier-known map matched by map_uid against a bound
timer/wq/task_work object (bpf_wq_init, bpf_task_work_schedule*), and an
opaque 'struct bpf_map *' used only at runtime (bpf_arena_*), which may be a
map fd or a PTR_TO_BTF_ID struct bpf_map (e.g. a bpf_map iterator's ctx->map).
That combined path only accepted the btf map form due to type confusion. The
'if (!reg->map_ptr)' check reads reg->map_ptr, which aliases reg->btf in the
bpf_reg_state union. A PTR_TO_BTF_ID register always has a non-NULL reg->btf,
so the guard silently passed and validation fell through to
process_kf_arg_ptr_to_btf_id(). It also recorded PTR_TO_BTF_ID info in
meta->map, which would be meaningless.
Split the annotation to avoid such type confusion and to align with
helper:
- '__const_map' -> KF_ARG_CONST_MAP_PTR: verifier-known map, handled by
process_map_ptr_arg() like helper ARG_CONST_MAP_PTR.
- '__map' -> KF_ARG_PTR_TO_BTF_ID: opaque struct bpf_map, validated by
process_kf_arg_ptr_to_btf_id(). A map fd still matches via
reg2btf_ids[CONST_PTR_TO_MAP], so bpf_arena_alloc_pages(&map) keeps
working.
Signed-off-by: Amery Hung <ameryhung@gmail.com>
Reviewed-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://lore.kernel.org/bpf/20260801074633.1595644-4-ameryhung@gmail.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Both the helper ARG_CONST_MAP_PTR and the kfunc KF_ARG_PTR_TO_MAP
recorded the map pointer in meta->map and, when a map was already
bound by a preceding timer/workqueue/task_work argument, rejected a
mismatching map.
Factor the logic into a single process_map_ptr_arg() used by both
paths. The bound-object name (timer, workqueue, or bpf_task_work) is
derived from the bound map's btf_record, and the register numbers in
the message are computed from the map argument position instead of
being hard-coded.
Signed-off-by: Amery Hung <ameryhung@gmail.com>
Reviewed-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://lore.kernel.org/bpf/20260801074633.1595644-3-ameryhung@gmail.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Drop process_timer_{helper,kfunc}() since bpf_call_arg_meta is now
shared by helper and kfunc. Call process_timer_func() directly.
Signed-off-by: Amery Hung <ameryhung@gmail.com>
Reviewed-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://lore.kernel.org/bpf/20260801074633.1595644-2-ameryhung@gmail.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
If the SCX_ENABLING -> SCX_ENABLED cmpxchg at the tail of
scx_root_enable_workfn() fails, the function jumps to err_disable
without setting ret. At that point ret still holds the return value
of the last successful __scx_init_task() call, which is 0, so the
err_disable fallback reports the meaningless message:
scx_root_enable() failed (0)
Set ret = -EBUSY, consistent with the other enable-state guards at
the top of the same function, so the fallback always reports a real
errno.
Signed-off-by: Liang Luo <luoliang@kylinos.cn>
Signed-off-by: Tejun Heo <tj@kernel.org>
The kernel-doc comment for sched_ext_ops::sub_cgroup_id uses the old
@cgroup_id name, which no longer matches the struct member. This
produces two kernel-doc warnings:
Warning: struct member sub_cgroup_id not described in sched_ext_ops
Warning: Excess struct member cgroup_id description in sched_ext_ops
Update the @param name to match the actual member.
Signed-off-by: Liang Luo <luoliang@kylinos.cn>
Signed-off-by: Tejun Heo <tj@kernel.org>
The trailing semicolon belongs at the point of use, not in the macro
definition. All uses have been verified to have their own semicolons.
This was found using the following Coccinelle semantic patch:
@r@
identifier i : script:ocaml() { String.lowercase_ascii i = i };
expression e;
@@
*#define i(...) e;
Signed-off-by: Julia Lawall <Julia.Lawall@inria.fr>
Signed-off-by: Tejun Heo <tj@kernel.org>
A robust futex unlock stores 0 over the whole futex value - wiping
FUTEX_WAITERS - and wakes a single waiter. That wakeup is a one-shot
notification: the protocol relies on its recipient to either acquire the
futex (and eventually unlock while aware of the remaining contention) or
re-arm FUTEX_WAITERS before sleeping again. If the woken waiter is killed
before it can do either, the kernel must jump in and wake the next task
down the line.
This is a known complication of the futex protocol with a previous
partial fix in commit ca16d5bee5 ("futex: Prevent robust futex exit
race"). Unfortunately, that fix is insufficient.
If a third task re-acquired the futex through the uncontended fast
path in the meantime, the notification is lost: robust exit processing
sees that it is owned by another task and does nothing, while the new
owner sees no FUTEX_WAITERS when it unlocks and wakes nobody.
The remaining waiters sleep forever behind a free futex:
A owns the futex, B and C sleep in FUTEX_WAIT
uval == A | FUTEX_WAITERS
A robust unlock: store 0, FUTEX_WAKE(1) wakes B
uval == 0
D fast path acquire: cmpxchg(0 -> D)
uval == D, no FUTEX_WAITERS
B killed before acting on the wakeup
B exit walk, pending op: owner D != B -> no action
D unlock: no FUTEX_WAITERS -> no wake
C sleeps forever
This is clearly a shortcoming in the implementation, which fails to keep
the FUTEX_WAITERS bit consistent.
Work around this by augmenting the robust list exit processing to also
perform the extra wakeup if the futex word is owned by another thread but
FUTEX_WAITERS is not set.
This does not fix the problem of a non-contended take over/release and free
sequence, which has been discussed for years and has been addressed by
commit 3ca9595d9f ("futex: Add support for unlocking robust futexes") and
subsequent changes, but failed to take the problem described above into
account.
A more complete solution which is based on the in kernel unlock of
contended robust futexes has been discussed in the context of this change
and should show up in mainline sooner than later.
[ tglx: Amend change log slightly and fixup coding style ]
Fixes: ca16d5bee5 ("futex: Prevent robust futex exit race")
Signed-off-by: Keno Fischer <keno@juliahub.com>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Assisted-by: ClaudeCode:claude-fable-5 tla+
Cc: stable@vger.kernel.org
Link: https://patch.msgid.link/20260730194705.38981-1-keno@juliacomputing.com
- Reset dropped_count in mmio_reset_data()
When mmio_reset_data() is called, it does not reset the dropped_count
so that subsequent runs will have incorrect reporting.
- Add NULL check for mmio_trace_array in logging functions
The functions __trace_mmiotrace_rw() and __trace_mmiotrace_map()
may have the 'tr' variable passed to it as NULL. But they both
dereference it without checking if it is NULL first.
- Check return value of __register_event() in trace_module_add_events()
If __register_event() fails, the call after it (__add_event_to_tracers())
will create a file for it. If the module fails to load and its memory
is freed, the file will still point to it and it will not be removed
as the registering of the event did not complete. Only call
__add_event_to_tracers() if the __register_event() was successful.
- Fix false positive match in regex_match_full()
The regex full matching uses a strncmp() to test against the match
string and the value. It should not match if value is a prefix of
the string to match. Check to make sure the length of the strings
match before comparing.
- Fix reader page read offset for remote buffers
A page swapped in by __rb_get_reader_page_from_remote() retains its
stale read offset, causing subsequent reads to skip events or read
past valid data.
- Fix memory leak of subbuf_ids in rb_allocate_cpu_buffer()
Remote buffers allocate a subbuf_ids array. If the allocator function
fails after it is allocated, it does not free it, resulting in a
memory leak.
-----BEGIN PGP SIGNATURE-----
iIkEABYKADIWIQRRSw7ePDh/lE+zeZMp5XQQmuv6qgUCam1A0RQccm9zdGVkdEBn
b29kbWlzLm9yZwAKCRAp5XQQmuv6qlW/AQDCpddDwAMwN80i3tn5uiqec9JLZT5I
DR6G+0zEF9a5DAD2IxiD3gH/mPVZ3CdiqfElBqjc0Pq1dx414xmzXBv+CA==
=GUpD
-----END PGP SIGNATURE-----
Merge tag 'trace-v7.2-rc5' of git://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace
Pull tracing fixes from Steven Rostedt:
- Reset dropped_count in mmio_reset_data()
When mmio_reset_data() is called, it does not reset the dropped_count
so that subsequent runs will have incorrect reporting.
- Add NULL check for mmio_trace_array in logging functions
The functions __trace_mmiotrace_rw() and __trace_mmiotrace_map() may
have the 'tr' variable passed to it as NULL. But they both
dereference it without checking if it is NULL first.
- Check return value of __register_event() in trace_module_add_events()
If __register_event() fails, the __add_event_to_tracers() call after
it will create a file for it. If the module fails to load and its
memory is freed, the file will still point to it and it will not be
removed as the registering of the event did not complete.
Only call __add_event_to_tracers() if the __register_event() was
successful.
- Fix false positive match in regex_match_full()
The regex full matching uses a strncmp() to test against the match
string and the value. It should not match if value is a prefix of the
string to match. Check to make sure the length of the strings match
before comparing.
- Fix reader page read offset for remote buffers
A page swapped in by __rb_get_reader_page_from_remote() retains its
stale read offset, causing subsequent reads to skip events or read
past valid data.
- Fix memory leak of subbuf_ids in rb_allocate_cpu_buffer()
Remote buffers allocate a subbuf_ids array. If the allocator function
fails after it is allocated, it does not free it, resulting in a
memory leak.
* tag 'trace-v7.2-rc5' of git://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace:
ring-buffer: Fix subbuf_ids memory leak in rb_allocate_cpu_buffer() error path
ring-buffer: Fix reader page read offset for remote buffers
tracing/filters: Fix false positive match in regex_match_full()
tracing: Check return value of __register_event() in trace_module_add_events()
tracing/mmiotrace: Add NULL check for mmio_trace_array in logging functions
tracing/mmiotrace: Reset dropped_count in mmio_reset_data()
btf_distill_func_proto() builds the function model used for the
fentry/fexit/fmod_ret/fsession trampolines and struct_ops. It has
accepted a 16-byte __int128 return value since the trampoline was
introduced: __get_type_size() returns the integer's type size, and the
return-type check only rejected ret < 0.
But the BPF trampoline preserves only 8 bytes of the return value (RAX on
x86, i.e. R0). For an attach type that reads the target's return value the
second half (RDX / R3) is neither saved nor restored, so a program
attached to a function returning a 16-byte value corrupts the value seen
by the real caller and itself observes only half of it. struct_ops
trampolines have the same limitation.
This affects the attach types that read the target's return value: fexit,
fmod_ret and fsession (plus the _multi variants of fexit and fsession),
and struct_ops. fentry/fentry_multi run before the target returns and are
unaffected.
Reject a >8 byte return value for these attach types in
bpf_check_attach_target() and bpf_check_attach_btf_id_multi(), and for
struct_ops in bpf_struct_ops_desc_init().
Fixes: fec56f5890 ("bpf: Introduce BPF trampoline")
Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
Reviewed-by: Eduard Zingerman <eddyz87@gmail.com>
Acked-by: Leon Hwang <leon.hwang@linux.dev>
Link: https://lore.kernel.org/bpf/20260729050159.2585809-1-yonghong.song@linux.dev
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
In rb_allocate_cpu_buffer(), cpu_buffer->subbuf_ids is allocated using
kcalloc() when buffer->remote is non-NULL. If a subsequent page allocation
fails (e.g., ring_buffer_desc_page() returns NULL or rb_allocate_pages()
fails), execution jumps to fail_free_reader.
While __free(kfree) automatically frees the outer cpu_buffer structure
at scope exit, kfree(cpu_buffer) does not recursively free nested heap
pointers such as cpu_buffer->subbuf_ids, resulting in a memory leak.
Fix this by explicitly freeing cpu_buffer->subbuf_ids in the
fail_free_reader error unwinding path when cpu_buffer->remote is set.
Link: https://patch.msgid.link/178550740672.380917.6067449683620196150.stgit@devnote2
Fixes: 2e67fabd8b ("ring-buffer: Introduce ring-buffer remotes")
Assisted-by: Antigravity:gemini-3.6-flash
Signed-off-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
Reviewed-by: Vincent Donnefort <vdonnefort@google.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
The untrusted PTR_TO_MEM early return skips pointer offset tracking
because accesses go through probe-read handling. Moving it after full
pointer-state propagation ensures scalar += untrusted_pointer leaves the
destination as PTR_TO_MEM instead of an unrelated scalar.
Fixes: f2362a57ae ("bpf: allow void* cast using bpf_rdonly_cast()")
Signed-off-by: Yiyang Chen <chenyy23@mails.tsinghua.edu.cn>
Tested-by: Daniel Wade <danjwade95@gmail.com>
Link: https://patch.msgid.link/20260729-c3-035-public-bpf-v4-v4-3-8ee297e2346b@mails.tsinghua.edu.cn
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
When scalar += pointer is handled in adjust_ptr_min_max_vals(), the
destination register inherits the pointer state from the source pointer.
Copying only selected fields is fragile because pointer provenance is
tracked by several bpf_reg_state fields.
Use the caller's temporary offset register to preserve the scalar operand
while replacing the destination with the full pointer state. This preserves
the frame number for PTR_TO_STACK registers and keeps parent identity
fields consistent.
Fixes: f4d7e40a5b ("bpf: introduce function calls (verification)")
Signed-off-by: Yiyang Chen <chenyy23@mails.tsinghua.edu.cn>
Tested-by: Daniel Wade <danjwade95@gmail.com>
Acked-by: Shung-Hsi Yu <shung-hsi.yu@suse.com>
Link: https://patch.msgid.link/20260729-c3-035-public-bpf-v4-v4-2-8ee297e2346b@mails.tsinghua.edu.cn
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
The sanitize_err() function is called when:
- ptr += scalar
- scalar += ptr
- scalar += scalar
ALU operations are processed.
This commit drops offset and pointer registers parameters from its
signature to simplify the follow-up changes for 'scalar += ptr' case.
regs[src].type is safe to access, as it is not mutated by the callers.
Signed-off-by: Yiyang Chen <chenyy23@mails.tsinghua.edu.cn>
Acked-by: Shung-Hsi Yu <shung-hsi.yu@suse.com>
Link: https://patch.msgid.link/20260729-c3-035-public-bpf-v4-v4-1-8ee297e2346b@mails.tsinghua.edu.cn
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Validate the functionality of LTL monitors by injecting events in a
controlled environment (KUnit) and expecting reactions, just like it is
done in DA monitors.
Reviewed-by: Nam Cao <namcao@linutronix.de>
Link: https://lore.kernel.org/r/20260723074534.43521-15-gmonaco@redhat.com
Signed-off-by: Gabriele Monaco <gmonaco@redhat.com>
Some monitors do not only rely on tracepoint arguments but also on the
currently executing task.
This makes it more challenging to mock events in KUnit.
Define wrapper functions around current, the functionality is mocked
only during KUnit, an additional function call is avoided using a static
branch unless any (even unrelated) KUnit test is running.
Rely on a global mock_current variable that is set only by the RV KUnit
tests and cleared on teardown. Unrelated KUnit tests that happen to
trigger RV handlers would see it null and use current.
Reviewed-by: Nam Cao <namcao@linutronix.de>
Reviewed-by: Wen Yang <wen.yang@linux.dev>
Link: https://lore.kernel.org/r/20260723074534.43521-14-gmonaco@redhat.com
Signed-off-by: Gabriele Monaco <gmonaco@redhat.com>
Validate the functionality of DA monitors by injecting events in a
controlled environment (KUnit) and expecting reactions.
Events handlers are exported directly from the monitor source files
without using system events and with dummy arguments (e.g. no real
tasks). If the provided sequence of events incurs a violation, the test
expects the stub version of rv_react() to be called.
This testing method can validate the entire monitor implementation since
it sits between the monitor and the system (in place of the
tracepoints). All sorts of system and timing events can be emulated
without affecting the running kernel.
Handlers and monitor functions are exported as part of a struct to
simplify the process of running KUnit tests from kernel modules.
Reviewed-by: Nam Cao <namcao@linutronix.de>
Link: https://lore.kernel.org/r/20260723074534.43521-13-gmonaco@redhat.com
Signed-off-by: Gabriele Monaco <gmonaco@redhat.com>
Export rv_get_task_monitor_slot, rv_put_task_monitor_slot, and rv_react
to GPL modules so they can be accessed by KUnit and future monitors
built as kernel modules.
Reviewed-by: Nam Cao <namcao@linutronix.de>
Link: https://lore.kernel.org/r/20260723074534.43521-12-gmonaco@redhat.com
Signed-off-by: Gabriele Monaco <gmonaco@redhat.com>
* pm-cpufreq:
cpufreq/amd-pstate: handle missing policy in dynamic EPP callbacks
cpufreq/amd-pstate: Cache the firmware programmed EPP value
cpufreq/amd-pstate: Toggle auto_sel in active mode on shared memory systems
cpufreq/amd-pstate: Fix EPP return type and handle errors during initialization
cpufreq: amd-pstate-ut: Skip tests when amd-pstate driver is not active
cpufreq: schedutil: Replace sprintf() with sysfs_emit() in sysfs show
cpufreq: schedutil: Fix self-contradictory comment in sugov_iowait_apply()
Documentation: admin-guide: cpufreq: fix sampling_rate example command
cpufreq: intel_pstate: Move two functions closer to callers
cpufreq: intel_pstate: Consolidate frequency values computation
cpufreq: intel_pstate: Introduce intel_pstate_update_freq_limits()
cpufreq: intel_pstate: Fix setting minimum P-state at init time
cpufreq: intel_pstate: Rename INTEL_PSTATE_HWP_BROADWELL
cpufreq: intel_pstate: Simplify HWP handling on Broadwell
cpufreq: intel_pstate: Adjust the .adjust_perf() driver callback
cpufreq: intel_pstate: Rearrange checks in hybrid_get_cost()
Align the rv_monitor variable name in LTL to the generic rv_this as it
is already done for DA/HA monitors. This improves consistency and eases
assumptions across model classes.
Reviewed-by: Nam Cao <namcao@linutronix.de>
Link: https://lore.kernel.org/r/20260723074534.43521-2-gmonaco@redhat.com
Signed-off-by: Gabriele Monaco <gmonaco@redhat.com>
No caller checks the return value from stop_one_cpu_nowait(). All
callers require the callback to run and arrange for the target CPU's
stopper to remain enabled while queuing the work. In particular, commit
f0498d2a54 ("sched: Fix stop_one_cpu_nowait() vs hotplug") added
preemption protection to the scheduler callers so that queuing must
succeed once the target CPU has been observed online.
Therefore, a failure is an unrecoverable violation rather than a condition
individual callers can recover from. Diagnose it with WARN_ON_ONCE() in
stop_one_cpu_nowait(). A check in the common helper covers current and
future callers consistently, while individual checks would duplicate
the same non-recoverable handling at every call site.
Make the function return void because there is no longer a meaningful
result for callers to consume.
On UP, warn if the supplied CPU is not the current CPU because the work
cannot be scheduled in that case.
Signed-off-by: Yury Norov <ynorov@nvidia.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Bradley Morgan <include@grrlz.net>
Reviewed-by: Shrikanth Hegde <sshegde@linux.ibm.com>
Link: https://patch.msgid.link/20260729022355.325058-1-ynorov@nvidia.com
KCSAN reports a data race between readers_active_check() and a
concurrently executing reader:
BUG: KCSAN: data-race in readers_active_check / percpu_down_write
race at unknown origin, with read to 0xffff9f3eb5bf5f30 of 4 bytes
by task 1271 on cpu 14:
readers_active_check+0x...
percpu_down_write+0x152/0x1f0
value changed: 0xfffffff9 -> 0xfffffff8
readers_active_check() calls per_cpu_sum(*sem->read_count), which
iterates over all CPUs and reads each CPU's per-CPU read_count
variable. Concurrently, a reader on a remote CPU is modifying its own
CPU's read_count via this_cpu_inc() / this_cpu_dec() as it enters and
exits the critical section. These are plain reads and writes to the
same per-CPU storage, hence KCSAN flags a data race.
This race is benign. readers_active_check() is called from the
percpu_down_write() wait loop (rcuwait_wait_event) after sem->block is
already set. At this point:
- New readers must immediately back out (they see block set, decrement
their counter, and wake the writer), so counters can only decrease.
- If the sum catches a reader's increment before its decrement,
readers_active_check() sees a non-zero sum and returns false. The
writer merely iterates the wait loop again -- a harmless retry.
- A false zero (observing sum == 0 while a reader is still active)
cannot happen: per_cpu_sum() reads each CPU's counter, and each
per-CPU int read is atomic on all architectures, so an active
reader's counter is always seen as non-zero.
Annotate the read with data_race() to suppress the KCSAN warning and
document the intentional nature of this unlocked access.
Signed-off-by: Sun Shaojie <sunshaojie@kylinos.cn>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/20260623104132.505117-1-sunshaojie@kylinos.cn
register_lock_class() can return NULL when the lock class pool is
exhausted, graph_lock() fails, or key validation fails. However,
__lock_set_class() uses the return value directly in pointer arithmetic
without a NULL check:
class = register_lock_class(lock, subclass, 0);
hlock->class_idx = class - lock_classes;
If class is NULL, this computes a wild offset that corrupts
hlock->class_idx. The subsequent reacquire_held_locks() call will
invoke hlock_class() with this corrupted index, leading to a NULL or
out-of-bounds pointer dereference.
Add the missing NULL check, consistent with how __lock_acquire() already
handles this case at the same call site.
Fixes: 64aa348edc ("lockdep: lock_set_subclass - reset a held lock's subclass")
Signed-off-by: Naveen Kumar Chaudhary <naveen.osdev@gmail.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Waiman Long <longman@redhat.com>
Reviewed-by: Dmitry Ilvokhin <d@ilvokhin.com>
Link: https://patch.msgid.link/h2kfw43n4527x6mgi2lwpz2rieqnfzgictpv4wr5nyfjkc47co@2r5vz4uz44db
Forking a task that has a pending uretprobe can oops the kernel with a
NULL pointer dereference in the clone() path:
BUG: kernel NULL pointer dereference, address: 0000000000000018
Oops: 0002 [#1] SMP NOPTI
RIP: 0010:hprobe_expire
CR2: 0000000000000018
Call Trace:
uprobe_copy_process
copy_process
kernel_clone
__x64_sys_clone
do_syscall_64
entry_SYSCALL_64_after_hwframe
This was found on real hosts on Meta fleet.
I've got the impression that this is what is happening:
CPU 1 CPU 2 (traced task)
----- -------------------
hit uprobe, prepare_uretprobe():
hprobe LEASED, refcount >= 1
uprobe_unregister()
put_uprobe(): refcount -> 0
fork() -> dup_utask()
hprobe_expire(hprobe, true)
try_get_uprobe() -> NULL
get_uprobe(NULL) <-- Oops
Only take the extra reference when the uprobe is non-NULL; a NULL means
it is gone and is the correct value to return.
Fixes: dd1a756778 ("uprobes: SRCU-protect uretprobe lifetime (with timeout)")
Signed-off-by: Breno Leitao <leitao@debian.org>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Acked-by: Oleg Nesterov <oleg@redhat.com>
Acked-by: Andrii Nakryiko <andrii@kernel.org>
Cc: stable@vger.kernel.org
Link: https://patch.msgid.link/20260729-uprobe-v1-1-61896b87c867@debian.org
SWIOTLB_FORCE has no remaining in-tree users. Forced bouncing is now
controlled through the swiotlb=force command line option via
swiotlb_force_bounce.
Remove the unused flag and simplify the force_bounce initialization.
Reviewed-by: Jason Gunthorpe <jgg@nvidia.com>
Signed-off-by: Aneesh Kumar K.V (Arm) <aneesh.kumar@kernel.org>
Link: https://lore.kernel.org/r/20260717180442.110954-24-aneesh.kumar@kernel.org
Signed-off-by: Marek Szyprowski <m.szyprowski@samsung.com>
Check the return value when converting swiotlb pools between encrypted and
decrypted mappings. If the default pool cannot be decrypted after early
initialization, mark the pool fully used so it cannot satisfy future bounce
allocations.
For late initialization, return the `set_memory_decrypted()` failure. For
restricted DMA pools, fail device initialization if the reserved pool
cannot be decrypted.
This prevents swiotlb from using pools whose encryption attributes do not
match their metadata, and avoids returning pages with uncertain encryption
state back to the allocator.
Reviewed-by: Jason Gunthorpe <jgg@nvidia.com>
Tested-by: Michael Kelley <mhklinux@outlook.com>
Tested-by: Mostafa Saleh <smostafa@google.com>
Reviewed-by: Petr Tesarik <ptesarik@suse.com>
Signed-off-by: Aneesh Kumar K.V (Arm) <aneesh.kumar@kernel.org>
Link: https://lore.kernel.org/r/20260717180442.110954-23-aneesh.kumar@kernel.org
Signed-off-by: Marek Szyprowski <m.szyprowski@samsung.com>
swiotlb_dyn_free() is used after removing a dynamic swiotlb pool from
RCU-protected lists. It can call swiotlb_free_tlb(), which may need to
restore the encryption state of an unencrypted pool with
set_memory_encrypted() before freeing the pages.
RCU callbacks run in atomic context, but set_memory_encrypted() is not
guaranteed to be atomic-safe on all architectures. For example, page
attribute updates may allocate page tables or take sleeping locks.
Use queue_rcu_work() for dynamic pool freeing instead. This keeps the RCU
grace period before freeing a published pool, while running the actual pool
teardown from workqueue context. Use the same helper for the transient-pool
error path, since that path may also be reached from atomic DMA mapping
context.
Tested-by: Michael Kelley <mhklinux@outlook.com>
Tested-by: Mostafa Saleh <smostafa@google.com>
Reviewed-by: Petr Tesarik <ptesarik@suse.com>
Signed-off-by: Aneesh Kumar K.V (Arm) <aneesh.kumar@kernel.org>
Link: https://lore.kernel.org/r/20260717180442.110954-22-aneesh.kumar@kernel.org
Signed-off-by: Marek Szyprowski <m.szyprowski@samsung.com>
ret in dma_direct_alloc() and dma_direct_alloc_pages() holds the returned
CPU mapping, not a generic return value. Rename it to cpu_addr and update
the remaining uses to match.
This makes the allocation paths easier to follow and keeps the local naming
consistent with what the variable actually represents.
Reviewed-by: Jason Gunthorpe <jgg@nvidia.com>
Tested-by: Michael Kelley <mhklinux@outlook.com>
Tested-by: Mostafa Saleh <smostafa@google.com>
Reviewed-by: Petr Tesarik <ptesarik@suse.com>
Signed-off-by: Aneesh Kumar K.V (Arm) <aneesh.kumar@kernel.org>
Link: https://lore.kernel.org/r/20260717180442.110954-21-aneesh.kumar@kernel.org
Signed-off-by: Marek Szyprowski <m.szyprowski@samsung.com>
Make the dma-direct helpers derive the DMA address encoding from
__DMA_ATTR_ALLOC_CC_SHARED instead of implicitly relying on
force_dma_unencrypted() inside phys_to_dma_direct()
Pass an explicit unencrypted/decrypted state into phys_to_dma_direct(),
make the alloc paths return DMA addresses that match the requested buffer
encryption state. Also only call dma_set_decrypted() when
__DMA_ATTR_ALLOC_CC_SHARED is actually set.
Reviewed-by: Jason Gunthorpe <jgg@nvidia.com>
Tested-by: Jiri Pirko <jiri@nvidia.com>
Tested-by: Michael Kelley <mhklinux@outlook.com>
Tested-by: Mostafa Saleh <smostafa@google.com>
Signed-off-by: Aneesh Kumar K.V (Arm) <aneesh.kumar@kernel.org>
Reviewed-by: Mostafa Saleh <smostafa@google.com>
Link: https://lore.kernel.org/r/20260717180442.110954-20-aneesh.kumar@kernel.org
Signed-off-by: Marek Szyprowski <m.szyprowski@samsung.com>
Devices that are DMA non-coherent and require a remap were skipping
dma_set_decrypted(), leaving DMA buffers encrypted even when the device
requires unencrypted access. Move the call after the if (remap) branch
so that both the direct and remapped allocation paths correctly mark the
allocation as decrypted (or fail cleanly) before use.
Fix dma_direct_alloc() and dma_direct_free() to apply set_memory_*() to the
linear-map alias of the backing pages instead of the remapped CPU address.
Also disallow highmem pages for __DMA_ATTR_ALLOC_CC_SHARED, because highmem
buffers do not provide a usable linear-map address.
Reviewed-by: Jason Gunthorpe <jgg@nvidia.com>
Tested-by: Jiri Pirko <jiri@nvidia.com>
Tested-by: Michael Kelley <mhklinux@outlook.com>
Tested-by: Mostafa Saleh <smostafa@google.com>
Signed-off-by: Aneesh Kumar K.V (Arm) <aneesh.kumar@kernel.org>
Link: https://lore.kernel.org/r/20260717180442.110954-19-aneesh.kumar@kernel.org
Signed-off-by: Marek Szyprowski <m.szyprowski@samsung.com>
Teach dma_direct_map_phys() to select the DMA address encoding based on
DMA_ATTR_CC_SHARED.
Use phys_to_dma_unencrypted() for decrypted mappings and
phys_to_dma_encrypted() otherwise. If a device requires unencrypted DMA
but the source physical address is still encrypted, force the mapping
through swiotlb so the DMA address and backing memory attributes remain
consistent.
Update the arm64, x86, s390 and powerpc secure-guest setup to not use
swiotlb force option
Tested-by: Jiri Pirko <jiri@nvidia.com>
Tested-by: Michael Kelley <mhklinux@outlook.com>
Tested-by: Mostafa Saleh <smostafa@google.com>
Signed-off-by: Aneesh Kumar K.V (Arm) <aneesh.kumar@kernel.org>
Link: https://lore.kernel.org/r/20260717180442.110954-18-aneesh.kumar@kernel.org
[mszyprow: rebased onto latest changes in arch/arm64/mm/init.c]
Signed-off-by: Marek Szyprowski <m.szyprowski@samsung.com>
dma_direct_map_phys() is too large to benefit from being inlined. Move its
implementation to direct.c and leave the declaration in direct.h.
No functional change in this patch
Signed-off-by: Aneesh Kumar K.V (Arm) <aneesh.kumar@kernel.org>
Link: https://lore.kernel.org/r/20260717180442.110954-17-aneesh.kumar@kernel.org
Signed-off-by: Marek Szyprowski <m.szyprowski@samsung.com>
Teach dma_capable() about DMA_ATTR_CC_SHARED so the capability
check can reject encrypted DMA addresses for devices that require
unencrypted/shared DMA.
Also propagate DMA_ATTR_CC_SHARED in swiotlb_map() when the selected
SWIOTLB pool is decrypted so the capability check sees the correct DMA
address attribute.
Reviewed-by: Jason Gunthorpe <jgg@nvidia.com>
Tested-by: Jiri Pirko <jiri@nvidia.com>
Tested-by: Michael Kelley <mhklinux@outlook.com>
Tested-by: Mostafa Saleh <smostafa@google.com>
Reviewed-by: Petr Tesarik <ptesarik@suse.com>
Signed-off-by: Aneesh Kumar K.V (Arm) <aneesh.kumar@kernel.org>
Link: https://lore.kernel.org/r/20260717180442.110954-16-aneesh.kumar@kernel.org
Signed-off-by: Marek Szyprowski <m.szyprowski@samsung.com>
Fold encrypted/decrypted pgprot selection into dma_pgprot() so callers
do not need to adjust the page protection separately.
Update dma_pgprot() to apply pgprot_decrypted() when DMA_ATTR_CC_SHARED or
__DMA_ATTR_ALLOC_CC_SHARED is set and pgprot_encrypted() otherwise Convert
the dma-direct mmap paths to pass DMA_ATTR_CC_SHARED instead of open-coding
force_dma_unencrypted() handling around dma_pgprot().
Reviewed-by: Jason Gunthorpe <jgg@nvidia.com>
Tested-by: Jiri Pirko <jiri@nvidia.com>
Tested-by: Michael Kelley <mhklinux@outlook.com>
Tested-by: Mostafa Saleh <smostafa@google.com>
Signed-off-by: Aneesh Kumar K.V (Arm) <aneesh.kumar@kernel.org>
Link: https://lore.kernel.org/r/20260717180442.110954-15-aneesh.kumar@kernel.org
Signed-off-by: Marek Szyprowski <m.szyprowski@samsung.com>
Teach swiotlb to distinguish between encrypted and decrypted bounce
buffer pools, and make allocation and mapping paths select a pool whose
state matches the requested DMA attributes.
Add a cc_shared flag to io_tlb_mem, initialize it for the default and
restricted pools, and propagate __DMA_ATTR_ALLOC_CC_SHARED into swiotlb
pool allocation. Reject swiotlb alloc/map requests when the selected pool
does not match the required encrypted/decrypted state.
Also return DMA addresses with the matching phys_to_dma_{encrypted,
unencrypted} helper so the DMA address encoding stays consistent with the
chosen pool.
Reviewed-by: Jason Gunthorpe <jgg@nvidia.com>
Tested-by: Jiri Pirko <jiri@nvidia.com>
Tested-by: Michael Kelley <mhklinux@outlook.com>
Tested-by: Mostafa Saleh <smostafa@google.com>
Signed-off-by: Aneesh Kumar K.V (Arm) <aneesh.kumar@kernel.org>
Link: https://lore.kernel.org/r/20260717180442.110954-14-aneesh.kumar@kernel.org
Signed-off-by: Marek Szyprowski <m.szyprowski@samsung.com>
Change swiotlb_tbl_map_single() to take the DMA mapping attributes by
reference and update the direct callers accordingly.
This is a preparatory change for a follow-up patch which updates the
attributes based on the selected swiotlb pool. Keeping the signature change
separate makes the follow-up patch easier to review.
No functional change in this patch.
Reviewed-by: Jason Gunthorpe <jgg@nvidia.com>
Tested-by: Michael Kelley <mhklinux@outlook.com>
Tested-by: Mostafa Saleh <smostafa@google.com>
Reviewed-by: Petr Tesarik <ptesarik@suse.com>
Signed-off-by: Aneesh Kumar K.V (Arm) <aneesh.kumar@kernel.org>
Link: https://lore.kernel.org/r/20260717180442.110954-13-aneesh.kumar@kernel.org
Signed-off-by: Marek Szyprowski <m.szyprowski@samsung.com>
Teach the atomic DMA pool code to distinguish between encrypted and
unencrypted pools, and make pool allocation select the matching pool based
on DMA attributes.
Introduce a dma_gen_pool wrapper that records whether a pool is
unencrypted, initialize that state when the atomic pools are created, and
use it when expanding and resizing the pools. Update dma_alloc_from_pool()
to take attrs and skip pools whose encrypted state does not match
__DMA_ATTR_ALLOC_CC_SHARED. Update dma_free_from_pool() accordingly.
Also pass __DMA_ATTR_ALLOC_CC_SHARED from the swiotlb atomic allocation
path so decrypted swiotlb allocations are taken from the correct atomic
pool.
Tested-by: Jiri Pirko <jiri@nvidia.com>
Tested-by: Michael Kelley <mhklinux@outlook.com>
Tested-by: Mostafa Saleh <smostafa@google.com>
Reviewed-by: Mostafa Saleh <smostafa@google.com>
Signed-off-by: Aneesh Kumar K.V (Arm) <aneesh.kumar@kernel.org>
Link: https://lore.kernel.org/r/20260717180442.110954-12-aneesh.kumar@kernel.org
Signed-off-by: Marek Szyprowski <m.szyprowski@samsung.com>
Propagate force_dma_unencrypted() into __DMA_ATTR_ALLOC_CC_SHARED in the
dma-direct allocation path and use the attribute to drive the related
decisions.
This updates dma_direct_alloc(), dma_direct_free(), and
dma_direct_alloc_pages() to fold the forced unencrypted case into attrs.
Reviewed-by: Jason Gunthorpe <jgg@nvidia.com>
Tested-by: Jiri Pirko <jiri@nvidia.com>
Tested-by: Michael Kelley <mhklinux@outlook.com>
Tested-by: Mostafa Saleh <smostafa@google.com>
Reviewed-by: Petr Tesarik <ptesarik@suse.com>
Signed-off-by: Aneesh Kumar K.V (Arm) <aneesh.kumar@kernel.org>
Link: https://lore.kernel.org/r/20260717180442.110954-11-aneesh.kumar@kernel.org
Signed-off-by: Marek Szyprowski <m.szyprowski@samsung.com>
Move swiotlb allocation out of __dma_direct_alloc_pages() and handle it in
dma_direct_alloc() / dma_direct_alloc_pages().
This is needed for follow-up changes that simplify the handling of
memory encryption/decryption based on the DMA attribute flags.
swiotlb backing pages are already mapped decrypted by
swiotlb_update_mem_attributes() and rmem_swiotlb_device_init(), so
dma-direct should not call dma_set_decrypted() on allocation nor
dma_set_encrypted() on free for swiotlb-backed memory.
Update alloc/free paths to detect swiotlb-backed pages and skip
encrypt/decrypt transitions for those paths. Keep the existing highmem
rejection in dma_direct_alloc_pages() for swiotlb allocations.
Only for "restricted-dma-pool", we currently set `for_alloc = true`, while
rmem_swiotlb_device_init() decrypts the whole pool up front. This pool is
typically used together with "shared-dma-pool", where the shared region is
accessed after remap/ioremap and the returned address is suitable for
decrypted memory access. So existing code paths remain valid.
Reviewed-by: Jason Gunthorpe <jgg@nvidia.com>
Tested-by: Jiri Pirko <jiri@nvidia.com>
Tested-by: Michael Kelley <mhklinux@outlook.com>
Tested-by: Mostafa Saleh <smostafa@google.com>
Signed-off-by: Aneesh Kumar K.V (Arm) <aneesh.kumar@kernel.org>
Reviewed-by: Mostafa Saleh <smostafa@google.com>
Link: https://lore.kernel.org/r/20260717180442.110954-8-aneesh.kumar@kernel.org
Signed-off-by: Marek Szyprowski <m.szyprowski@samsung.com>
swiotlb_alloc_tlb() can allocate from the DMA atomic pool when a decrypted
pool is needed from atomic context. With CONFIG_DMA_DIRECT_REMAP, the
atomic pool is backed by remapped virtual addresses, which are not the same
as the direct-map addresses returned by phys_to_virt().
swiotlb_init_io_tlb_pool() currently reconstructs the pool virtual address
from the physical start address. For atomic-pool backed allocations this
stores the wrong address in pool->vaddr. Later, swiotlb_free_tlb() passes
that address to dma_free_from_pool(), which will fail to recognize the
chunk
Pass the virtual address returned by the allocation path into
swiotlb_init_io_tlb_pool(), and store that address in pool->vaddr. This
keeps the pool free path using the same virtual address as the allocator.
Fixes: 79636caad3 ("swiotlb: if swiotlb is full, fall back to a transient memory pool")
Reviewed-by: Jason Gunthorpe <jgg@nvidia.com>
Tested-by: Michael Kelley <mhklinux@outlook.com>
Tested-by: Mostafa Saleh <smostafa@google.com>
Reviewed-by: Petr Tesarik <ptesarik@suse.com>
Signed-off-by: Aneesh Kumar K.V (Arm) <aneesh.kumar@kernel.org>
Reviewed-by: Mostafa Saleh <smostafa@google.com>
Link: https://lore.kernel.org/r/20260717180442.110954-6-aneesh.kumar@kernel.org
Signed-off-by: Marek Szyprowski <m.szyprowski@samsung.com>
dma_direct_alloc_pages() may satisfy atomic allocations from the coherent
atomic pools. The pool allocation is keyed by the virtual address stored in
the gen_pool, but the pages API returns only the backing struct page.
On architectures with CONFIG_DMA_DIRECT_REMAP, atomic pool chunks are added
to the gen_pool using their remapped virtual address.
dma_direct_free_pages() reconstructs a linear-map address with
page_address(page) and passes that to dma_free_from_pool(). That address
does not match the gen_pool virtual range, so the pool lookup can fail and
the code can fall through to freeing a pool-owned page through the normal
page allocator path.
Add a page-based pool free helper that looks up the owning pool chunk by
physical address, translates it back to the gen_pool virtual address, and
frees that address to the pool. Use it from dma_direct_free_pages() while
keeping the existing virtual-address helper for coherent allocation frees.
Tested-by: Michael Kelley <mhklinux@outlook.com>
Tested-by: Mostafa Saleh <smostafa@google.com>
Signed-off-by: Aneesh Kumar K.V (Arm) <aneesh.kumar@kernel.org>
Link: https://lore.kernel.org/r/20260717180442.110954-5-aneesh.kumar@kernel.org
Signed-off-by: Marek Szyprowski <m.szyprowski@samsung.com>
atomic_pool_expand() frees the allocated pages from the remove_mapping
error path only when CONFIG_DMA_DIRECT_REMAP is enabled.
When CONFIG_DMA_DIRECT_REMAP is disabled, failures after page allocation,
such as gen_pool_add_virt(), jump to remove_mapping and return without
freeing the pages.
Move __free_pages(page, order) out of the CONFIG_DMA_DIRECT_REMAP block so
that cleanup paths always release the allocation.
Reviewed-by: Jason Gunthorpe <jgg@nvidia.com>
Tested-by: Michael Kelley <mhklinux@outlook.com>
Tested-by: Mostafa Saleh <smostafa@google.com>
Signed-off-by: Aneesh Kumar K.V (Arm) <aneesh.kumar@kernel.org>
Link: https://lore.kernel.org/r/20260717180442.110954-3-aneesh.kumar@kernel.org
Signed-off-by: Marek Szyprowski <m.szyprowski@samsung.com>
Commit 5b138c534f ("dma-direct: factor out a dma_direct_alloc_from_pool
helper") changed dma_direct_alloc_from_pool() to return the CPU address
from dma_alloc_from_pool(). That fits dma_direct_alloc(), but
dma_direct_alloc_pages() also uses the helper and expects a struct page *.
Fix this by making dma_direct_alloc_from_pool() return the struct page *
again, and pass the CPU address back through an out-parameter for the
dma_direct_alloc() caller.
Fixes: 5b138c534f ("dma-direct: factor out a dma_direct_alloc_from_pool helper")
Cc: stable@vger.kernel.org
Tested-by: Michael Kelley <mhklinux@outlook.com>
Tested-by: Mostafa Saleh <smostafa@google.com>
Reviewed-by: Jason Gunthorpe <jgg@nvidia.com>
Signed-off-by: Aneesh Kumar K.V (Arm) <aneesh.kumar@kernel.org>
Reviewed-by: Mostafa Saleh <smostafa@google.com>
Link: https://lore.kernel.org/r/20260717180442.110954-2-aneesh.kumar@kernel.org
Signed-off-by: Marek Szyprowski <m.szyprowski@samsung.com>
Replace silent WARN_ON_ONCE with WARN_ONCE that prints the actual error
code from bpf_trampoline_unlink_prog(). This aids debugging of race
conditions during link teardown, while keeping the warning rate limited
to avoid log flooding.
This will be very helpful for speeding up trouble-shooting of some crash
UAF due to bpf_trampoline_unlink_prog failures.
No change to unlink behavior.
Signed-off-by: Xu Xin <xu.xin16@zte.com.cn>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Reviewed-by: Emil Tsalapatis <emil@etsalapatis.com>
Acked-by: Leon Hwang <leon.hwang@linux.dev>
Link: https://lore.kernel.org/bpf/20260729141159128mEJmS_aujBKr-cBu1p_UI@zte.com.cn
In bpf_link_show_fdinfo and bpf_link_get_info_by_fd, link->prog is
accessed without holding any locks. If the prog is concurrently replaced
via bpf_link_update, the old prog can be freed, leading to a potential
UAF issue.
Fix this by accessing link->prog under RCU protection to safely fetch
the pointer and guarantee its lifetime while reading its fields.
Fixes: 0c991ebc8c ("bpf: Implement bpf_prog replacement for an active bpf_cgroup_link")
Reported-by: Sashiko <sashiko-bot@kernel.org>
Signed-off-by: Pu Lehui <pulehui@huawei.com>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Reviewed-by: Emil Tsalapatis <emil@etsalapatis.com>
Reviewed-by: Amery Hung <ameryhung@gmail.com>
Acked-by: Leon Hwang <leon.hwang@linux.dev>
Link: https://lore.kernel.org/bpf/f87b53c0-8f00-45a6-82db-8242fa9b143f@huaweicloud.com [0]
Link: https://lore.kernel.org/bpf/20260728025457.2814876-1-pulehui@huaweicloud.com
In bpf_netns_link_update_prog, the checks for old_prog and prog type
are currently performed locklessly before acquiring netns_bpf_mutex.
This creates a race condition that can lead to a UAF issue.
If two threads concurrently execute BPF_LINK_UPDATE on the same netns
link, the following execution path can trigger a UAF:
CPU0 CPU1
bpf_netns_link_update_prog
if (old_prog && old_prog != link->prog)
return -EPERM;
bpf_netns_link_update_prog
if (old_prog && old_prog != link->prog)
...
old_prog = xchg(&link->prog, new_prog);
bpf_prog_put(old_prog);
if (new_prog->type != link->prog->type) <-- trigger UAF
Fix this by moving the old_prog and prog->type checks inside the
netns_bpf_mutex critical section. Meanwhile, use guard() to simplify
lock management and avoid all the goto jumping.
Fixes: 7f045a49fe ("bpf: Add link-based BPF program attachment to network namespace")
Reported-by: Sashiko <sashiko-bot@kernel.org>
Signed-off-by: Pu Lehui <pulehui@huawei.com>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Reviewed-by: Amery Hung <ameryhung@gmail.com>
Reviewed-by: Emil Tsalapatis <emil@etsalapatis.com>
Link: https://lore.kernel.org/bpf/f87b53c0-8f00-45a6-82db-8242fa9b143f@huaweicloud.com [0]
Link: https://lore.kernel.org/bpf/20260728023259.2813482-1-pulehui@huaweicloud.com
-----BEGIN PGP SIGNATURE-----
iQJIBAABCgAyFiEES0KozwfymdVUl37v6iDy2pc3iXMFAmprvkoUHHBhdWxAcGF1
bC1tb29yZS5jb20ACgkQ6iDy2pc3iXOc2hAAyvfOofGyPzcGVjyYdN9mla/eQ0Av
qIDji0nXNF+cj07TGea/af4cWboc3iZnjF9mNNitI2cawMCsN53LdF3Cktc538ij
cnGHXVRueG2btoCGvCD3carbo4dmEGFzB3+3HvPaUFYHrw2sft3+tuxqyssXx63f
9MjMiky05WAsgKrZ5EAttJPJ6NeSR+Lbh5bFtRqHr3wPqvW2J3vEyowHIBz2dEjC
y9zhgY6HoFWivi/8ftY9Xor9+RwHjH1AYRA/w9ZEzv28Uzcn1HWE+oR+vcOqquCN
qagtqXtapA8RV1EtBd23Htwxg469zbaQiIjzQWqxje4wnXfw5eIriKXUY3hNH73S
iaDjoEhX2wvlPk6rRKjEZTZCStkNTmQ/fTicuT9UcRW9WsLa0BGZNx7aomThwS6e
5idm9KtvKPwRgtXbZlS4VTRqRu6haySevRxMZLe6CcsyutOFv0n/YcPfSWDJGvLQ
Zzx6FamPQYk/xkyBGueFibw809hSf2Pi8jMbyIbMkunbukLzwJTCpa+LfECMWSi7
pqG0H8KGEJUEW6VFDVtUtq0SaeVJN43oG920ExG2qRfLA2Wwr7hSnh93hreLCEDB
YRgQn8YrmIeRGolA+UJNe1mFDxGnQzkcOQYs2XC1+MyIfUJNgCVHnd0mD69KcbXw
exz8eKyXARs/LM8=
=oqQe
-----END PGP SIGNATURE-----
Merge tag 'audit-pr-20260730' of git://git.kernel.org/pub/scm/linux/kernel/git/pcmoore/audit
Pull audit fixes from Paul Moore:
- Fix potential integer overflows in audit_log_n_string()
Similar to the earlier fix to audit_log_n_hex() that you merged
earlier in July. Expect a cleaner, and generally better fix for these
functions in an upcoming merge window, but this addresses the problem
in a small patch that should be easy for people to backport.
- Fix potential use-after-free in audit_del_rule()
* tag 'audit-pr-20260730' of git://git.kernel.org/pub/scm/linux/kernel/git/pcmoore/audit:
audit: fix potential use-after-free in audit_del_rule()
audit: fix potential integer overflow in audit_log_n_string()
- Allow fast frequency switching in the ACPI CPPC library only when
every supported control used by the driver callback has an address
space already accepted for fast access (Christian Loehle)
- Skip writes to unsupported performance controls in the ACPI CPPC
library (Christian Loehle)
- Update cppc_cpufreq_update_perf_limits() to read policy->min and
policy->max once and, if the lockless snapshot is inconsistent,
reduce the minimum to the observed maximum, along the lines of
cpufreq_driver_resolve_freq() (Christian Loehle)
- Fix a possible memory leak in the powernowk8_cpu_init() error
paths (Abdun Nihaal)
- Loosen the requirement on lowest nonlinear frequency != min freq in
the amd-pstate driver that is too tight for new systems some of
which actually have the lowest nonlinear frequency identical to the
minimum frequency (Mario Limonciello)
- Prevent amd-pstate from loading on unsupported hardware (Rong Zhang)
- Address an initialization race in the schedutil governor when it
runs on multi-CPU cpufreq policies, by making it initialize all
per-CPU structures first and only then publish the per-CPU
utilization update hooks (Zhongqiu Han)
-----BEGIN PGP SIGNATURE-----
iQFGBAABCAAwFiEEcM8Aw/RY0dgsiRUR7l+9nS/U47UFAmprmRgSHHJqd0Byand5
c29ja2kubmV0AAoJEO5fvZ0v1OO1Ro0H/3llnBhbu9nq7QcFmlPqcRh2yqWozPoG
oJTYEc/FEvPTEy65b6YxPFm64heiVTpv2gSZk0NVLnsXmDc3qNfoVFN7BvY8soPV
X4hEZbVi/ZE84l2gmc3t9HjK+j9mxfyWiWj/QDqZvZzf3v0QN8Klo9SvzlwXlnXX
1KdBpPPZYv7H0xIIOiL7K17JbvVTAXIwsV2SidOkm+29xArMYnHnzxqaIoh70kx7
vvRLR0NwubHzPZt66L5TFQCwixa+O71YaQ2l1k6TOzZ5G2BEtriXjAaimAd88iAu
Ogpie91bcERZdmoBIGtBCg9jSrHIqJE8ItugBsZw21BnIBsAKpgpJgw=
=U/2b
-----END PGP SIGNATURE-----
Merge tag 'pm-7.2-rc6' of git://git.kernel.org/pub/scm/linux/kernel/git/rafael/linux-pm
Pull power management fixes from Rafael Wysocki:
"These fix issues related to cpufreq, in the ACPI CPPC library and the
generic CPPC cpufreq driver, in the powernow-k8 and amd-pstate
drivers, and in the schedutil governor:
- Allow fast frequency switching in the ACPI CPPC library only when
every supported control used by the driver callback has an address
space already accepted for fast access (Christian Loehle)
- Skip writes to unsupported performance controls in the ACPI CPPC
library (Christian Loehle)
- Update cppc_cpufreq_update_perf_limits() to read policy->min and
policy->max once and, if the lockless snapshot is inconsistent,
reduce the minimum to the observed maximum, along the lines of
cpufreq_driver_resolve_freq() (Christian Loehle)
- Fix a possible memory leak in the powernowk8_cpu_init() error paths
(Abdun Nihaal)
- Loosen the requirement on lowest nonlinear frequency != min freq in
the amd-pstate driver that is too tight for new systems some of
which actually have the lowest nonlinear frequency identical to the
minimum frequency (Mario Limonciello)
- Prevent amd-pstate from loading on unsupported hardware (Rong
Zhang)
- Address an initialization race in the schedutil governor when it
runs on multi-CPU cpufreq policies, by making it initialize all
per-CPU structures first and only then publish the per-CPU
utilization update hooks (Zhongqiu Han)"
* tag 'pm-7.2-rc6' of git://git.kernel.org/pub/scm/linux/kernel/git/rafael/linux-pm:
cpufreq: powernow-k8: Fix possible memory leak in powernowk8_cpu_init()
ACPI: CPPC: Skip writes to unsupported performance controls
cpufreq/amd-pstate: Prevent the driver from loading on unsupported hardware
cpufreq/amd-pstate: Loosen requirement on lowest nonlinear frequency != min freq
cpufreq: schedutil: Publish util hooks only after all sg_cpu are initialized
cpufreq: cppc: Sanitize lockless policy limit snapshots
ACPI: CPPC: Check all controls for fast switching
The comments for call_rcu_tasks() and call_rcu_tasks_rude() referred
to "RCU read-side critical sections", which is ambiguous. specify
"rcu-tasks" and "rcu-tasks rude" respectively to match the actual
flavor being described.
Signed-off-by: Zqiang <qiang.zhang@linux.dev>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
This 'commit 1612160b91 ("rcu-tasks: Eliminate deadlocks involving
do_exit() and RCU tasks")' has reimplemented the rcu_tasks_postscan(),
and the synchronize_srcu() has been removed, this commit therefore
rename tasks_rcu_exit_srcu_stall_timer to tasks_rcu_exit_stall_timer.
Signed-off-by: Zqiang <qiang.zhang@linux.dev>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
The rdp->cpu_no_qs.s field is accessed only by the current CPU,
but can be accessed both at task level and from interrupt handlers.
All accesses from interrupts-enabled code must therefore be marked.
This commit therefore converts from __this_cpu_read() to this_cpu_read(),
but only in interrupts-enabled code, that is, the rcu_qs() function.
KCSAN located this issue.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
When CONFIG_KASAN=y and CONFIG_KASAN_STACK=y builds, the
show_rcu_gp_kthreads() exceeds the 1024-byte frame-size limit:
make kernel/rcu/tree.o KCFLAGS="-fstack-usage"
DESCEND objtool
DESCEND bpf/resolve_btfids
INSTALL libsubcmd_headers
CC kernel/rcu/tree.o
In file included from kernel/rcu/tree.c:4998:
kernel/rcu/tree_stall.h: In function 'show_rcu_gp_kthreads':
kernel/rcu/tree_stall.h:994:1: warning: the frame size of 1656 bytes is larger than 1024 bytes [-Wframe-larger-than=]
grep show_rcu kernel/rcu/tree.su
tree_nocb.h:1622:13:show_rcu_nocb_state 896 dynamic,bounded
tree_stall.h:933:6:show_rcu_gp_kthreads 1784 dynamic,bounded
tree_stall.h:1102:13:sysrq_show_rcu 16 static
Wrap the pr_info() into two noinline_for_stack helpers function:
show_rcu_state() print rcu_state status, and show_rcu_node()
print single rcu_node status.
After apply this change:
grep show_rcu kernel/rcu/tree.su
tree_stall.h:955:22:show_rcu_node 696 dynamic,bounded
tree_stall.h:930:22:show_rcu_state 872 dynamic,bounded
tree_nocb.h:1622:13:show_rcu_nocb_state 896 dynamic,bounded
tree_stall.h:972:6:show_rcu_gp_kthreads 544 static
tree_stall.h:1113:13:sysrq_show_rcu 16 static
Signed-off-by: Zqiang <qiang.zhang@linux.dev>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
rcu_all_qs() and rcu_note_context_switch() read/clear the per-CPU
->rcu_urgent_qs and ->rcu_need_heavy_qs flags with plain raw_cpu_read()
and this_cpu_write(), while the RCU core clears them with WRITE_ONCE() in
rcu_disable_urgency_upon_qs(). KCSAN flags the resulting same-CPU race:
BUG: KCSAN: data-race in rcu_all_qs / rcu_disable_urgency_upon_qs
It is benign -- the flags are advisory and rcu_all_qs() re-reads
->rcu_urgent_qs with smp_load_acquire() before acting on it -- but these
are the last unmarked accesses to the two flags; every other access
already uses READ_ONCE()/WRITE_ONCE()/smp_*. Mark them to match. No
functional change.
Reproduced on a PREEMPT_NONE, CONFIG_KCSAN_INTERRUPT_WATCHER=y kernel with
a pthreads program whose threads (two per CPU) loop reading a large file:
for (;;) {
int fd = open("/proc/kallsyms", O_RDONLY);
while (read(fd, buf, sizeof(buf)) > 0)
;
close(fd);
}
The read()s drive cond_resched() -> rcu_all_qs() while the busy CPUs keep
the grace period urgent, so the RCU core clears the flags concurrently.
Fixes: 2dba13f0b6 ("rcu: Switch urgent quiescent-state requests to rcu_data structure")
Signed-off-by: Itai Handler <itai.handler@gmail.com>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
The rcu_check_gp_start_stall() works entirely on rnp parameter,
and never uses it's rdp parameter. this commit therefore drop it,
updating both callers and the declaration.
No functional change.
Signed-off-by: Zqiang <qiang.zhang@linux.dev>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
The rdp->cpu_no_qs.b.norm field is accessed only by the current CPU,
but can be accessed both at task level and from interrupt handlers.
All accesses from interrupts-enabled code must therefore be marked.
This commit therefore converts from __this_cpu_read() to this_cpu_read(),
from __this_cpu_write() to this_cpu_write(), and plain C-language accesses
to READ_ONCE() and WRITE_ONCE(), but only in interrupts-enabled code.
KCSAN located this issue.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Commit 14a8570564 ("sched/deadline: Use revised wakeup rule for
dl_server") applies the revised wakeup rule to any server, as a result
servers that are not running (dl_defer_running == 0) and start with a
deadline overflow get enqueued and can boost tasks as if they were
running, invalidating the defer rule and the documented state model.
Apply the revised wakeup rule only for deferrable servers that are
marked as running.
Fixes: 14a8570564 ("sched/deadline: Use revised wakeup rule for dl_server")
Signed-off-by: Gabriele Monaco <gmonaco@redhat.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Acked-by: Juri Lelli <juri.lelli@redhat.com>
Tested-by: Andrea Righi <arighi@nvidia.com>
Link: https://patch.msgid.link/20260522125833.264145-1-gmonaco@redhat.com
A page swapped in by __rb_get_reader_page_from_remote() retains its
stale read offset, causing subsequent reads to skip events or read
past valid data. Fix it.
Link: https://patch.msgid.link/20260729133609.4022734-1-vdonnefort@google.com
Fixes: fbd1743ecb ("ring-buffer: Add non-consuming read for ring-buffer remotes")
Signed-off-by: Vincent Donnefort <vdonnefort@google.com>
Reviewed-by: Keir Fraser <keirf@google.com>
Tested-by: Keir Fraser <keirf@google.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
`audit_del_rule()` destroys `e->rule.exe` via `audit_remove_mark_rule()`
before unlinking the rule from RCU-visible filter lists and waiting for a
grace period. Concurrent readers in `audit_filter()` and
`audit_filter_rules()` still dereference `e->rule.exe`, while the fsnotify
mark can be freed on an independent lifetime path. This creates a
use-after-free window during rule deletion.
Fix this by unlinking the rule from the RCU-visible lists and invoking
`synchronize_rcu()` before calling `audit_remove_mark_rule()` (and other
rule removal helpers). This ensures that all existing RCU readers have
exited the critical section before any underlying resources are destroyed.
Cc: stable@vger.kernel.org
Fixes: 34d99af52a ("audit: implement audit by executable")
Reported-by: Vega <vega@nebusec.ai>
Assisted-by: Codex:gpt-5.4
Signed-off-by: Luxiao Xu <rakukuip@gmail.com>
Signed-off-by: Ren Wei <enjou1224z@gmail.com>
Signed-off-by: Paul Moore <paul@paul-moore.com>
audit_log_n_string() computes new_len as "slen + 3" (enclosing quotes
plus the NUL terminator) and stores it into an int, while slen is a
size_t. For a sufficiently large slen the addition can overflow and/or
the result be truncated when assigned to the int new_len, so the
"new_len > avail" check can be bypassed and the subsequent
memcpy(ptr, string, slen) can write past the skb tail.
This is the same class of bug that was fixed for the hex sibling in
commit 65dfde57d1 ("audit: fix potential integer overflow in
audit_log_n_hex()"); both helpers are reached through
audit_log_n_untrustedstring() with the same length source.
Make new_len a size_t and use check_add_overflow() to catch the
overflow, mirroring the audit_log_n_hex() fix. No functional change for
the in-tree callers, which all pass bounded lengths.
Cc: stable@vger.kernel.org
Fixes: 168b717395 ("AUDIT: Clean up logging of untrusted strings")
Signed-off-by: Zhan Xusheng <zhanxusheng@xiaomi.com>
Signed-off-by: Paul Moore <paul@paul-moore.com>
regex_match_full() calls strncmp(str, r->pattern, len) where len is the
target field buffer size. When len is smaller than r->len (the filter
pattern length), strncmp() checks only len bytes of r->pattern against
str. If those len bytes match, strncmp() returns 0, resulting in a
false-positive match where a shorter string in a fixed-size field
matches a longer filter pattern.
For example, a 4-byte static string field containing "abcd" matched the
filter pattern "abcdefgh" because strncmp("abcd", "abcdefgh", 4)
returned 0. In this case, @len does NOT include '\0' because it is
fixed-size array.
Fix this by returning 0 (no match) early when len < r->len.
Fixes: 1889d20922 ("tracing/filters: Provide basic regex support")
Cc: stable@vger.kernel.org
Link: https://patch.msgid.link/178528488779.124250.5571741156199253769.stgit@devnote2
Assisted-by: Antigravity:gemini-3.5-flash
Signed-off-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
trace_module_add_events() ignores the return value of __register_event()
and unconditionally calls __add_event_to_tracers() for each event.
If __register_event() fails (for example, if event_init() fails), the
trace_event_call is not added to ftrace_events list, but
__add_event_to_tracers() still creates a trace_event_file pointing to it.
If module loading subsequently fails and module memory is freed, tracing
state retains a stale trace_event_call pointer in trace_event_file,
leading to a use-after-free when tracefs or tracing subsystem operations
are later executed.
Fix this by checking the return value of __register_event() and only
calling __add_event_to_tracers() if event registration succeeded.
Fixes: ae63b31e4d ("tracing: Separate out trace events from global variables")
Cc: stable@vger.kernel.org
Link: https://patch.msgid.link/178528487878.124250.14170824576025743236.stgit@devnote2
Assisted-by: Antigravity:gemini-3.5-flash
Signed-off-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
mmio_trace_rw() and mmio_trace_mapping() retrieve mmio_trace_array into
tr and pass it to __trace_mmiotrace_rw() and __trace_mmiotrace_map().
If these functions are invoked while mmio_trace_array is NULL (e.g. before
initialization or after disabled), accessing tr->array_buffer.buffer will
result in a NULL pointer dereference crash.
Fix this by adding an explicit NULL check for tr at the beginning of
__trace_mmiotrace_rw() and __trace_mmiotrace_map().
Link: https://patch.msgid.link/178524300062.56416.8362487250709962380.stgit@devnote2
Fixes: f984b51e07 ("ftrace: add mmiotrace plugin")
Assisted-by: Antigravity:gemini-3.6-flash
Signed-off-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
mmio_reset_data() is called during tracer initialization, reset, and
start. While it resets overrun_detected and prev_overruns, it neglects
to reset dropped_count. Consequently, dropped event counts from prior
tracing sessions persist in dropped_count and corrupt overrun reports
in subsequent runs.
Fix this by explicitly calling atomic_set(&dropped_count, 0) in
mmio_reset_data().
Link: https://patch.msgid.link/178524299122.56416.16277704230639425172.stgit@devnote2
Fixes: 173ed24ee2 ("mmiotrace: count events lost due to not recording")
Assisted-by: Antigravity:gemini-3.6-flash
Signed-off-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
In register_fprobe(), get_ips_from_filter() resolves target function
addresses and increments module reference counts via try_module_get() for
symbols in kernel modules. If get_ips_from_filter() fails on the second
pass and returns an error, register_fprobe() returned directly without
releasing module references acquired up to that point.
Fix this by ensuring the cleanup loop executing module_put() runs even when
get_ips_from_filter() returns a negative error.
Link: https://lore.kernel.org/all/178528125360.101985.4144133640239273153.stgit@devnote2/
Fixes: d24fa977ee ("tracing: fprobe: Fix to lock module while registering fprobe")
Assisted-by: Antigravity:gemini-3.6-flash
Signed-off-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
The futex address must be naturally aligned and this is checked via
"address % size" where `address' is the supplied address and `size' is
the expected size of futex. It is guaranteed that `size' is power of two
but the compiler does not see it and creates here a `div' operation
(x86, arm, gcc-15).
We can take advantage of the pow2 property and rewrite it as
"address & (size-1)".
As per testing, the command
|perf bench futex hash -f 1 -b 16384 -t 1 -r 30
improved from
| [thread 0] futex: 0x5619f931f740 [ 7001583 ops/sec ]
to
| [thread 0] futex: 0x55da173e5740 [ 7376137 ops/sec ]
or by 5.3%
Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/20260701161736.xYYizA0e@linutronix.de
Runtime constify the read-only after init data __futex_shift(shift_32),
__futex_mask(mask_32), and __futex_queues(ptr) used in __futex_hash()
hot path to avoid referencing global variable.
This also allows __futex_queues to be allocated dynamically to
"nr_node_ids" slots instead of reserving config dependent MAX_NUMNODES
(1 << CONFIG_NODES_SHIFT) worth of slots upfront.
Runtime constants are initialized before their first access and
runtime_const_init() provides necessary barrier to ensure subsequent
accesses are not reordered against their initialization.
No functional changes intended.
perf bench futex on a 3rd Gen EPYC (2 x 64C/128T):
+----------------+-----------+-----------+-----------+--------------+
| Benchmark | Kernel 1 | Kernel 2 | Unit | % Improvement|
| | (avg/5) | (avg/5) | | (K2 vs K1) |
+----------------+-----------+-----------+-----------+--------------+
| Wake-parallel | 0.01614 | 0.00456 | ms | +71.75% |
| Requeue | 0.26394 | 0.24644 | ms | +6.63% |
| Lock-pi | 34.0 | 57.2 | ops/sec | +68.24% |
+----------------+-----------+-----------+-----------+--------------+
Performance testing on a 144-thread Intel(R) Xeon(R) CPU E7-8890 v3 (4 NUMA nodes):
+-------------------------------------------------------------+
| perf bench futex hash -b 0 |
+----------------------+------------+------------+------------+
| Configuration | As-is | Patched | Delta |
+----------------------+------------+------------+------------+
| 1 thread, 1 futex | 6,449,632 | 6,532,004 | +1.28% |
| 144 threads, 1024 fx | 2,111,486 | 2,139,685 | +1.34% |
+----------------------+------------+------------+------------+i
[ prateek: Dynamically allocate __futex_queues, mark the global data
__ro_after_init since they are constified after futex_init(). ]
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reported-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de> # MAX_NUMNODES bloat
Signed-off-by: K Prateek Nayak <kprateek.nayak@amd.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Charlie Jenkins <thecharlesjenkins@gmail.com>
Tested-by: Charlie Jenkins <thecharlesjenkins@gmail.com>
Link: https://patch.msgid.link/20260227161841.GH606826@noisy.programming.kicks-ass.net
Link: https://patch.msgid.link/20260728052540.4728-9-kprateek.nayak@amd.com
kho_preserved_memory_reserve() calculates the size of a preservation by
doing 1 << (order + PAGE_SHIFT). Since the '1' is a 32-bit integer, it
can only be shifted by 31. That is, it will only work for preservations
up to 2 GiB. Larger preservations will trigger undefined behaviour.
While preservations larger than 2 GiB can't be obtained via folios
currently, they can be obtained via kho_preserve_pages().
For example, memblock reserve_mem uses kho_preserve_pages().
Reservations larger than 2 GiB are valid and will trigger this bug if
properly aligned.
Fix it by using 1UL for shifting.
Fixes: fc33e4b44b ("kexec: enable KHO support for memory preservation")
Reported-by: Sashiko <sashiko-bot@kernel.org>
Cc: stable@vger.kernel.org
Signed-off-by: Pratyush Yadav (Google) <pratyush@kernel.org>
Link: https://patch.msgid.link/20260727150240.889555-1-pratyush@kernel.org
Signed-off-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
The nodemasks in these structures may come from a variety of sources,
including tasks and cpusets - and should never be modified by any code
when being passed around inside another context.
Link: https://lore.kernel.org/20260609002919.3967782-1-gourry@gourry.net
Signed-off-by: Gregory Price <gourry@gourry.net>
Reviewed-by: Lorenzo Stoakes <ljs@kernel.org>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Reviewed-by: Barry Song <baohua@kernel.org>
Acked-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Tested-by: SeongJae Park <sj@kernel.org>
Acked-by: SeongJae Park <sj@kernel.org>
Acked-by: Waiman Long <longman@redhat.com>
Acked-by: Zi Yan <ziy@nvidia.com>
Cc: Axel Rasmussen <axelrasmussen@google.com>
Cc: Baoquan He <baoquan.he@linux.dev>
Cc: Brendan Jackman <jackmanb@google.com>
Cc: Chris Li <chrisl@kernel.org>
Cc: David Rientjes <rientjes@google.com>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Kairui Song <kasong@tencent.com>
Cc: Kemeng Shi <shikemeng@huaweicloud.com>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Michal Koutný <mkoutny@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Nhat Pham <nphamcs@gmail.com>
Cc: Shakeel Butt <shakeel.butt@linux.dev>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Tejun Heo <tj@kernel.org>
Cc: Wei Xu <weixugc@google.com>
Cc: Yuanchu Xie <yuanchu@google.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
The BUG_ON(axp->pid_count >= AUDIT_AUX_PIDS) check in
audit_signal_info_syscall() will panic the kernel if the signal target
pid array overflows, which is too severe for this situation.
Replace it with a WARN_ON_ONCE() and return of -EINVAL, instead.
Signed-off-by: Ricardo Robaina <rrobaina@redhat.com>
Signed-off-by: Paul Moore <paul@paul-moore.com>
The BUG_ON(!mutex_is_locked(&audit_filter_mutex)) check in
audit_add_to_parent() will panic the kernel if the mutex is not held,
which is too severe for this situation.
Replace it with lockdep_assert_held(), instead.
Signed-off-by: Ricardo Robaina <rrobaina@redhat.com>
Signed-off-by: Paul Moore <paul@paul-moore.com>
enable_trace_fprobe() sets the file link or the TP_FLAG_PROFILE flag and
then registers each trace_fprobe in the probe list. If
__register_trace_fprobe() fails partway through, the function returns
immediately without unregistering the trace_fprobes it already registered
or undoing the file link / flag it set, leaving the event half-enabled and
leaking the registered fprobe(s).
enable_trace_kprobe() already handles this with a rollback path. Do the
same for fprobe: on failure, unregister all probes and clear the file link
or profile flag.
Link: https://lore.kernel.org/all/20260724064208.480030-1-raushan.jhon@gmail.com/
Fixes: 334e5519c3 ("tracing/probes: Add fprobe events for tracing function entry and exit.")
Cc: stable@vger.kernel.org
Signed-off-by: Raushan Patel <raushan.jhon@gmail.com>
Signed-off-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
traceprobe_expand_meta_args() parses $argN with simple_strtoul() and
calls sprint_nth_btf_arg(n - 1, ...). For $arg0, n is 0 so the index is
-1. Because ctx->nr_params is signed, the "idx >= nr_params" guard in
sprint_nth_btf_arg() does not catch the negative index, and
ctx->params[-1].name_off is read out of bounds.
The normal per-argument path (parse_probe_vars()) already rejects
$arg0 via its argument-number check, but meta-argument expansion runs
before per-argument parsing and substitutes the value first, bypassing
that check.
Reject $arg0 explicitly during expansion.
Link: https://lore.kernel.org/all/20260724054435.146279-1-raushan.jhon@gmail.com/
Fixes: 18b1e870a4 ("tracing/probes: Add $arg* meta argument for all function args")
Cc: stable@vger.kernel.org
Signed-off-by: Raushan Patel <raushan.jhon@gmail.com>
Signed-off-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
MAX_COMMON_HEAD_LEN (63) was used to allocate a temporary buffer for
formatting command heads in trace_kprobe_match_command_head() and
trace_uprobe_match_command_head(). However, the buffer size is too
short for some longer symbols. Especially, with rust code, the symbol
can be mangled and become very long.
Refactor trace_kprobe_match_command_head() to perform direct string
comparisons using strcmp() and strncmp(), eliminating the need for a
temporary buffer and removing the MAX_COMMON_HEAD_LEN string length
restriction on probe symbol names.
For trace_uprobe_match_command_head(), since tu->filename is already
matched via strncmp(), use a fixed 64-byte stack buffer solely for
formatting offset and ref_ctr_offset (which requires at most 39 bytes).
With all users converted, remove the MAX_COMMON_HEAD_LEN definition from
trace_probe.h.
Link: https://lore.kernel.org/all/178521361102.34226.9650586522488974115.stgit@devnote2/
Reported-by: Zhan Xusheng <zhanxusheng1024@gmail.com>
Link: https://lore.kernel.org/all/20260724023317.624074-1-zhanxusheng@xiaomi.com/
Signed-off-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
In preparation for removing the strlcat API[1],
replace the string concatenation logic with a struct seq_buf,
which tracks the current position and the remaining space internally.
Use seq_buf_str() to NUL-terminate before passing to early_enable_events().
Link: https://github.com/KSPP/linux/issues/370 [1]
Link: https://patch.msgid.link/20260713045249.69942-1-woradorn.laon@gmail.com
Signed-off-by: Woradorn Laodhanadhaworn <woradorn.laon@gmail.com>
[ Moved placement of #include <linux/seq_buf.h> ]
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
Replace strcpy() with strscpy() in __trace_find_cmdline() for
consistency with the existing strscpy() call in the same function,
and to avoid potential buffer overflow as flagged by the Kernel
Self Protection Project.
Link: https://patch.msgid.link/20260705173648.5418-1-posheng.lin.tw@gmail.com
Signed-off-by: Po-Sheng Lin <posheng.lin.tw@gmail.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
Currently on boot up and when modules are loaded, the trace event
infrastructure will examine the TP_printk's of every event looking to see
if it dereferences pointers on the ring buffer via printk formats like
"%pB" and such. What it doesn't do is check if the arguments themselves
do a dereference from a pointer.
This was brought with a fix[1] to the fsl_edma event that had in the
arguments of the TP_printk(): "__entry->edma->membase"
The __entry->edma is a pointer saved in the ring buffer. The dereference
from TP_printk() happens when the user reads the "trace" file which can be
seconds, minutes, hours, days, weeks, or even months later! There is no
guarantee that the __entry->edma pointer will still be pointing to what it
was when it was recorded, and could crash the kernel when a user reads the
event.
Add logic to the test_event_printk() that also checks for this case and
warn if the event dereferences a pointer from the ring buffer.
[1] https://lore.kernel.org/all/20260630200022.1826420-1-martin@kaiser.cx/
Link: https://patch.msgid.link/20260630184836.74d477b6@gandalf.local.home
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
Reviewed-by: Martin Kaiser <martin@kaiser.cx>
Reviewed-by: Vinod Koul <vkoul@kernel.org>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
preemptirq_delay_test accepts cpu_affinity as a module parameter and,
when it is non-negative, writes that CPU directly into a temporary
cpumask from the worker thread. Values outside nr_cpu_ids can set a bit
outside the allocated cpumask before the test reports a normal affinity
error.
Validate the requested CPU in preemptirq_delay_run() before setting it
in the temporary cpumask. Invalid affinity requests are reported by
the test thread and skipped before cpumask_set_cpu() can touch an
out-of-range bit.
Link: https://patch.msgid.link/20260628131021.2208632.6a5c6c959813.preemptirq-delay-test-invalid-cpu-affinity@trailofbits.com
Assisted-by: Codex:gpt-5.5-cyber-preview
Signed-off-by: Samuel Moelius <sam.moelius@trailofbits.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
A single closing curly bracket should be put into a trace sequence buffer.
Thus use the corresponding function “trace_seq_putc”.
The source code was transformed by using the Coccinelle software.
Link: https://patch.msgid.link/d215fa89-9a62-4067-86ec-833290f35c80@web.de
Signed-off-by: Markus Elfring <elfring@users.sourceforge.net>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
A single string should be put into a sequence within a loop.
Thus use the corresponding function “seq_puts” for one selected call.
The source code was transformed by using the Coccinelle software.
Link: https://patch.msgid.link/1cf327f0-49a6-477f-a06f-2b22a167db24@web.de
Signed-off-by: Markus Elfring <elfring@users.sourceforge.net>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
Single characters should be put into a sequence. Thus use the corresponding
function “seq_putc” for selected calls.
The source code was transformed by using the Coccinelle software.
Link: https://patch.msgid.link/6bcaa4da-05c6-4097-90f5-3969f8a1dfbc@web.de
Signed-off-by: Markus Elfring <elfring@users.sourceforge.net>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
expr_str() allocates a fixed MAX_FILTER_STR_VAL buffer and then builds
expression names with a series of raw strcat() appends. Nested operands,
constants, field flags, and generated field names can push the rendered
string past that fixed limit before the name is attached to the hist
field.
Build expression strings with seq_buf and return -E2BIG when the
rendered name would exceed MAX_FILTER_STR_VAL. This keeps the existing
tracing-side limit while replacing the raw append logic with bounded
construction.
Link: https://patch.msgid.link/20260611055945.22348-4-pengpeng@iscas.ac.cn
Signed-off-by: Pengpeng Hou <pengpeng@iscas.ac.cn>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
expr_str() currently reports all failure cases as NULL, so callers cannot
distinguish invalid recursion depth from allocation failure or later
string construction errors.
Return ERR_PTR()-encoded errors from expr_str() and make parse_unary()
and parse_expr() propagate them. Clear expr->name before destroying the
hist field so the error pointer is not freed as a string.
Link: https://patch.msgid.link/20260611055945.22348-3-pengpeng@iscas.ac.cn
Signed-off-by: Pengpeng Hou <pengpeng@iscas.ac.cn>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
expr_str() allocates a temporary expression buffer and manually frees it
on some error paths.
Convert the buffer to __free(kfree) and return it with return_ptr() on
success. This keeps ownership handling separate from the later ERR_PTR()
conversion and string-bound change.
Link: https://patch.msgid.link/20260611055945.22348-2-pengpeng@iscas.ac.cn
Signed-off-by: Pengpeng Hou <pengpeng@iscas.ac.cn>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
The HIST_FIELD_FL_CONST path uses the fixed "u64" type string.
Point hist_field->type directly to the string literal, matching the
HIST_FIELD_FL_HITCOUNT path. The release path already uses kfree_const(),
so no duplication is needed.
Link: https://patch.msgid.link/20260527023450.2137639-1-pengyu@kylinos.cn
Signed-off-by: Yu Peng <pengyu@kylinos.cn>
Acked-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
Add events/<sys>/<event>/btf_ids, a per-template file that exposes
the BTF ids resolve_btfids fills in for each tracepoint:
btf_obj_id BTF object owning the ids below
raw_btf_id FUNC_PROTO of __bpf_trace_<call> (named args), consumed
by raw_tp / tp_btf BPF programs
tp_btf_id trace_event_raw_<call> ring-buffer record, consumed by
classic BPF_PROG_TYPE_TRACEPOINT programs
DECLARE_EVENT_CLASS now emits a 2-entry BTF_ID_LIST (FUNC __bpf_trace_*
and STRUCT trace_event_raw_*) and stores the pointer in
trace_event_class.
Per-syscall events under syscalls/ share the handcrafted classes
event_class_syscall_{enter,exit} instead of going through
DECLARE_EVENT_CLASS. Wire those classes to the BTF id lists
generated for sys_enter / sys_exit so all ~700 per-syscall
events expose the shared dispatcher prototype and record.
The per-syscall events do not own their own tracepoint
(they share sys_enter/sys_exit), so raw_btf_id is reported as 0
on those events; the meaningful raw_btf_id is exposed on
raw_syscalls/sys_{enter,exit}/btf_ids where raw_tp / tp_btf
programs can actually attach.
Link: https://patch.msgid.link/20260518-generic_tracepoint-v2-2-b755a5cf67bb@meta.com
Signed-off-by: Mykyta Yatsenko <yatsenko@meta.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
Drop the static qualifier and add prototypes to <linux/btf.h> so the
tracing core can look up module BTF and translate ids stored by
resolve_btfids (which are local to a module's split BTF) into the
runtime ids used by the kernel.
Used by the upcoming events/<sys>/<event>/btf_ids tracefs interface.
Link: https://patch.msgid.link/20260518-generic_tracepoint-v2-1-b755a5cf67bb@meta.com
Signed-off-by: Mykyta Yatsenko <yatsenko@meta.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
- new driver mm81x for an S1G device
- new driver nxpwifi for NXP devices
(mostly forked off from mwifiex)
- ath12k: much kernel infrastructure integration work
- brcmfmac: DPP support, some Cypress part update
- nl80211: per-link statistics support
-----BEGIN PGP SIGNATURE-----
iQIzBAABCgAdFiEEpeA8sTs3M8SN2hR410qiO8sPaAAFAmpl5OQACgkQ10qiO8sP
aAC64g//azU0JD8z/CLmEfPiG4sHQL7WvtkKKkkdxt5kle9wcjRh/8DDhGgF8Qb1
Knq90Ees6wJAhZkGVKQW4cf3yWXSzp7sdg/M3bQQeelCJo4Di0xiKDS8Rjxplyh0
q/ukqck7qvzYsMvnOvmnsJcpZ6kIvICC1Xi3/KMok9h3gtUHdJqhc1tEZz1Xt7Kl
5qtEQWaMTcrQjf/RtGaesD3ai/asxjUBCOqIdlPM0FktQuDZ7nyZOYlT1pQi3ReN
4tODKGYNG8EkNi4p7Pfo4miZsfx13ZwSs/qGaSeFSddUWcttqLf1oe7O5BvPOjoV
nN33NIurdiBGky5sV8krjPar3C2OCoqdi8FASg3BJZYgtd3B950267ShsVn6Mp0B
yTbo90JqX86PIGDkzaMosNkc7OFSfHmh78tuSeiM0FV30TbnjQEasLZdx/kZ9CP2
01A8zurfL8Q81tZNiywzIAnD4PhIzi8zvG/nqJGWukizAGiZhUEwbfgXvd8vEOns
njoprYVBETT8awkV05vgiJhGxDf88kpD58s/8g36XNys06gHzXM46/IspNVZQu+9
qOxdHRL14B+ANdJirRgeJIZj6Wll7pNx+aqPndI6h11ezqfzR8mDYrjB0Kl38z8u
BOeZ7FE51AtwjLMnMofgZCZoAkJqA/y8n1WrGX7lUtS2uqDIBVw=
=eCJ1
-----END PGP SIGNATURE-----
Merge tag 'wireless-2026-07-26' of https://git.kernel.org/pub/scm/linux/kernel/git/wireless/wireless-next
Johannes Berg says:
====================
wireless-next-2026-07-26
Mostly driver changes this time:
- new driver mm81x for an S1G device
- new driver nxpwifi for NXP devices
(mostly forked off from mwifiex)
- ath12k: much kernel infrastructure integration work
- brcmfmac: DPP support, some Cypress part update
- nl80211: per-link statistics support
====================
Link: https://patch.msgid.link/20260726105205.942922-60-johannes@sipsolutions.net
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
There is no particular reason to keep bpf_res_spin_lock() disabled in
tracing programs, since it is safe against reentrancy and deadlocks.
Remove the restriction for tracing programs covered by the predicate
is_tracing_prog_type().
This is a prerequisite before the definition of is_tracing_prog_type()
is updated to include raw_tp, fentry, fexit, and fmod_ret. Existing
tracing programs will be updated to use bpf_res_spin_lock() instead when
it is available.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Reviewed-by: Emil Tsalapatis <emil@etsalapatis.com>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://lore.kernel.org/bpf/20260719113551.1294284-2-memxor@gmail.com
scx_link_sched() carries each failure out of the locked section through
err_msg and ret because scx_error() used to take scx_sched_lock and couldn't
be called under it. That restriction is gone, so report each failure at the
site it's detected and return directly. The scx_error() here claims the exit
on the sched being linked, which has no descendants yet, and the locked
propagation walk is deferred, so nothing reacquires scx_sched_lock inline.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
scx_hardlockup() defers the abort to an irq_work because exit claiming used
to take scx_sched_lock and couldn't run from NMI. The deferral is now
unnecessary - claiming is NMI-safe and asserting ->aborting is exactly what
breaks the live-locks that hard-lock CPUs. Call handle_lockup() directly and
drop the irq_work. This also makes the self-detected case recoverable: the
perf watchdog fires on the hard-locked CPU itself, where a queued irq_work
never runs with IRQs off.
Also fix the return value: %true used to be returned whenever sched_ext was
loaded, suppressing the kernel's hardlockup report even when the abort was
refused. Return %true only when this call initiated the abort.
Fixes: bd2d76455b ("sched_ext: Defer scx_hardlockup() out of NMI")
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
The per-cpu kick lists are protected by IRQ masking which doesn't stop NMIs,
so scx_bpf_kick_cpu() from NMI silently drops the kick after a one-time
warning. A dropped kick can leave a CPU idle when the scheduler believes it
was woken, which is a correctness problem for the scheduler even if the
kernel is fine. Now that scx_error() works from NMI, abort the scheduler
instead so that the bug is surfaced deterministically. The warned_nmi_kick
tracking is no longer needed.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
The bstr exit kfuncs format the message into a shared static buffer under a
raw spinlock before initiating the exit. The lock can't be taken from NMI
and needlessly serializes all bstr exits system-wide.
Now that exit claiming is lock-free, reverse the order: claim the exit first
and format directly into the exit_info message buffer which the claim winner
owns exclusively. The new scx_exit_bstr() implements the sequence, replacing
scx_bstr_format(), and the shared buffer and lock are deleted; the formatter
itself is what bpf_trace_printk() already runs from NMI. scx_prog_sched()
callers were relying on the lock for RCU protection, which is now provided
explicitly.
A malformed format no longer changes or fails the requested operation:
scx_bpf_exit_bstr() keeps its graceful exit kind and scx_bpf_sub_kill_bstr()
still kills the child, with a fallback message carrying the formatting
errno, while the sched that supplied the bad format is aborted for its bug.
Before this and the previous patch, an "any" category kfunc called from NMI
context could trigger scx_error() and deadlock - e.g. a tracing prog
attached to a function running in NMI calling scx_bpf_dsq_peek() on a
non-existent DSQ would try to grab scx_sched_lock, which may be held by the
interrupted CPU. This and the previous patch fix the deadlock: scx_error()
and the bstr exit kfuncs, and thus scx_bpf_error() and scx_bpf_exit(), are
now safe to call from any context including NMI.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
scx_claim_exit() claims descendants' exits by walking the subtree under
scx_sched_lock, making exit claiming, and thus scx_error(), unusable from
NMI and from under scx_sched_lock. However, kfuncs raising errors can run
from NMI-attached BPF progs, the hardlockup handler runs in NMI, and
scx_link_sched() wants to report failures under the lock.
The walk does two things with different urgencies: ->aborting must be
asserted synchronously to break IRQs-off dispatch-path live-locks, while the
descendants' exit_kind claims can happen later. Split them: sweep ->aborting
locklessly under RCU to unwedge the system and defer the locked
SCX_EXIT_PARENT walk to a new irq_work, both of which are NMI-safe.
The sweep stores each node's ->aborting and then reads its children list
while scx_link_sched() inserts and then checks the parent's ->aborting, the
two sides paired by full barriers - one side always sees the other. A link
that sees ->aborting undoes its insert and fails. As the undo's
list_del_rcu() leaves ->sibling non-empty, list_empty() can no longer
identify a never-linked sched during teardown - add sch->linked instead.
trace_sched_ext_exit can now fire from NMI and is called after the
->aborting stores so that its callbacks don't hold up live-lock recovery.
The exit backtrace is skipped for NMI exits as stack_trace_save()'s
NMI-safety is arch-dependent and undocumented.
v2: Move trace_sched_ext_exit() after the ->aborting stores (Andrea).
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
New HW no longer implements a separate class for latency-sensitive QPs
and advertises this by a new cap bit.
Signed-off-by: Leon Romanovsky <leon@kernel.org>
For when a device is not DMA capable, the max mapping size would be 0, so
make dma_max_mapping_size() reflect that.
Signed-off-by: John Garry <john.g.garry@oracle.com>
Reviewed-by: Robin Murphy <robin.murphy@arm.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Link: https://patch.msgid.link/20260629085310.2298552-2-john.g.garry@oracle.com
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
Unlike local reenqueues, cap rejections have no repeat limit. A
malfunctioning scheduler can keep re-inserting a task to a cid it lacks caps
on, cycling the task through reject and reenqueue. This was assumed safe
because a task that never runs trips the stall watchdog. However, the
reenqueue irq_work re-arms itself and outranks the timer vector, blocking
everything else on the CPU including stall detection and recovery, until the
NMI hardlockup detector fires.
Local reenqueues already have a repeat cap, SCX_REENQ_LOCAL_MAX_REPEAT,
which needs generalizing to cover all reenqueues. It also has an attribution
problem. Counted per-cpu on root, it tears down the whole hierarchy even
when a sub-scheduler caused the repeated reenqueues.
Generalize by bounding every reenqueue with one per-task counter. reenq_cnt
is bumped in scx_do_enqueue_task() on each SCX_ENQ_REENQ, the single path
every reenqueue producer passes through, and cleared in clr_task_runnable()
when the task is picked to run and in scx_disable_task() when it leaves the
scheduler's control. Past SCX_REENQ_MAX_REPEAT the task's owning scheduler
is ejected with a new SCX_EXIT_ERROR_REENQ and the task is left stranded to
be picked up during sched exit.
The SCX_EV_REENQ_LOCAL_REPEAT event becomes SCX_EV_REENQ_REPEAT, counting
repeat reenqueues from all sources.
v2: Count SCX_EV_REENQ_REPEAT only when a reenqueue leads to another
reenqueue, not on every reenqueue.
v3: - Also clear reenq_cnt in scx_disable_task() so that the count doesn't
carry over to the next owner across sched class switches, scheduler
replacement or sub-scheduler rehoming (Andrea Righi).
- Update the stale SCX_EV_REENQ_LOCAL_REPEAT references in sched-ext.rst
(Andrea Righi).
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
- Move rb_desc->nr_page_va before updating dynamic array
The rb_descr->page_va is a dynamic array counted by nr_page_va. But the
updating of the page_va[] is done before the nr_page_va is incremented
causing a build with CONFIG_UBSAN_BOUNDS to flag it as an overflow.
Move the increment of the counted by value before the array element is
updated.
- Propagate errors from remote event bulk updates
The return value of trace_remote_enable_event() was not being checked by
remote_events_dir_enable_write() where it would silently fail. Have it
check the return value and propagate that back up to user space.
- Fix resource leak on mmiotrace trace_pipe close
The mmiotrace tracer was created in 2008 before the trace_pipe had a close
callback to allow tracers to do clean up from trace_pipe open. The
trace_pipe close cleanup callback was added in 2009 but the mmiotrace
tracer was not updated. It had a hack to do the cleanup in the read call,
where it may leak if user space did not read the entire buffer.
Add a callback to mmiotrace trace_pipe close do to the cleanup properly.
- Fix a possible NULL pointer dereference in the mmiotrace tracer
If the mmio_pipe_open() fails to find a PCI device, it will set the
hiter->dev pointer to NULL. The read function will blindly dereference
that pointer. Fix the read call to check to see if that pointer is
populated before dereferencing it.
- Fix union collision of module and refcnt for dynamic events
In 'struct trace_event_call', the 'module' pointer and the 'refcnt' atomic
variable share the same memory space in a union. The filter on module
logic only checked if the 'module' was set to determine if the event
belonged to the module. As dynamic events are always builtin, it doesn't
need the 'module' field of the structure and used a refcount. But the
module filtering logic would then mistaken these dynamic events as a
module and call module_name(event->module) on it.
Add a check to see if the event is a dynamic event and if so, do not check
it for being part of the given module.
- Reset the top level buffer in selftests before running instances
The ftracetest selftest initializes each instance before executing the
tests. But it does not reset the top level buffer. Dynamic events are only
added and removed by the top level so any left over dynamic events will
not be removed by the reset in the instances.
Left over dynamic events can cause the tests to incorrectly fail. Reset
the top level buffer before running the instances.
- Make the context_switch counter 64 bit
The code to read user space for a system call trace event or for a
trace_marker will disable migration, enable preemption, read user space
into a per CPU buffer, disable preemption and enable migration again.
It checks if the per CPU context switch counter to see if it changed, and
if it did not, it would know that the per CPU buffer was not touched by
another task.
But the save counter was 32 bit and it would compare it to the 64 bit
context_switch variable. A long running system could have the
context_switch variable greater that 1<<32 in which case the compare will
always fail. The compare will promote the 32 bit int saved value to 64 bit
and compare it to the full 64 bit counter. Since the top 32 bits of the
saved value was zero, it would never match.
- Fix a use-after-free of the event_enable trigger
The event_enable trigger allows for enabling one event when another event
is triggered. When the trigger is removed, it must go through a
synchronization phase to make sure it is not triggered again. The trigger
itself is delayed by the "bulk delay" logic that was recently added.
But the code that frees the event_enable data used to rely on the trigger
code to do the synchronization. Now that the code uses the call RCU
functions (and a workqueue), that delay no longer is there.
Add a callback private_data_free() function that allows triggers to clean
up data after the synchronization phase has completed.
- Move the module_ref counter into the delay callback
Since an event of the event_enable trigger can enable an event for a
module, it ups the module ref count for that event's module. This prevents
the event from trying to enable an event that no longer exists and cause a
use-after-free bug.
The ref counter was set back down when the trigger was removed but not
after thy synchronization phase. This could lead to the module data being
accessed after module was unloaded.
Move the module ref decrement into the private_data_free() callback of the
event_enable trigger.
- Add mutex to protect parser in ftrace filtering
The set_ftrace_filter file uses a parsing descriptor that is allocated at
open and modified by writes. If multiple threads were to write to the
descriptor at the same time, it can corrupt the parser.
Add a mutex around the modifications of the parser descriptor.
- Fix possible corruption in perf syscall tracing
The perf system call trace events can now read user space. To do so, the
reads of user space enable preemption and disables it again. During this
time that preemption is enabled, the task can migrate. The perf event list
head is assigned via a per CPU pointer. It is done before the user space
part is called. If the user space reading migrates the task to another
CPU, then the head pointer is no longer valid.
Re-assign the head pointer after the reading of user space to keep it
using the correct data.
-----BEGIN PGP SIGNATURE-----
iIoEABYKADIWIQRRSw7ePDh/lE+zeZMp5XQQmuv6qgUCamYDGhQccm9zdGVkdEBn
b29kbWlzLm9yZwAKCRAp5XQQmuv6qkhYAP9G5wDTVQQzitl900iWp9rvQ2Qm5UWN
JDnK2HO1elmj0AD8CRiHBI5W3O2yUmoO4bOFZ9YFXz+DqJ1jwkDs5FfqjAU=
=EMD9
-----END PGP SIGNATURE-----
Merge tag 'trace-v7.2-rc4' of git://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace
Pull tracing fixes from Steven Rostedt:
- Move rb_desc->nr_page_va before updating dynamic array
The rb_descr->page_va is a dynamic array counted by nr_page_va. But
the updating of the page_va[] is done before the nr_page_va is
incremented causing a build with CONFIG_UBSAN_BOUNDS to flag it as an
overflow.
Move the increment of the counted by value before the array element
is updated.
- Propagate errors from remote event bulk updates
The return value of trace_remote_enable_event() was not being checked
by remote_events_dir_enable_write() where it would silently fail.
Have it check the return value and propagate that back up to user
space.
- Fix resource leak on mmiotrace trace_pipe close
The mmiotrace tracer was created in 2008 before the trace_pipe had a
close callback to allow tracers to do clean up from trace_pipe open.
The trace_pipe close cleanup callback was added in 2009 but the
mmiotrace tracer was not updated. It had a hack to do the cleanup in
the read call, where it may leak if user space did not read the
entire buffer.
Add a callback to mmiotrace trace_pipe close do to the cleanup
properly.
- Fix a possible NULL pointer dereference in the mmiotrace tracer
If the mmio_pipe_open() fails to find a PCI device, it will set the
hiter->dev pointer to NULL. The read function will blindly
dereference that pointer. Fix the read call to check to see if that
pointer is populated before dereferencing it.
- Fix union collision of module and refcnt for dynamic events
In 'struct trace_event_call', the 'module' pointer and the 'refcnt'
atomic variable share the same memory space in a union. The filter on
module logic only checked if the 'module' was set to determine if the
event belonged to the module. As dynamic events are always builtin,
it doesn't need the 'module' field of the structure and used a
refcount. But the module filtering logic would then mistaken these
dynamic events as a module and call module_name(event->module) on it.
Add a check to see if the event is a dynamic event and if so, do not
check it for being part of the given module.
- Reset the top level buffer in selftests before running instances
The ftracetest selftest initializes each instance before executing
the tests. But it does not reset the top level buffer. Dynamic events
are only added and removed by the top level so any left over dynamic
events will not be removed by the reset in the instances.
Left over dynamic events can cause the tests to incorrectly fail.
Reset the top level buffer before running the instances.
- Make the context_switch counter 64 bit
The code to read user space for a system call trace event or for a
trace_marker will disable migration, enable preemption, read user
space into a per CPU buffer, disable preemption and enable migration
again. It checks if the per CPU context switch counter to see if it
changed, and if it did not, it would know that the per CPU buffer was
not touched by another task.
But the save counter was 32 bit and it would compare it to the 64 bit
context_switch variable. A long running system could have the
context_switch variable greater that 1<<32 in which case the compare
will always fail. The compare will promote the 32 bit int saved value
to 64 bit and compare it to the full 64 bit counter. Since the top 32
bits of the saved value was zero, it would never match.
- Fix a use-after-free of the event_enable trigger
The event_enable trigger allows for enabling one event when another
event is triggered. When the trigger is removed, it must go through a
synchronization phase to make sure it is not triggered again. The
trigger itself is delayed by the "bulk delay" logic that was recently
added. But the code that frees the event_enable data used to rely on
the trigger code to do the synchronization. Now that the code uses
the call RCU functions (and a workqueue), that delay no longer is
there.
Add a callback private_data_free() function that allows triggers to
clean up data after the synchronization phase has completed.
- Move the module_ref counter into the delay callback
Since an event of the event_enable trigger can enable an event for a
module, it ups the module ref count for that event's module. This
prevents the event from trying to enable an event that no longer
exists and cause a use-after-free bug.
The ref counter was set back down when the trigger was removed but
not after thy synchronization phase. This could lead to the module
data being accessed after module was unloaded.
Move the module ref decrement into the private_data_free() callback
of the event_enable trigger.
- Add mutex to protect parser in ftrace filtering
The set_ftrace_filter file uses a parsing descriptor that is
allocated at open and modified by writes. If multiple threads were to
write to the descriptor at the same time, it can corrupt the parser.
Add a mutex around the modifications of the parser descriptor.
- Fix possible corruption in perf syscall tracing
The perf system call trace events can now read user space. To do so,
the reads of user space enable preemption and disables it again.
During this time that preemption is enabled, the task can migrate.
The perf event list head is assigned via a per CPU pointer. It is
done before the user space part is called. If the user space reading
migrates the task to another CPU, then the head pointer is no longer
valid.
Re-assign the head pointer after the reading of user space to keep it
using the correct data.
* tag 'trace-v7.2-rc4' of git://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace:
tracing: perf: Fix stale head for perf syscall tracing
ftrace: Add global mutex to serialize trace_parser access
tracing: Delay module ref count for "enable_event" trigger
tracing: Fix use-after-free freeing trigger private data
tracing: Fix context switch counter truncation
selftests/ftrace: Reset triggers at top level before instance loop
tracing: Fix union collision of module and refcnt for dynamic events
tracing: Fix mmiotrace possible NULL dereferencing of hiter->dev
tracing: Fix resource leak on mmiotrace trace_pipe close
tracing: Propagate errors from remote event bulk updates
tracing/remotes: Fix page_va[] access before counter update in trace_remote_alloc_buffer()
RDMA device names are unique only within a network namespace, but an
RDMA cgroup can account resources for devices from multiple namespaces.
Duplicate names therefore make cgroup output ambiguous and can cause
limit writes to select the wrong device.
Use the system-wide RDMA device index to distinguish duplicate names
while preserving the existing UAPI for unique names. Reject ambiguous
name-only writes with -ENOTUNIQ and expose a complete device view to
administrators.
Signed-off-by: Jiri Pirko <jiri@nvidia.com>
Link: https://patch.msgid.link/20260716132316.1495242-9-jiri@resnulli.us
Signed-off-by: Leon Romanovsky <leon@kernel.org>
dsq->first_task is __rcu for the lockless scx_bpf_dsq_peek(). The task
removal path compares it against the departing task with a plain load, which
sparse flags. The comparison runs under the dsq lock and only tests
identity, so rcu_access_pointer() is the fit.
Signed-off-by: Tejun Heo <tj@kernel.org>
scx_root is __rcu and naked accesses were left as transitional markers for
the multi-scheduler transition, to be converted to accesses through the
associated scheduler instances. Most accesses have since been converted to
resolve the sched from the program or task at hand. The remaining naked
sites divide into ones that semantically always want the root sched, which
this patch resolves, and one that is left to a later patch.
The resolved sites:
- The SCX_OPS_TID_TO_TASK validation and the ecaps sync kick already hold a
sched whose ancestors[] pins the root as entry 0 with plain pointers
stable for the sched's lifetime. Reach the root through the sched at hand.
- The dispatch entry, class switch, idle notification and fork init paths
only execute while the scheduler is live and scx_root never changes inside
the live window, so no update can race them. Add scx_root_protected_live()
which documents that invariant and resolves with a plain load.
- The hotplug path, including the ecaps reseeds, runs with the hotplug lock
held, which excludes the scx_root writers. Add scx_root_protected(), which
accepts either the hotplug lock or scx_enable_mutex.
- Is-root tests use a zero level instead of comparing against the global.
touch_core_sched_dispatch() stays naked, to be resolved by a later patch.
Signed-off-by: Tejun Heo <tj@kernel.org>
cgrp->scx_sched is __rcu and published with rcu_assign_pointer() but every
reader loads it with a plain access, so sparse flags all of them. The reads
are lock-protected: enable/disable paths rewrite the field under all of
scx_enable_mutex, scx_fork_rwsem and cgroup_mutex, and cgroup creation
inherits the parent's sched under cgroup_mutex before the new cgroup is
reachable, so holding any one of the three locks makes the read stable.
Add scx_cgroup_sched() which states the protection with
rcu_dereference_check() and convert the readers. No functional changes.
Signed-off-by: Tejun Heo <tj@kernel.org>
While booting with KHO, the following crash was observed:
BUG: unable to handle page fault for address: ff19164fffff8328
RIP: 0010:__free_one_page+0x1a1/0x6b0
Call Trace:
<TASK>
[<ffffffff913208bf>] free_one_page+0xaf/0x240
[<ffffffff93973288>] deferred_free_pages+0xa8/0xd0
[<ffffffff93971b4f>] deferred_init_memmap_chunk+0x10f/0x1b0
[<ffffffff9396e265>] padata_mt_helper+0x65/0xa0
[<ffffffff90fac402>] process_scheduled_works+0x202/0x410
[<ffffffff90fae739>] worker_thread+0x1f9/0x2d0
[<ffffffff90fb62fd>] kthread+0x27d/0x2f0
[<ffffffff90fae540>] ? __pfx_worker_thread+0x10/0x10
[<ffffffff90fb6080>] ? __pfx_kthread+0x10/0x10
[<ffffffff90efdc55>] ret_from_fork+0x145/0x280
[<ffffffff90fb6080>] ? __pfx_kthread+0x10/0x10
[<ffffffff90e2e46a>] ret_from_fork_asm+0x1a/0x30
</TASK>
deferred_init_memmap_chunk() interleaves initialization of struct pages
with freeing them. This works fine without KHO because free regions
will never be buddy neighbors. However, with KHO, free memory will be split
into (free && scratch) and (free && !scratch), that can be buddy neighbors.
KHO scratch is aligned to CMA_MIN_ALIGNMENT_PAGES pages but buddy looks
at the neighborhood of MAX_ORDER_NR_PAGES pages. These values are
configurable but CMA_MIN_ALIGNMENT_PAGES is always less or equal to
MAX_ORDER_NR_PAGES. In the crashing configuration they were set as
follows:
CMA_MIN_ALIGNMENT_PAGES = 1 << 9
MAX_ORDER_NR_PAGES = 1 << 10
So while freeing one chunk, buddy accessed uninitialized struct pages
from another chunk, tried to merge the blocks and crashed.
To fix this, let's just align KHO scratch to MAX_ORDER_NR_PAGES pages.
Fixes: c6073743d0 ("kho: make preserved pages compatible with deferred struct page init")
Signed-off-by: Michal Clapinski <mclapinski@google.com>
Link: https://patch.msgid.link/20260717134028.2880508-1-mclapinski@google.com
[rppt: massaged the changelog]
Signed-off-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
LUO keeps track of successful retrieve attempts on an FLB. It does so
to avoid multiple retrievals of the same FLB. Multiple retrievals cause
problems because once the FLB is retrieved, the serialized data
structures are likely freed and the FLB is likely in a very different
state from what the code expects.
All this works well when retrieve succeeds. When it fails,
luo_flb_retrieve_one() returns the error immediately, without ever
storing anywhere that a retrieve was attempted or what its error code
was. If the user attempts to retrieve another file registered with the
same FLB, LUO will attempt to call the FLB's retrieve() callback again.
The retry is problematic for much of the same reasons listed above. The
FLB is likely in a very different state than what the retrieve logic
normally expects (e.g. some KHO pages may have already been restored and
freed).
There is no sane way of attempting the retrieve again. Remember the
error retrieve returned and directly return it on a retry.
This is done by changing the retrieved bool to a retrieve_status
integer. A value of 0 means retrieve was never attempted, a positive
value means it succeeded, and a negative value means it failed and the
error code is the value.
This is similar to commit f85b1c6af5 ("liveupdate: luo_file: remember
retrieve() status") which did the same for LUO files.
Fixes: cab056f2aa ("liveupdate: luo_flb: introduce File-Lifecycle-Bound global state")
Assisted-by: Gemini:gemini-3-pro-preview
Signed-off-by: David Matlack <dmatlack@google.com>
Reviewed-by: Pasha Tatashin <pasha.tatashin@soleen.com>
Reviewed-by: Pratyush Yadav (Google) <pratyush@kernel.org>
Link: https://patch.msgid.link/20260528174140.1921129-3-dmatlack@google.com
Signed-off-by: Pasha Tatashin <pasha.tatashin@soleen.com>
Signed-off-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Increment the outgoing FLB refcount in liveupdate_flb_get_outgoing() so
that the FLB structure cannot be freed while the caller is actively
using it. Add an additional liveupdate_flb_put_outgoing() function so
the caller can explicitly indicate when it is done using the outgoing
FLB.
During a Live Update, the kernel may need to fetch the outgoing FLB
outside of the scope of a file handler's preserve() and unpreserve()
callbacks. In that situation there is no way for the caller to protect
itself against the outgoing FLB from being freed while it is using it.
Incrementing the reference count in liveupdate_flb_get_outgoing()
ensures it cannot be freed.
This change also aligns the outgoing FLB lifecycle management with the
incoming FLB, since the latter uses the same get/put semantics.
Fixes: cab056f2aa ("liveupdate: luo_flb: introduce File-Lifecycle-Bound global state")
Assisted-by: Gemini:gemini-3-pro-preview
Signed-off-by: David Matlack <dmatlack@google.com>
Reviewed-by: Pasha Tatashin <pasha.tatashin@soleen.com>
Link: https://patch.msgid.link/20260528174140.1921129-2-dmatlack@google.com
Signed-off-by: Pasha Tatashin <pasha.tatashin@soleen.com>
Signed-off-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
The UAPI documents liveupdate_session_finish::reserved as requiring zero,
but luo_session_finish() currently ignores it and finishes the session.
Accepting nonzero values prevents the field from being safely repurposed
by a future extension.
Reject nonzero reserved values before changing session state, matching
LIVEUPDATE_SESSION_GET_NAME.
Fixes: 16cec0d265 ("liveupdate: luo_session: add ioctls for file preservation")
Assisted-by: Codex:gpt-5.6-sol
Reviewed-by: Pratyush Yadav (Google) <pratyush@kernel.org>
Signed-off-by: Jackie Liu <liuyun01@kylinos.cn>
Link: https://patch.msgid.link/20260716012607.22020-2-liu.yun@linux.dev
Signed-off-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
In ftrace, the trace_parser structure is allocated and initialized when
a trace file is opened, and is subsequently used across write and release
handlers to parse user input.
The affected handler paths and their specific functions are:
- Open paths: ftrace_regex_open(), ftrace_graph_open()
- Write paths: ftrace_regex_write(), ftrace_graph_write()
- Release paths: ftrace_regex_release(), ftrace_graph_release()
If userspace opens a trace file descriptor and shares it across multiple
threads, concurrent write calls will race on the parser's internal state,
specifically the 'idx', 'cont', and 'buffer' fields, leading to corrupted
input or undefined behavior.
Fix this by adding a global mutex, parser_lock, to serialize all access
to trace_parser across write and release paths, preventing concurrent
corruption of parser state.
Fixes: e704eff3ff ("ftrace: Have set_graph_function handle multiple functions in one write")
Fixes: 689fd8b65d ("tracing: trace parser support for function and graph")
Cc: stable@vger.kernel.org
Link: https://patch.msgid.link/20260725024721.1983675-1-wutengda@huaweicloud.com
Signed-off-by: Tengda Wu <wutengda@huaweicloud.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
- Fix tcp_bpf_sendmsg() error path mistaking a concurrently-freed
sk_psock->cork for the local temporary message and freeing it
again. (Chengfeng Ye)
- Reject passing scalar NULL to nonnull arg of a global subprog.
Previously the verifier did not account for the cases directly
passing scalars to a global subprog, e.g.: 'global_func(0);'
would pass even if 'global_func' argument was marked nonnull.
(Amery Hung)
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
-----BEGIN PGP SIGNATURE-----
iHUEABYKAB0WIQRINyQBqoQUC24dy5htleuBPyPTXgUCamPsBwAKCRBtleuBPyPT
Xo/uAP0YR6/9em1W/sZnss6Wkwfwd9NdmmlrJqSlt7aq+A+W7AD/V5HbigM4pJ76
CZFkKvnzhhU3dSXm/dBT1/GGJbL4pwo=
=uzKM
-----END PGP SIGNATURE-----
Merge tag 'bpf-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf
Pull bpf fixes from Eduard Zingerman:
- Fix tcp_bpf_sendmsg() error path mistaking a concurrently-freed
sk_psock->cork for the local temporary message and freeing it again
(Chengfeng Ye)
- Reject passing scalar NULL to nonnull arg of a global subprog.
Previously the verifier did not account for the cases directly
passing scalars to a global subprog, e.g.: 'global_func(0);' would
pass even if 'global_func' argument was marked nonnull (Amery Hung)
* tag 'bpf-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf:
bpf, sockmap: Fix cork use-after-free in tcp_bpf_sendmsg()
selftests/bpf: Test passing scalar NULL to nonnull global subprog
bpf: Reject passing scalar NULL to nonnull arg of a global subprog
Add 'P' flag to the diagnostic line to indicate whether any per-cpu
rtpcp's lazy_timer is pending. this helps diagnose stalls where
rcu-task callbacks are queued but the kthread stay sleep because
the lazy_timer has not yet fired and no grace period has started.
The output is as follows:
[ 31.319540][ T77] call_rcu_tasks() has failed boot-time tests.
[ 31.320205][ T77] rcu_tasks: RTGS_WAIT_CBS(11) since 7518 g:4 i:0 kCuUP l:150000
Signed-off-by: Zqiang <qiang.zhang@linux.dev>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
This commit updates the comment "We can't create the thread unless
interrupts are enabled." from 'commit 4929c913bd ("rcu: Make
call_rcu_tasks() tolerate first call with irqs disabled")' to be more
clear and also to cover deferred wakeup and to take into account the fact
that kthread creation has been move to core_initcall() time by 'commit
c63eb17ff0 ("rcu: Create call_rcu_tasks() kthread at boot time")'.
Signed-off-by: Zqiang <qiang.zhang@linux.dev>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Although the rdp->rcu_cpu_has_work field is accessed only by the
corresponding CPU, it can be accessed by both interrupt handlers via
invoke_rcu_core_kthread() and at task level via rcu_cpu_kthread().
This means that we need this_cpu_read() rather than __this_cpu_read(),
this_cpu_write() rather than __this_cpu_write(), and READ_ONCE()
rather than plain C-language loads. The exception is the boot-time
rcu_spawn_core_kthreads(), which cannot race with kthreads that have
not yet been spawned.
This commit therefore makes it so.
KCSAN located this issue.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Ever since the kvfree_rcu() tracing moved out of the callback-enqueue
path, rcutree_enqueue() no longer looks at the callback function
pointer: By the time it is invoked, __call_rcu_common() has already
stored the function into rhp->func, and the enqueue path only adds
the rcu_head to the segmented callback list and emits tracepoints
that do not take the function pointer.
Nevertheless, the function pointer is still threaded through
call_rcu_core(), call_rcu_nocb(), and rcutree_enqueue(), forcing
each level to carry a dead argument.
Remove the parameter from all three functions, from the no-CBs stub,
and from the corresponding declarations. Anything needing the
callback function can still get it from rhp->func.
No functional change.
Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
The ->expedited_need_qs counter was part of the old expedited
grace-period machinery that spun waiting for each CPU to check in.
The current implementation instead tracks holdout CPUs via the
rcu_node structures' ->expmask fields and waits on the ->exp_wq[]
wait queues, so nothing reads or writes ->expedited_need_qs any
longer.
Remove the field to avoid wasting space in rcu_state and to keep
readers of the expedited code from searching for nonexistent users.
Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Currently raw_cpu_write() is used to clear the ->rcu_need_heavy_qs
field of the per-CPU rcu_data structure. However, on x86 this is a
normal assignment, which does not play well with concurrent accesses.
This commit therefore upgrades the uses of raw_cpu_write() to its
concurrency-safe counterpart WRITE_ONCE() of an rdp pointer obtained
from this_cpu_ptr(&rcu_data).
KCSAN located this issue.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Currently __this_cpu_read() and __this_cpu_write() is used to access the
->cpu_no_qs.b.exp field of the per-CPU rcu_data structure. However,
this can fail when the accesses can happen in interrupt handlers, as
recently started being exercised by rcutorture. This commit therefore
upgrades the uses of __this_cpu_read() and __this_cpu_write() to their
interrupt-safe counterparts this_cpu_read() and this_cpu_write().
KCSAN located this issue.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Add nexp and exp_interval parameters to rcuscale that spawn kthreads
running synchronize_rcu_expedited() in a loop. This generates concurrent
expedited GP load while the normal writers measure GP or callback
latency.
When combined with gp_async=1 (which uses call_rcu() for writers), this
tests how effectively callbacks benefit from expedited grace periods.
With RCU callback expedited GP tracking, the async callbacks should
complete faster because they piggyback on the expedited GPs rather than
waiting for normal GPs.
Reviewed-by: Paul E. McKenney <paulmck@kernel.org>
Signed-off-by: Puranjay Mohan <puranjay@kernel.org>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Even when rcu_pending() triggers rcu_core(), the normal callback
advancement path through note_gp_changes() -> __note_gp_changes() bails
out when rdp->gp_seq == rnp->gp_seq (no normal GP change). Since
expedited GPs do not update rnp->gp_seq, rcu_advance_cbs() is never
called and callbacks remain stuck in RCU_WAIT_TAIL.
Add a direct callback advancement block in rcu_core() that checks for GP
completion via rcu_segcblist_nextgp() combined with
poll_state_synchronize_rcu_full(). When detected, trylock rnp and call
rcu_advance_cbs() to move completed callbacks to RCU_DONE_TAIL. Wake the
GP kthread if rcu_advance_cbs() requests a new grace period.
Uses trylock to avoid adding contention on rnp->lock. If the lock is
contended, callbacks will be advanced on the next tick.
Reviewed-by: Paul E. McKenney <paulmck@kernel.org>
Signed-off-by: Puranjay Mohan <puranjay@kernel.org>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
rcu_pending() decides whether rcu_core() should run on the current CPU's
timer tick. It does not account for expedited grace periods: after an
expedited GP completes, a non-offloaded CPU's callbacks remain in
RCU_WAIT_TAIL (not yet advanced to RCU_DONE_TAIL) and rcu_core() is
never invoked to advance them.
Detect that case via rcu_segcblist_nextgp() combined with a new
memory-ordering-free poll variant,
poll_state_synchronize_rcu_full_unordered(). This keeps rcu_pending()
cheap: it runs on every tick that has pending callbacks, so it must
not pay for the two memory barriers in
poll_state_synchronize_rcu_full(). The check is only a hint to run
rcu_core(); the ordered re-check and the actual callback advancement
happen there.
Signed-off-by: Puranjay Mohan <puranjay@kernel.org>
Reviewed-by: Frederic Weisbecker <frederic@kernel.org>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
When an expedited grace period completes, rcu_exp_wait_wake() wakes
waiters on rnp->exp_wq[] but does not notify the NOCB rcuog kthreads. An
rcuog kthread that is waiting for a grace period sleeps on the leaf
rcu_node's ->nocb_gp_wq[] with a wait condition based on the grace-period
state, so without a wakeup, callbacks on offloaded CPUs that could
benefit from the expedited GP wait until the rcuog kthread wakes for some
other reason (e.g. the next normal GP or a timer).
Make the rcuog grace-period wait honour expedited GPs and wake it when
one completes:
- nocb_gp_wait() now records the grace period to wait for as a struct
rcu_gp_seq (both normal and expedited), tracks the earliest pending
normal and expedited sequence across the group, and releases the wait
via poll_state_synchronize_rcu_full() so it wakes for whichever
completes first. ->nocb_gp_seq is widened to struct rcu_gp_seq
accordingly.
- rcu_exp_wait_wake() calls the new rcu_nocb_exp_cleanup() on leaf
nodes, which wakes both ->nocb_gp_wq[0] and ->nocb_gp_wq[1] (the
expedited sequence does not share parity with the normal ->gp_seq the
waiter indexed with). Both this path and rcu_nocb_gp_cleanup() use
the shared rcu_nocb_cleanup_wake() helper, which checks swait_active()
first; the smp_mb() in rcu_gp_cleanup()/rcu_exp_wait_wake() orders the
grace-period state update before that check.
A stub rcu_nocb_exp_cleanup() is provided for CONFIG_RCU_NOCB_CPU=n.
Signed-off-by: Puranjay Mohan <puranjay@kernel.org>
Reviewed-by: Frederic Weisbecker <frederic@kernel.org>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Update documentation comments throughout the RCU callback infrastructure
to reflect the transition from a single grace-period sequence number to
the full struct rcu_gp_seq that tracks both normal and expedited grace
periods.
The ->gp_seq[] array documentation in rcu_segcblist.h is updated to
describe dual (normal and expedited) GP tracking. The
rcu_segcblist_advance(), rcu_segcblist_accelerate(), and
rcu_advance_cbs() comments are updated to refer to the struct rcu_gp_seq
state (gsp) instead of the old bare grace-period sequence number (seq).
Reviewed-by: Paul E. McKenney <paulmck@kernel.org>
Signed-off-by: Puranjay Mohan <puranjay@kernel.org>
Reviewed-by: Frederic Weisbecker <frederic@kernel.org>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Currently, RCU callbacks only track normal grace-period sequence
numbers. This means callbacks must wait for normal grace periods to
complete even when expedited grace periods have already elapsed.
Use the full struct rcu_gp_seq (which tracks both the normal and
expedited grace-period sequences) throughout the callback
infrastructure.
rcu_segcblist_advance() now checks both normal and expedited GP
completion via poll_state_synchronize_rcu_full(), and becomes
parameterless since it reads the grace-period state internally.
rcu_segcblist_accelerate() stores the full state (both sequences)
instead of just the normal one. rcu_accelerate_cbs() and
rcu_accelerate_cbs_unlocked() use get_state_synchronize_rcu_full() to
capture both sequences, and the NOCB advance checks use
poll_state_synchronize_rcu_full() instead of comparing only the normal
sequence.
srcu_segcblist_advance() becomes a standalone implementation because it
compares SRCU sequences directly and cannot use
poll_state_synchronize_rcu_full(), which reads RCU-specific globals.
srcu_segcblist_accelerate() sets the ->exp field to
RCU_GET_STATE_NOT_TRACKED so that poll_state_synchronize_rcu_full()
compares only ->norm and ignores ->exp.
Signed-off-by: Puranjay Mohan <puranjay@kernel.org>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Triggers are now delayed from freeing, but can still be triggered until
after the RCU grace period has ended. The freeing of the enable_event data
is put into the private_data_free() callback, but the put of the module
refcount is done immediately.
It is possible that if a module is removed that has an event that would
enable (or disable) it is still active, it can read the data of the module
after it is removed causing a use-after-free bug.
Move the trace_event_put_ref() that releases the module into the delayed
callback so that the module can not be removed until any reference to its
events are finished.
Cc: stable@vger.kernel.org
Link: https://patch.msgid.link/20260724132415.1b5005db@gandalf.local.home
Reported-by: Sashiko <sashiko-bot@kernel.org>
Link: https://sashiko.dev/#/patchset/20260724030523.19081-1-devnexen%40gmail.com
Fixes: 61d445af0a ("tracing: Add bulk garbage collection of freeing event_trigger_data")
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
Commit 61d445af0a ("tracing: Add bulk garbage collection of freeing
event_trigger_data") moved the kfree() of event_trigger_data to a kthread
that runs tracepoint_synchronize_unregister() before freeing. That removed
the synchronization the trigger .free callbacks used to get implicitly and
inline from trigger_data_free().
event_hist_trigger_free(), event_hist_trigger_named_free() and
event_enable_trigger_free() free their satellite data (hist_data, cmd_ops,
enable_data) right after trigger_data_free() returns. With the
synchronization now deferred to the kthread, a concurrent tracepoint
handler can still reach that data through the list_del_rcu()'d trigger,
causing a use-after-free.
The histogram teardown must stay synchronous: remove_hist_vars() and
unregister_field_var_hists() have to detach a synthetic event from the
histogram before the trigger-removal write returns, otherwise a following
command races in and the synthetic-event removal fails with -EBUSY, as the
trigger-synthetic-eprobe.tc selftest catches. Make those callbacks wait
with the correct barrier - tracepoint_synchronize_unregister(), matching
the free kthread - before freeing.
The enable trigger has no such synchronous requirement, and a blocking
synchronize there would re-serialize the path that commit deliberately
deferred. Give it an optional private_data_free() callback that the free
kthread runs after its grace period, and free enable_data from there.
Link: https://patch.msgid.link/20260724030523.19081-1-devnexen@gmail.com
Suggested-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
Suggested-by: Steven Rostedt <rostedt@goodmis.org>
Fixes: 61d445af0a ("tracing: Add bulk garbage collection of freeing event_trigger_data")
Signed-off-by: David Carlier <devnexen@gmail.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
scx_bpf_cidperf_set() reaches cpufreq with no cap check, so any cid-form
sub-sched can steer the frequency of any cid in its view, including ones it
holds nothing on.
Gate it behind a new SCX_CAP_PERF rather than SCX_CAP_BASE: hardware control
is a separate axis from queue access - a parent may well delegate scheduling
on a cid without handing over its frequency. PERF neither implies nor is
implied by the other caps. The check runs under the target rq's lock, which
ecaps updates are also folded under, so it is authoritative - a write can
never land after a revoke has taken effect. Denials are counted in
SCX_EV_SUB_CIDPERF_DENIED.
The operation is synchronous and the outcome is reported to the caller:
scx_bpf_cidperf_set() now returns 0 or -errno, -EACCES on denial. The
cid-form interface is still under initial development, so the signature is
changed in place without versioning.
scx_qmap grants PERF alongside its existing cid grants so the cpuperf demo
keeps working in sub-scheds.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
Factor the cpuperf target write out of scx_bpf_cpuperf_set() into
scx_cpuperf_set() which takes the acting sched and returns 0 or -errno, and
flatten the nested validation into early returns. No functional change.
Prep for gating the write behind a cap and reporting the outcome from the
cid-form kfunc.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
kick_one_cpu() silently skips a kick when the kicking sub-sched lacks
SCX_CAP_BASE on the target cid, as does kick_one_cpu_if_idle() for idle
kicks. The skips are sound with the same logic as the reenq gate but are
invisible today, unlike the preempt degradation counted in
SCX_EV_SUB_PREEMPT_DENIED. Count them in a new SCX_EV_SUB_KICK_DENIED event
so every cap denial is observable.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
scx_bpf_dsq_reenq() with an SCX_DSQ_LOCAL_ON target schedules deferred reenq
work on the cid's cpu, raising an IPI when the target rq isn't the locked
one. Nothing checks caps along the way, so a sub-sched holding no cap at all
on a cid can force its cpu to take IPIs and rq lock cycles at will. The
analogous scx_bpf_kick_cid() path gates delivery on SCX_CAP_BASE in
kick_one_cpu() to prevent exactly this.
Apply the same rule at the reenq scheduling point: if the calling sched
lacks SCX_CAP_BASE on the target cid, drop the reenq and count it in the new
SCX_EV_SUB_REENQ_DENIED event. The check is lockless, which is fine: a reenq
slipping through right after a revoke is harmless, and a wrong denial can't
happen - if the caller has seen its ownership of the cpu, the check sees it
too.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
The trampoline could be corrupted by the blindly
'tr->flags = BPF_TRAMP_F_TAIL_CALL_CTX' in verifier.
1. A fexit attached to a tail_call_reachable prog. 'tr->flags' became
'BPF_TRAMP_F_CALL_ORIG | BPF_TRAMP_F_TAIL_CALL_CTX'. And, the
trampoline would poke the target prog's nop insn using jmp insn instead
of call insn.
2. Another fexit loaded with the same tail_call_reachable prog target.
'tr->flags' became 'BPF_TRAMP_F_TAIL_CALL_CTX'.
3. Close the first fexit link. Due to no BPF_TRAMP_F_CALL_ORIG in
'tr->flags', the trampoline will fail to restore the prog's nop insn
using call insn.
[ 3.410719] WARNING: kernel/bpf/syscall.c:3551 at bpf_tracing_link_release+0x53/0x60, CPU#1: test_progs/98
...
[ 3.428793] bpf_link_free+0x58/0x130
[ 3.429293] bpf_link_release+0x23/0x30
Fix the warning by updating 'tr->flags' with '|=' and lock.
Fixes: 2b5dcb31a1 ("bpf, x64: Fix tailcall infinite loop")
Signed-off-by: Leon Hwang <leon.hwang@linux.dev>
Reviewed-by: Pu Lehui <pulehui@huawei.com>
Acked-by: Jiri Olsa <jolsa@kernel.org>
Link: https://lore.kernel.org/bpf/20260722151909.69142-2-leon.hwang@linux.dev
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
BPF subprograms use the bpf_callback_t ABI, but task work invokes the
callback through a three-argument function pointer. This trips kCFI.
Store and invoke the callback as bpf_callback_t.
Fixes: 38aa7003e3 ("bpf: task work scheduling kfuncs")
Signed-off-by: Mykyta Yatsenko <yatsenko@meta.com>
Link: https://lore.kernel.org/bpf/20260724-task_work_cfi-v1-1-2616691781ed@meta.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
trace_user_fault_read() samples nr_context_switches_cpu() before enabling
preemption and retries the user copy if the counter changes. The helper
returns unsigned long long because rq->nr_switches is u64, but the saved
value is unsigned int.
Once a CPU has performed 2^32 context switches, assigning the counter to
cnt discards its upper bits. The comparison after the copy promotes cnt
back to unsigned long long, but the lost bits remain zero, so it reports a
change even when the task was never scheduled out. Every retry then fails
the same way until the 100-try guard warns and the user copy is abandoned.
This affects long-running systems and workloads with high context-switch
rates. A CPU switching 1,000 times per second takes about 50 days.
Store the sampled count in unsigned long long so the full value is
preserved.
Cc: stable@vger.kernel.org
Fixes: 64cf7d058a ("tracing: Have trace_marker use per-cpu data to read user space")
Link: https://patch.msgid.link/20260717173252.3431565-1-usama.arif@linux.dev
Reported-by: Breno Leitao <leitao@debian.org>
Signed-off-by: Usama Arif <usama.arif@linux.dev>
Acked-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
Reviewed-by: Breno Leitao <leitao@debian.org>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
In 'struct trace_event_call', the 'module' pointer and the 'refcnt'
atomic variable share the same memory space in a union. For dynamic
events, the union member is 'refcnt', which acts as an active
reference counter.
When a dynamic event (such as kprobe, uprobe, fprobe, eprobe, or
wprobe) has a non-zero reference count (e.g. due to active event
triggers or perf attachments), its 'call->module' evaluates to a
small non-zero integer instead of NULL.
When filtering or setting events for a specific module (e.g., writing
':mod:<module>' to 'set_event'), the code in
'__ftrace_set_clr_event_nolock()' and 'update_event_fields()' reads
'call->module' directly without checking whether the event is dynamic.
This causes the kernel to treat the small integer (refcnt) as a
'struct module' pointer, leading to a NULL/invalid pointer dereference
(Oops) when dereferencing the module name.
Fix this by ensuring that the 'TRACE_EVENT_FL_DYNAMIC' flag is checked
before treating 'call->module' as a valid pointer in these code paths.
Cc: stable@vger.kernel.org
Link: https://patch.msgid.link/178425670947.84440.11344393611899824907.stgit@devnote2
Fixes: 4c86bc531e ("tracing: Add :mod: command to enabled module events")
Assisted-by: Antigravity:gemini-3.5-flash
Signed-off-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
The flags passed to the pick-idle kfuncs are values from the
scx_pick_idle_cpu_flags enum, whose members are prefixed
SCX_PICK_IDLE_ (SCX_PICK_IDLE_CORE, SCX_PICK_IDLE_IN_NODE).
Three kernel-doc comments in idle.c erroneously used
%SCX_PICK_IDLE_CPU_* which does not correspond to any defined flag
name, while the adjacent scx_bpf_pick_idle_cpu_node() correctly
documents %SCX_PICK_IDLE_*.
Fix the three occurrences to use the correct SCX_PICK_IDLE_* prefix.
Signed-off-by: Liang Luo <luoliang@kylinos.cn>
Signed-off-by: Tejun Heo <tj@kernel.org>
Add a wakeup monitor to detect a lower-priority task waking up a
higher-priority task.
The rtapp/sleep monitor already detects this. However, that monitor
triggers an error in the context of the wakee task and user only gets
the stacktrace of that task. It is also extremely useful to get the
stacktrace of the waker task, which this monitor offers. In other
words, this monitor complements the rtapp/sleep monitor.
Signed-off-by: Nam Cao <namcao@linutronix.de>
Reviewed-by: Gabriele Monaco <gmonaco@redhat.com>
Link: https://lore.kernel.org/r/ba5658fa13e49ada466b84a2c211f233037180b5.1781852967.git.namcao@linutronix.de
Signed-off-by: Gabriele Monaco <gmonaco@redhat.com>
The rtapp/sleep monitor's primary purpose is detecting common mistakes
with user-space real-time design. Monitoring real-time issues with
kernel threads is a bonus.
However, accomodating kernel threads complicates the monitor due to
the edge cases which is seen by the monitor as lower-priority task
waking higher-priority task:
- kthread_stop() wakes up the task in order to stop it.
- The rcu thread and migration thread can be woken by any task.
- The ktimerd thread is woken near the end of irq_exit_rcu(), where
the preempt counter is "broken" and falsely says this is task
context. This requires the monitor to use the hardirq_context flag
instead of the preempt counter.
Beside complicating the monitor, the final case also requires enabling
CONFIG_TRACE_IRQFLAGS (so that "hardirq_context" can be used). This
adds overhead to the kernel even when the monitor is not active. This
may be an obstacle to enabling this monitor in distros' kernels.
Furthermore, kernel threads usually are started before the monitor is
enabled. Consequently, the threads' states (i.o.w. the monitor's
atomic propositions for the threads) are not fully known to the
monitor. As a result, the kernel threads mostly cannot be monitored.
Overall, the downsides of accomodating kernel threads outweights the
benefits. Thus, exclude kernel threads to simplify the monitor.
Signed-off-by: Nam Cao <namcao@linutronix.de>
Reviewed-by: Gabriele Monaco <gmonaco@redhat.com>
Link: https://lore.kernel.org/r/eec2ca5224bcdacc45b8e1eb2f0e68109e1cae7a.1781852967.git.namcao@linutronix.de
Signed-off-by: Gabriele Monaco <gmonaco@redhat.com>
CLOCK_REALTIME is the only clock that often is misused in real-time
applications. The other clocks either are safe for real-time uses
(CLOCK_TAI, CLOCK_MONOTONIC, CLOCK_BOOTTIME) or are unlikely to be misused
(CLOCK_AUX, CLOCK_PROCESS_CPUTIME_ID).
Update the monitor to only warn about CLOCK_REALTIME.
While at it, update the out-of-sync documentation.
Signed-off-by: Nam Cao <namcao@linutronix.de>
Reviewed-by: Gabriele Monaco <gmonaco@redhat.com>
Link: https://lore.kernel.org/r/c7ceb5c6263ee8f43a2676acae669cf486b0d903.1781852967.git.namcao@linutronix.de
Signed-off-by: Gabriele Monaco <gmonaco@redhat.com>
The rtapp/sleep monitor detects real-time tasks which go to sleep in an
real-time-unsafe manner. If this happen, the monitor triggers a trace event
in the sched_wakeup tracepoint's handler.
However, the invoking context of that trace event is not the most
informative, because of the stack trace of that event is the wakeup's code
path which is not very helpful:
74.669317: rv:error_sleep: condvar[254]: violation detected
ltl_validate+0x345 ([kernel.kallsyms])
handle_sched_wakeup+0x34 ([kernel.kallsyms])
ttwu_do_activate+0xff ([kernel.kallsyms])
sched_ttwu_pending+0x104 ([kernel.kallsyms])
__flush_smp_call_function_queue+0x15b ([kernel.kallsyms])
__sysvec_call_function_single+0x18 ([kernel.kallsyms])
sysvec_call_function_single+0x66 ([kernel.kallsyms])
asm_sysvec_call_function_single+0x1a ([kernel.kallsyms])
pv_native_safe_halt+0xf ([kernel.kallsyms])
default_idle+0x9 ([kernel.kallsyms])
default_idle_call+0x33 ([kernel.kallsyms])
do_idle+0x234 ([kernel.kallsyms])
cpu_startup_entry+0x24 ([kernel.kallsyms])
start_secondary+0xf8 ([kernel.kallsyms])
common_startup_64+0x13e ([kernel.kallsyms])
What would be much more valuable is the stack trace of the task itself.
Instead of using the sched_wakeup tracepoint, use the sched_exit
tracepoint. This makes the event happen in the task's context, making
the stack trace far more informative for user:
rv:error_sleep: condvar[254]: violation detected
ltl_validate+0x345 ([kernel.kallsyms])
handle_sched_exit+0x39 ([kernel.kallsyms])
__schedule+0x80f ([kernel.kallsyms])
schedule+0x22 ([kernel.kallsyms])
futex_do_wait+0x33 ([kernel.kallsyms])
__futex_wait+0x8c ([kernel.kallsyms])
futex_wait+0x73 ([kernel.kallsyms])
do_futex+0xc6 ([kernel.kallsyms])
__x64_sys_futex+0x121 ([kernel.kallsyms])
do_syscall_64+0xf3 ([kernel.kallsyms])
entry_SYSCALL_64_after_hwframe+0x77 ([kernel.kallsyms])
__futex_abstimed_wait_common64+0xc6 (inlined)
__futex_abstimed_wait_common+0xc6 (/usr/lib/x86_64-linux-gnu/libc.so.6)
Signed-off-by: Nam Cao <namcao@linutronix.de>
Reviewed-by: Gabriele Monaco <gmonaco@redhat.com>
Link: https://lore.kernel.org/r/d97b4b5c476e5792b6875ec9bbf8dc214f999516.1781852967.git.namcao@linutronix.de
Signed-off-by: Gabriele Monaco <gmonaco@redhat.com>
A global subprogram argument tagged __arg_nonnull is set up as a
non-nullable PTR_TO_MEM. However the verifier does not check against a
scalar NULL, leading to real NULL pointer dereference. Reject it as
well.
Fixes: 94e1c70a34 ("bpf: support 'arg:xxx' btf_decl_tag-based hints for global subprog args")
Signed-off-by: Amery Hung <ameryhung@gmail.com>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://patch.msgid.link/20260723221815.367797-1-ameryhung@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Add missing function parameter descriptions and use the correct
function name in kernel-doc comments to avoid kernel-doc warnings:
Warning: kernel/sched/ext/ext.c:2692 function parameter 'sch' not described in 'finish_dispatch'
Warning: kernel/sched/ext/ext.c:5309 function parameter 'stalled_mask' not described in 'scx_rcu_cpu_stall'
Warning: kernel/sched/ext/ext.c:5405 function parameter 'cpu' not described in 'scx_hardlockup'
Warning: kernel/sched/ext/ext.c:8470 expecting prototype for scx_bpf_dsq_insert(). Prototype was for scx_bpf_dsq_insert___v2() instead
Warning: kernel/sched/ext/ext.c:8784 expecting prototype for scx_bpf_dsq_move_to_local(). Prototype was for scx_bpf_dsq_move_to_local___v2() instead
Warning: kernel/sched/ext/ext.c:9498 expecting prototype for scx_bpf_reenqueue_local(). Prototype was for scx_bpf_reenqueue_local___v2() instead
Signed-off-by: Randy Dunlap <rdunlap@infradead.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
For the smp_mb() after kthread_run() in rcu_spawn_tasks_kthread_generic()
from these commits:
'commit eacd6f04a1 ("rcu-tasks: Move Tasks RCU to its own file")'
'commit 84a8f446ff ("rcu: Defer rcu_tasks_kthread() creation till
first call_rcu_tasks()")'
the memory order pairing as follows:
rcu_spawn_tasks_kthread()
->t = kthread_run(rcu_tasks_kthread, ...);
->smp_mb(); /* Ensure others see full kthread. */
->WRITE_ONCE(rcu_tasks_kthread_ptr, t);
call_rcu_tasks()
->if (READ_ONCE(rcu_tasks_kthread_ptr))
->wake_up(&rcu_tasks_cbs_wq)
->try_to_wake_up()
lock pi_lock
->smp_mb__after_spinlock()
//see full kthread
Because the 'commit d119357d07 ("rcu-tasks: Treat only synchronous
grace periods urgently")' moved the kthread_ptr assignment into the
rcu_tasks_kthread() function, the following memory order pairings are
sufficient:
The runq's raw_spinlock/unlock(or smp_mb__after_spinlock()) from
wake_up_process() in kthread_run() and __schedule() provides memory order
barrier when the kthread is first scheduled, this ensures the kthread's
func observes all of the kthread's initialization.
The kthread's smp_store_release(&rtp->kthread_ptr, ...) in
rcu_tasks_kthread() and smp_load_acquire(&rtp->kthread_ptr)
in call_rcu_tasks_generic() compose release/acquire pairing, the
cumulativity of smp_store_release() propagates visibility of the kthread's
initialization through the scheduler chain.
This commit therefore remove smp_mb() in rcu_spawn_tasks_kthread_generic().
Signed-off-by: Zqiang <qiang.zhang@linux.dev>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
This commit adds a dump RCU task status function to rcu tasks tests,
used to obtain more information to help debug when the RCU tasks
boot-time tests failed.
Signed-off-by: Zqiang <qiang.zhang@linux.dev>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
The RCU tasks trace has been reimplemented by 'commit c27cea4416
("rcu: Re-implement RCU Tasks Trace in terms of SRCU-fast")', the
rcu_tasks structure's->n_ipis_fails is no longer used, this commit
therefore remove it.
Signed-off-by: Zqiang <qiang.zhang@linux.dev>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Paul McKenney noted that a softirq (or irq_work) handler arming for a
deferred QS can fire and find rcu_preempt_depth() > 0 -- the task is
still inside its outer reader, so rcu_preempt_need_deferred_qs() bails
without reporting the QS. At that point the queued mechanism has been
consumed but ->defer_qs_pending stays in DEFER_QS_PENDING.
In the meantime, the only remaining path back to a quiescent state on
this CPU may be a local_irq_disable()/_enable() pair that does not
call preempt_check_resched() (it is just `sti`/`cli`). patch 6's
unconditional set_need_resched_current() makes need_resched true, but
without an irq_work being raised the next outer rcu_read_unlock_special()
hits the P-gate at the arming code:
if (rdp->defer_qs_pending != DEFER_QS_PENDING) {
rdp->defer_qs_pending = DEFER_QS_PENDING;
irq_work_queue_on(...); // <-- skipped
}
so no irq_work is queued for the hardirq-exit preempt_schedule_irq()
path either. The deferred QS now waits until the next timer tick (or
similar preempt-safe boundary), needlessly extending expedited grace
period latency.
Clear ->defer_qs_pending in the bail-out path of rcu_preempt_deferred_qs()
when rcu_preempt_depth() > 0. The recursion guard semantics introduced
by commit b41642c877 ("rcu: Fix rcu_read_unlock() deadloop due to IRQ
work").
The clear is also safe against fresh recursion at this exact program
point: rcu_preempt_depth() > 0 guarantees we are still inside an outer
reader, so any inner rcu_read_unlock() from tracing infrastructure
brings nesting back to outer (>0), never to 0. The slow path of
rcu_read_unlock_special() is structurally unreachable under that
condition, so no recursive raise_softirq_irqoff()/irq_work_queue_on()
can be triggered by the clear. Essentially, the mechanism will work to
prevent the following recursion which Xiongfeng had previously reported:
irq_exit() -> __irq_exit_rcu()
-> tick_irq_exit() -> tick_nohz_irq_exit() -> tick_nohz_stop_sched_tick()
-> trace_tick_stop() // BPF prog hooked here
-> rcu_read_unlock_special()
-> irq_work_queue_on(&rdp->defer_qs_iw, rdp->cpu) // self-IPI re-enters irq_exit
Reported-by: Paul E. McKenney <paulmck@kernel.org>
Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Currently rdp->defer_qs_pending transitions from DEFER_QS_PENDING to
DEFER_QS_IDLE at two sites: rcu_preempt_deferred_qs_irqrestore() and
rcu_preempt_deferred_qs_handler() (depth>0 reset). Both write the
IDLE value directly.
Introduce a single inline helper rcu_defer_qs_clear() in tree.h and
route both sites through it. This becomes the single
PENDING->IDLE transition point for upcoming work.
Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
RCU stall warnings currently print task states as raw hexadecimal
values, requiring developers to manually decode them.
Use task_state_to_char() so that stall warnings show the same symbolic
task-state representation used elsewhere in the kernel.
For example:
->state=0x402 becomes ->state=I
->state=0x0 becomes ->state=R
->state=0x2 becomes ->state=D
This improves readability while preserving the underlying diagnostic
information.
Suggested-by: Zqiang <qiang.zhang@linux.dev>
Co-developed-by: Wang Lian <lianux.mm@gmail.com>
Signed-off-by: Wang Lian <lianux.mm@gmail.com>
Signed-off-by: Kunwu Chan <kunwu.chan@gmail.com>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
SRCU and Tasks RCU do not track expedited grace periods. When their
callback state is checked via poll_state_synchronize_rcu_full(), the
uninitialized or zeroed exp field could cause false-positive
completion detection.
This commit adds an RCU_GET_STATE_NOT_TRACKED sentinel value (0x2) that
these subsystems can place into exp to indicate that expedited GP
tracking is not applicable. The expedited sequence check in
poll_state_synchronize_rcu_full() is guarded to skip entries marked with
this sentinel.
This is needed to allow rcu_segcblist_advance() and rcu_accelerate_cbs()
to work with both normal and expedited grace periods via
get_state_synchronize_rcu_full() and poll_state_synchronize_rcu_full().
Reviewed-by: Paul E. McKenney <paulmck@kernel.org>
Signed-off-by: Puranjay Mohan <puranjay@kernel.org>
Reviewed-by: Frederic Weisbecker <frederic@kernel.org>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Change the type of the per-segment ->gp_seq[] array in struct
rcu_segcblist from unsigned long to struct rcu_gp_seq. This prepares the
callback tracking infrastructure to record both normal and expedited
grace periods per segment.
The rcu_segcblist_nextgp(), rcu_segcblist_advance(), and
rcu_segcblist_accelerate() helpers now take a struct rcu_gp_seq * instead
of an unsigned long, and all callers use the .norm field for comparisons
and assignments. The SRCU and Tasks RCU wrappers construct a struct
rcu_gp_seq with only .norm set and forward to the core helpers.
No functional change: only the .norm field is used.
Signed-off-by: Puranjay Mohan <puranjay@kernel.org>
Reviewed-by: Frederic Weisbecker <frederic@kernel.org>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
This commit extracts the tail-pointer cleanup and segment compaction
logic from rcu_segcblist_advance() into a new static helper function,
rcu_segcblist_advance_compact(). This shared logic will be reused by the
upcoming srcu_segcblist_advance() standalone implementation, which
cannot call the core rcu_segcblist_advance() because that function will
use RCU-specific globals.
No functional change.
Reviewed-by: Paul E. McKenney <paulmck@kernel.org>
Signed-off-by: Puranjay Mohan <puranjay@kernel.org>
Reviewed-by: Frederic Weisbecker <frederic@kernel.org>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Add srcu_segcblist_advance() and srcu_segcblist_accelerate() wrappers
that forward to the core rcu_segcblist_advance() and
rcu_segcblist_accelerate() functions, and switch all SRCU (srcutree.c)
and Tasks RCU (tasks.h) callers to use these wrappers.
This isolates SRCU and Tasks RCU from upcoming changes to the core
advance/accelerate functions, which will switch to struct
rcu_gp_seq for dual normal/expedited GP tracking. Because SRCU and
Tasks RCU use only normal GP sequences, their wrappers will maintain the
existing unsigned long interface.
No functional change.
Reviewed-by: Paul E. McKenney <paulmck@kernel.org>
Signed-off-by: Puranjay Mohan <puranjay@kernel.org>
Reviewed-by: Frederic Weisbecker <frederic@kernel.org>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
The polled grace-period state structure rcu_gp_oldstate holds a snapshot
of the normal (and, on SMP, expedited) grace-period sequence numbers.
Upcoming changes store this structure in the callback segment list, where
the "oldstate" name reads poorly: there it represents the grace period a
segment is waiting on and is also compared against the current
grace-period state.
Rename struct rcu_gp_oldstate to the more neutral struct rcu_gp_seq, and
shorten its members rgos_norm and rgos_exp to norm and exp. Local
variables and parameters of this type are renamed from rgosp/rgos to
gsp/gs accordingly.
While at it, provide a single definition of the structure in rcupdate.h
rather than separate Tiny-RCU and Tree-RCU definitions, and give it the
->exp field unconditionally. Tiny RCU does not track expedited grace
periods and leaves ->exp unused, but a single definition that always has
->exp lets the shared callback code in rcu_segcblist.c reference it
without CONFIG_SMP guards, including on !SMP builds.
No functional change.
Signed-off-by: Puranjay Mohan <puranjay@kernel.org>
Reviewed-by: Frederic Weisbecker <frederic@kernel.org>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
The trace_rcu_this_gp() wrapper forwards only the rcu_node structure's
fields and the requested grace-period sequence number to the
rcu_future_grace_period tracepoint. Its rcu_data pointer parameter
has no users, but every one of the ten call sites must nevertheless
come up with an rcu_data pointer to pass in.
Remove the parameter and update all callers. This also allows
rcu_future_gp_cleanup() to drop the local rcu_data pointer that
existed solely to feed this trace call.
No functional change.
Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Each scftorture_invoker() thread decrements n_started, which is
initialized to the number of threads, and is then supposed to wait
until all of its siblings have also checked in before starting the
test proper. However, the wait loop is guarded by
!atomic_dec_return(&n_started), which is true only for the final
thread to arrive, and by then n_started is already zero, so the
final thread does not wait either. The side-effect (possibly positive)
is that no thread ever waits and the start-synchronization barrier is
dead code, with early threads beginning to hammer smp_call_function*()
while later threads are still being spawned.
Invert the test so that every thread other than the last spins until
n_started reaches zero, making the threads start testing together as
intended. The existing torture_must_stop() check in the wait loop
continues to bound the wait during shutdown.
We can also drop the spinning entirely if the intent is to leave it as
dead code, however for the current intent, this patches fixes the code.
Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
scf_torture_stats_print() aggregates each invoker thread's counters
into a local scf_statistics structure before printing, but the
n_single_rpc_ofl field is missing from the aggregation loop. As a
result, the "single_rpc_ofl" value printed in the statistics line is
always zero, even when smp_call_function_single() invocations for the
RPC test have failed due to offline CPUs and been counted by the
invoker threads.
Add the missing accumulation so that the printed value reflects the
actual counts.
Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
The mmiotrace tracer was added May 12th 2008. At that time, resources
created in pipe_open() could not be freed because there was not
pipe_close function pointer of the tracer. The pipe_close function pointer
was added in December 7th, 2009, but the mmiotrace tracer was not updated.
mmio_pipe_open() allocates a header_iter and takes a pci_dev reference
when trace_pipe is opened. mmio_close() frees them, but it was only
wired to the tracer's .close callback.
tracing_release_pipe() invokes .pipe_close, not .close, when the
trace_pipe file is released. As a result, closing trace_pipe with the
mmiotrace tracer active leaked the header_iter allocation and left a
stale pci_dev reference.
Set .pipe_close to mmio_close, matching how function_graph wires both
callbacks to the same handler.
Note, if the trace_pipe is read to completion, it will clean up the
resources, but if one were to run:
# head -n 1 /sys/kernel/tracing/trace_pipe
VERSION 20070824
Over and over again, it would trigger a massive leak.
Cc: stable@vger.kernel.org
Fixes: c521efd170 ("tracing: Add pipe_close interface)
Link: https://patch.msgid.link/20260715143604.14481-1-gaikwad.dcg@gmail.com
Signed-off-by: deepakraog <gaikwad.dcg@gmail.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
remote_events_dir_enable_write() ignores the return value from
trace_remote_enable_event(). If a remote rejects an event state change,
the write therefore reports success even though the affected event remains
in its previous state.
Keep trying all events, but retain and return the first error. This matches
__ftrace_set_clr_event_nolock(), which permits partial updates while
notifying userspace when an operation fails.
Cc: stable@vger.kernel.org
Link: https://patch.msgid.link/20260715074455.3897-1-liu.yun@linux.dev
Fixes: 775cb093bc ("tracing: Add events/ root files to trace remotes")
Assisted-by: Codex:gpt-5.6-sol
Signed-off-by: Jackie Liu <liuyun01@kylinos.cn>
Reviewed-by: Vincent Donnefort <vdonnefort@google.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
Introduce the "dpm_watchdog_enabled" module parameter to allow the DPM
watchdog to be enabled or disabled at boot time and runtime.
Additionally, introduce the CONFIG_DPM_WATCHDOG_ENABLED Kconfig option
to set the default value of the module parameter at compile time.
The primary motivation for this configurability resolves around Android
GKI (Generic Kernel Image). We want to enable CONFIG_DPM_WATCHDOG in
the GKI so the feature is available. However, because the GKI is shared
across many different devices, we don't want to inadvertently affect
devices that are unaware of this feature. This provides a way to
compile it in, but keep it disabled by default for those devices via the
kernel command line or module parameters.
To maintain backward compatibility, CONFIG_DPM_WATCHDOG_ENABLED relies
on `default y`. Previously, the DPM watchdog was always active if
CONFIG_DPM_WATCHDOG was set. Defaulting this new option to 'y' ensures
that the behavior remains unchanged for existing users and defconfigs
when they upgrade.
Signed-off-by: Tzung-Bi Shih <tzungbi@kernel.org>
Link: https://patch.msgid.link/20260720030821.2780257-3-tzungbi@kernel.org
Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
Delay allocating and linking the next swap_map_page until another image
page actually needs to be recorded.
The previous code linked and wrote a new swap map page as soon as the
current one became full. When the image size was an exact multiple of
MAP_PAGE_ENTRIES, that left an empty final map page that existed only to
terminate the on-disk chain.
Instead, keep a full map page in memory and only allocate the next map page
when the next image page arrives. This preserves the resume chain while
avoiding an unnecessary swap slot allocation and write for the empty
swap map page.
Signed-off-by: Haesung Kim <mattkim513@gmail.com>
Link: https://patch.msgid.link/20260714072627.3165744-1-mattkim513@gmail.com
Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
When memory_bm_create() succeeds for copy_bm but fails for zero_bm,
the function returns without freeing the resources allocated for
copy_bm. This results in a memory leak that includes radix tree nodes,
zone structures, and page lists.
Fix this by calling memory_bm_free() to release copy_bm's resources
before returning the error code when zero_bm allocation fails.
Fixes: 005e8dddd4 ("PM: hibernate: don't store zero pages in the image file")
Signed-off-by: Malaya Kumar Rout <malayarout91@gmail.com>
Acked-by: Brian Geffon <bgeffon@google.com>
Link: https://patch.msgid.link/20260711145246.8625-1-malayarout91@gmail.com
Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
The slot array already tracks allocation and task_monitor_count
duplicates that state. On an invalid second release, the old code
warns but still decrements the counter, corrupting later allocations.
Use the slot array as the sole source of truth. Return after warning
about an unused slot, and return -EBUSY when no slot is free.
Reviewed-by: Gabriele Monaco <gmonaco@redhat.com>
Signed-off-by: Li Qiang <liqiang01@kylinos.cn>
Link: https://lore.kernel.org/r/20260715015825.1413822-1-liqiang01@kylinos.cn
Signed-off-by: Gabriele Monaco <gmonaco@redhat.com>
* Avoid running unit tests without amd-pstate
* Fixes for EPP on shared memory systems
* Fixes for dynamic EPP callbacks
* Avoid loading on guests
* Allow lowest nonlinear == minimum freq
-----BEGIN PGP SIGNATURE-----
iQJHBAABCgAxFiEECwtuSU6dXvs5GA2aLRkspiR3AnYFAmphGlcTHHN1cGVybTFA
a2VybmVsLm9yZwAKCRAtGSymJHcCdkotD/wMShVE4ozDlaNXjQq2LdQB4LK/hbvy
A4Z1BzHVn8ljw8eCkyZB/sB4pjNA0aXnXtTSkimbvwkiQhYbOqoNF5YFA1JWcqNK
IjhUDTXbs+mwVDIU8aRuynkioGa1+jMPlLOExUkngCMTdraftPRUfSIH5I7cQ80q
+qlSMJvhT2S3AuZYhy+HBKJa1Hbl2i36viMuxqmSicz8AwKXEHnZ2EFaf2u+ssat
keF9/S2jvD9g8B/jxiVsK3zCxlH3RBwDQGLWRG7DaKSHLOugLV8BHtv37cNSYyiq
e+TTSOeziUi9A8rGl39WYbIj5qUwrYvJ7c8aLeZ28RT3zS0IlqSrkFGeAw2XfInH
/Kp9khIQdI09Jv5RIKsDrEjCws8zGB9SghxxnEDasagIfCoj4RIEAOvZeBdjvPr+
BMWYe5XLw7VzJIVX6bCBYC3QOld/2YZjV/duu611Cs9ijqYoxWuF9uM/Eorqx889
aj1kDcy/4PtkBM2CwhU6PnM2zwAQr45jIQ9Nc2Scm5PkBc8NQvYtO1L2qRRCoI8s
khfcywX4iOnqoNswskji5c8N9gQUh/eTChOhgJoMkYRZv+yO0IcpDoQpUJUAz1wO
BcTC0ZPESvNb/tAbm6WZFPoxVm33CwnGQr7WdOArx4K/ZyKvC25D/sfzJth/BSI7
g+uEfZuopnuxSw==
=rxGv
-----END PGP SIGNATURE-----
Merge tag 'amd-pstate-v7.3-2026-07-22' of ssh://gitolite.kernel.org/pub/scm/linux/kernel/git/superm1/linux
Merge amd-pstate content for 7.3 (07/22/26) from Mario Limonciello:
"* Avoid running unit tests without amd-pstate
* Fixes for EPP on shared memory systems
* Fixes for dynamic EPP callbacks
* Avoid loading on guests
* Allow lowest nonlinear == minimum freq"
* tag 'amd-pstate-v7.3-2026-07-22' of ssh://gitolite.kernel.org/pub/scm/linux/kernel/git/superm1/linux: (923 commits)
cpufreq/amd-pstate: handle missing policy in dynamic EPP callbacks
cpufreq/amd-pstate: Cache the firmware programmed EPP value
cpufreq/amd-pstate: Toggle auto_sel in active mode on shared memory systems
cpufreq/amd-pstate: Fix EPP return type and handle errors during initialization
cpufreq: amd-pstate-ut: Skip tests when amd-pstate driver is not active
cpufreq/amd-pstate: Prevent the driver from loading on unsupported hardware
cpufreq/amd-pstate: Loosen requirement on lowest nonlinear frequency != min freq
Linux 7.2-rc4
Revert "drm/amd/display: Restore 5s vbl offdelay for NV3x+ DGPUs"
drm/amd/display: check GRPH_FLIP status before sending event
drm/amd/display: consolidate DCN vblank/flip handling onto vupdate_no_lock
drm/amd: Create a device link between APU display and XHCI devices
drm/amd/display: wire DCN42B mcache programming callback
drm/amd/display: set new_stream to NULL after release
drm/amd/display: Force PWM backlight on Lenovo Legion 5 15ARH05
drm/amdkfd: free MQD managers on DQM init failures
drm/amdgpu/ttm: Consider concurrent VM flushes for buffer entities
drm/amd/pm/smu7: Fix AC/DC switch notification
drm/amdgpu: Disable PCIe dynamic speed switching on Ryzen Pinnacle Ridge
drm/amdgpu: always emit the job vm fence
...
page_va[] is annotated __counted_by(nr_page_va), so nr_page_va must
cover an index before that element is accessed. The allocation loop
writes page_va[id] while nr_page_va is still id and increments it only
afterwards, so every write is one element past the declared count.
The store is out of bounds with respect to the annotation: a build with
CONFIG_UBSAN_BOUNDS on a toolchain that honours __counted_by
(clang >= 20.1, gcc >= 15.1) flags it as an array-index overflow.
Increment nr_page_va before writing the element it now covers. A failed
allocation then leaves the slot counted but NULL; the error path frees
it with free_page(0), which is a no-op.
Cc: stable@vger.kernel.org
Link: https://patch.msgid.link/20260713072823.2668323-1-fuad.tabba@linux.dev
Fixes: 96e43537af ("tracing: Introduce trace remotes")
Signed-off-by: Fuad Tabba <fuad.tabba@linux.dev>
Reviewed-by: Vincent Donnefort <vdonnefort@google.com>
Tested-by: Vincent Donnefort <vdonnefort@google.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
These callbacks are sysfs show paths. Use sysfs_emit() and
cpumask_pr_args() to emit the masks.
This prepares for removing cpumap_print_to_pagebuf().
Link: https://lore.kernel.org/all/akANJ-AT7nHpRMq-@yury/
Acked-by: Robin Murphy <robin.murphy@arm.com>
Signed-off-by: Yury Norov <ynorov@nvidia.com>
Commit b0473dcd4b ("smp: Improve smp_call_function_single()
CSD-lock diagnostics") changed smp_call_function_single() so that,
when CSD lock debugging is enabled, async !wait calls use the
destination CPU csd_data. That improves diagnostics, but it also removes
the single-writer property that made the old csd_lock() safe: multiple
CPUs can now prepare the same destination CPU CSD concurrently.
csd_lock() currently waits for CSD_FLAG_LOCK to clear and then sets the
bit with a non-atomic read-modify-write. Two senders can both see an
unlocked CSD, set the bit, overwrite the callback fields, and enqueue
the same llist node. Re-adding a node that is already the queue head can
make node->next point to itself, leaving the target CPU stuck walking
call_single_queue. Later synchronous work, such as a TLB shootdown, can
then remain queued and trigger soft-lockup warnings or panics.
Keep the single csd_lock() implementation, but when CSD lock debugging is
enabled, acquire CSD_FLAG_LOCK with try_cmpxchg_acquire(). This makes the
destination CPU CSD a real atomic lock in the only configuration where it
can be shared by multiple remote senders, while preserving the existing
non-debug fast path.
Fixes: b0473dcd4b ("smp: Improve smp_call_function_single() CSD-lock diagnostics")
Signed-off-by: Chuyi Zhou <zhouchuyi@bytedance.com>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Link: https://patch.msgid.link/20260716004539.13983-2-paulmck@kernel.org
Commit b0473dcd4b ("smp: Improve smp_call_function_single()
CSD-lock diagnostics") made smp_call_function_single() use the destination
CPU's csd_data when CSD lock debugging is enabled. That lets the debug code
associate a stuck CSD lock with the target CPU, but it also means the CPU
argument is used in per_cpu_ptr() before generic_exec_single() has a chance
to validate it.
This becomes unsafe when smp_call_function_any() cannot find an online CPU
in the supplied mask. In that case the selected CPU can be nr_cpu_ids, and
the !wait path calls get_single_csd_data(cpu) before generic_exec_single()
returns -ENXIO. With csdlock_debug_enabled set, that indexes the per-CPU
offset array with an invalid CPU number.
Use the destination CPU's csd_data only when the CPU number is within
nr_cpu_ids. For invalid CPU numbers, fall back to the local CPU's csd_data
and let generic_exec_single() perform the existing validation and return
-ENXIO.
Fixes: b0473dcd4b ("smp: Improve smp_call_function_single() CSD-lock diagnostics")
Signed-off-by: Chuyi Zhou <zhouchuyi@bytedance.com>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Reviewed-by: Paul E. McKenney <paulmck@kernel.org>
Acked-by: Muchun Song <muchun.song@linux.dev>
Link: https://patch.msgid.link/20260716004539.13983-1-paulmck@kernel.org
* Fix validation of LIVEUPDATE_SESSION_GET_NAME ioctl argument caused by a
wrong resolution of a merge conflict during the last merge window
-----BEGIN PGP SIGNATURE-----
iQEzBAABCgAdFiEEeOVYVaWZL5900a/pOQOGJssO/ZEFAmpg7E4ACgkQOQOGJssO
/ZHdXQf+KPPnhNyDqHgHtnsp/sd4wQ5vGVjfmAwtX7nxpxPenGSwhZ8yKShClMNM
2s0KDQmorqKK1j3MMy+xzI+MUl7uHMnHB4cpOsESrILzfqWUgo5VmEU4beCCYXXs
1JD4eKjAs+7F59g0HE75pwmqyXzrD2nq29xWlVxQCOxsTkGe+j28B/uqw0MI8CTd
66LZzyFRT/VBjyaZYSU03KN5heJsFsckYaoPCxK4bZqaiFi65bCyd/YZ7t6JgqDl
PLx1fwjFpmCz6HArFfw3EnwVzkMoZ/qep4h+h+WMH3jiXUUKFHuJWpHYoXYN1C4r
xuQm3Rpq3HmwR/8VHtwLTJSeNXtfBg==
=KyUS
-----END PGP SIGNATURE-----
Merge tag 'liveupdate-fixes-2026-07-22' of git://git.kernel.org/pub/scm/linux/kernel/git/liveupdate/linux
Pull liveupdate fix from Mike Rapoport:
- Fix validation of LIVEUPDATE_SESSION_GET_NAME ioctl argument caused
by a wrong resolution of a merge conflict during the last merge
window
* tag 'liveupdate-fixes-2026-07-22' of git://git.kernel.org/pub/scm/linux/kernel/git/liveupdate/linux:
liveupdate: fix GET_NAME ioctl argument validation
The nesting depth check and the cgroup online check in
scx_sub_enable_workfn() reach err_disable without setting ret, so
the fallback error added by commit db4e9defd2 ("sched_ext: Record
an error on errno-only sub-enable failure") reports
"scx_sub_enable() failed (0)".
This is currently harmless because both paths record their own
scx_error() first and the first error wins, but it leaves the
fallback broken for these paths. Set -EINVAL and -ENODEV there
so the fallback always reports a real errno.
v2: The validate_ops() path from v1 is already fixed in for-7.3
(sub.c already has ret = scx_validate_ops()), so only the two
remaining paths are addressed.
Signed-off-by: Cui Jian <cjian720@163.com>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
SCX's built-in idle CPU tracking is imperfect and can be out-of-sync
with the actual idle state of CPUs, especially immediately after
enabling SCX due to scx_idle_enable() marking all online CPUs idle.
scx_select_cpu_dfl() skips marking the selected CPU as busy if
the selected CPU is the waker CPU in the SCX_WAKE_SYNC case.
If the waker CPU was marked idle by SCX, it will still be marked idle
after CPU selection and potentially even after switching to the wakee.
In the allowed_cpus selftest, this can manifest as the test failing with
the following message in dmesg:
allowed_cpus.bpf.c:21: CPU 0 should be marked as busy
This patch explicitly marks the waker CPU as busy. With this patch,
the test failure no longer reproduces. There are still some pretty
unlikely races that could make the test fail (e.g. pick_task_idle()
marking the selected CPU idle between selection and validation), but
these can't be fixed easily.
Signed-off-by: Kuba Piecuch <jpiecuch@google.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
There has been quite a bit of activity across the ath drivers.
Significant changes in ath12k include:
Align with new Qualcomm generic Peripheral Authentication Service (PAS).
Ongoing infrastructure changes to support the QCC2072 platform.
Ongoing infrastructure changes to support the IPQ5332 platform.
Enhance datapath statistics.
Tuning of datapath parameters.
In addition, an assortment of cleanups and minor bug fixes across ath6kl,
ath10k, ath11k, ath12k, and carl9170.
-----BEGIN PGP SIGNATURE-----
iHUEABYKAB0WIQQ/mtSHzPUi16IfDEksFbugiYzLewUCamDp7AAKCRAsFbugiYzL
ezyJAQC6D29OEEnel7C3H4gPsLdF8oI3pszt9MHFAhKpqCh/7AEAzzuvfNVfewXU
WeP+bkfi+fLPSChX4UxWsw8xx4vVHgM=
=A8cC
-----END PGP SIGNATURE-----
Merge tag 'ath-next-20260722' of git://git.kernel.org/pub/scm/linux/kernel/git/ath/ath
Jeff Johnson says:
==================
ath.git patches for v7.3 (PR #1)
There has been quite a bit of activity across the ath drivers.
Significant changes in ath12k include:
Align with new Qualcomm generic Peripheral Authentication Service (PAS).
Ongoing infrastructure changes to support the QCC2072 platform.
Ongoing infrastructure changes to support the IPQ5332 platform.
Enhance datapath statistics.
Tuning of datapath parameters.
In addition, an assortment of cleanups and minor bug fixes across ath6kl,
ath10k, ath11k, ath12k, and carl9170.
==================
Signed-off-by: Johannes Berg <johannes.berg@intel.com>
arch_bpf_trampoline_size() allocates a buffer to get actual size required
for a trampoline.
This buffer must be in the module address space because
__arch_prepare_bpf_trampoline() calculates rel32 offsets relatively to
that buffer.
In preparation for enabling ROX mode for EXECMEM_BPF make sure that the
allocated memory is writable.
Add bpf_jit_alloc_exec_rw() wrapper for execmem_alloc_rw() and use it for
buffer allocation in arch_bpf_trampoline_size().
Signed-off-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Acked-by: Song Liu <song@kernel.org>
Link: https://lore.kernel.org/bpf/20260716-execmem-x86-rox-bpf-v0-v3-4-4e76158c01c5@kernel.org
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
execmem_alloc() can return ROX memory that is already filled with
architecture defined trapping instructions.
In preparation for enabling this mode for BPF on x86, make sure that there
is no redundant management of the ROX memory.
There is no need to fill allocated memory with trapping instructions, to
request permissions reset on free and to set ROX permissions as this all
is handled by execmem_alloc().
Add bpf_jit_mem_is_rox() wrapper for execmem_is_rox(), use it to check if
execmem_alloc() returns ROX memory and skip the redundant steps in that
case.
Signed-off-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Acked-by: Song Liu <song@kernel.org>
Link: https://lore.kernel.org/bpf/20260716-execmem-x86-rox-bpf-v0-v3-3-4e76158c01c5@kernel.org
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
bpf_jit_alloc_exec() and bpf_jit_free_exec() are wrappers for the
corresponding execmem APIs.
Architectures define the properties of the memory range needed by BPF in
their initialization of execmem and don't need to override neither of
them.
Drop the __weak qualifier from bpf_jit_alloc_exec() and
bpf_jit_free_exec().
Signed-off-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Acked-by: Song Liu <song@kernel.org>
Link: https://lore.kernel.org/bpf/20260716-execmem-x86-rox-bpf-v0-v3-2-4e76158c01c5@kernel.org
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
bpf_dispatcher->rw_image is a temporary writable buffer that
arch_prepare_bpf_dispatcher() fills and then copies into
bpf_dispatcher->image using bpf_arch_text_copy().
The rel32 offsets emitted by emit_bpf_dispatcher() are calculated against
->image, so ->rw_image does not need to live in the module address range.
Allocate ->rw_image with vzalloc() to avoid permissions dance when
EXECMEM_BPF will be backed by ROX caches.
Using vzalloc() rather than vmalloc() ensures that the memory that
bpf_dispatcher_update() unconditionally copies into the executable buffer
is zeroed, which is not ideal but still better than random memory returned
by the existing bpf_jit_alloc_exec() or plain vmalloc().
Switching from bpf_jit_alloc_exec() to vzalloc() also saves a bit of
space in the more scarce module address space.
Signed-off-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Acked-by: Song Liu <song@kernel.org>
Link: https://lore.kernel.org/bpf/20260716-execmem-x86-rox-bpf-v0-v3-1-4e76158c01c5@kernel.org
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Use %pe format specifier instead of %ld with PTR_ERR() to print
error pointers as a symbolic error name (e.g. -ENOMEM) instead
of a raw integer value.
Signed-off-by: Ronan Marchal <ronanmarchal29@gmail.com>
[ rjw: Subject rewrite ]
Link: https://patch.msgid.link/20260615191832.75923-1-ronanmarchal29@gmail.com
Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
Use sysfs_emit() instead of sprintf() in rate_limit_us_show().
sysfs_emit() is the preferred API for sysfs output as it provides
PAGE_SIZE bounds checking and ensures proper sysfs formatting.
No functional change intended.
Signed-off-by: Zhongqiu Han <zhongqiu.han@oss.qualcomm.com>
Link: https://patch.msgid.link/20260716131546.1159644-1-zhongqiu.han@oss.qualcomm.com
Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
Commit 16a03c71bb ("cpufreq: schedutil: Merge initialization code of
sg_cpu in single loop") merged the per-CPU initialization and the
utilization-hook registration into a single loop in sugov_start().
For a shared cpufreq policy this re-introduces the race originally fixed
by commit ab2f7cf141 ("cpufreq: schedutil: Fix sugov_start() versus
sugov_update_shared() race").
The scheduler's util path reaches the hook under RCU-sched and never takes
policy->rwsem, so the rwsem held across sugov_start() cannot serialize the
two. Once the first CPU's hook is published, sugov_update_shared() may run
and, via sugov_next_freq_shared(), read/write each sibling sugov_cpu
(iowait_boost, util, bw_min, ...) concurrently with the memset() still
initializing them, with no lock common to both sides: the update side holds
sg_policy->update_lock while the init side holds only policy->rwsem, which
the scheduler's util path never takes.
The walk only accesses scalar members, never a pointer like ->sg_policy,
so it does not crash today; it merely uses stale (or zero on first start)
values that skew the frequency selection and tracepoints. It is still a
genuine data race, and a latent crash once any pointer member is
dereferenced there.
Restore the two-phase approach: initialize all per-CPU structures first,
and only then publish the per-CPU utilization update hooks.
Fixes: 16a03c71bb ("cpufreq: schedutil: Merge initialization code of sg_cpu in single loop")
Cc: stable@vger.kernel.org
Signed-off-by: Zhongqiu Han <zhongqiu.han@oss.qualcomm.com>
Reviewed-by: Christian Loehle <christian.loehle@arm.com>
Link: https://patch.msgid.link/20260716115159.848403-1-zhongqiu.han@oss.qualcomm.com
Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
Several helpers in snapshot.c are introduced with kernel-doc (/**)
comment blocks but do not describe their parameters with @param
tags.This emits warnings when building
with extra warnings enabled (make W=1), for example:
kernel/power/snapshot.c:469: warning: Function parameter or member
'zone' not described in 'add_rtree_block'
kernel/power/snapshot.c:469: warning: Function parameter or member
'gfp_mask' not described in 'add_rtree_block'
kernel/power/snapshot.c:469: warning: Function parameter or member
'safe_needed' not described in 'add_rtree_block'
kernel/power/snapshot.c:469: warning: Function parameter or member
'ca' not described in 'add_rtree_block'
These are file-local implementation details, not part of the
exported kernel API documented under Documentation/. Replace the
kernel-doc markers with plain block comments for the affected
functions.
Properly documented symbols such as
alloc_rtree_node(), snapshot_read_next() and snapshot_write_next()
remain unchanged.
Signed-off-by: Adi Nata <adinata.softwareengineer@gmail.com>
[ rjw: Subject rewrite ]
Link: https://patch.msgid.link/20260609231626.38839-1-adinata.softwareengineer@gmail.com
Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
The cid tables are visible to the cid kfuncs while being modified: the
first enable publishes the global pointers before filling them,
ops.init_cids() overrides rewrite them in place, and re-enables rebuild
them in place. A racing TRACING or SYSCALL program can read unfilled
entries, including uninitialized memory in the kmalloc'd tables, or torn
topo updates.
Tie the tables' lifetimes to the root sched instead: each root enable
builds a fresh set privately and publishes the per-table __rcu globals once
the layout is final, and root disable unpublishes and RCU-frees the set. A
non-NULL global is now always a fully built table which stays valid for the
reader's RCU read section, and lookups stay two loads. Kfuncs treat NULL as
no-mapping, also after the scheduler exits instead of reporting the stale
last mapping.
The cid kfuncs are available whether the root scheduler is cid-form or
cpu-form, the latter to allow gradual migration to cids. Every root
therefore builds and publishes a default mapping.
Every reader must either be gated on scheduler liveness or NULL-check
inside an RCU read section. Fix the two kfuncs that were neither:
scx_bpf_this_cid() read the table with no RCU or preemption protection and
scx_bpf_task_cid() relied on KF_RCU, which doesn't put a sleepable program
in an RCU read section. The hotplug callbacks are instead serialized by
retiring the tables inside the cpus_read_lock() section that clears
scx_root.
v2: Document why every root builds the tables (desc + cid.c comment).
Reported-by: Andrea Righi <arighi@nvidia.com>
Closes: https://lore.kernel.org/r/al3tLtPZZkFjMveK@gpd4
Reviewed-by: Andrea Righi <arighi@nvidia.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
select_task_rq_scx() falls into the default path when the scheduler has no
ops.select_cpu or is bypassing. There it calls scx_select_cpu_dfl() and
direct-dispatches to the picked CPU's local DSQ.
While bypassing, neither does anything: the enqueue path routes the task to
a bypass DSQ before consulting the direct-dispatch target, so the direct
dispatch never happens, and the CPU pick at most shifts which CPU's bypass
DSQ receives the task. Worse, when the scheduler does its own idle tracking,
the built-in idle cpumasks the pick consults are not even updated, so it
doesn't work anyway.
Return prev_cpu without the default selection while bypassing and let the
bypass enqueue place the task.
Reviewed-by: Andrea Righi <arighi@nvidia.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
check_rq_for_timeouts() blames a runnable stall on the task's owner. Under
a sub-scheduler hierarchy the stalled task can be sitting on a DSQ that a
different scheduler has to drain, e.g. an ancestor's bypass DSQ while the
owner is bypassing. The drainer then escapes blame while the owner is
exited, and when the owner's exit is already claimed, nothing actionable is
reported at all.
Blame the DSQ's owning scheduler instead. The local DSQ is consumed by the
cpu itself and keeps blame on the owner. Detection keeps the owner's timeout
and single-scheduler behavior is unchanged.
Reviewed-by: Andrea Righi <arighi@nvidia.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
BPF_REFCOUNT is not marked as a unique field, while it should be. Fix
this oversight.
Fixes: d54730b50b ("bpf: Introduce opaque bpf_refcount struct and add btf_record plumbing")
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Link: https://patch.msgid.link/20260719153634.2908692-4-memxor@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
btf_find_struct_field() initializes a fresh seen mask for every recursive
descent. Unique special fields in different levels of the same aggregate
therefore do not see one another. The duplicate fields can reach
btf_parse_fields(), where they trigger an invariant WARN_ON_ONCE(). A
crafted user BTF can consequently trigger the warning before map creation
checks capabilities.
Initialize the seen mask once in btf_find_field() and pass the same pointer
through struct, datasec, and nested-struct walks. This gives the entire field
traversal one shared uniqueness state.
Fixes: 64e8ee8148 ("bpf: look into the types of the fields of a struct type recursively.")
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Link: https://patch.msgid.link/20260719153634.2908692-3-memxor@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Sashiko pointed out correctly that the case statement for
BPF_RES_SPIN_LOCK incorrectly checks offset for BPF_SPIN_LOCK.
Fix it by checking res_spin_lock_off instead.
Fixes: 0de2046137 ("bpf: Implement verifier support for rqspinlock")
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Link: https://patch.msgid.link/20260719153634.2908692-2-memxor@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Queue and stack pop/peek helpers accept an uninitialized output buffer
because the verifier expects the helper to initialize it. The empty-map
error path clears the buffer, but a failed lock acquisition returns
-EBUSY without writing it.
Clear the output before returning -EBUSY so BPF programs cannot observe
uninitialized stack contents after a failed helper call.
Fixes: a34a9f1a19 ("bpf: Avoid deadlock when using queue and stack maps from NMI")
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Reviewed-by: Emil Tsalapatis <emil@etsalapatis.com>
Link: https://lore.kernel.org/bpf/20260719125419.1782196-1-memxor@gmail.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Accept only one "region value" pair entry for the dmem.max, dmem.min,
dmem.low files.
This changes the UAPI that otherwise accepted multiple lines for setting
multiple entries in one write. No existing user is known to rely on
writing multiple regions in a single write.
Processing multiple regions in dmemcg_limit_write() could quietly change
first limits before failing on a later one and returning an error to the
writer, with no indication some changes occurred.
Acked-by: Tejun Heo <tj@kernel.org>
Signed-off-by: Eric Chanudet <echanude@redhat.com>
Reviewed-by: Maxime Ripard <mripard@kernel.org>
Reviewed-by: Thadeu Lima de Souza Cascardo <cascardo@igalia.com>
Tested-by: Thadeu Lima de Souza Cascardo <cascardo@igalia.com>
Reviewed-by: Natalie Vock <natalie.vock@gmx.de>
Link: https://patch.msgid.link/20260608-cgroup-dmem-write-single-region-v2-1-b0cd6c4ccf1b@redhat.com
Signed-off-by: Maarten Lankhorst <dev@lankhorst.se>
kprobe_perf_func() calls memset(&entry[1], 0, dsize) prior to calling
store_trace_args(). store_trace_args() populates the entry buffer and
handles dynamic data fields.
Remove this redundant memset call to align with kretprobe_perf_func() and
other probe perf functions.
Link: https://lore.kernel.org/all/178454429309.296567.17379087988750549247.stgit@devnote2/
Assisted-by: Antigravity:gemini-3.5-flash
Signed-off-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
fentry_perf_func() calls memset(&entry[1], 0, dsize) prior to calling
store_trace_args(). store_trace_args() populates the entry buffer and
handles dynamic data fields.
Furthermore, passing dsize (the dynamic data byte length) to memset at
&entry[1] (the start of fixed trace arguments) is inaccurate as it zeroes
from the fixed args area rather than the dynamic data region.
Remove this redundant memset call to align with fexit_perf_func() and other
probe perf functions.
Link: https://lore.kernel.org/all/178454428366.296567.16465331441301815980.stgit@devnote2/
Assisted-by: Antigravity:gemini-3.5-flash
Signed-off-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
trace_fprobe_match_command_head() copies trace_fprobe_symbol(tf) into a
local buffer 'buf' of size MAX_COMMON_HEAD_LEN + 1 using snprintf before
comparing with argv[0].
Since trace_fprobe_symbol(tf) already returns a null-terminated string,
comparing it directly with argv[0] via strcmp() avoids stack buffer usage
and potential symbol truncation at MAX_COMMON_HEAD_LEN.
Link: https://lore.kernel.org/all/178454427449.296567.12336315661120939938.stgit@devnote2/
Assisted-by: Antigravity:gemini-3.5-flash
Signed-off-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
In store_trace_entry_data(), edata is cast to unsigned long for pointer
offset arithmetic before being cast back to unsigned long *. Cast edata
to u8 * instead.
Link: https://lore.kernel.org/all/178454425622.296567.16952341966130280432.stgit@devnote2/
Assisted-by: Antigravity:gemini-3.5-flash
Signed-off-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
In trace_probe_compare_arg_type(), prior to entering the comparison loop,
a->nr_args and b->nr_args are checked for equality. Since the loop
condition is i < a->nr_args, i is guaranteed to be less than b->nr_args
inside the loop.
Remove the redundant (b->nr_args <= i) check.
Link: https://lore.kernel.org/all/178454423769.296567.6694636865644203423.stgit@devnote2/
Assisted-by: Antigravity:gemini-3.5-flash
Signed-off-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
eprobe_dyn_event_match() checks if the target event system in argv[0]
matches ep->event_system using strncmp(ep->event_system, argv[0], len).
However, if ep->event_system is longer than len (e.g. "eprobes" vs
"ep/event"), strncmp() still returns 0 because the first len characters
match.
Check that ep->event_system[len] is '\0' to ensure exact system name
matching.
Link: https://lore.kernel.org/all/178454235856.290363.14872590900774231133.stgit@devnote2/
Fixes: 7d5fda1c84 ("tracing: Fix event probe removal from dynamic events")
Cc: stable@vger.kernel.org
Assisted-by: Antigravity:gemini-3.5-flash
Signed-off-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
In __set_print_fmt(), LEN_OR_ZERO is defined as (len ? len - pos : 0).
If len is non-zero but smaller than pos, len - pos evaluates to a negative
integer. When passed as a size argument to snprintf(), this negative value
is cast to a large unsigned size_t, bypassing buffer size limits.
Ensure len > pos before subtracting to avoid integer underflow.
Link: https://lore.kernel.org/all/178454234934.290363.15247317871499514139.stgit@devnote2/
Fixes: 5bf652aaf4 ("tracing/probes: Integrate duplicate set_print_fmt()")
Cc: stable@vger.kernel.org
Assisted-by: Antigravity:gemini-3.5-flash
Signed-off-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
If trace_probe_log.argc is 0 in __trace_probe_log_err(), the loop
constructing the command string will not execute and p will remain equal to
command. Writing to *(p - 1) will cause an out-of-bounds access before
command. This should not happen, but better to be treated.
Reject if trace_probe_log.argc is 0.
Link: https://lore.kernel.org/all/178454233992.290363.18323091580600697731.stgit@devnote2/
Fixes: ab105a4fb8 ("tracing: Use tracing error_log with probe events")
Cc: stable@vger.kernel.org
Assisted-by: Antigravity:gemini-3.5-flash
Signed-off-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
In trace_probe_match_command_args(), a stack buffer buf[MAX_ARGSTR_LEN + 1]
(256 bytes) is used to format "<name>=<comm>". However, since name can
be up to 32 bytes (MAX_ARG_NAME_LEN) and comm up to 255 bytes
(MAX_ARGSTR_LEN), the formatted string can exceed 256 bytes and get
truncated by snprintf(), causing spurious argument matching failures.
Instead of formatting into a temporary buffer on stack, compare the
argument name, the '=' delimiter, and the comm expression directly.
Link: https://lore.kernel.org/all/178454233010.290363.10428767141343428804.stgit@devnote2/
Fixes: eb5bf81330 ("tracing/kprobe: Add per-probe delete from event")
Cc: stable@vger.kernel.org
Assisted-by: Antigravity:gemini-3.5-flash
Signed-off-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
Reread the syscall number from pt_regs and stop returning the eventually
modified syscall number.
That moves the reread to the end of syscall_trace_enter() and prepares for
moving it to the call site.
No functional change.
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Tested-by: Michal Suchánek <msuchanek@suse.de>
Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com>
Link: https://patch.msgid.link/20260712141346.639115923@kernel.org
Move it out of line and let it reread the syscall number on it's own. That
makes the low level entry code denser and allows to move the reread to the
call site of syscall_trace_enter() once the tracer is fixed up.
To prevent the compiler from putting audit_context() out of line and
thereby breaking dead code elimination, mark audit_context()
__always_inline.
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Tested-by: Michal Suchánek <msuchanek@suse.de>
Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com>
Link: https://patch.msgid.link/20260712141346.576865340@kernel.org
If a watcher deliberately disables interrupts (either by user choice, or
because we're dealing with a scoped reordered access) to avoid detecting
any data races in interrupts, NMIs are still able to fire.
When we set up a watchpoint on a scoped reordered access, we disabled
interrupts because the same CPU cannot observe reordering of its own
accesses. To ensure we observe no false positives from NMIs, disable
access checking for interrupt contexts as well.
Fixes: 69562e4983 ("kcsan: Add core support for a subset of weak memory modeling")
Signed-off-by: Marco Elver <elver@google.com>
Running kernel-doc on timer_migration.h reports:
Warning: kernel/time/timer_migration.h:105 bad line:
The empty line separating the @available paragraph from the @idle
member documentation in the struct tmigr_cpu kernel-doc block lacks
the " *" line prefix that kernel-doc requires on every line inside a
block. The header is not scanned by the build-time kernel-doc checks,
so the warning only shows up when kernel-doc is invoked on the file
directly.
Add the missing prefix. The empty line was introduced when the
@available documentation was expanded by commit 45a13ba52c
("timers/migration: Update stale @online doc to @available").
No functional change.
Signed-off-by: Babanpreet Singh <bbnpreetsingh@gmail.com>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Assisted-by: Claude:claude-sonnet-5
Link: https://patch.msgid.link/20260717035947.7-1-bbnpreetsingh@gmail.com
Hybrid automata monitors's clock variables have two different
representations:
- The invariant representation, which is the timestamp when the invariant
expires
- The guard representation, which is the timestamp when the clock is last
reset
This dual representation makes the logic quite difficult to follow (well,
at least for me). It also complicates the monitors and the generation tool,
as it requires conversion back and forth between the representation.
Simplify by using the clock variables for a single purpose: storing the
time stamp since the clock is last reset.
This also allows simplifying rvgen, which will be done in a follow-up
commit.
Reviewed-by: Gabriele Monaco <gmonaco@redhat.com>
Signed-off-by: Nam Cao <namcao@linutronix.de>
Link: https://lore.kernel.org/r/c0f600dcbf3d8b487c944406851a39146f4d91fa.1781847583.git.namcao@linutronix.de
Signed-off-by: Gabriele Monaco <gmonaco@redhat.com>
EXPORT_SYMBOL_GPL(msi_domain_free_irqs_all) was mistakenly placed after
msi_domain_free_irqs_range() instead of after its intended function
msi_domain_free_irqs_all().
Signed-off-by: Li RongQing <lirongqing@baidu.com>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Reviewed-by: Radu Rendec <radu@rendec.net>
Link: https://patch.msgid.link/20260717143157.1718-1-lirongqing@baidu.com
Two unrelated things go by "cgroup" in the cid form. Sub-schedulers attach
to cgroups, and the cgroup_*() ops deliver cpu controller events. While the
ops names suggest cgroup2 hierarchy, they actually operate on the cpu
controller.
Rename them to cpuctl_* in struct sched_ext_ops_cid, which has no users
outside scx_qmap yet. The cpu form is deployed ABI and keeps the old names.
The layout is unchanged and the kernel keeps calling through the cpu-form
union view.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
Sub-schedulers don't get cgroups yet: every task_group is inited on the root
sched and the routing added by the previous patches always resolves to it.
Add the handover: an enabling sub-scheduler takes over the cgroups in its
subtree and a disabling one returns them to its parent.
scx_cgroup_claim_subtree() runs while the sub enables, after the subtree's
cgrp->scx_sched's are set and before any task is claimed. It inits each
subtree task_group on the sub, exits it from the parent and updates
tg->scx.sched. A failed ops.cgroup_init() unwinds the sub-side inits and
aborts the enable with the parent untouched.
Disabling reverses it with scx_cgroup_return_subtree(): exit each cgroup
from the sub, then re-init it on the parent with the current tg->scx.*
values, resyncing weight and bandwidth changes made while the sub had it.
When a re-init fails, the parent is failed and the remaining task_groups
still transfer uninited and get no cgroup ops - the same punting done for
tasks. The dying parent's own disable moves them onward.
The handover walks include dying but not yet offlined task_groups, the same
as root's bulk walks: a removed cgroup keeps hosting scheduling events until
its dying tasks finish their final context switches, and its
ops.cgroup_exit() must follow the last of them. tg on/offlining is excluded
through cgroup_lock(), so either ordering against an rmdir of a subtree
cgroup delivers balanced init/exit pairs.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
With sub-schedulers claiming cgroup subtrees, cgroup ops must be delivered
to each task_group's sched rather than always to root. Add tg->scx.sched to
track which sched initialized the task_group. It is set and cleared together
with SCX_TG_INITED.
Deliver the ops accordingly:
- ops.cgroup_exit() goes to the sched whose ops.cgroup_init() it pairs with.
- ops.cgroup_prep_move/move/cancel_move() go to the task's sched, and only
for moves that don't re-home the task. A re-homing move is reported
through the ops.exit_task/init_task() pair instead. The cgroups passed to
the move ops can be outside the sched's inited set as the cpu controller
can be coarser than the sub-scheduler topology.
- Knobs of a cgroup belong to the parent, so ops.set_weight/idle/bandwidth()
go to the parent task_group's sched.
All task_groups currently resolve to the root sched, so no behavior changes
until sub-schedulers start claiming cgroups.
While at it, scx_cgroup_init() is restructured so both paths share the
recording.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
A task's sched (p->scx.sched) must match its cgroup's owner
(cgrp->scx_sched). cgroup migration breaks the invariant:
scx_cgroup_move_task() only fires root's ops.cgroup_move() and never
re-homes the task, leading to wrong-sched scheduling and, once the stale
sched is freed, a use-after-free.
Hook into the new cgroup task migration events and re-home each task whose
destination cgroup is owned by a different sched. The events map naturally
to the transfer: MIGRATING runs the fallible init for the destination sched,
letting it reject the migration the same way ops.cgroup_prep_move() can,
MIGRATED does the re-home, which can't fail, and CANCELED undoes the init
when the migration falls through.
Pre-commit, the task's task_group still reflects the source, so
__scx_init_task() grows an explicit cgroup argument for the migration path
to hand ops.init_task() the destination cgroup.
Signed-off-by: Tejun Heo <tj@kernel.org>
Closes: https://lore.kernel.org/r/alnxrsexEe_nQwqL@gpd4
Reviewed-by: Andrea Righi <arighi@nvidia.com>
scx_cgroup_enabled is in the CONFIG_EXT_GROUP_SCHED block. The upcoming
cgroup migration re-homing needs the gate outside the block. Move the
definition and flag flips outside CONFIG_EXT_GROUP_SCHED. No functional
changes.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
Factor out scx_rehome_task() and scx_punt_task() from the sub-disable
re-home loop and scx_fail_parent(). The upcoming cgroup migration re-homing
also needs scx_rehome_task(). No functional changes.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
A subsystem can attach to the cgroup hierarchy itself, independent of which
controllers are enabled where - BPF hooks already behave this way and
sched_ext sub-schedulers do too. Controller callbacks can't track task
migrations for them: sched_ext must re-home a task whose migration crosses a
sub-scheduler boundary, but the cpu controller's attach callbacks fire only
when the task_group changes and miss moves whenever the controller topology
is coarser than the sub-scheduler topology.
Add cgroup_task_notifier with per-task migration events mirroring the
can_attach/attach/cancel_attach phases so that a consumer which prepares
per-task state can also veto a migration: CGROUP_TASK_MIGRATING fires
pre-commit, CGROUP_TASK_MIGRATED post-commit and
CGROUP_TASK_MIGRATE_CANCELED unwinds a failed migration. Only migrations
that change a task's dfl cgroup are reported.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
The BPF_MOV64_PERCPU_REG insn requires JIT to emit native code to for
'dst_reg = src_reg + <percpu_base_off>'.
However, the interpreter ignores the 'off' at its ALU64_MOV_X label.
The 'off' indicates the insn is BPF_MOV64_PERCPU_REG insn. Then, when
the interpreter loads memory from the register, it will hit a page
fault.
[ 2.545572] BUG: unable to handle page fault for address: ffffffffacaaf034
[ 2.546485] #PF: supervisor read access in kernel mode
[ 2.547167] #PF: error_code(0x0000) - not-present page
[ 2.547850] PGD 134e63067 P4D 134e63067 PUD 134e64063 PMD 10021c063 PTE 800ffffeca550062
[ 2.548912] Oops: Oops: 0000 [#1] SMP PTI
Set jit_required as true in order to disallow interpreter fallback in
core.c::__bpf_prog_select_runtime(), if any BPF_ADDR_PERCPU insn is
patched to the prog.
BTW, rename the helper bpf_map_supports_cpu_flags() to
bpf_map_is_percpu_map().
Fixes: 7bdbf74463 ("bpf: add special internal-only MOV instruction to resolve per-CPU addrs")
Signed-off-by: Leon Hwang <leon.hwang@linux.dev>
Link: https://lore.kernel.org/bpf/20260715141122.15783-4-leon.hwang@linux.dev
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
The interpreter does not recognize the BPF_JMP|BPF_JA|BPF_X insn, which
is used for insn_array map. Thereafter, it would hit the BUG_ON() in
___bpf_prog_run() at run time.
[ 2.563726] BPF interpreter: unknown opcode 0d (imm: 0x0)
[ 2.564557] ------------[ cut here ]------------
[ 2.565206] kernel BUG at kernel/bpf/core.c:2349!
[ 2.565882] Oops: invalid opcode: 0000 [#1] SMP PTI
Set jit_required as true when insn_array map is used in the prog in
order to disallow interpreter fallback for gotox insn in
core.c::__bpf_prog_select_runtime().
Fixes: 493d9e0d60 ("bpf, x86: add support for indirect jumps")
Signed-off-by: Leon Hwang <leon.hwang@linux.dev>
Link: https://lore.kernel.org/bpf/20260715141122.15783-3-leon.hwang@linux.dev
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Since the interpreter does not support the arena-related insns,
interpreter fallback should not be allowed for these insns in
core.c::__bpf_prog_select_runtime().
Currently, when the interpreter executes the arena ST/LDX/STX insns,
it would hit the BUG_ON() in ___bpf_prog_run() at run time.
[ 2.579196] BPF interpreter: unknown opcode a2 (imm: 0x0)
[ 2.579998] ------------[ cut here ]------------
[ 2.580652] kernel BUG at kernel/bpf/core.c:2349!
[ 2.581314] Oops: invalid opcode: 0000 [#1] SMP PTI
Set jit_required as true when arena map is used in the prog to disallow
interpreter fallback for arena-related insns.
Fixes: 6082b6c328 ("bpf: Recognize addr_space_cast instruction in the verifier.")
Signed-off-by: Leon Hwang <leon.hwang@linux.dev>
Link: https://lore.kernel.org/bpf/20260715141122.15783-2-leon.hwang@linux.dev
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
bpf_arena_free_pages() accepts scalar arena addresses. The runtime
masks the address to the low 32 bits and reconstructs a full user
address from the arena base before returning the range to the arena
free tree.
When the scalar value is below the low 32 bits of the arena base,
full_uaddr falls below user_vm_start. The existing upper-end clipping
then turns this into an out-of-range free-tree offset. A later
allocation can reuse that offset and return an address below the arena
mapping.
Reject such frees before computing the clipped range.
Fixes: 317460317a ("bpf: Introduce bpf_arena.")
Signed-off-by: Yiyang Chen <chenyy23@mails.tsinghua.edu.cn>
Reviewed-by: Emil Tsalapatis <emil@etsalapatis.com>
Link: https://lore.kernel.org/bpf/20260717-c10-031-public-bpf-next-v2-b4-v2-1-54b555443a7c@mails.tsinghua.edu.cn
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
arena is the only map type whose map_mem_usage() still returns 0, so
"bpftool map show" and fdinfo always showed 0 memlock for an arena no
matter how many pages it had.
Count the pages that are actually mapped into the arena: bump a counter in
apply_range_set_cb() when a page goes in and drop it in
apply_range_clear_cb() when a page goes out, both under the arena spinlock.
map_mem_usage() then just returns nr_pages << PAGE_SHIFT.
Only real data pages are counted, not the scratch page.
Signed-off-by: Jiayuan Chen <jiayuan.chen@linux.dev>
Reviewed-by: Emil Tsalapatis <emil@etsalapatis.com>
Link: https://lore.kernel.org/bpf/20260717114117.350851-3-jiayuan.chen@linux.dev
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Replace the scratch_page field in the pte-callback data with the arena
pointer; later patches use other arena fields from these callbacks. No
functional change.
Signed-off-by: Jiayuan Chen <jiayuan.chen@linux.dev>
Reviewed-by: Emil Tsalapatis <emil@etsalapatis.com>
Link: https://lore.kernel.org/bpf/20260717114117.350851-2-jiayuan.chen@linux.dev
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Pull to receive:
477869bfaf ("sched_ext: Reject setting disallow from init_task outside the enable path")
5f8b69642d ("sched_ext: Take cgroup_lock() first in scx_cgroup_lock()")
8c13364db9 ("sched_ext: Skip sub-disable teardown for never-linked sub-schedulers")
5cdc928598 ("sched_ext: Don't enable non-ext tasks in the sub-sched task loops")
as dependencies for the upcoming cgroup migration patchset and to
resolve the conflicts with the ext.c/sub.c split on for-7.3.
5f8b69642d comments scx_cgroup_lock() which for-7.3 exported for
sub.c. Resolved by keeping the exported version with the comment.
8c13364db9 and 5cdc928598 patch the pre-split sub-sched enable and
disable paths in ext.c which for-7.3 moved to sub.c. Resolved by
applying the never-linked teardown skip and the class gates to sub.c.
Signed-off-by: Tejun Heo <tj@kernel.org>
Root enable and scx_post_fork() enable a task only if it's on the ext class.
Tasks on other classes, possible under an SCX_OPS_SWITCH_PARTIAL root, are
left READY and enabled by switching_to_scx() when they switch over. The sub
enable-commit pass and the sub-disable re-home loop enable unconditionally,
so a fair-class READY task in the subtree becomes ENABLED while not on
sched_ext. A later switch to SCHED_EXT then trips the task state validation
WARN (ENABLED with the previous state not READY) and calls ops.enable() a
second time.
Gate scx_enable_task() on the task's class in both loops.
Fixes: 337ec00b1d ("sched_ext: Implement cgroup sub-sched enabling and disabling")
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
A sub-scheduler enable can fail before scx_link_sched() links the sched into
the hierarchy, e.g. when the parent is already being disabled, and cleanup
still runs the full scx_sub_disable().
That is racy against root disable: drain_descendants() is the only ordering
between a sub's disable-time task walk and root disable's all-task teardown,
and an unlinked sub is invisible to it. Root's teardown can thus run between
the never-linked sub's drain and its walk, exiting every task to no
scheduler.
The walk then trips the membership WARN and re-homes the exited tasks onto
the dying hierarchy, a use-after-free.
Skip the cgroup ownership reset and the task walk if @sch was never linked,
indicated by the empty ->sibling as unlinking only happens later in the same
function. The membership WARN remains valid: a linked sub is always waited
on by an ancestor's drain.
Fixes: 337ec00b1d ("sched_ext: Implement cgroup sub-sched enabling and disabling")
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
scx_cgroup_lock() write-locks scx_cgroup_ops_rwsem and then takes
cgroup_lock(), which can deadlock through kernfs:
scx enable/disable cgroup rmdir cpu.weight write
------------------ ------------ ----------------
cgroup_lock()
percpu_down_write(rwsem)
cgroup_lock()
kernfs_get_active()
percpu_down_read(rwsem)
kernfs_drain()
The enable path waits for the rmdir to release cgroup_mutex. The rmdir,
deactivating the cpu controller's files, waits in kernfs_drain() for the
write's active reference. The write, in scx_group_set_weight(), waits for
the rwsem behind the pending writer.
Take cgroup_lock() first. The set_* paths take no cgroup locks inside the
read side, so a pending write-lock then only waits for read sections that
always run to completion, and no dependency from the rwsem back to
cgroup_mutex remains.
Fixes: a5bd6ba30b ("sched_ext: Use cgroup_lock/unlock() to synchronize against cgroup operations")
Cc: stable@vger.kernel.org # v6.18+
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
The p->scx.disallow revert assumes the root enable path, where the switching
loop reads the reverted policy right afterwards and leaves the task off SCX.
The sub-scheduler disable path also reaches it when re-initializing the
returned tasks on a root parent. Nothing reads the policy there: the task is
enabled on root anyway and keeps running on the ext class with a silently
rewritten policy.
Kill the sched instead, matching the fork and non-root branches, and update
the disallow documentation, which equated !fork with the load path and
pointed at a stale debugfs path for nr_rejected.
Fixes: 337ec00b1d ("sched_ext: Implement cgroup sub-sched enabling and disabling")
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
check_kfunc_call() reads meta.func_name when bpf_fetch_kfunc_arg_meta()
returns -EACCES, but that error can come from fetch_kfunc_meta() (e.g.
fd_array_get_btf() rejecting BTF binding for a signed program) before
meta is memset(), leaving it uninitialized and risking a garbage deref
in verbose().
Move the memset() to the start of bpf_fetch_kfunc_arg_meta() so meta is
zeroed on every error return. The intended "not allowed" -EACCES path
still sets func_name first, so its message is unchanged.
Signed-off-by: Amery Hung <ameryhung@gmail.com>
Acked-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Link: https://patch.msgid.link/20260715172127.2416388-3-ameryhung@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
For a bloom filter, the value argument of bpf_map_peek_elem() is always
an input. Therefore, the verifier should not allow passing uninitialized
stack memory to it to avoid information leak.
bpf_map_peek_elem() tags its value argument ARG_PTR_TO_MAP_VALUE |
MEM_UNINIT, telling the verifier the callee fills the buffer. This holds
for queue/stack maps, but not for a bloom filter, which reads the buffer
as an input to test set membership and never writes it.
As a result, a program can pass an uninitialized stack buffer to
bpf_map_peek_elem() on a bloom filter. The verifier accepts it and marks
the buffer initialized on return, letting the program read back leftover
kernel stack memory. Bloom maps require CAP_BPF to create, so this is a
CAP_BPF-gated stack infoleak that bypasses the boundary CAP_BPF is meant
to enforce (arbitrary kernel reads are gated behind CAP_PERFMON).
Signed-off-by: Amery Hung <ameryhung@gmail.com>
Acked-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Link: https://patch.msgid.link/20260715172127.2416388-2-ameryhung@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
- Fix a UAF in socket clone early bailout paths (Matt Bobrowski)
- Reject unhashed UDP sockets on sockmap update to prevent refcount leaks
(Michal Luczaj)
- Account for receive queue data in FIONREAD on sockmap sockets without a
verdict program (Mattia Meleleo)
- Reject negative constant offsets for verifier buffer pointers (Sun Jian)
- Fix for tracing of kfuncs with implicit arguments (Ihor Solodrai)
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
-----BEGIN PGP SIGNATURE-----
iHUEABYKAB0WIQR0/74CM8h5+1E9pgtHLTd7Y1QvgwUCalp3/AAKCRBHLTd7Y1Qv
g9J/AQCJzidAmStp1i074sZC7sC1uq+aYIvwycpMTp6hVgyP+AEAw7oKcnGkPGXs
jeEDctaLtxBnPMXGAR5UaeqdNqu9ywQ=
=zQgn
-----END PGP SIGNATURE-----
Merge tag 'bpf-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf
Pull bpf fixes from Kumar Kartikeya Dwivedi:
- Fix a UAF in socket clone early bailout paths (Matt Bobrowski)
- Reject unhashed UDP sockets on sockmap update to prevent refcount
leaks (Michal Luczaj)
- Account for receive queue data in FIONREAD on sockmap sockets without
a verdict program (Mattia Meleleo)
- Reject negative constant offsets for verifier buffer pointers (Sun
Jian)
- Fix for tracing of kfuncs with implicit arguments (Ihor Solodrai)
* tag 'bpf-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf:
selftests/bpf: Cover tracing implicit kfunc args
bpf: Fix tracing of kfuncs with implicit args
selftests/bpf: Cover negative buffer pointer offsets
bpf: Reject negative const offsets for buffer pointers
selftests/bpf: Test FIONREAD on a sockmap socket without a verdict program
bpf, sockmap: Account for receive queue in FIONREAD without a verdict program
selftests/bpf: Fail unbound UDP on sockmap update
selftests/bpf: Adapt sockmap update error handling
bpf, sockmap: Reject unhashed UDP sockets on sockmap update
selftests/bpf: Ensure UDP sockets are bound
bpf: Fix UAF in sock clone early bailouts
The function try_enable_preferred_console() currently has the
non-obvious side effect of returning success for consoles that are
already pre-enabled. This obscures the logic flow during console
registration.
Move the check for pre-enabled consoles directly into the top-level
register_console(). This change makes the handling of pre-enabled
consoles explicit and easier to follow.
Furthermore, this separation lays the groundwork for future cleanups
where try_enable_preferred_console() can be restricted to cases where
an entry actually exists in the console_cmdline[] array.
Also it fixes a possible out-of-bound access when the console_cmdline[]
array is full and no console matched. In fact, the check of
c->user_specified did not make much sense. The new console either
matched and was handled in the for-cycle. Or it did not match
and then *c pointed to an unused entry.
Possible behavior change:
try_enable_preferred_console() will newly be called also with
@user_specified parameter set to "false" when it failed with the "true"
variant. But it looks like the right way to do. It will allow to call
newcon->setup() when the console was preferred by some platform
specific code.
Reported-by: Naveen Kumar Chaudhary <naveen.osdev@gmail.com> # out-of-bound access
Closes: https://lore.kernel.org/r/7sq4tr2nmlz32tvkf6vpsghv6exvqfghsrlvywjcqihzsqqbf7@bspclmti5xg4
Reviewed-by: John Ogness <john.ogness@linutronix.de>
Link: https://patch.msgid.link/20260604101459.393162-2-pmladek@suse.com
Signed-off-by: Petr Mladek <pmladek@suse.com>
The SWIOTLB bounce buffer pool size is hardcoded at 64 MB via
IO_TLB_DEFAULT_SIZE with no compile-time knob to adjust it. On
memory-constrained embedded or mobile platforms equipped with a
hardware IOMMU (e.g., ARM SMMU) covering most DMA-capable devices,
reserving 64 MB at boot is unnecessarily wasteful — the SWIOTLB is
only exercised for devices that bypass the IOMMU or have restricted
DMA address ranges.
Introduce CONFIG_SWIOTLB_DEFAULT_SIZE_MB, an integer Kconfig option
(range 1–64 MB, default 64) that allows platforms to set a smaller
compile-time default. IO_TLB_DEFAULT_SIZE is updated to derive from
this value when CONFIG_SWIOTLB is enabled, preserving the existing
64 MB default when the option is not configured.
The runtime "swiotlb=<nslabs>" kernel parameter override remains
fully supported and takes precedence over the compile-time default.
Signed-off-by: Jagadeesh Pagadala <jpagadal@qti.qualcomm.com>
Signed-off-by: Bibek Kumar Patro <bibek.patro@oss.qualcomm.com>
Reviewed-by: Michael Kelley <mhklinux@outlook.com>
Link: https://lore.kernel.org/r/20260702-swiotlb-v2-1-9205f3ba5408@oss.qualcomm.com
Signed-off-by: Marek Szyprowski <m.szyprowski@samsung.com>
Some JIT compilers, such as x86_64, rely on a register to pass the TCC.
When subprograms of synchronous callback invoke tailcall, C helpers
invoking bpf callback clobber this register, and the corrupted TCC may
bypass the TCC limit, leading to infinite tailcall.
Fix this by rejecting tailcall inside all subprogs of sync callback.
This also cleanly consolidates the existing async and exception callback
checks into a single unified `is_cb` check.
Reported-by: Sashiko <sashiko-bot@kernel.org>
Reported-by: Björn Töpel <bjorn@kernel.org>
Signed-off-by: Pu Lehui <pulehui@huawei.com>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://patch.msgid.link/20260716120157.835937-3-pulehui@huaweicloud.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Currently in check_max_stack_depth_subprog, when the verifier enters a
new callee branch, the local tail_call_reachable is not properly
synchronized with the callee's state.
Consider a main prog branching into multiple subprogs:
subprog0 -> tailcall
main <
subprog1 -> subprog2
When the verifier finishes checking subprog0 and backtracks to main
prog, the local tail_call_reachable state is left as true. As it
proceeds to subprog1, this uncleared state leaks into the new branch,
falsely marking subprog1 and subprog2 as tailcall reachable.
Fix this by explicitly syncing tail_call_reachable with the callee's
has_tail_call state on entry. The caller's state is safely preserved and
restored via the existing backtracking logic.
Fixes: ebf7d1f508 ("bpf, x64: rework pro/epilogue and tailcall handling in JIT")
Reported-by: Sashiko <sashiko-bot@kernel.org>
Signed-off-by: Pu Lehui <pulehui@huawei.com>
Link: https://patch.msgid.link/20260716120157.835937-2-pulehui@huaweicloud.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
A kfunc marked with KF_IMPLICIT_ARGS flag takes implicit arguments
(such as bpf_prog_aux) that the verifier injects at load time.
resolve_btfids strips those from the kfunc's BTF-visible prototype and
keeps the real kernel ABI in a counterpart _impl prototype [1].
fentry/fexit/fmod_ret/fsession programs may attach to the BPF kernel
functions, including those with implicit args. However
bpf_check_attach_target() and bpf_check_attach_btf_id_multi() extract
the struct btf_func_model from the wrong BTF prototype of the
kfunc. The btf_func_model is later read to construct the trampoline,
which then causes the injected implicit argument to be clobbered and
the kfunc dereferencing garbage.
Add btf_attach_func_proto() to resolve the real ABI prototype of the
kfunc the way the call site does: by looking up the _impl prototype
for a KF_IMPLICIT_ARGS kfunc. Use it at both attach-target model
construction sites.
To enable this, make two supporting changes:
* pass bpf_verifier_log instead of bpf_verifier_env to
find_kfunc_impl_proto(), so it can be reused from the attach path
* add btf_kfunc_check_flag() to test a flag across all of a kfunc's
hook sets, because a program attaching to a kfunc is not in the
kfunc's call-set
KF_IMPLICIT_ARGS must be consistent across the sets, so
btf_kfunc_check_flag() returns -EINVAL on inconsistency.
btf_kfunc_check_flag() reads the kfunc's flags from the target's
kfunc_set_tab. For a module BTF that table is stable only after the
module is live, so take a module reference around the read, mirroring
how the kfunc call path gates the same lookup with btf_try_get_module().
The remaining call sites of btf_distill_func_proto() are safe as
is. The BPF_TRACE_ITER case distills a registered iterator's
prototype, and bpf_struct_ops_desc_init() distills the
function-pointer members of a struct_ops type. Neither is a kfunc, and
so can't have implicit arguments.
[1] https://lore.kernel.org/all/20260120222638.3976562-1-ihor.solodrai@linux.dev/
Fixes: 64e1360524 ("bpf: Verifier support for KF_IMPLICIT_ARGS")
Reported-by: Tejun Heo <tj@kernel.org>
Signed-off-by: Ihor Solodrai <ihor.solodrai@linux.dev>
Link: https://github.com/sched-ext/scx/issues/3687#issuecomment-4906694106
Link: https://patch.msgid.link/20260713235223.1639022-2-ihor.solodrai@linux.dev
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
With cgroup v2 migration of a multithreaded process having threads
in different cgroups of a threaded subtree, it is possible that
cpuset_can_attach() can be called with tasks that are not migrating with
respect to cpuset if cpuset controller is not enabled in some of the
subtree cgroups. IOW, the old cpuset can be the same as the new one. This
can cause problem when we need to track the set of old cpusets and the
new cpusets in singly linked lists as a cpuset cannot be in both lists.
As reported by Tejun, the following is an example threaded subtree with
partial cpuset delegation that can cause this issue to show up.
P (+cpuset)
|- R (cpuset) <- destination
| `- C (no cpuset) -> effective cpuset == R
`- W (cpuset)
Group leader in R, thread_a in C, thread_b in W; migrate the whole
process into R (echo $PID > R/cgroup.procs). thread_a moves C->R:
its cgroup changes so compare_css_sets() keeps it in the taskset, but
its cpuset css is unchanged (C inherits R's), so task_cs() == cs ==
R. cpuset is in ss_mask because thread_b (W->R) changed. can_attach()
then tags R as a source (thread_a) and the destination (thread_b):
Handle this special case by skipping tasks that are not migrating in
cpuset_can_attach() and avoid calling cpuset_can_attach_check() in this
case. By doing so, the destination cpuset will not be put into source
cpuset linked list.
As the source cpuset cannot be easily determined in cpuset_attach(),
unnecessary work can be performed if a task is not actually
migrating. However, no harm will be done except wasting some
CPU cycles. If it happens that none of the tasks is migrating,
attach_ctx.old_cs will be NULL and task iteration won't be needed.
Reported-by: Tejun Heo <tj@kernel.org>
Closes: https://lore.kernel.org/lkml/e254af713b5345aec3d086771ecf1e71@kernel.org
Signed-off-by: Waiman Long <longman@redhat.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
The only case where the cgroup_taskset structure requires task migration
to multiple cpusets is when enabling a cpuset controller in cgroup v2
where the newly created child cpusets inherits the same effective CPUs
and memory nodes from the parent. In that case, task migration can happen
directly with no update to tasks' CPU and memory nodes assignment and no
further work needed from the cpuset side except updating nr_deadline_tasks
when DL tasks are involved and setting old_mems_allowed in the child
cpusets.
Do that by tracking all the destination cpusets with a new dst_cs_head
singly linked list. The reset_migrate_dl_data() function is integrated
into clear_attach_data() so that it can be used for both source and
destination cpusets.
A warning will be printed if there are multiple destination cpusets but
it is not on default hierarchy or when the CPUs or memory nodes change.
Signed-off-by: Waiman Long <longman@redhat.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
The smp_call*() functions handle their required preemption and CPU
pinning internally. The explicit preempt_disable() in
scftorture_invoke_one() is therefore no longer required for correctness.
Keeping the outer preempt_disable() would also prevent scftorture from
exercising the narrowed internal preemption-disabled regions during IPI
dispatch.
Removing the outer preemption protection can expose a CPU hotplug race in
the test validation when use_cpus_read_lock is false. For multicast
operations, SCF_PRIM_MANY or SCF_PRIM_ALL, if only one CPU is online,
smp_call_function_many() correctly skips sending IPIs and leaves scfc_out
false. Without preemption disabled, a CPU hotplug thread can preempt the
test thread, bring a second CPU online and increment num_online_cpus().
When the test thread resumes, the validation check can observe
num_online_cpus() > 1 and falsely trigger the memory-ordering warning,
leaking the scfcp structure.
Remove the preempt_disable() and preempt_enable() pairs around the
smp_call*() invocations in scftorture_invoke_one(). Restrict the
num_online_cpus() > 1 validation to the use_cpus_read_lock=true case,
where the CPU count is stable during the evaluation.
Signed-off-by: Chuyi Zhou <zhouchuyi@bytedance.com>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Tested-by: Paul E. McKenney <paulmck@kernel.org>
Reviewed-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Link: https://patch.msgid.link/20260709122933.4021501-10-zhouchuyi@bytedance.com
smp_call_function_many_cond() handles the preemption and CPU pinning
requirements internally. on_each_cpu_cond_mask() only builds the call
flags and forwards the request to that helper.
Remove the outer preempt_disable() and preempt_enable() pair from
on_each_cpu_cond_mask().
Signed-off-by: Chuyi Zhou <zhouchuyi@bytedance.com>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Tested-by: Paul E. McKenney <paulmck@kernel.org>
Reviewed-by: Muchun Song <muchun.song@linux.dev>
Reviewed-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Link: https://patch.msgid.link/20260709122933.4021501-9-zhouchuyi@bytedance.com
smp_call_function_many_cond() handles the preemption and CPU pinning
requirements internally. smp_call_function() only forwards the request to
that helper for cpu_online_mask and does not access CPU-local state on
its own.
Remove the outer preempt_disable() and preempt_enable() pair from
smp_call_function().
Signed-off-by: Chuyi Zhou <zhouchuyi@bytedance.com>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Tested-by: Paul E. McKenney <paulmck@kernel.org>
Reviewed-by: Muchun Song <muchun.song@linux.dev>
Reviewed-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Link: https://patch.msgid.link/20260709122933.4021501-8-zhouchuyi@bytedance.com
smp_call_function_many_cond() still has to keep the caller pinned to the
current CPU while the remote IPI request is built and dispatched. This
protects the queueing state and CPU-hotplug boundary that are required
before the synchronous wait starts:
- It protects the current CPU's per-CPU scratch cpumask,
cfd->cpumask_ipi. Another task running on the same CPU could otherwise
enter smp_call_function_many_cond() and reuse that scratch cpumask
before the current caller has finished building and sending the IPI
request.
- It provides the CPU-hotplug exclusion required by the CSD queueing
side. New CSDs must not be queued after smpcfd_dying_cpu() has flushed
the outgoing CPU's callback queue. Keeping preemption disabled until
all required CSDs have been queued and the corresponding IPIs have
been sent prevents CPU offline from crossing that boundary in the
middle of the queueing operation.
The CSD acquisition side also relies on that caller-side CPU pinning.
csd_lock() waits for CSD_FLAG_LOCK to clear and then marks the CSD busy
with a regular store, so another task on the same CPU must not be
allowed to acquire and reinitialize the same per-CPU CSD concurrently.
After the callbacks have been queued and the IPIs have been sent, the
caller only performs the final csd_lock_wait() completion wait. If it is
preempted there, another task running on the original CPU may enter
smp_call_function_many_cond(), but any attempt to reuse the same per-CPU
CSD will block in csd_lock() until the previous callback clears
CSD_FLAG_LOCK. The final csd_lock_wait() does not acquire or reinitialize
the CSD, so it does not need the same caller-side preemption-disabled
protection.
The wait mask is task-local, so it cannot be overwritten by another task
on the original CPU. The per-CPU CSD storage also remains allocated
across CPU offline, so csd_lock_wait() can safely dereference it even if
the target CPU is offlined after the caller is unpinned.
With those requirements satisfied, enable preemption before the
synchronous csd_lock_wait() loop. This makes the potentially long wait
preemptible and migratable while keeping the CPU-pinned section around
the remote CPU selection and IPI dispatch.
Signed-off-by: Chuyi Zhou <zhouchuyi@bytedance.com>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Tested-by: Paul E. McKenney <paulmck@kernel.org>
Reviewed-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Link: https://patch.msgid.link/20260709122933.4021501-7-zhouchuyi@bytedance.com
smp_call_function_many_cond() uses per-CPU CSD objects when queueing
callbacks to remote CPUs, and the wait path later dereferences those CSDs
from csd_lock_wait().
Making the wait path preemptible allows the initiating task to be
preempted or migrated before it waits for completion. A target CPU can be
offlined in that window. If smpcfd_dead_cpu() frees the target CPU's
per-CPU CSD storage, csd_lock_wait() can later dereference freed memory.
One way to protect the CSD storage is to free it via RCU or after a
synchronization step in the CPU offline path, but that would add
unnecessary complexity and can delay CPU shutdown.
Allocate the per-CPU CSD storage the first time a CPU comes up and keep
it allocated when the CPU is offlined. This allows csd_lock_wait() to
access the CSD even when the target CPU is offlined after preemption is
re-enabled and before the wait is invoked.
Signed-off-by: Chuyi Zhou <zhouchuyi@bytedance.com>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Tested-by: Paul E. McKenney <paulmck@kernel.org>
Reviewed-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Acked-by: Muchun Song <muchun.song@linux.dev>
Link: https://patch.msgid.link/20260709122933.4021501-6-zhouchuyi@bytedance.com
smp_call_function_many_cond() uses the per-CPU cfd->cpumask as the list
of remote CPUs to wait for. That is safe while the caller remains pinned
to the current CPU for the whole operation, because another task cannot
run on the same CPU and reuse the per-CPU mask.
The synchronous wait is the long-latency part of the operation. To make
that wait preemptible, the mask iterated by csd_lock_wait() must remain
stable even if the task is preempted or migrates. If the wait used the
per-CPU cfd->cpumask after dropping CPU pinning, another task scheduled
on the original CPU could enter smp_call_function_many_cond() and
overwrite the mask while the first task is still iterating it.
Give each task private IPI cpumask storage and use it as the wait mask in
smp_call_function_many_cond(). Other cpumask storage choices do not fit
this use case:
- Per-CPU storage is the state that becomes unsafe once the wait is
made preemptible. After the caller drops CPU pinning, another task
scheduled on the original CPU can enter smp_call_function_many_cond()
and reuse the same per-CPU mask.
- Stack storage is not suitable for large NR_CPUS or
CONFIG_CPUMASK_OFFSTACK=y configurations. The wait mask needs to
scale with cpumask_size(), and putting that storage on the stack is
not acceptable on large systems.
- Allocating the mask inside smp_call_function_many_cond() would put an
allocation and a failure path in the generic IPI path. A sleeping
allocation is not suitable because callers have historically only
provided a preempt-disabled context, not a sleepable one. GFP_ATOMIC
would avoid sleeping, but a failure fallback would make the latency
improvement opportunistic instead of guaranteed.
The users are not limited to a small, pre-identifiable class of tasks. On
x86, ordinary tasks can reach this path through TLB flushes during exit,
unmap and reclaim, so allocating the mask only for a known subset of
tasks is not straightforward.
The memory cost is explicit: one word is added to task_struct. When
cpumask_size() fits in that word, the mask is stored inline and no
separate allocation is needed. Larger systems allocate cpumask_size() per
task; on x86-64 NR_CPUS=8192 this is 1 KiB per task. For context,
x86 already carries several KiB of per-task architecture and FPU state,
depending on the enabled features and configuration. That does not make
the extra cpumask free, but it puts the large-NR_CPUS case in
perspective.
Signed-off-by: Chuyi Zhou <zhouchuyi@bytedance.com>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Tested-by: Paul E. McKenney <paulmck@kernel.org>
Reviewed-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Link: https://patch.msgid.link/20260709122933.4021501-5-zhouchuyi@bytedance.com
smp_call_function_any() disables preemption across the entire operation:
selecting a target CPU, enqueueing the IPI, and synchronously waiting for
the remote CPU. smp_call_function_single() already re-enables preemption
before the synchronous csd_lock_wait(), so callers of
smp_call_function_any() should benefit from the same shorter
preemption-disabled section.
Simply removing get_cpu() and put_cpu() from smp_call_function_any()
would leave the preemption disablement entirely to
smp_call_function_single(). That opens a preemption window between
selecting the remote CPU, for example via sched_numa_find_nth_cpu(), and
dispatching the IPI in smp_call_function_single(). If the selected CPU is
fully offlined in that window, smp_call_function_single() fails its
cpu_online() check and returns -ENXIO to the caller, violating the
guarantee that smp_call_function_any() executes on any online CPU in the
mask.
Move the remote CPU selection into a common
__smp_call_function_single() helper. Keep the target CPU selection and
IPI dispatch within the same preemption-disabled region, while still
allowing the wait path to use the shorter preemption-disabled section
provided by smp_call_function_single().
Signed-off-by: Chuyi Zhou <zhouchuyi@bytedance.com>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Tested-by: Paul E. McKenney <paulmck@kernel.org>
Reviewed-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Link: https://patch.msgid.link/20260709122933.4021501-4-zhouchuyi@bytedance.com
smp_call_function_single() disables preemption while it validates the
target CPU, prepares the call single data, queues the callback and sends
the IPI.
For the !wait case, preemption protects the per-CPU csd_data from
concurrent modification by another task on the same CPU. For the wait
case, the CSD is stack allocated and no other task can reuse it. CPU
pinning is still required until the callback has been queued and the IPI
has been sent, to ensure that the target CPU cannot be offlined after the
online check but before dispatch.
After generic_exec_single() has queued the callback, the synchronous
csd_lock_wait() invocation at the end of the execution does not require
the caller to remain pinned to the current CPU.
Enable preemption before csd_lock_wait() to shorten the
preemption-disabled section.
Signed-off-by: Chuyi Zhou <zhouchuyi@bytedance.com>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Tested-by: Paul E. McKenney <paulmck@kernel.org>
Reviewed-by: Muchun Song <muchun.song@linux.dev>
Reviewed-by: Steven Rostedt (Google) <rostedt@goodmis.org>
Reviewed-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Link: https://patch.msgid.link/20260709122933.4021501-3-zhouchuyi@bytedance.com
The CSD lock wait debugging code in __csd_lock_wait() must run with
preemption disabled. The smp function call mechanisms which invoke it
currently keep preemption disabled across the wait, so the debugging code
inherits that guarantee from its callers.
Keeping preemption disabled across the whole smp function call operation
can induce large scheduling latencies. Shortening the caller-side
preemption-disabled region would invoke __csd_lock_wait() with preemption
enabled.
Prepare for that by disabling preemption explicitly around the CSD lock
wait debugging code in __csd_lock_wait().
Signed-off-by: Chuyi Zhou <zhouchuyi@bytedance.com>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Tested-by: Paul E. McKenney <paulmck@kernel.org>
Reviewed-by: Steven Rostedt (Google) <rostedt@goodmis.org>
Reviewed-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Acked-by: Muchun Song <muchun.song@linux.dev>
Link: https://patch.msgid.link/20260709122933.4021501-2-zhouchuyi@bytedance.com
LIVEUPDATE_SESSION_GET_NAME was developed in the liveupdate/next branch
while the session type validation change was carried in liveupdate-fixes.
When the conflict between the two branches was resolved, the GET_NAME
operation descriptor picked up the structure and last member from
RETRIEVE_FD.
This makes both its known size and minimum size 16 bytes rather than 72.
A zero-initialized request still succeeds because luo_session_get_name()
writes the full name before luo_ucmd_respond() copies the full GET_NAME
response to userspace. However, copy_struct_from_user() treats the
output-only name field as unknown trailing data and rejects the request
with -E2BIG if any byte in that field is nonzero.
Use the GET_NAME structure and its name field in the descriptor.
Link: https://lore.kernel.org/all/ahWlYXNjGUbkKoHy@sirena.org.uk/
Assisted-by: Codex:gpt-5.6-sol
Reviewed-by: Pratyush Yadav (Google) <pratyush@kernel.org>
Signed-off-by: Jackie Liu <liuyun01@kylinos.cn>
Link: https://patch.msgid.link/20260716012607.22020-1-liu.yun@linux.dev
Signed-off-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
With CONFIG_EXT_SUB_SCHED=y but no sub-scheduler attached - the common case
- hot paths still pay for sub-sched bookkeeping. Gate it behind
__scx_has_subs, a static key counting live sub-schedulers, so that a
root-only system stops paying.
Most conversions are simple skip-if-no-sub tests. scx_idle_notify() is
special - it's a hierarchy walk, so give it a fast path which notifies the
root directly using the same tests as the walk. A pending
SCX_RQ_SUB_IDLE_RENOTIFY can be ignored as no sub can be owed one and the
caller clears the flag either way.
Suggested-by: Andrea Righi <arighi@nvidia.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
rq->scx.sub_dispatch_prev is sub-sched-only but was left unconditional. Move
it into the CONFIG_EXT_SUB_SCHED block next to ecaps_to_sync and gate its
updates.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
scx_dispatch_sched() is common dispatch machinery and looks out of place in
sub.h, but it needs scx_cpu_arg() from cid.h and can't move into internal.h
without creating a circular include. Add inlines.h on top of internal.h and
cid.h, and move the function there. The function was sub.h's only cid.h
user, so drop that include. Pure code move, no functional change.
v2: Host the function in a new inlines.h instead of at internal.h's tail,
which formed a circular include with cid.h. Drop sub.h's now-unused
cid.h include. (sashiko AI)
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
scx_discard_ecaps_to_sync() waited for balance_one() to consume a dying
sched's queued ecaps sync, polling with resched_cpu() + msleep(). The wait
is unbounded - the ext dl_server forces picks through sustained fair or RT
load only while ext tasks are queued, so an ext-idle cpu monopolized by a
higher class can stall the teardown indefinitely.
Remove the node directly instead: take all queued nodes, drop the dying
sched's and resplice the rest. Consumption runs under the rq lock and batch
nodes read as on-list throughout, so the producer-side dedup stays correct.
A node that an in-flight scx_process_sync_ecaps() batch holds across a
dispatch-induced rq unlock still needs a wait, but one bounded by that batch
completing rather than by a future balance.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
This is needed by rust/helpers/srcu.c which now adds
rust_helper_srcu_readers_active() as a wrapper around the SRCU helper
for Rust callers.
To achive this:
1- Move the srcu_readers_active() implementation from
"kernel/rcu/srcutree.c" to "include/linux/srcutree.h".
2- Implement a matching srcu_readers_active() in
"include/linux/srcutiny.h" and use it on the existing open-coded
WARN_ON() check in cleanup_srcu_struct().
Signed-off-by: Onur Özkan <work@onurozkan.dev>
Reviewed-by: Gary Guo <gary@garyguo.net>
Reviewed-by: Alice Ryhl <aliceryhl@google.com>
Reviewed-by: Boqun Feng <boqun@kernel.org>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Restructure the SRCU initialization functions so it always follows
one direction:
init_srcu_struct() -> __init_srcu_struct() -> lockdep or generic
This uses the same wrapper style as mutex. It avoids the old confusing
style where init_srcu_struct() and __init_srcu_struct() called each
other in different configs. It also helps Rust side to have simpler
helper for SRCU initialization.
Signed-off-by: Onur Özkan <work@onurozkan.dev>
Reviewed-by: Gary Guo <gary@garyguo.net>
Reviewed-by: Alice Ryhl <aliceryhl@google.com>
Reviewed-by: Boqun Feng <boqun@kernel.org>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
This commit abstracts the open-coded dumping of reader segments in the
rcu_torture_cleanup() function into a new rcu_torture_dump_read_segs()
function. This abstraction will allow reader segments to be dumped for
other purposes.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Now that Tasks Trace RCU is implemented in terms of SRCU, it no longer
has any particular need for the IRQ_WORK Kconfig option. This commit
therefore removes the "select IRQ_WORK" from the TASKS_TRACE_RCU Kconfig
option.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
The kerneldoc of sugov_iowait_apply() says the IO boost value is increased
in sugov_iowait_apply() and, in the same sentence, that it is decreased by
the same function. That is self-contradictory, and the first part is wrong:
sugov_iowait_apply() only decreases the boost.
The boost is actually increased in sugov_iowait_boost(). Fix the comment to
name sugov_iowait_boost() as the place where the boost is increased, so it
matches the code.
No functional change.
Fixes: fd7d5287fd ("cpufreq: schedutil: Cleanup and document iowait boost")
Signed-off-by: Zhongqiu Han <zhongqiu.han@oss.qualcomm.com>
Reviewed-by: Christian Loehle <christian.loehle@arm.com>
Link: https://patch.msgid.link/20260703092433.4080165-1-zhongqiu.han@oss.qualcomm.com
Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
In some cases, the processor may not actually stick to the "desired"
performance level programmed through the driver's .adjust_perf()
callback and may go above it, which may not be desirable (for instance,
there may be a UCLAMP_MAX limit set for the task currently running on
the given CPU which should be respected).
Address that by adjusting the .adjust_perf() callback to take an
additional argument, max_perf, representing the maximum allowed
performance level of the CPU and update the intel_pstate driver to
take that argument into account as appropriate.
Accordingly, adjust cpufreq_driver_adjust_perf() and the other existing
user of .adjust_perf(), which is the amd-pstate driver (but the behavior
of that driver is not changed).
While at it, also update the cpufreq_driver_adjust_perf()
documentation to reflect this change and some previous code
changes that have not been taken into account in it.
Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
Acked-by: Viresh Kumar <viresh.kumar@linaro.org>
Reviewed-by: Zhongqiu Han <zhongqiu.han@oss.qualcomm.com>
Reviewed-by: Mario Limonciello (AMD) <superm1@kernel.org>
Link: https://patch.msgid.link/6277654.lOV4Wx5bFT@rafael.j.wysocki
[ rjw: Adjusted Rust function formatting ]
Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
psi_schedule_rtpoll_work() is called locklessly from the scheduler hotpath
and can race psi_trigger_destroy() taking down the last rtpoll trigger under
rtpoll_trigger_lock:
psi_schedule_rtpoll_work() psi_trigger_destroy()
rcu_read_lock();
task = rcu_dereference(rtpoll_task);
rcu_assign_pointer(rtpoll_task, NULL);
timer_delete(&rtpoll_timer);
mod_timer(&rtpoll_timer, ...);
rcu_read_unlock();
synchronize_rcu();
kthread_stop(task_to_destroy);
The group can then be freed with the re-armed timer still pending, and
poll_timer_fn() runs on freed memory.
461daba06b ("psi: eliminate kthread_worker from psi trigger scheduling
mechanism") deleted the timer synchronously after the synchronize_rcu(),
which prevented this but raced trigger creation instead: the deletion could
cancel the timer that a new trigger set armed during the grace period and,
as creation also reinitialized the timer at the time, corrupt it.
8f91efd870 ("psi: Fix race between psi_trigger_create/destroy") moved the
initialization into group_init() and the deletion into the locked section,
trading the creation races for the window above.
Neither placement in the destruction path works. A pending timer firing
while the group is alive is harmless though. poll_timer_fn() just wakes the
rtpoll waitqueue and doesn't re-arm itself. Bind the timer to the group's
lifetime instead and shut it down in psi_cgroup_free(). Nothing can arm it
by then. timer_shutdown_sync() because the timer is never armed again.
Fixes: 8f91efd870 ("psi: Fix race between psi_trigger_create/destroy")
Cc: stable@vger.kernel.org # v5.10+
Reported-by: Sashiko AI <sashiko-bot@kernel.org>
Closes: https://lore.kernel.org/all/20260711000434.36C4A1F000E9@smtp.kernel.org/
Signed-off-by: Tejun Heo <tj@kernel.org>
Acked-by: Johannes Weiner <hannes@cmpxchg.org>
Tested-by: Matt Fleming <mfleming@cloudflare.com>
Acked-by: Suren Baghdasaryan <surenb@google.com>
a5b98009f1 ("sched/psi: fix race between file release and pressure write")
made pressure_write() hold cgroup_mutex across psi_trigger_create(), which
forks the psimon kthread for the first rtpoll trigger. As kthread creation
depends on the whole fork path, the commit inadvertently created a lot of
unwanted locking dependencies from cgroup_mutex.
sched_ext got hit by one: its enable path blocks forks and then grabs
cgroup_mutex, so a pressure write racing a scheduler enable deadlocks, with
every other fork piling up behind.
Fix it by splitting trigger creation so that the worker is forked with
cgroup_mutex dropped and the kernfs active reference left broken. The latter
matters because rmdir and cgroup.pressure writes drain active references
under cgroup_mutex. Publishing the trigger last keeps error reporting
synchronous and preserves the of->priv lifetime rules.
The trigger registered in the first stage pins the group's rtpoll machinery
across the unlocked window, leaving only creation races to resolve. The
catch-up poll on installation covers scheduling attempts dropped while there
was no worker.
v2: Retagged sched/psi (was cgroup).
Fixes: a5b98009f1 ("sched/psi: fix race between file release and pressure write")
Cc: stable@vger.kernel.org
Cc: Edward Adam Davis <eadavis@qq.com>
Cc: Chen Ridong <chenridong@huaweicloud.com>
Reported-by: Matt Fleming <mfleming@cloudflare.com>
Closes: https://lore.kernel.org/all/20260710100441.2653477-1-matt@readmodwrite.com/
Signed-off-by: Tejun Heo <tj@kernel.org>
Acked-by: Johannes Weiner <hannes@cmpxchg.org>
Tested-by: Matt Fleming <mfleming@cloudflare.com>
Acked-by: Suren Baghdasaryan <surenb@google.com>
The verifier rejects variable offsets for PTR_TO_TP_BUFFER and PTR_TO_BUF
accesses, but it currently accepts a constant negative offset produced by
pointer arithmetic.
Commit 022ac07508 ("bpf: use reg->var_off instead of reg->off for
pointers") moved constant pointer offsets from reg->off to reg->var_off.
However, __check_buffer_access() continued to check only the instruction
offset. An access with reg->var_off equal to -8 and an instruction offset
of zero therefore passes verification.
For writable raw tracepoints, the access end is also calculated from the
unsigned reg->var_off.value. An eight-byte access starting at -8 wraps
the calculated end to zero, allowing the program to load and attach
without increasing max_tp_access.
After ensuring that reg->var_off is constant, calculate the effective
access start using signed arithmetic and reject it when it is negative.
Use the validated start to calculate the access end for both
PTR_TO_TP_BUFFER and PTR_TO_BUF.
Fixes: 022ac07508 ("bpf: use reg->var_off instead of reg->off for pointers")
Signed-off-by: Sun Jian <sun.jian.kdev@gmail.com>
Acked-by: Shung-Hsi Yu <shung-hsi.yu@suse.com>
Cc: stable@vger.kernel.org # 5.2.0
Link: https://patch.msgid.link/20260714093846.18159-2-sun.jian.kdev@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Helper and kfunc argument checking carried two separate meta structs: the
verifier-local struct bpf_call_arg_meta and bpf_kfunc_call_arg_meta.
Merge them into a single struct bpf_call_arg_meta. This is groundwork for
sharing argument checking between helpers and kfuncs.
While merging, drop the btf_id field from the helper meta since it is
never used.
No functional change.
Signed-off-by: Amery Hung <ameryhung@gmail.com>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://lore.kernel.org/bpf/20260715064047.1793790-8-ameryhung@gmail.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
meta->pkt_access is only ever a copy of fn->pkt_access, assigned once in
check_helper_call() and read back in may_access_direct_pkt_data(). Have
may_access_direct_pkt_data() take the bpf_func_proto and read
fn->pkt_access directly, and drop the meta field along with its
assignment.
The only non-NULL caller, check_func_arg(), already has fn in scope; the
remaining callers pass NULL and are unaffected.
No functional change intended.
Suggested-by: Eduard Zingerman <eddyz87@gmail.com>
Signed-off-by: Amery Hung <ameryhung@gmail.com>
Link: https://lore.kernel.org/bpf/20260715064047.1793790-7-ameryhung@gmail.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
The constant "size of the PTR_TO_MEM returned in R0" argument is handled by
both helpers (ARG_CONST_ALLOC_SIZE_OR_ZERO) and kfuncs (__rdonly_buf_size /
__rdwr_buf_size), each with its own meta field (meta->mem_size,
meta->r0_size) and duplicated validation. Add struct arg_alloc_mem_desc
and a shared process_const_alloc_mem_size(), and replace both fields with
meta->arg_alloc_mem.
The desc records presence with a 'found' flag instead of using a non-zero
size as the sentinel. This also fixes a pre-existing bug on the kfunc
return path: "no size argument" was tested as r0_size == 0, so an explicit
__rdonly_buf_size/__rdwr_buf_size of 0 was treated as absent and fell
through to btf_resolve_size(), giving R0 the size of the pointed-to return
type instead of 0. With 'found', an explicit zero size is honored and
btf_resolve_size() is used only when no size argument was passed.
The size is stored in a u32, matching regs[R0].mem_size. The U32_MAX
check now apply to both helper and kfunc through
process_const_alloc_mem_size().
Fold bpf_session_cookie return size assignment into current kfunc return
size resolution path.
Note that verifier saves kfunc return size through r0_size instead of
mem_size. The later has no active readers so remove it.
Signed-off-by: Amery Hung <ameryhung@gmail.com>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://lore.kernel.org/bpf/20260715064047.1793790-5-ameryhung@gmail.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
check_func_arg() only ever handles register arguments (the caller
loops over the first MAX_BPF_FUNC_REG_ARGS arguments), so a single
argno_t built from the register number identifies the argument for
every callee. Remove the duplicated argno_from_reg() calls to
simplify check_func_arg().
'regno' is still kept for the few places that need the raw register
number directly (register reads, verbose R%d messages) and for
referring to the neighbouring size/memory argument in the
ARG_CONST_SIZE{,_OR_ZERO} cases.
No functional change intended.
Suggested-by: Eduard Zingerman <eddyz87@gmail.com>
Signed-off-by: Amery Hung <ameryhung@gmail.com>
Link: https://lore.kernel.org/bpf/20260715064047.1793790-4-ameryhung@gmail.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
To prepare for unifying the helper and kfunc call_arg_meta, group the
scattered MEM_UNINIT ("raw") memory argument fields (raw_mode, regno and
access_size) into a new struct arg_raw_mem_desc. The intention is to
make it clear about when these are set and used instead of fields with
overly generic names.
Identify the raw argument once, up front, in check_raw_mode_ok() (like
check_proto_release_reg() does for release_regno), recording its regno.
check_stack_range_initialized() now recognizes the raw buffer by matching
that regno, so the separate raw_mode flag is no longer needed, and the
per-argument "meta->raw_mode = arg_type & MEM_UNINIT" assignments in
check_func_arg() go away with it.
A raw memory argument can be tagged either ARG_PTR_TO_MEM | MEM_UNINIT or
ARG_PTR_TO_MAP_VALUE | MEM_UNINIT (the output buffer of bpf_map_pop_elem()
and bpf_map_peek_elem()). Either may be passed as a PTR_TO_STACK, which
reaches check_stack_range_initialized() through check_helper_mem_access(),
so both must be treated as raw. Extend arg_type_is_raw_mem() to match the
map value case as well; otherwise check_raw_mode_ok() would not record the
regno for it and an uninitialized stack buffer passed to those helpers
would be wrongly rejected for programs without CAP_PERFMON.
No functional change intended. This patch does not enable raw_mode
memory access for kfunc (i.e., uninit stack will not be allowed to be
passed to kfunc for unprivileged programs). Existing kfuncs with arguments
tagged with __uninit are either priviledged or dynptr kfuncs, which take
another path to make sure the access is checked by check_mem_access().
Signed-off-by: Amery Hung <ameryhung@gmail.com>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://lore.kernel.org/bpf/20260715064047.1793790-3-ameryhung@gmail.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
For a MEM_UNINIT ("raw mode") helper argument,
check_stack_range_initialized() open-coded a scan that rejected any
STACK_DYNPTR slot in the range with "potential write to dynptr". This
duplicated, and was stricter than, the handling that runs when the
buffer is actually marked initialized. check_helper_call() later replays
the write byte by byte via check_mem_access(), which goes through
destroy_if_dynptr_stack_slot(), which rejects overwritting a referenced
dynptr. Therefore drop the redundant scan and rely on check_mem_access().
Signed-off-by: Amery Hung <ameryhung@gmail.com>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://lore.kernel.org/bpf/20260715064047.1793790-2-ameryhung@gmail.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
luo_session_alloc() calls INIT_LIST_HEAD(&session->file_set.files_list)
followed immediately by luo_file_set_init(), which also performs
INIT_LIST_HEAD() on the same list head. Remove the duplicate call.
Signed-off-by: Hongfu Li <lihongfu@kylinos.cn>
Reviewed-by: Pratyush Yadav <pratyush@kernel.org>
Link: https://patch.msgid.link/20260714074056.95335-1-hongfu.li@linux.dev
Signed-off-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
To avoid potential stack overflows on limited kernel stacks, convert
parse_probe_arg() from a recursive function into a loop-based
implementation with a simple local state stack.
Since recursion is eliminated using a loop with a fixed-size
stack in the context, this restricts the dereference nesting
depth. The maximum nesting depth of dereferences is now restricted
to the same limit as typecasts (TRACEPROBE_MAX_NESTED_LEVEL, which
is 8) and reports the same TOO_MANY_NESTED error. Update ftrace
selftests to reflect this restriction and simplify the checks.
Note that this change slightly alters the behavior of nested
dereferencing in fetcharg. Previously, dereferencing without BTF
allowed for up to 14 levels of nesting, whereas dereferencing
with BTF was limited to 3 levels. With this change, the nesting
depth is now limited to 8 levels in both cases.
Link: https://lore.kernel.org/all/178399141396.27810.5390060618628718661.stgit@devnote2/
Assisted-by: Antigravity:gemini-3.5-flash
Signed-off-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
To support BTF argument parsing (such as accessing fields within nested
structures via typecasting), the maximum argument string length needs
to be extended. Extend MAX_ARGSTR_LEN from 63 to 255.
Since MAX_ARGSTR_LEN was previously reused to format command heads in
trace_*probe_match_command_head() functions, introduce a dedicated
MAX_COMMON_HEAD_LEN (63) macro for matching command heads and switch
these functions to use the new macro.
Link: https://lore.kernel.org/all/178399140457.27810.11387684872148824707.stgit@devnote2/
Assisted-by: Antigravity:gemini-3.5-flash
Signed-off-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
Decompose parse_probe_vars() by extracting retval, stack, current task
struct, and function argument parsing logic into dedicated static
helper functions (parse_probe_var_retval, parse_probe_var_stack,
parse_probe_var_current, and parse_probe_var_arg). This simplifies
parse_probe_vars() and improves its readability.
Link: https://lore.kernel.org/all/178399137612.27810.14869178899276413372.stgit@devnote2/
Assisted-by: Antigravity:gemini-3.5-flash
Signed-off-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
fp pointer and unsigned long have the same size on all relevant
architectures that build Linux. Furthermore this struct is only used in
architectures that do not set ARCH_DEFINE_ENCODE_FPROBE_HEADER which is
set only for 64bit architectures (apart from LoongArch).
Both fields are aligned on these architectures so the struct with
__packed and without it are the same.
Remove the __packed as it is unnecessary.
[Masami: Removed Fixes tag because this is not fixing any problem.]
Link: https://lore.kernel.org/all/20260428-topic-fprobe-packed-v7-1-v1-1-9abc9b866b4c@baylibre.com/
Signed-off-by: Markus Schneider-Pargmann (The Capable Hub) <msp@baylibre.com>
Signed-off-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
When tracing the kernel local variables, sometimes we need to get the
CPU local variables. To access it, current simple dereference is not
enough.
Thus, introduce a special this_cpu_read() dereference to access per-cpu
variable for the current CPU (accessing other CPU variable may race with
updates on other CPUs). Also this_cpu_ptr() is for accessing per-cpu
pointer.
Those are working as same as the kernel percpu macro.
Link: https://lore.kernel.org/all/178271367680.1176915.4711734074448973989.stgit@devnote2/
Signed-off-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
Since we can use the BTF to cast value to a structure pointer type,
it is useful to introduce "$current" special variable support to
fetcharg.
User can define a fetcharg to access current task_struct properties
using BTF info. e.g.
$current->cpus_ptr
Link: https://lore.kernel.org/all/178271366709.1176915.15320906169981578568.stgit@devnote2/
Signed-off-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
Add a field specifier option for the typecast. This works like
container_of() macro.
(STRUCT[,FIELD[.FIELD2...]])VAR
This is equivalent to :
container_of(VAR, struct STRUCT, FIELD[.FIELD2...])
For example:
echo "f tick_nohz_handler next_tick=(tick_sched,sched_timer)timer->next_tick" >> dynamic_events
This will trace tick_nohz_handler() with its tick_sched::next_tick which
is converted from @timer by contianer_of(tick, struct tick_sched, sched_timer).
So, if you enabkle both fprobes:tick_nohz_handler__entry and
timer:hrtimer_expire_entry events, we will see something like:
<idle>-0 [002] d.h1. 3778.087272: hrtimer_expire_entry: hrtimer=00000000d63db328 f
unction=tick_nohz_handler now=3777450051040
<idle>-0 [002] d.h1. 3778.087281: tick_nohz_handler__entry: (tick_nohz_handler+0x4
/0x140) next_tick=3777450000000
Link: https://lore.kernel.org/all/178271365745.1176915.725923927180862257.stgit@devnote2/
Signed-off-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
This allows type casting to various fetchargs without parentheses
by recursively calling parse_probe_arg on the target when type
casting is used.
For example, this allows the following expressions:
- (STRUCT)%REG->FIELD
- (STRUCT)$stackN->FIELD
- (STRUCT)@SYM->FIELD
Note that @SYM+/-OFFSET with typecast needs parentheses like:
- (STRUCT)(@SYM-8)->FIELD
Link: https://lore.kernel.org/all/178271364804.1176915.17600229399446264813.stgit@devnote2/
Signed-off-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
When we hit an open parenthesis right after typecast closing
parenthesis, it means we have nested typecast. This allows us to
typecast a generic data member in a structure to a pointer to
another structure.
For example, to cast a DATA_MEMBER of VAR structure to STRUCT pointer
and get MEMBER value.
(STRUCT)(VAR->DATA_MEMBER)->MEMBER
Also, we can nest typecast.
(STRUCT1)((STRUCT2)$ARG->FIELD2)->FIELD1
Currently the max nest level is limited to 3.
This also allows user to use typecasting for registers or stacks on
kprobe events. e.g.
(STRUCT)(%ax)->MEMBER
(STRUCT)($stack0)->MEMBER
Link: https://lore.kernel.org/all/178271363855.1176915.16793301788257446529.stgit@devnote2/
Signed-off-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
Support BTF typecast feature on other probe events, but only if it is
kernel function entry or return, and must use function parameter name
or $retval. This means you can do:
(STRUCT)PARAM->MEMBER
Note: you can not use other variables like $stackN, %reg etc. That
needs nesting support.
To support other probe events, we just need to use last_struct type
when we find a function parameter in parse_btf_arg().
This also updates <tracefs>/README file to show struct typecast.
Link: https://lore.kernel.org/all/178271362928.1176915.12235759508786922490.stgit@devnote2/
Signed-off-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
A cid-form scheduler can grant caps to and revoke them from its child
sub-schedulers but has no way to tear one down. Add scx_bpf_sub_kill() to
evict a direct child with a printf-style reason that reaches the child's
scx_exit_info. No exit code is taken because the child is a separate
scheduler whose exit-code semantics the parent cannot know. The child and
its subtree are disabled through the usual async path under a new exit kind,
SCX_EXIT_PARENT_KILL.
The bstr formatting infrastructure in ext.c is exposed through internal.h
with scx_ prefixes so the kfunc, which lives in sub.c, can format the
reason.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
scx_process_sync_ecaps() consumes ecaps syncs while the sched is bypassing
without delivering ops.sub_ecaps_updated(), leaving reported_ecaps stale.
Nothing re-queued a sync when bypass lifted, so a cid whose caps never
change again would never be notified. Attach-time initial grants hit this
every time: they are consumed during the enable bypass window, so a sched
never learned its initial effective caps through the callback.
Re-queue a sync for every (sched, cpu) with an undelivered delta at the
per-cpu bypass exit in scx_bypass(), next to the idle renotify catch-up. The
next balance on the cpu then delivers the pending delta with proper dispatch
context.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
__scx_update_idle() notified only the root scheduler. A sub-scheduler that
holds a cid needs that cid's idle state to place and kick on it.
Deliver ops.update_idle() to every scheduler that holds SCX_CAP_BASE on the
transitioning cid. The root holds every cap, so a real transition always
reaches it.
Real transitions are not enough on their own. A cid that is already idle
when a sub-sched gains baseline access produces no transition, so the new
holder would never learn it is idle. The ecaps sync arms a re-notify on the
gain, and the next idle pick delivers ops.update_idle() to just that sched,
leaving holders that already track the cpu untouched. A matching loss of
baseline access drops any pending re-notify.
Bypass suppresses ops.update_idle() too, so a cpu that goes idle during a
bypass window and stays idle yields no transition to re-deliver on
un-bypass. Arm the same re-notify for every sched leaving bypass. The acute
case is a child granted cids during its own ops.sub_attach(). The grant
lands while the child is bypassed and the notify walk skips it, so on
un-bypass it holds cids it never saw go idle. The root is owed the same and
is armed through a separate per-rq flag, which keeps this working when
sub-schedulers are compiled out.
v2: Gate the idle catch-up in pick_task_idle() to avoid a double ops.update_idle(). (sashiko AI)
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
scx_local_or_reject_dsq() authorizes a local-DSQ insert against the caps
of the scheduler doing the insert. On the consume/dispatch paths that is
the scheduler running balance_one(), passed down through
scx_consume_dispatch_q() and move_local_task_to_local_dsq(), so the check
is correct.
The remote-move path loses it. move_remote_task_to_local_dsq()
re-activates @p on the destination rq through enqueue_task_scx(), which
reconstructs the scheduler from the task, i.e. @p's owner. When an
ancestor places a descendant's task - e.g. draining a bypassed
sub-scheduler - the owner is a sub-scheduler of the placer, so
authorizing against the owner checks a narrower cap set and can
spuriously reject a task the placer is entitled to run.
Carry the placing scheduler across the activate_task() boundary the same
way enq_flags already are, via a per-rq field set only for the duration
of the re-activation, and have scx_local_or_reject_dsq() authorize
against it. The placer's caps are a superset of the owner's, so this
admits what the placer may run and keeps rejecting what it may not.
v2: Document @sch in move_remote_task_to_local_dsq()'s kerneldoc. (Andrea)
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
A kick forces a scheduling event on the target cpu, and a preemption also
evicts the running task. Gate both on caps. Any kick requires baseline
access on the cid, and preempting a task the sub-sched does not own -
whether by a SCX_ENQ_PREEMPT insert or a SCX_KICK_PREEMPT kick - requires
the new SCX_CAP_PREEMPT. Gating either alone would leave a hole - the
weakest cap authorizing preempting kicks, or plain kicks disturbing cpus the
kicker has no access to.
Preempting the sched's own subtree is always allowed, and the cap extends
the right to any task on the cid. PREEMPT implies ENQ, and so ENQ_IMMED.
A preempting insert tests the running task under the target rq lock and is
rejected and reenqueued unless the victim is in the inserter's subtree or it
holds PREEMPT. A migration-disabled task is admitted regardless, but with
SCX_ENQ_PREEMPT stripped.
Kicks are enforced on the delivery path, where the effective caps can be
read coherently under the target rq's lock. A kick from a sub-sched lacking
SCX_CAP_BASE on the cid is dropped, and a SCX_KICK_PREEMPT kick without
PREEMPT for a task outside the kicker's subtree degrades to a plain
reschedule.
Unlike the enqueue caps, PREEMPT is checked only at the instant of the
insert or kick, never as a standing property of a queued task.
v2: Clear SCX_ENQ_PREEMPT on the offline and migration_pending force-admits.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
Add SCX_CAP_ENQ, which gates inserting tasks onto a cid's local DSQ. Unlike
IMMED enqueue, plain enqueues can pile up, so ENQ is the stronger cap and
implies ENQ_IMMED. Losing ENQ also triggers the reenq scan. The scan tests
each queued task and the running task against the cap each needs via
scx_caps_for_task(), so an ENQ-only loss reenqueues plain tasks, evicting a
running one, while IMMED tasks, which need only ENQ_IMMED, stay put.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
A task's slice grants it cpu occupancy - how long it holds its cpu. In a
sub-scheduler hierarchy cpu access is delegated through revocable
capabilities, so a task's occupancy must follow them. Only its own scheduler
sets its slice, and extending the slice is allowed only while that scheduler
holds baseline cpu access (SCX_CAP_BASE) on the cpu. Otherwise a scheduler
could keep occupying a cpu it has been denied simply by handing out long
slices.
The cap check reads effective caps, which are coherent only under the task's
rq lock, and the kernel decrements the slice under that lock as the task
runs, so a running task's slice can be changed only there while a queued
task's can be set directly. Make scx_bpf_task_set_slice() apply the slice
under the rq lock. Synchronously when the caller already holds it, otherwise
by stashing it in the new p->scx.slice_oob, tagged with the scheduler's id
so a request that outlived a reassignment is dropped. Whether the caller
holds @p's current rq lock is tested with p->scx.runnable_cpu.
Revocation is enforced through the same grant. When a cpu's effective caps
lose SCX_CAP_BASE, the cap-revoke reenq scan also checks the running task
and zeroes its slice to evict it. The scan runs as a balance callback after
the pick, so this catches both the task that was running when the revoke
landed and a capless task the pick just promoted off the local DSQ. The
paths that keep a task on its cpu - holding on to the last runnable task in
balance, the ENQ_LAST reinsertion and the slice refill on pick - skip tasks
lacking baseline access. A migration-disabled task is exempt, mirroring its
capless admission on insert.
v4: Test rq ownership with p->scx.runnable_cpu, closing a remote-wakeup TOCTOU. (sashiko AI)
v3: Keep a pending out-of-band slice request across refill and preserve. (sashiko AI)
v2: Only write slice directly when @p is queued on the held rq. (sashiko AI)
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
Add p->scx.runnable_cpu, the cpu @p is runnable on, or -1 when it is not.
It is stamped as @p joins the runnable_list (set_task_runnable()) and
cleared as it leaves (clr_task_runnable()), both under the rq lock.
task_cpu() can't answer "is @p on this rq" reliably: a remote wakeup changes
it under @p's pi_lock alone, without the source rq lock, so it can read as
the locked rq while @p is really elsewhere. runnable_cpu changes only under
the rq lock, so a caller holding an rq lock can compare against it to know
whether that is @p's current rq.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
A later change makes set_task_slice() also drop a pending out-of-band slice
request, so the BPF-triggered writes to p->scx.slice need to funnel through
one place. Introduce set_task_slice() and route those writes through it.
update_curr_scx() decrements curr->scx.slice directly for accounting and is
left alone. No functional change - the helper only assigns p->scx.slice.
v2: Reword the set_task_slice comment to "BPF-triggered writes". (Andrea)
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
Neither a scx_sched pointer nor its cgroup id uniquely identifies a
scheduler instance. A freed sched's memory can be reallocated, and a cgroup
can detach one sched and attach another. Add a monotonic, never-reused u64
id. A later patch compares it to drop a slice request that outlived a change
of a task's owning scheduler.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
Replace the __SCX_CAP_DUMMY placeholder with SCX_CAP_ENQ_IMMED, which gates
inserting IMMED tasks onto a cid's local DSQ. An IMMED enqueue is guaranteed
to either get its task running on the cpu at once or hand it back to the
scheduler, so IMMED work can never pile up on the cpu's queue and a cpu can
be shared across sub-scheds through IMMED access without any of them
swamping it.
That makes ENQ_IMMED the natural baseline, the minimal cap to make any use
of a cpu. SCX_CAP_BASE aliases it so gates on basic cpu access can state the
intention instead of naming ENQ_IMMED.
Enforcement covers inserts and queued tasks. An insert without the cap is
diverted to the reject DSQ, and queued tasks are reenqueued when the cap is
lost. scx_bpf_sub_dispatch() skips a child that lacks the cap on the cpu, as
its inserts would only be rejected. Vacating the running task on cap loss
lands in a later patch.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
A SAVE/RESTORE requeue re-inserts a running task in place and is immediately
followed by set_next_task_scx(). It is not a real scheduling event: the task
is already admitted to its cid and must return to the local DSQ
unconditionally.
scx_caps_for_enq() maps an enqueue to the cap its local-DSQ insert requires.
Add SCX_ENQ_IGNORE_CAPS, set it on the RESTORE-in-place branch of
enqueue_task_scx(), and have scx_caps_for_enq() require no caps for it, so
the cid admission gate never diverts an in-place restore to the reject DSQ.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
When a sub-scheduler dispatches a task to a CPU it lacks the required
capability on, the task must be rejected rather than allowed to run.
Add the machinery for that. Each rq gets a reject DSQ, a kernel-internal
holding queue that is never run and that the BPF scheduler cannot reach. An
insert that must be refused is diverted there instead of the local DSQ, and
a deferred requeue then hands the parked tasks back to the BPF scheduler to
re-decide. A cap revoke extends this to already-queued tasks. When the
revoke reaches the cpu's effective caps, the cpu scans its local DSQ and
reenqueues the tasks that no longer qualify.
A migration-disabled task must run on its cpu, so a capless one is admitted
anyway and counted in the new SCX_EV_SUB_FORCED_ADMIT event.
This is preparation for the actual sub-sched cap enforcement. The divert is
wired but inert here.
v2: Admit offline-rq and migration_pending inserts to local, not reject. (sashiko AI)
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
The local DSQ is synchronized by the containing rq lock rather than its own
dsq->lock. A later patch adds a second such DSQ. In preparation, factor the
"rq owns the lock" test into dsq_is_rq_owned() and rename
local_dsq_post_enq() to rq_owned_post_enq(), taking @rq explicitly.
No behavior change.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
A sub-scheduler that gains or loses effective caps on a cpu may want to act
on it right away - e.g. place or preempt on a newly usable cpu. The existing
ops.sub_caps_updated() doesn't fit as it is delivered asynchronously to
scheduling operations and can arrive before the per-cpu effective caps go
live.
Add ops.sub_ecaps_updated(cid, before, after), a cid-form callback fired
from scx_process_sync_ecaps() when a sub-sched's effective caps on a cid
change. It runs in dispatch context so the sched can insert, kick or preempt
on the cid directly. @before is the caps as of the last delivery.
Cpu hotplug rides the same machinery. Going down zeroes each sched's ecaps
on the cpu's cid, with queued syncs discarded at consumption while the cpu
is inactive. Coming back up queues a sync for every sched. reported_ecaps is
kept across the down/up cycle, so the resync fires the callback only if
ownership actually changed while the cpu was down.
v2: Compute cid below the active-cpu guard; discard queued syncs on !cpu_active(). (sashiko AI)
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
Checking a sched's caps on a cid would need to test several cap bits against
caps[] to account for implied caps. Also, caps[] modifications aren't
synchronized against scheduling operations on each cpu, which can lead to
awkward race conditions.
Collect them per cpu instead. caps[] under pshard->lock stays the target
configuration. scx_sched_pcpu->ecaps is added, the transposed effective
copy: the set of cap bits the sched holds on that cpu which can be accessed
with a single read. It is stable under the rq lock. It can also be read
locklessly with READ_ONCE().
Grant and revoke only mutate caps[]. They queue a sync request on the target
cpu's rq->scx.ecaps_to_sync and kick it, and the cpu recomputes the queued
scheds' ecaps from caps[] in balance_one() under its own rq lock. A dying
sched runs the sync directly to retire its queued request before freeing. As
held references can defer the freeing past the enclosing root scheduler's
lifetime, root enable discards leftover sync requests before going live.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
Wire up ops_cid.sub_caps_updated() to notify sub-scheds of cap changes.
Three constraints shape the design:
1. Static memory. Deliveries use a fixed-size buffer, both for runtime
efficiency and so notifications can't be lost under memory pressure.
2. High-frequency updates. Grant/revoke can mutate caps in bursts, and the
notifier path must absorb that without amplifying it.
3. Recursive grant/revoke from the callback. A child receiving a
notification can call grant/revoke on its own children, which can
cascade recursively down its subtree.
(1) and (2) lead to coalescing into a fixed payload. Each delivery carries a
single (cmask, caps) pair covering every change since the previous one.
Direction (set vs cleared) isn't encoded as it doesn't fit in the fixed-size
summary. The callback queries scx_bpf_sub_caps() for current state. Only one
delivery is in flight per shard. Further changes fold into the same buffer
and ship as the next callback, so a shard's callbacks fire in order.
(3) leads to deferred delivery. Events accumulate during grant/revoke and
are delivered after the shard lock is released.
v2:
- Request a private stack for ops.sub_caps_updated(). (sashiko AI)
- Build cmask_arena_out via scx_cmask_ref, not by re-reading its header.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
Caps are per-cid permissions parents delegate to direct children via
scx_bpf_sub_grant() / scx_bpf_sub_revoke(). A child's cap set is always a
subset of its parent's. Sub-scheds check their caps locally, and cross-sched
communication is needed only when the delegation set itself changes.
Caps will be used to implement sub-sched scheduling on the enqueue path.
Picking a cid for a task at a leaf depends on which cids the leaf is allowed
to use, and resolving that programmatically on every enqueue would mean a
cross-sched round-trip call chain, possibly retrying if the request can't be
granted as-is. The dispatch path is different - it runs as top-down
recursion via scx_bpf_sub_dispatch().
Locking is per shard. cid space is split into shards, and each sub-sched has
its own pshard->lock for each shard. Operations are broken up on shard
boundaries. Different shards never contend. Shards are expected to be
topology-aligned and likely to serve as the locality unit when cids are
allocated to schedulers, so per-shard lock granularity scales naturally with
the allocation pattern.
This patch adds the framework with a single dummy cap. Real caps land in
later patches.
The enable path is reordered for pshards. scx_arena_pool_init() moves ahead
of scx_link_sched() so the pshards are allocated before the sched becomes
reachable - scx_alloc_pshards() skips allocation when the arena pool isn't
initialized.
- scx_bpf_sub_grant(): Per-cid all-or-nothing grant to direct child.
- scx_bpf_sub_revoke(): Clear caps on @cmask across @child and its subtree.
- scx_bpf_sub_caps(): Lockless snapshot of caps on a cid range.
/sys/kernel/sched_ext/SCHED/caps shows the caps each scheduler currently
holds.
v4: Move the pshard[] full build/publish and the err_disable scx_error() recording to earlier patches. (sashiko AI)
v3: Build pshard[] fully before publishing it, read it with READ_ONCE. (sashiko AI)
v2: Validate ops before scx_link_sched() publishes the sub. (sashiko AI)
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
A scheduler's BPF programs can outlive it. A timer it armed or a tracing
program it loaded can fire after ops.exit() has run, before the programs are
unloaded, and scx_prog_sched() still resolves the program to its scheduler
through ops->priv. Harmless while kfuncs touch only lifetime-stable state,
but a hazard once a kfunc reads global state a newly loaded scheduler can
change underneath it.
Add scx_sched->dead, set right after ops.exit() and drained with
synchronize_rcu(). It follows exit() rather than preceding it so exit()'s own
kfunc calls still resolve to @sch. scx_prog_sched() returns NULL for a dead
scheduler, so every kfunc's existing !sch bail rejects it at one choke
point.
v2: Check dead in the CONFIG_EXT_SUB_SCHED=n scx_prog_sched() too. (sashiko AI)
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
Factor the sibling/ancestor portion of scx_next_descendant_pre() out as
scx_skip_subtree_pre(), a pre-order walk primitive that skips @pos's
subtree, and call it from scx_next_descendant_pre(). Same locking rules as
the existing primitive.
Used in a follow-up to fast-skip subtrees that have nothing to do during a
descendant walk.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
Future kfuncs need to walk descendants without scx_sched_lock. Make the
walker RCU-safe so that they can. A sub-sched's fields are initialized
before it is linked, so a walk that observes a linked node also observes its
setup. In-place changes after linking carry their own ordering.
Switch the children/sibling list ops to RCU and expand the descendant walker
to accept rcu_read_lock as a valid read-side context. Walkers that mutate
keep scx_sched_lock.
A sub-sched can be linked while an ancestor is bypassing, after the bypass
walk that propagates the depth has passed its parent. Bypass state is a
per-cpu flag plus a depth count and can't be established atomically at link
time, so refuse to link under a bypassing ancestor. Take scx_bypass_lock
across linking to check the parent's bypass state coherently.
v3: Reject linking under a bypassing ancestor instead of inheriting bypass_depth. (sashiko AI)
v2: Inherit bypass_depth before publishing @sch on the RCU sibling list.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
scx_call_op_set_cpumask() builds a per-cpu cmask in the set_cmask scratch,
which lives in BPF-writable arena. A scheduler can corrupt the scratch's
inline header (base, nr_cids, alloc_words) from another cpu, so sizing and
indexing the write from it risks an out-of-bounds write.
Drive the build from kernel-known geometry instead.
scx_cmask_ref_init_kern() imposes base and nr_cids rather than reading them,
and scx_cmask_ref_from_cpumask() fills the scratch from the ref. Neither
reads the header back.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
kfuncs taking struct scx_cmask * from BPF arena memory have two problems.
The pointer can be any value the BPF prog hands in, and the header (@base,
@nr_cids, @alloc_words) can be mutated by the prog concurrently with kernel
access.
Add scx_cmask_ref, a validated handle. _init() normalizes the input pointer
into the arena's kern_vm range via scx_arena_to_kaddr() and snapshots the
header, rejecting a range outside the machine or a nr_cids whose words
exceed the declared @alloc_words. Downstream sizing uses the snapshot, not
the live header. _shard() reads slices while _or() and _copy() write back,
all bounded by the snapshot. No callers yet.
struct scx_cmask's bits[] carried __counted_by(alloc_words), so
UBSAN_BOUNDS and FORTIFY_SOURCE bound accesses to the array. That bound is
read from @alloc_words at the access. For an arena cmask @alloc_words is
BPF-writable. A prog that sets it larger than the real allocation makes the
check pass on a genuine overrun, so the annotation catches nothing, and it
only runs under those debug configs. Drop it - _init() validates
@alloc_words explicitly, and kernel-owned cmasks set it themselves.
v2: Validate @alloc_words in _init(), drop __counted_by. (Andrea, sashiko AI)
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
Add struct scx_pshard and sch->pshard[] indexed by shard_idx, each entry
allocated on its shard's NUMA node from scx_shard_node[si]. The struct
starts empty (one dummy field). Follow-up patches will grow it as
shard-local state lands. Only cid-type schedulers with an arena pool get
pshards.
Allocation happens after ops.init_cids() returns so any
scx_bpf_cid_override() it issues has finalized scx_nr_cid_shards and
scx_shard_node[]. sch->nr_pshards records the array size for the async RCU
free path, which may run after a later scheduler's scx_cid_init() has
rewritten the global.
v3: Build and publish pshard[] fully-formed here rather than a later patch.
v2: Free the partially-allocated pshard array on alloc failure. (sashiko AI)
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
Split kobject_init_and_add() in scx_alloc_and_add_sched(): only
kobject_init() runs there. A new scx_sched_sysfs_add() helper does
kobject_add() (and creates sub_kset when the scheduler implements
ops.sub_attach), called by both enable workfns once @sch is linked and its
sysfs-visible state is initialized. Prep so a future caps attribute can rely
on @sch being fully built by the time it's sysfs-visible. Add early enough
that a stall later in enable still leaves sysfs inspectable.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
An overridden cid mapping invalidates the auto-generated shard layout, so
the override call has to provide both. Extend scx_bpf_cid_override() with a
shard_start[] array that lists the first cid of each shard (starting at 0,
strictly increasing, last shard implicitly extends to num_possible_cpus()).
A scheduler that wants only custom shards with the auto-generated cid
mapping can read the current mapping and pass it back unchanged.
Overridden shards can span NUMA nodes, so scx_shard_node[] is rebuilt by
majority count: each shard is assigned to the node that owns the most cpus
in it.
v2: Snapshot the caller's cpu_to_cid/shard_start arrays before validating. (sashiko AI)
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
Sub-sched operations need a scalable locking / work domain smaller than the
whole cid space. Carve the cid space into topology-respecting shards: each
shard is a contiguous cid range that stays within one LLC, and LLCs larger
than the per-shard cap (default 24 cids, configurable via
ops.cid_shard_size) split into enough shards to fit. A hard cap of
SCX_CID_SHARD_MAX_CPUS prevents pathological sizes under custom
configurations.
No-topo cids pack into their own shards so every cid has a shard assignment.
Also build scx_cid_shard_ranges[] for O(1) shard-to-cid-range lookup and
scx_shard_node[] so callers can size or place work by NUMA without walking
cids. Auto-built shards inherit their LLC's node. No-topo shards carry
NUMA_NO_NODE.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
A cid-form scheduler that calls scx_bpf_cid_override() to install a custom
cid layout can only do so from ops.init(). Enable-path setup that depends on
the cid layout thus has to run after ops.init(), and ops.init() itself can't
use anything derived from the final layout, which turned out to be too
restrictive.
Add an ops.init_cids() callback dedicated to finalizing the cid layout. It
runs before the rest of the enable-path setup, so the final layout is in
effect for everything that follows including ops.init(), which now runs
after the arena pool and cmask scratch allocations.
scx_bpf_cid_override() is restricted to ops.init_cids() at load time. It
sits in a kfunc set gated by SCX_KF_ALLOW_INIT_CIDS, a flag set only on the
init_cids op, so the verifier rejects a call from any other context. The
runtime root-only check is dropped as ops.init_cids() only runs during root
enable.
The qmap demo moves its override into a dedicated qmap_init_cids() and,
while at it, introduces an enum for the cid override modes instead of
hard-coded integers.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
The kick machinery kept its targets in rq->scx shared by every sched on the
cpu. A preempt kick carried no record of which scheduler requested it.
A later patch needs preempt kicks scoped to the requesting scheduler so a
sub-scheduler can preempt only tasks in its own subtree. Move the kick masks
into the per-sched per-cpu scx_sched_pcpu and have scx_kick_cpu() link the
sched onto a per-cpu list (rq->scx.sched_pcpus_to_kick). The cpu's single
kick irq_work walks that list and kicks each sched's targets on its behalf,
so a kick stays attributed to its scheduler.
The SCX_KICK_WAIT sync set (cpus_to_sync, the kick_sync snapshot and the
balance-callback trigger) stays in rq->scx: the waiter is the cpu, not the
scheduler, and its only writers, the kick irq_work and the wait balance
callback, are cpu-local.
On disable, free_kick_syncs() flushes each cpu's pending kick irq_work
before clearing @ksyncs, so a late kick unlinks its to_kick_node instead of
early-returning on a NULL @ksyncs and leaving the node linked at free.
v3: Flush the kick irq_work in free_kick_syncs() before clearing @ksyncs. (sashiko AI)
v2: Warn once per sched on scx_bpf_kick_cpu() from NMI.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
A per-task op must be dispatched on the scheduler that owns the task.
SCX_CALL_OP_TASK() and its _RET twin take @sch explicitly, and a caller that
passes the wrong scheduler would silently run the op on it. Add a
WARN_ON_ONCE() that @sch matches the task's owner so such a mismatch is
caught rather than hidden.
Two sites legitimately target a scheduler other than the task's owner:
cgroup_move() runs on the root sched, and scx_sub_init_cancel_task() fires
exit_task() on a task not yet associated with @sch. Both switch to the inner
__SCX_CALL_OP_TASK(), which dispatches on the explicit @sch without the
assert.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
Improve readability of the sg_dma_len assignment in the P2PDMA cases by
removing duplicated code, which was a direct result of the d0d08f4bd7
("dma-direct: Fix missing sg_dma_len assignment in P2PDMA bus mappings")
fix. No functional change.
Suggested-by: Leon Romanovsky <leon@kernel.org>
Link: https://lore.kernel.org/all/20260604071856.GA245424@unreal/
Reviewed-by: Pranjal Shrivastava <praan@google.com>
Reviewed-by: Leon Romanovsky <leon@kernel.org>
Reviewed-by: Logan Gunthorpe <logang@deltatee.com>
Link: https://lore.kernel.org/r/20260713083022.1110993-1-m.szyprowski@samsung.com
Signed-off-by: Marek Szyprowski <m.szyprowski@samsung.com>
- A cpuset that never set its memory nodes could divide by zero when a
task's mempolicy rebinds on CPU hotplug. Rebind against the effective
nodes, which are always populated.
- Documentation fixes for memory.stat, io.stat, and the misc and v1 RDMA
controllers.
-----BEGIN PGP SIGNATURE-----
iIQEABYKACwWIQTfIjM1kS57o3GsC/uxYfJx3gVYGQUCalVWkg4cdGpAa2VybmVs
Lm9yZwAKCRCxYfJx3gVYGet9AQD3/qE93E+PctxXa+rmRfSSjIzqqv9NUZa4THjs
YOUqSAEApVSLPkhg3xHF61q9l+FuocLLY378Uf6LYGWet1SghAQ=
=1Gp+
-----END PGP SIGNATURE-----
Merge tag 'cgroup-for-7.2-rc3-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/cgroup
Pull cgroup fixes from Tejun Heo:
- A cpuset that never set its memory nodes could divide by zero when a
task's mempolicy rebinds on CPU hotplug. Rebind against the effective
nodes, which are always populated
- Documentation fixes for memory.stat, io.stat, and the misc and v1
RDMA controllers
* tag 'cgroup-for-7.2-rc3-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/cgroup:
Docs/admin-guide/cgroup-v2: note blkcg_debug_stats gates io.latency stats
Docs/admin-guide/cgroup-v1: document rdma.peak, rdma.events and rdma.events.local
Docs/admin-guide/cgroup-v2: drop stale misc interface file count
cgroup/cpuset: rebind mm mempolicy to effective_mems, not mems_allowed
Docs/admin-guide/cgroup-v2: fix memory.stat doc details
- Lifecycle fixes for the new sub-scheduler support: two use-after-frees
and an enable-failure path that left a half-initialized sub-scheduler
linked.
- Two dispatch-path locking bugs: a spurious scheduler abort from a
migration race, and a lockdep splat from stale runqueue-lock tracking.
- Callback and task-state fixes: stale scheduler-owned state on a task
leaving SCX, a weight callback running after disable, and a bogus
warning on core-scheduling forced idle.
- On nohz_full, finite-slice tasks could miss the tick that expires their
slice. Enable it when such a task is picked, with a selftest.
- Smaller fixes: userspace CPU-mask helpers, ratelimited deprecation
warnings, docs and a sparse annotation.
-----BEGIN PGP SIGNATURE-----
iIQEABYKACwWIQTfIjM1kS57o3GsC/uxYfJx3gVYGQUCalVWgw4cdGpAa2VybmVs
Lm9yZwAKCRCxYfJx3gVYGTmUAP0QHX7Ae/g7yMPEB4slURpnSj/wE7hRCI5jTmay
2iIEJgD/RAhpwUUAp4Abozt0mjQMWh9UyEVjOxPCVNKaNh1XfwU=
=EBCm
-----END PGP SIGNATURE-----
Merge tag 'sched_ext-for-7.2-rc3-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/sched_ext
Pull sched_ext fixes from Tejun Heo:
- Lifecycle fixes for the new sub-scheduler support: two
use-after-frees and an enable-failure path that left a
half-initialized sub-scheduler linked.
- Two dispatch-path locking bugs: a spurious scheduler abort from a
migration race, and a lockdep splat from stale runqueue-lock
tracking.
- Callback and task-state fixes: stale scheduler-owned state on a task
leaving SCX, a weight callback running after disable, and a bogus
warning on core-scheduling forced idle.
- On nohz_full, finite-slice tasks could miss the tick that expires
their slice. Enable it when such a task is picked, with a selftest.
- Smaller fixes: userspace CPU-mask helpers, ratelimited deprecation
warnings, docs and a sparse annotation.
* tag 'sched_ext-for-7.2-rc3-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/sched_ext:
sched_ext: Skip ops.set_weight() for disabled tasks
tools/sched_ext: scx - Fix cmask_subset(), cmask_equal() and cmask_weight()
sched_ext: Fix premature ops->priv publication in scx_alloc_and_add_sched()
sched_ext: Record an error on errno-only sub-enable failure
selftests/sched_ext: Verify nohz_full tick behavior
sched_ext: Enable tick for finite slices on nohz_full
sched_ext: Preserve rq tracking across local DSQ dispatch
sched_ext: Documentation: Fix ops table header reference
sched_ext: Don't warn on core-sched forced idle in put_prev_task_scx()
sched_ext: Pin parent scx_sched across a child sub-scheduler's lifetime
sched_ext: Annotate ksyncs with __rcu in alloc/free_kick_syncs()
sched_ext: Check remote rq eligibility under task's rq lock
sched_ext: Reset dsq_vtime and slice when a task leaves SCX
sched_ext: Avoid flooding the log with deprecation warnings
Update DMA reserved memory pool allocation log messages to display
sizes in KiB instead of MiB. Using MiB caused allocations less than
1 MiB to be logged as 0 MiB due to integer truncation. KiB provides
better precision for smaller memory regions specified in the Device Tree.
Signed-off-by: Vova Sharaienko <sharaienko@google.com>
Signed-off-by: Marek Szyprowski <m.szyprowski@samsung.com>
Link: https://lore.kernel.org/r/20260629223759.2637162-1-sharaienko@google.com
restore_dm_crypt_keys_to_thread_keyring() gets a reference to the user
keyring before restoring the saved dm-crypt keys.
The same keyring reference is then passed to add_key_to_keyring() for each
saved key, but add_key_to_keyring() drops that reference on every call.
This is only balanced when exactly one key is restored. With multiple
keys, the keyring reference is dropped too many times and may trigger a
refcount underflow or use-after-free.
When more than five keys are restored, a refcount underflow/use-after-free
warning can be triggered.
The early error paths after lookup_user_key() also return without dropping
the keyring reference.
Keep ownership of the keyring reference in
restore_dm_crypt_keys_to_thread_keyring(), drop it once on all exit paths,
and make add_key_to_keyring() only use the reference without consuming it.
Fixes: 62f17d9df6 ("crash_dump: retrieve dm crypt keys in kdump kernel")
Signed-off-by: Guangshuo Li <lgs201920130244@gmail.com>
Reviewed-and-tested-by: Coiby Xu <Coiby.Xu@gmail.com>
Acked-by: Baoquan He <baoquan.he@linux.dev>
Reviewed-by: Bradley Morgan <include@grrlz.net>
Link: https://patch.msgid.link/20260704112509.3717884-1-lgs201920130244@gmail.com
Signed-off-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
- Free field in error path of synthetic event parse
In __create_synth_event() the field was allocated but was not freed in the
error path.
- Fix ring_buffer_event_length() on 8 byte aligned architectures
On architectures with CONFIG_HAVE_64BIT_ALIGNED_ACCESS set to y,
the ring_buffer_event_length() may return the wrong size. This
is because archs with that config set will always use the "big
event meta header" as that is 8 bytes keeping the payload 8 bytes
aligned, even when a 4 byte header could hold the size of the
event.
But ring_buffer_event_length() doesn't take this into account and only
subtracts 4 bytes for the meta header in the length when it should have
subtracted 8 bytes.
- Have osnoise wait for a full rcu synchronization on unregister
osnoise_unregister_instance() used to call synchronize_rcu() before
freeing its copy of the instance but was switched to kfree_rcu().
The osniose tracer has code that traverses the instances that it
uses, and inst is just a pointer to that instance. By using kfree_rcu()
instead of synchronize_rcu(), the instance that the inst pointer is
pointing to can be freed while the osnoise code is still referencing it.
That is, a rmdir on an instance first unregisters the tracer. When the
unregister finishes, the rmdir expects that the tracer is finished with
the instance that it is using. By putting back the synchronize_rcu()
in osnoise_unregister_instance() the unregistering of osnoise will now
return when all the users of the instance have finished.
- Remove an unused setting of "ret" in tracing_set_tracer()
- Fix ring_buffer_read_page() copying events
The commit that changed ring_buffer_read_page() to show dropped events
from the buffer itself, split the "commit" variable between the commit
value (with flags) and "size" that holds the size of the sub-buffer.
A cut and paste error changed the test of the reading from checking the
size of the buffer to the size of the event causing reads to only read one
event at a time.
- Make tracepoint_printk a static variable
When the tracing sysctl knobs were move from sysctl.c to trace.c, the
variable tracepoint_printk no longer needed to be global. Make it static.
- Fix some typos
- Fix NULL pointer dereference in func_set_flag()
The flags update of the function tracer first checks if the value of the
flag is the same and exits if they are, and then it checks if the current
tracer is the function tracer and exits if it isn't. The problem is that
these checks need to be in a reversed order, as if the tracer isn't the
function tracer, then the flag being checked may not exist. Reverse the
order of these checks.
- Fix ufs core trace events to not dereference a pointer in TP_printk()
The TP_printk() part of the TRACE_EVENT() macro is called when the user
reads the "trace" file. This can be seconds, minutes, hours, days, weeks,
and even months after the data was recorded into the ring buffer. Thus,
saving a pointer to an object into the ring buffer and then dereferencing
it from TP_printk() can cause harm as the object the pointer is pointing
to may no longer exist.
Fix all the trace events in ufs core to save the device name in the ring
buffer instead of dereferencing the device descriptor from TP_printk().
- Prevent out-of-bound reads in glob matching of trace events
The filter logic of events allows simple glob logic to add wild cards to
filter on strings. But some events have fields that may not have a
terminating 'nul' character. This may cause the glob matching to go beyond
the string. Change the logic to always pass in the length of the field
that is being matched.
- Add no-rcu-check version of trace_##event##_enabled()
The trace_##event##_enabled() usually wraps trace events to do extra work
that is only needed when the trace event is enabled. But this can hide
events that are placed in locations where RCU is not watching, and can
make lockdep not see these bugs when the event is not enabled.
The trace_##event##_enabled() was updated to always test to make sure RCU
is watching to catch locations that may call events without RCU being
active.
This caused a false positive for the irq_disabled() and related events. As
that use trace_irq_disabled_enabled() to force RCU to be watching when the
event is enabled via the ct_irq_enter() function, calls the event, and
then calls ct_irq_exit() to put RCU back to its original state.
The trace_irq_disabled_enabled() should not trigger a warning when RCU is
not watching because the code within its block handles the case properly.
Make a __trace_##event##_enabled() version for this event to use that
doesn't check RCU is watching as it handles the case when it isn't.
- Fix use-after-free in user_event_mm_dup()
When the enabler is removed from the link list, it is freed immediately.
But it is protected via RCU and needs to be freed after an RCU
grace period. Use queue_rcu_work() so that the event_mutex can also
be taken as user_event_put() takes the mutex on the last reference
is released.
- Free type string in error path of parse_synth_field()
There's an error path in parse_synth_field() where the allocated type
string is not freed.
- Add selftest that tests deferred event teardown
- Fix leak in error path of trace_remote_alloc_buffer()
If page allocation fails, the desc->nr_cpus is not incremented for the
current CPU and the allocations done for it are not freed.
- Fix allocation length in trace_remote_alloc_buffer()
The logic to calculate the struct_len was doing a double count and setting
the value too large. Calculate the size upfront to fix the error and
simplify the logic.
- Fix sparse CPU masks in ring_buffer_desc()
If there are sparse CPUs (gaps in the numbering), the ring_buffer_desc()
will fail as it tests the CPU number against the number of CPUs that are
used.
-----BEGIN PGP SIGNATURE-----
iQJIBAABCgAyFiEEXtmkj8VMCiLR0IBM68Js21pW3nMFAmpTv/cUHHJvc3RlZHRA
Z29vZG1pcy5vcmcACgkQ68Js21pW3nN80w/9HEbliUUrJVNw8QCY8BmeBEzfCRhZ
2CeDzTg+UrLMb2BtqUhA9EjQnRPawe8mPxAfpZ750SdFnCkRhbfNwYmKar3M/8iW
y2Bhsvd3Hz6nRiHvKhG/1RkflgKOWLYf/TQSByKlakYLV8t4uffa0fe0N0zcnikK
XPGrhrTeRDJ7s94vb1u7p7mSSp2iy4ZhfXbagk966X3I4zvseIYNn5IIzyM3IIPT
M7mE60SxvgSRz0QqTJqihmqjXYSeNOQ2iU18wnvp4QfcT0IVuLKm027JnK/YpjWS
ZasMNGzO1kp5uo6hkXLrMlT3LjaNEMFTGogs1m9o9auhbMf443LK8GtZ1m1rwnTG
cXn1PocBevQIaXWPbnxMrSwIuN7YMUqjgX6SCxbXpDxSSuE5i/x1VmE08CjxnSm6
TdAdom9bE44FvMgaAz9+KhIUgm013c2rFDNiFroYtfkma5FqpfnfglA4/TkJvexY
e2L75cR3lrDEDiXXMQDndmvxNMhppwNlAH3mOZLiaCmGyFQFHhgAaG250K2Y6a5K
HtIfJkT6RQ3rWifV0E0+zlyRiOrvVCCA8WwNArWb1dTWrVU86Hr7q58jAq9p0Yrz
FnhpjdE3eWoUHZ5f28AMR2k5lMG3o6mOGT55JyUVCLw+rPSG5358UIg2xZiEo22k
RFlT26FFHbMW50Y=
=+//w
-----END PGP SIGNATURE-----
Merge tag 'trace-v7.2-rc2' of git://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace
Pull tracing fixes from Steven Rostedt:
- Free field in error path of synthetic event parse
In __create_synth_event() the field was allocated but was not freed
in the error path
- Fix ring_buffer_event_length() on 8 byte aligned architectures
On architectures with CONFIG_HAVE_64BIT_ALIGNED_ACCESS set to y, the
ring_buffer_event_length() may return the wrong size. This is because
archs with that config set will always use the "big event meta
header" as that is 8 bytes keeping the payload 8 bytes aligned, even
when a 4 byte header could hold the size of the event
But ring_buffer_event_length() doesn't take this into account and
only subtracts 4 bytes for the meta header in the length when it
should have subtracted 8 bytes
- Have osnoise wait for a full rcu synchronization on unregister
osnoise_unregister_instance() used to call synchronize_rcu() before
freeing its copy of the instance but was switched to kfree_rcu(). The
osniose tracer has code that traverses the instances that it uses,
and inst is just a pointer to that instance. By using kfree_rcu()
instead of synchronize_rcu(), the instance that the inst pointer is
pointing to can be freed while the osnoise code is still referencing
it
That is, a rmdir on an instance first unregisters the tracer. When
the unregister finishes, the rmdir expects that the tracer is
finished with the instance that it is using. By putting back the
synchronize_rcu() in osnoise_unregister_instance() the unregistering
of osnoise will now return when all the users of the instance have
finished
- Remove an unused setting of "ret" in tracing_set_tracer()
- Fix ring_buffer_read_page() copying events
The commit that changed ring_buffer_read_page() to show dropped
events from the buffer itself, split the "commit" variable between
the commit value (with flags) and "size" that holds the size of the
sub-buffer. A cut and paste error changed the test of the reading
from checking the size of the buffer to the size of the event causing
reads to only read one event at a time
- Make tracepoint_printk a static variable
When the tracing sysctl knobs were move from sysctl.c to trace.c, the
variable tracepoint_printk no longer needed to be global. Make it
static
- Fix some typos
- Fix NULL pointer dereference in func_set_flag()
The flags update of the function tracer first checks if the value of
the flag is the same and exits if they are, and then it checks if the
current tracer is the function tracer and exits if it isn't. The
problem is that these checks need to be in a reversed order, as if
the tracer isn't the function tracer, then the flag being checked may
not exist. Reverse the order of these checks
- Fix ufs core trace events to not dereference a pointer in TP_printk()
The TP_printk() part of the TRACE_EVENT() macro is called when the
user reads the "trace" file. This can be seconds, minutes, hours,
days, weeks, and even months after the data was recorded into the
ring buffer. Thus, saving a pointer to an object into the ring buffer
and then dereferencing it from TP_printk() can cause harm as the
object the pointer is pointing to may no longer exist
Fix all the trace events in ufs core to save the device name in the
ring buffer instead of dereferencing the device descriptor from
TP_printk()
- Prevent out-of-bound reads in glob matching of trace events
The filter logic of events allows simple glob logic to add wild cards
to filter on strings. But some events have fields that may not have a
terminating 'nul' character. This may cause the glob matching to go
beyond the string. Change the logic to always pass in the length of
the field that is being matched
- Add no-rcu-check version of trace_##event##_enabled()
The trace_##event##_enabled() usually wraps trace events to do extra
work that is only needed when the trace event is enabled. But this
can hide events that are placed in locations where RCU is not
watching, and can make lockdep not see these bugs when the event is
not enabled
The trace_##event##_enabled() was updated to always test to make sure
RCU is watching to catch locations that may call events without RCU
being active
This caused a false positive for the irq_disabled() and related
events. As that use trace_irq_disabled_enabled() to force RCU to be
watching when the event is enabled via the ct_irq_enter() function,
calls the event, and then calls ct_irq_exit() to put RCU back to its
original state
The trace_irq_disabled_enabled() should not trigger a warning when
RCU is not watching because the code within its block handles the
case properly. Make a __trace_##event##_enabled() version for this
event to use that doesn't check RCU is watching as it handles the
case when it isn't
- Fix use-after-free in user_event_mm_dup()
When the enabler is removed from the link list, it is freed
immediately. But it is protected via RCU and needs to be freed after
an RCU grace period. Use queue_rcu_work() so that the event_mutex can
also be taken as user_event_put() takes the mutex on the last
reference is released
- Free type string in error path of parse_synth_field()
There's an error path in parse_synth_field() where the allocated type
string is not freed
- Add selftest that tests deferred event teardown
- Fix leak in error path of trace_remote_alloc_buffer()
If page allocation fails, the desc->nr_cpus is not incremented for
the current CPU and the allocations done for it are not freed
- Fix allocation length in trace_remote_alloc_buffer()
The logic to calculate the struct_len was doing a double count and
setting the value too large. Calculate the size upfront to fix the
error and simplify the logic
- Fix sparse CPU masks in ring_buffer_desc()
If there are sparse CPUs (gaps in the numbering), the
ring_buffer_desc() will fail as it tests the CPU number against the
number of CPUs that are used
* tag 'trace-v7.2-rc2' of git://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace:
ring-buffer: Allow sparse CPU masks in ring_buffer_desc()
tracing/remotes: Fix struct_len in trace_remote_alloc_buffer()
tracing/remotes: Fix leak in trace_remote_alloc_buffer() error path
selftests/user_events: Wait for deferred event teardown after unregister
tracing/synthetic: Free type string on error path
tracing/user_events: Fix use-after-free in user_event_mm_dup()
tracing: Add a no-rcu-check version of trace_##event##_enabled()
tracing: Prevent out-of-bounds read in glob matching
ufs: core: tracing: Do not dereference pointers in TP_printk()
tracing: Fix NULL pointer dereference in func_set_flag()
samples: ftrace: Fix typos in benchmark comment
tracing: Make tracepoint_printk static as not exported
ring-buffer: Fix ring_buffer_read_page() copying only one event per page
tracing: Remove unused ret assignment in tracing_set_tracer()
tracing/osnoise: Call synchronize_rcu() when unregistering
ring-buffer: Fix event length with forced 8-byte alignment
tracing/synthetic: Free pending field on error path
The return value of __secure_computing() currently uses 0 to indicate
that a system call should be allowed, and -1 to indicate that it should
be blocked/killed. This 0/-1 pattern is non-intuitive for a security
check function and makes the control flow at the call sites less readable.
Furthermore, any potential future changes to these return values would
require a high-risk, error-prone audit of all its users across different
architectures.
Sanitize this logic by converting the return type of __secure_computing()
to a proper boolean, where 'true' explicitly means 'allow' and 'false'
means 'fail/deny'.
Update all the two dozen or so call sites across the tree to align with
this new boolean semantic. No functional changes are intended, as the
callers still return -1 to the lower-level assembly entry code upon
seccomp denial.
Rename the function to __seccomp_permit_syscall() so that the purpose is
entirely clear.
[ tglx: Rename the function ]
Suggested-by: Thomas Gleixner <tglx@kernel.org>
Suggested-by: Mark Rutland <mark.rutland@arm.com>
Signed-off-by: Jinjie Ruan <ruanjinjie@huawei.com>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Tested-by: Mukesh Kumar Chaurasiya (IBM) <mkchauras@gmail.com>
Reviewed-by: Mukesh Kumar Chaurasiya (IBM) <mkchauras@gmail.com>
Acked-by: Oleg Nesterov <oleg@redhat.com>
Link: https://patch.msgid.link/20260707190254.230735780@kernel.org
Since tracing_multi link does not set ftrace_managed, it would fail to
release the tracing_multi link when attaching tracing_multi link and
then attaching fentry link.
[ 3.714215] WARNING: kernel/bpf/trampoline.c:1727 at bpf_trampoline_multi_detach+0x20b/0x240, CPU#1: test_progs/97
...
[ 3.733170] bpf_tracing_multi_link_release+0x14/0x30
[ 3.733890] bpf_link_free+0x58/0x130
[ 3.734414] bpf_link_release+0x23/0x30
Fix it by setting 'ftrace_managed = true' in register_fentry_multi().
Fixes: aef4dfa790 ("bpf: Add bpf_trampoline_multi_attach/detach functions")
Signed-off-by: Leon Hwang <leon.hwang@linux.dev>
Acked-by: Jiri Olsa <jolsa@kernel.org>
Link: https://lore.kernel.org/bpf/20260711124822.29406-2-leon.hwang@linux.dev
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
bpf_cpumask_populate() writes to its destination with bitmap_copy(), but
the destination is typed as struct cpumask *. That allows the verifier to
accept borrowed cpumask pointers returned by read-only kfuncs, such as
scx_bpf_get_online_cpumask(), as a writable destination.
Make the destination a struct bpf_cpumask * so populate follows the same
ownership rule as the other mutating cpumask kfuncs. Query kfuncs continue
to accept const struct cpumask * inputs.
Fixes: 950ad93df2 ("bpf: add kfunc for populating cpumask bits")
Signed-off-by: Nicholas Dudar <main.kalliope@gmail.com>
Acked-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Emil Tsalapatis <emil@etsalapatis.com>
Link: https://lore.kernel.org/bpf/20260709182800.2037938-2-main.kalliope@gmail.com
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
unearthed other races in the area (Thomas Gleixner)
Signed-off-by: Ingo Molnar <mingo@kernel.org>
-----BEGIN PGP SIGNATURE-----
iQJFBAABCgAvFiEEBpT5eoXrXCwVQwEKEnMQ0APhK1gFAmpSJ0gRHG1pbmdvQGtl
cm5lbC5vcmcACgkQEnMQ0APhK1hyKA//W5KqLrRgXjH9cS7tF6PLo9NhbZZn5FhH
jxThPvJosNyCj8UEVD+iCRjkqkNxiY9mzx3px2ZVlJg8K35ez2PeZ/zx0wvuSWlM
sD7k4S3JJQ7vAE4CkYvsf91TxCWqiOMw+NLyo7MgDhwsOdgPgUAAKlCdCyoFzRK7
rAV7FQWMQag24donwPiesk8bnU4pWrooq8BoY7XVCy6aHDwwPVxrs1yPyUEikIXq
AT2HiYFOu9ok6rSlJs+A2JQnmke6M6aIJ5dmUGR7gUDRAqlXesmM6jlwklzK0a4O
j6lpE05ouIhZpYzQVTDxvf8cytp0+rutTX8w0lPRlqxYkaeBVEpktxfBAoqFCsoe
GpK8hK0PM57W29zhnTygQib53QQAk9QBQetp0NwUjLSOu3ouhCAOC3wuoctJ4fKY
oNSKGPBhHwJN5932JuxlEll/xbo+bFyHfP87tqn1SA0bgKMSNet5q7+wYE+1gcTp
9LHQnoMpjyRBjuOupnvxqeOVbN3qlNdY6MINFwxBZuKG7UwmYpTH7l72fuUM/eNr
b4UHKgwv3vuQX/BfBoT1mLnzWQTltrfEywBCXHsEHUQ7KFhdRHEbnh3zVDZCT5S3
5N/RS8aFcryD/0/DhxWEhO2DBTgjDDvmOKplIPYFtikDI1dkjZ9dZ0BCI42u6wMH
by25mw2rF2s=
=2I60
-----END PGP SIGNATURE-----
Merge tag 'timers-urgent-2026-07-11' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
Pull timer fix from Ingo Molnar:
- Fix a subtle posix-cpu-timers vs. exec() race, which
unearthed other races in the area (Thomas Gleixner)
* tag 'timers-urgent-2026-07-11' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip:
posix-cpu-timers: Prevent UAF caused by non-leader exec() race
-----BEGIN PGP SIGNATURE-----
iQJIBAABCgAyFiEES0KozwfymdVUl37v6iDy2pc3iXMFAmpRNpEUHHBhdWxAcGF1
bC1tb29yZS5jb20ACgkQ6iDy2pc3iXN5+Q/+N1wsGx07LPmidKbPTx3R+yej3JCW
hoLhnmPt7AXxVL671W29vkazKZ8hyQ5hzRy3wITyue9hsiXTNIwNffE1iBKkqMVA
mchh38ziazJJ3h9PVWdiXdW/PoqOIAicFCUe0KJw8bV21Tdzjl+tCqVylU4j57k8
xpkGUifsK8FpkvDMzMlOfD5e4qShOqkPhacVz1OJDrW8gHV5Dh1AuTXdKz58dA1F
Plu7IrtNd+CM1AQwS8ucawDB5PXnxHa708evS5bId/9fG/x5L5ZPJw6+jCPVA+Ea
qUkWWalAJzOEp4jWDhvCyymtqiGhj9rFAydxCy32IAsF7TLTDCV8E+6Iq78wGtjg
Oep+Tn9wOqV5QMILqlZ0zQxeDWnQWA0MzVRqIOFAvMrpurPLUbYZLFbUyNWouGBD
cfU6O+AhADTmfkFfxxTFLsm0RQCEdnF1XeZEreXgwFuuF/A7SybmB2WTWE5wcmbu
Riks3O9NHK620vCLf+wNpDc7+BkNmfBBhLUsyk0NDzEsIpViWS24stumBAP7+Fge
0KnWaB+vOD8IVub1YGgrxTNf3DfK6p8CLgwGYZMBjYFczJF8gCYXTfKlqmRVpXzw
20WaBbriMeMKgC5azD3aPqCg+/vgfBdCcEHSXAIS2KZivMQqYaodlFtTUowoDKng
sn2hwA3SKfZ4aGw=
=wY2k
-----END PGP SIGNATURE-----
Merge tag 'audit-pr-20260710' of git://git.kernel.org/pub/scm/linux/kernel/git/pcmoore/audit
Pull audit fixes from Paul Moore:
"Two relatively small audit patches to fix potential data races with
the main audit backlog queue as well as possible integer overflows
when logging data as hex strings"
* tag 'audit-pr-20260710' of git://git.kernel.org/pub/scm/linux/kernel/git/pcmoore/audit:
audit: fix potential integer overflow in audit_log_n_hex()
audit: Fix data races of skb_queue_len() readers on audit_queue
No user currently relies on sparse CPU masks, but the descriptor logic already
supports them via linear fallback. Remove the arbitrary limitation.
Link: https://patch.msgid.link/20260709160017.1729517-4-vdonnefort@google.com
Fixes: 2e67fabd8b ("ring-buffer: Introduce ring-buffer remotes")
Reported-by: Sashiko <sashiko-bot@kernel.org>
Signed-off-by: Vincent Donnefort <vdonnefort@google.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
Pre-calculate desc->struct_len up-front in trace_remote_alloc_buffer()
with trace_buffer_desc_size() to fix double-counting.
While at it, use the accessor __first_ring_buffer_desc().
Link: https://patch.msgid.link/20260709160017.1729517-3-vdonnefort@google.com
Fixes: 96e43537af ("tracing: Introduce trace remotes")
Reported-by: Sashiko <sashiko-bot@kernel.org>
Signed-off-by: Vincent Donnefort <vdonnefort@google.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
If page allocation fails in trace_remote_alloc_buffer(), desc->nr_cpus
is not yet incremented for the current CPU. As a consequence, on error,
half-allocated rb_desc will not be freed in trace_remote_free_buffer().
Increment desc->nr_cpus as soon as the first allocation for the current
CPU has succeeded.
Link: https://patch.msgid.link/20260709160017.1729517-2-vdonnefort@google.com
Fixes: 96e43537af ("tracing: Introduce trace remotes")
Reported-by: Sashiko <sashiko-bot@kernel.org>
Signed-off-by: Vincent Donnefort <vdonnefort@google.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
When switching a task's sched_class away from sched_ext, we get the
following sequence of events in __sched_setscheduler():
sched_change_begin()
switched_from_scx()
scx_disable_task(p)
ops.disable(p)
__setscheduler_params()
set_load_weight()
reweight_task_scx(p)
ops.set_weight(p)
p->sched_class = next_class;
sched_change_end()
...
Notably, ops.set_weight() is called _after_ ops.disable().
This violates the expected semantics of the callbacks, the expectation
being that ops.disable() can only be followed by ops.exit_task() or
ops.enable().
Skipping the weight adjustment for disabled tasks should be harmless
since the weight will be recalculated in scx_enable_task() if the task
ever rejoins SCX.
Fixes: 637b068282 ("sched: Fold sched_class::switch{ing,ed}_{to,from}() into the change pattern")
Cc: stable@vger.kernel.org # v6.19+
Signed-off-by: Kuba Piecuch <jpiecuch@google.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
sched_ext schedulers have state in BPF programs and kernel. scx_dump
provides kernel state and BPF program state on error, but this is static
in what it can provide.
Add a sched_ext_exit tracepoint in scx_claim_exit() so that BPF programs
can dynamically inspect scheduler specific state at the moment of exit.
Pass the exiting scx_sched so attached programs can read its state, and,
since exits propagate through a hierarchy of sub-schedulers, identify
which scheduler each event belongs to.
Signed-off-by: Pat Somaru <patso@likewhatevs.io>
Signed-off-by: Tejun Heo <tj@kernel.org>
map_range() reads rb->aux_pages[], rb->aux_nr_pages and rb->aux_pgoff via
perf_mmap_to_page() while holding only event->mmap_mutex. Those fields are
serialized by rb->aux_mutex, and mmap_mutex is per event.
Thus, two events sharing one rb via PERF_EVENT_IOC_SET_OUTPUT can race
rb_alloc_aux() with map_range(), leading to a page-UAF scenario as follows:
CPU 0 CPU 1
===== =====
rb_alloc_aux() map_range()
[1]: allocate rb->aux_pages[0]
[2]: rb->aux_nr_pages++
[3]: perf_mmap_to_page()
returns rb->aux_pages[0]
[4]: map it as VM_PFNMAP
[5]: rb->aux_pgoff = 1
munmap the page
[6]: free rb->aux_pages[0]
Pages mapped as VM_PFNMAP have no refcount protection, so CPU 1 holds a
mapping to a freed physical frame.
Fix this by taking rb->aux_mutex across the page walk in map_range().
Fixes: b709eb872e ("perf: map pages in advance")
Signed-off-by: Lee Jia Jie <jiajie.lee@starlabs.sg>
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Cc: stable@vger.kernel.org
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Now that nothing inflates tick_length beyond tick_length_base (the
adjtime path was converted to use time_offset in the previous commit),
the two fields are always equal.
Remove tick_length_base and keep tick_length as the single field.
Remove the per-second reset and the delta update in
ntp_update_frequency() since there is no separate base to track.
No functional change intended.
Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Assisted-by: Kiro:claude-opus-4.6-1m
Link: https://patch.msgid.link/20260621220051.1030462-8-dwmw2@infradead.org
time_offset (the exponential PLL phase slew) and time_adjust (the
linear adjtime() slew) can be asked to move the clock in opposite
directions. second_overflow() folds only their *net* into the per-tick
skew_delta, so the cancelling overlap would never be drained from
either tracker by the per-tick code — and if they cancel exactly,
skew_delta is zero and neither converges at all.
Arguably we could just let one of them entirely cancel out the other
immediately, but that would be a change in userspace-visible behaviour.
Instead, preserve the existing behaviour by calculating the "conflict"
portion between the opposing skew each second, and transferring that
amount directly from one tracker to the other.
Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Assisted-by: Kiro:claude-opus-4.8
Link: https://patch.msgid.link/20260621220051.1030462-7-dwmw2@infradead.org
The legacy adjtime() slew (ADJ_OFFSET_SINGLESHOT) was the last user of
tick_length != tick_length_base: it slewed the clock by inflating
tick_length directly, which delivered the correction imprecisely (e.g.
delivering only 4997.5µs when asked for a 5ms skew).
Deliver it accurately through the same per-tick mechanism that is now
used for time_offset, allowing it to contribute to skew_delta and thus
drive the delivery through ntp_error and mult selection.
To allow for accurate accounting, store the sub-microsecond part of
time_adjust is separately, while keeping time_adjust in microseconds
as that's the external API.
Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Assisted-by: Kiro:claude-opus-4.8
Link: https://patch.msgid.link/20260621220051.1030462-6-dwmw2@infradead.org
Currently, the phase offset of time_offset and time_adjust is delivered
by adjusting tick_length in second_overflow(), and immediately draining
time_offset/time_adjust by the amount that the tick_length adjustment is
*estimated* to cause. This is fairly approximate, in part because it is
not always correct to assume that precisely NTP_INTERVAL_FREQ ticks will
occur between one call to second_overflow() and the next. It could also
over and under-run in the final second of delivery.
Instead of inflating tick_length, transfer the intended skew directly
into ntp_error each tick to achieve the desired rate.
In second_overflow(), calculate skew_delta which is the per-tick slew
rate, in the same units as time_offset: (ns << NTP_SCALE_SHIFT) / HZ.
In logarithmic_accumulation(), drain up to 'skew_delta' time units from
time_offset into ntp_error to drive the overall effective rate. The new
ntp_drain_skew() function returns the amount which is actually 'claimed'
by time_offset (and in a future patch, time_adjust). Any overrun which
is delivered by the changed 'mult' (as described below) but not claimed
by ntp_drain_skew() will remain in ntp_error to be corrected away in
subsequent ticks.
Simply transferring the precise amount from time_offset to ntp_error
would be sufficent to make the time *eventually* converge, however the
skew delivered is limited by the choice of { mult, mult+1 } each tick
and thus the convergence would be extremely slow.
In theory we could inflate ntp_err_mult with the magnitude of ntp_error
in the general case — but that would cause overcorrection in a tickless
kernel. Instead, in timekeeping_adjust(), take skew_delta into account
when calculating 'mult', such that the available {mult, mult+1} choices
bracket the overall effective rate *including* the skew, to avoid the
delta just building up in ntp_error.
The effect is that the inflated 'mult' causes ntp_error to grow because
xtime_interval is (e.g.) longer than the true tick_length. But then the
same delta is removed again as it's drained from time_offset.
This gives behaviour equivalent to the old tick_length += delta approach
but with exact per-tick accounting of the time_offset actually imparted
to the clock, and no overrun.
Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Assisted-by: Kiro:claude-opus-4.8
Link: https://patch.msgid.link/20260621220051.1030462-5-dwmw2@infradead.org
cycle_interval is an integer number of counter cycles per NTP interval,
so the real time it represents differs from the nominal
NTP_INTERVAL_LENGTH by up to half a counter period. For coarse
clocksources this is significant: the 3.579545 MHz ACPI PM timer at
HZ=1000 rounds 3579.545 cycles up to 3580, making each tick 1.000127 ms
(+127 PPM).
Commit a386b5af8e ("time: Compensate for rounding on odd-frequency
clocksources") introduced xtime_remainder to compensate for exactly
this, citing the same 127 PPM ACPI PM example. The compensation is
correct and necessary, but it was applied inside the timekeeping
accumulation in timekeeping.c: subtracted in the mult computation in
timekeeping_adjust() and folded into the ntp_error update in
logarithmic_accumulation(). That keeps the base rate correct and leaves
NTP its full symmetric +/-MAXFREQ range rather than +373/-627 PPM, but
the NTP code in ntp.c never sees it: tick_length is computed without the
correction, so ntp.c's notion of how long a tick is disagrees with the
rate timekeeping actually produces.
Make the offset an explicit part of the NTP tick_length instead. Add
ntp_data::cs_tick_adj, a fixed per-second addend that
ntp_update_frequency() includes alongside ntp_tick_adj and time_freq.
tk_setup_internals() computes it from the difference between the real
cycle_interval duration and the nominal interval, stores it in the
timekeeper, and hands it to NTP through a new argument to ntp_clear() --
which already recomputes the frequency and is invoked after every
clocksource (re)configuration. timekeeping_init() now uses TK_UPDATE_ALL
for this; clearing NTP there is otherwise redundant since ntp_init() has
just initialised it.
ntp.c now computes the true tick rate, giving a single source of truth.
Like ntp_tick_adj, cs_tick_adj stays internal to the kernel: userspace
still sees the nominal 1.000000 ms tick via adjtimex and is unaware of
the addends. timekeeping_adjust() and logarithmic_accumulation() use
ntp_tick / xtime_interval directly, and xtime_remainder is removed.
The base-rate arithmetic is unchanged: ntp_tick becomes
xtime_interval << ntp_error_shift, so the mult division yields the same
base mult and the ntp_error accumulation still nets to zero per tick.
Beyond the cleanup of treating all the tick_length contributions
(nominal interval, ntp_tick_adj, cs_tick_adj, time_freq) consistently
as addends in one place, it also prepares for feed-forward discipline:
a future timekeeping_set_reference() will set tick_length to track an
absolute external reference such as a vmclock, and that path needs
ntp.c to own a tick_length that already reflects the clocksource
quantisation, with no hidden correction applied elsewhere.
Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Assisted-by: Kiro:claude-opus-4.8
Acked-by: John Stultz <jstultz@google.com>
Link: https://patch.msgid.link/20260621220051.1030462-4-dwmw2@infradead.org
consume_remote_task() drops this_rq before unlinking a remote task from
its DSQ and locking src_rq. When called from ops.dispatch(),
scx_locked_rq() keeps pointing to this_rq across the lock dance. The
subsequent switch_rq_lock(src_rq, this_rq) cannot update the tracking
because its guard does not match, leaving the tracking stale while
src_rq is held.
Keep this_rq locked until the task has been unlinked and the DSQ lock
released, then use switch_rq_lock() to switch directly to src_rq. Use
the same helper to restore this_rq after losing the dequeue race. The
successful path already switches back through
move_remote_task_to_local_dsq(), so scx_locked_rq() now follows the
actually held rq throughout the consume path.
Suggested-by: Tejun Heo <tj@kernel.org>
Link: https://lore.kernel.org/all/455e701bca66bdecde530d225f4dba0a@kernel.org
Signed-off-by: Andrea Righi <arighi@nvidia.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
scx_bpf_kick_cid() returned an error code, but the value conveys nothing
actionable and no caller consumes it. The kick is asynchronous, so a
successful return only means it was queued. An invalid @cid is already
reported through scx_error() by scx_cid_to_cpu(), and a missing scheduler
leaves nothing to kick.
Make scx_bpf_kick_cid() return void to match scx_bpf_kick_cpu(). The
cid-form kfuncs are not in practical use yet, so the ABI change is safe.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
Direct writes to p->scx.slice and p->scx.dsq_vtime bypass
scx_bpf_task_set_slice/dsq_vtime() and the authority checks they carry.
Those checks exist for sub-schedulers, which attach only through the
cid-form struct_ops, so the direct writes only need to be closed there.
Give sched_ext_ops_cid its own verifier ops that reject the two fields.
cid-form is a new interface with no legacy users, so there is no
compatibility to keep. The cpu-form keeps direct writes, and the deprecation
warning they carried is dropped.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
scx_rq.extra_enq_flags carries scx-specific enqueue flags across the
activate_task() boundary during a cross-rq task move in
move_remote_task_to_local_dsq(). Rename it to remote_activate_enq_flags to
name that role, and fix the stale comment that referenced the old
move_task_to_local_dsq() name.
No functional change.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
The cid-form ops overlay their cpu-form siblings at the same struct slot.
Ops whose signature matches the sibling are invoked through the cpu-form
call sites unchanged, but set_cmask() takes an arena cmask address rather
than a cpumask, so scx_call_op_set_cpumask() calls ops_cid.set_cmask()
directly and hand-rolled the kf_tasks[] and locked_rq bracket that
SCX_CALL_OP_TASK() provides. The hand-rolled bracket reset locked_rq to
NULL on exit instead of restoring the saved value, so a nested call would
clobber the outer op's locked-rq tracking.
Parameterize the dispatch macros by the ops-table member and add
SCX_CALL_CID_OP_TASK(), which routes through sch->ops_cid. Convert
scx_call_op_set_cpumask() to it and drop the hand-rolled bracket. The only
behavioral change is that locked_rq is now saved and restored like every
other op call site.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
The cmask ops can operate on BPF-arena cmasks which BPF programs may read
and write concurrently. The _RACY op variants existed to make such lockless
reads explicit but this turned out to be too restrictive. Mark the word
accesses in all the two-cmask ops with READ_ONCE/WRITE_ONCE instead and drop
the _RACY variants.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
Pull to receive:
db4e9defd2 ("sched_ext: Record an error on errno-only sub-enable failure")
49b3378a75 ("sched_ext: Fix premature ops->priv publication in scx_alloc_and_add_sched()")
e6979d05c6 ("tools/sched_ext: scx - Fix cmask_subset(), cmask_equal() and cmask_weight()")
for further sub-sched changes and to resolve the conflicts with the
sub-sched updates on for-7.3.
db4e9defd2 adds scx_error() to the sub-enable err_disable sink which
for-7.3 moved from ext.c into sub.c. Resolved by applying the fix to
scx_sub_enable_workfn() in sub.c.
49b3378a75 drops RCU_INIT_POINTER() from an scx_alloc_and_add_sched()
unwind label whose body changed with for-7.3's stall_cpus addition.
Resolved by dropping the line from the updated unwind.
Signed-off-by: Tejun Heo <tj@kernel.org>
scx_alloc_and_add_sched() publishes @sch through ops->priv before allocating
the cgroup path. If that allocation fails, the unwind path clears ops->priv
and frees @sch immediately. scx_prog_sched() callers can dereference
ops->priv from RCU context the moment it is set, so freeing without a grace
period can use-after-free a concurrent kfunc caller.
Move the publication below the cgroup path allocation so that every failure
path after publication frees @sch through kobject_put(), whose release path
defers the freeing by a grace period.
Fixes: 105dcd005b ("sched_ext: Introduce scx_prog_sched()")
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
scx_sub_enable_workfn() has several failure paths that only return an errno
(e.g. -ENOMEM from an allocation) and jump to err_disable without calling
scx_error(). scx_flush_disable_work() runs the disable, and thus ops.exit(),
only when an error has been recorded, so an errno-only failure leaves the
half-initialized sub-scheduler linked.
Record an error at the err_disable sink so every errno-only failure runs the
disable path.
Fixes: ebeca1f930 ("sched_ext: Introduce cgroup sub-sched support")
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
The comment for scx_bpf_dsq_insert() references "@dsp_id" in the
description body, but the parameter is named "@dsq_id" in both the
parameter list and the function signature.
Signed-off-by: Liang Luo <luoliang@kylinos.cn>
Acked-by: Andrea Righi <arighi@nvidia.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
check_kfunc_args() detects a kfunc argument named rdonly_buf_size or
rdwr_buf_size and stores reg->var_off.value into meta->r0_size, a u64,
and does not bound it. check_kfunc_call() later copies that value into
the returned register's mem_size field:
meta->r0_size = reg->var_off.value;
...
regs[BPF_REG_0].mem_size = meta.r0_size;
regs[BPF_REG_0].mem_size is u32. A constant whose upper 32 bits are set
gets truncated instead of causing a load-time rejection, so the verifier
records a PTR_TO_MEM register with an approximately 4 GiB mem_size for
whatever allocation the kfunc returned. A later access check against
that register uses the truncated, wrong bound.
Reject rdonly_buf_size/rdwr_buf_size values that exceed U32_MAX at the
point meta->r0_size is set.
Fixes: eb1f7f71c1 ("bpf/verifier: allow kfunc to return an allocated mem")
Signed-off-by: Nicholas Dudar <main.kalliope@gmail.com>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://lore.kernel.org/bpf/20260709155837.1879230-2-main.kalliope@gmail.com
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
When a spilled scalar is filled back with a sign-extending narrowing load
(BPF_MEMSX), check_stack_read_fixed_off() copies the spilled register
including its scalar id, but coerce_reg_to_size_sx() then sign-extends the
filled register's value. If the same slot is also filled with a plain
zero-extending load (BPF_MEM), both destination registers share the id yet
hold different values. A later 'if <zext-reg> == const' then refines the
sign-extended register through sync_linked_regs() to a value it does not
have at runtime (e.g. the verifier believes 0x80000000 while the register
is 0xffffffff80000000), which can be turned into an out-of-bounds access.
Drop the shared scalar id at the sign-extension site in check_mem_access()
when sign extension actually changes the value, mirroring the BPF_MOVSX
handling in check_alu_op() (no_sext = reg_umax < 2^(size*8-1)).
Fixes: 3cd5c89065 ("bpf: Let the verifier assign ids on stack fills")
Reported-by: STAR Labs SG <info@starlabs.sg>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
When emitting a record via legacy printing, it is possible that a handover
to another legacy printing context occurs. When a context has performed a
handover, the console SRCU read lock is released and the pointer to the
console struct might now be invalid. Therefore, after calling
nbcon_legacy_emit_next_record() or console_emit_next_record(), it is
necessary to check if a handover occurred _before_ further @con usage.
Sashiko pointed out that console_flush_one_record() was not doing this.
In console_flush_one_record(), after emitting a record, move the further
usage of @con after the handover check.
Fixes: c158834b22 ("printk: nbcon: Use nbcon consoles in console_flush_all()")
Reported-by: Sashiko <sashiko-bot@kernel.org>
Closes: https://lore.kernel.org/lkml/20260630170903.099D61F000E9@smtp.kernel.org
Signed-off-by: John Ogness <john.ogness@linutronix.de>
Reviewed-by: Petr Mladek <pmladek@suse.com>
Link: https://patch.msgid.link/20260703141521.202813-1-john.ogness@linutronix.de
Signed-off-by: Petr Mladek <pmladek@suse.com>
Commit 5816bf4273 ("lsm,selinux: Add LSM blob support for BPF objects")
made the LSM hook wrappers for BPF object creation clean up the LSM
state internally upon denial, e.g. security_bpf_map_create() internally
calls security_bpf_map_free() when the bpf_map_create hook returns an
error. map_create() however still routes a denial to its free_map_sec
label, which invokes security_bpf_map_free() a second time, so the
bpf_map_free hook fires twice for a single denied map.
In-tree LSMs are unaffected in practice since the blob kfree() inside
security_bpf_map_free() is NULL-safe and idempotent and none of them
implement bpf_map_free, but a BPF LSM program attached to that hook
observes double invocations. Route the denial to free_map instead.
Fixes: 5816bf4273 ("lsm,selinux: Add LSM blob support for BPF objects")
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Link: https://lore.kernel.org/bpf/20260709073422.379247-1-daniel@iogearbox.net
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
When an architecture (such as LoongArch, ARM64, and RISC-V) implements
bpf_jit_inlines_helper_call(), the verifier skips rewriting the helper
call offset (insn->imm) in bpf_do_misc_fixups(). This is because the
helper is expected to be inlined by the JIT compiler later. Therefore,
insn->imm remains as the raw helper enum ID.
However, if JIT is disabled at runtime (net.core.bpf_jit_enable=0) or
if JIT compilation fails dynamically (e.g., due to OOM), the program
falls back to the BPF interpreter.
When the interpreter executes (__bpf_call_base + insn->imm) with the
unpatched raw ID, it jumps into an invalid address space, triggering
an instruction alignment fault or a kernel panic.
Although these helpers have valid C implementations in the kernel, the
omission of offset rewriting makes runtime interpreter fallback fatal.
Fix this by setting 'prog->jit_required = 1' when helper call rewriting
is skipped for JIT inlining. This ensures that such programs are safely
rejected if JIT is not available, preventing the runtime kernel panic.
Fixes: 2ddec2c80b ("riscv, bpf: inline bpf_get_smp_processor_id()")
Suggested-by: Alexei Starovoitov <ast@kernel.org>
Suggested-by: KaFai Wan <kafai.wan@linux.dev>
Acked-by: Leon Hwang <leon.hwang@linux.dev>
Signed-off-by: Tiezhu Yang <yangtiezhu@loongson.cn>
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Introduce a 'jit_required' bitfield flag in struct bpf_prog to track
whether a BPF program strictly requires the JIT compiler to run. This
prevents a dangerous runtime fallback to the interpreter for features
that are only implemented in the JIT compiler.
Currently, bpf_prog_has_kfunc_call() is used only for kernel function
calls, replace the kfunc-specific helper with the new 'jit_required'
flag. This makes it easy to support other JIT-only BPF features, such
as inlined helpers.
Suggested-by: Alexei Starovoitov <ast@kernel.org>
Suggested-by: KaFai Wan <kafai.wan@linux.dev>
Suggested-by: Leon Hwang <leon.hwang@linux.dev>
Acked-by: Leon Hwang <leon.hwang@linux.dev>
Signed-off-by: Tiezhu Yang <yangtiezhu@loongson.cn>
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
bpf_find_vma() reads task->mm and calls mmap_read_trylock(mm) without
holding a reference on the mm. On a foreign task, a concurrent exit_mm()
can free the mm_struct between the lockless read and the trylock,
resulting in a use-after-free. mm_struct is not SLAB_TYPESAFE_BY_RCU.
For the current task, task->mm is stable. For a foreign task, pin the mm
under task->alloc_lock and release it with mmput_async(), mirroring commit
d8e27d2d22 ("bpf: fix mm lifecycle in open-coded task_vma iterator").
Use spin_trylock() instead of get_task_mm() so BPF context does not block
on alloc_lock. Reject irqs-disabled contexts and !CONFIG_MMU on the
foreign-task path because dropping the mm reference is not safe there.
Race:
CPU0 (BPF program) CPU1 (exiting task)
============================ ==========================
bpf_find_vma(foreign_task):
mm = task->mm
exit_mm():
task->mm = NULL
mmput(mm) -> frees mm_struct
mmap_read_trylock(mm)
// UAF on mm
Fixes: 7c7e3d31e7 ("bpf: Introduce helper bpf_find_vma")
Signed-off-by: Sanghyun Park <sanghyun.park.cnu@gmail.com>
Reviewed-by: Puranjay Mohan <puranjay@kernel.org>
Acked-by: Yonghong Song <yonghong.song@linux.dev>
Link: https://lore.kernel.org/bpf/20260708072106.199637-2-sanghyun.park.cnu@gmail.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
bpf_prog_calc_tag() copies the instructions into a plain vmalloc() scratch
buffer to blind the map fds before hashing. The buffer scales with the
program, up to ~8MB at the 1M instruction limit, and is allocated on every
program load, but unlike the rest of the load-time scratch memory it is
not charged to the loader's memcg. Use GFP_KERNEL_ACCOUNT to account it
like the other allocations scoped to the verification/load.
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Link: https://lore.kernel.org/bpf/20260708211537.371874-5-daniel@iogearbox.net
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
The insn_aux_data array is allocated with a plain vzalloc(), while every
other allocation scoped to the verification - verifier states, explored
states, the cfg/scc arrays, liveness masks, jump history - is charged to
the loader's memcg via GFP_KERNEL_ACCOUNT.
At 136 bytes per instruction it is one of the largest verification-time
buffers, in the range of ~130MB for a program at the 1M instruction limit
(worst case), and it lives across the whole verification. The buffer is
also inconsistent with itself: when instruction patching grows it, the
vrealloc() in bpf_patch_insn_data() already passes GFP_KERNEL_ACCOUNT.
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Link: https://lore.kernel.org/bpf/20260708211537.371874-4-daniel@iogearbox.net
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
bpf_get_btf_vmlinux() serializes the lazy vmlinux BTF parse with
bpf_verifier_lock, the same mutex bpf_check() holds across the whole
verification of an unprivileged program (if enabled; it's disabled
by default). The latter can potentially stall the mutex holder for
a long time (e.g. via userfaultfd), and therefore block first-time
bpf_get_btf_vmlinux() caller from any context, including privileged
program loads.
Give the vmlinux BTF initialization a dedicated btf_vmlinux_lock so
it is independent of the unprivileged verification mutex. The parse
only needs mutual exclusion against itself.
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Link: https://lore.kernel.org/bpf/20260708211537.371874-3-daniel@iogearbox.net
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
bpf_get_btf_vmlinux() lazily parses the vmlinux BTF under the
bpf_verifier_lock, but publishes the result through a plain store
and re-checks it through a plain lockless load. Nothing orders
the stores initializing the struct btf inside btf_parse_vmlinux()
against the store publishing the pointer: On a weakly ordered
arch, a concurrent first-time caller taking the lockless fast
path could in principle observe the pointer before the parsed
contents are visible. The mutex_unlock() does not help such a
reader given it only synchronizes with a later acquisition of the
same lock. Thus, publish the pointer with smp_store_release()
and read it on the fast path with smp_load_acquire().
Acquire semantics are needed rather than a dependency-ordered
READ_ONCE(): btf_parse_vmlinux() also populates globals outside
the returned object (e.g. bpf_ctx_convert.t). An address
dependency would only order accesses performed through the
pointer and not cover other globals.
Fixes: 8580ac9404 ("bpf: Process in-kernel BTF")
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Link: https://lore.kernel.org/bpf/20260708211537.371874-2-daniel@iogearbox.net
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
set_next_task_scx() updates the tick dependency before __schedule()
updates rq->curr. When switching from a non-EXT task, such as idle, to
an EXT task with a finite slice, sched_update_tick_dependency() checks
the outgoing task and can allow the tick to remain stopped.
The dependency can also be lost without a slice-type transition. After a
finite-slice task leaves the CPU idle, the enqueue path can clear the
dependency against the idle rq->curr. SCX_RQ_CAN_STOP_TICK still records
a finite slice, so another finite task skips the transition block and
can run without the ticks needed to expire its slice.
The reverse mismatch can also happen when the last finite-slice EXT task
is dequeued: sub_nr_running() updates the dependency before rq->curr
changes, so the outgoing task state can keep the dependency set after
the CPU goes idle.
Fix this by unconditionally enabling the scheduler tick whenever a
finite-slice EXT task is selected on a nohz_full CPU. Moreover, when the
last runnable EXT task leaves, ignore the outgoing EXT slice state so
the generic scheduler can correctly re-evaluate and clear the tick
dependency.
Fixes: 22a920209a ("sched_ext: Implement tickless support")
Signed-off-by: Andrea Righi <arighi@nvidia.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
The function calculates new_len as len << 1 for hex encoding. This
has two overflow risks: the shift itself can overflow when len is
large, and the result can be truncated when assigned to new_len
(declared as int) from the size_t calculation.
Fix by using check_shl_overflow() to catch shift overflow and
changing new_len and loop counter i to size_t to prevent truncation.
Cc: stable@vger.kernel.org
Fixes: 168b717395 ("AUDIT: Clean up logging of untrusted strings")
Reviewed-by: Richard Guy Briggs <rgb@redhat.com>
Signed-off-by: Ricardo Robaina <rrobaina@redhat.com>
[PM: remove vertical whitspace noise]
Signed-off-by: Paul Moore <paul@paul-moore.com>
dispatch_to_local_dsq() can run from scx_bpf_dsq_move_to_local() while
ops.dispatch() has recorded the current rq. Moving a task to a local DSQ
may switch to the source or destination rq before synchronously invoking
ops.dequeue() through the following path:
SCX_CALL_OP(dispatch, rq)
ops.dispatch()
scx_bpf_dsq_move_to_local()
scx_flush_dispatch_buf()
finish_dispatch()
dispatch_to_local_dsq()
scx_dispatch_enqueue()
local_dsq_post_enq()
call_task_dequeue()
SCX_CALL_OP_TASK(dequeue, locked_rq, ...)
The nested callback saves the recorded rq and restores it on return. If
the rq tracking does not follow the lock switch, update_locked_rq() can
trigger the following lockdep assertion while restoring an rq which is
no longer held:
WARNING: kernel/sched/sched.h:1641 at call_task_dequeue+0x160/0x170
Call Trace:
scx_dispatch_enqueue+0x2b0/0x460
dispatch_to_local_dsq+0x138/0x230
scx_flush_dispatch_buf+0x1af/0x220
scx_bpf_dsq_move_to_local___v2+0xe2/0x1c0
bpf__sched_ext_ops_dispatch+0x4b/0xa7
do_pick_task_scx+0x3b6/0x910
__pick_next_task+0x105/0x1f0
__schedule+0x3e7/0x1980
Introduce switch_rq_lock() to update the tracking state together with
each rq lock handoff. Use it in dispatch_to_local_dsq(),
move_remote_task_to_local_dsq() and the in-balance paths of
scx_dsq_move(), ensuring that scx_locked_rq() consistently refers to the
rq whose lock is actually held throughout the lock dance.
Fixes: 7fb39e4eb4 ("sched_ext: Save and restore scx_locked_rq across SCX_CALL_OP")
Cc: stable@vger.kernel.org # 7.1+
Signed-off-by: Andrea Righi <arighi@nvidia.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
The signed gen_loader used to police its own metadata map from within
BPF: emit_signature_match() read the kernel-cached map->sha[] back
through hardcoded struct bpf_map offsets and compared it against a hash
that compute_sha_update_offsets() baked into the signed instructions,
after a BPF_OBJ_GET_INFO_BY_FD round-trip to populate map->sha[].
The kernel now verifies the metadata at BPF_PROG_LOAD time by folding
the frozen contents of the loader's exclusive fd_array maps into the
signature, so the loader no longer checks anything itself. Generated
loaders thus carry no verification logic of their own anymore: Nothing
in the signing chain depends on emitted loader bytecode doing the right
thing.
On the loading side, skel_internal.h now sets fd_array_cnt for a signed
load so the kernel scans fd_array for the exclusive metadata map -
still frozen, as the kernel requires - and the BPF_OBJ_GET_INFO_BY_FD
round-trip to populate map->sha[] is gone. The struct bpf_map layout
BUILD_BUG_ON()s on the kernel side are removed as well: they only
pinned the ABI for the in-BPF read of map->sha[] that is no longer
needed. Same for the map->excl member. Note: gen_hash is retained; it
still marks a loader as signed so an untrusted host cannot re-dimension
maps or override initial values now covered by the signature.
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Link: https://lore.kernel.org/bpf/20260708075343.358712-4-daniel@iogearbox.net
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
A signed gen_loader program carries the programs, maps and relocations
it installs in a metadata array map. The loader instructions are covered
by the PKCS#7 signature, but the metadata map is not: Today the loader
compares the map contents from within BPF against a hash baked into its
(signed) instructions, using the kernel-cached map hash. The kernel
itself never actually attests that the metadata the loader installs is
the metadata that was signed.
This split is the core of the long-standing objection to the BPF signing
scheme from the LSM / integrity side: the integrity check of a light
skeleton only completes once the loader program runs, that is, after the
security_bpf_prog_load() hook, so at admission time an LSM observes a
program whose payload has not yet been verified. Auditing the chain
link is also not a purely cryptographic operation: whoever signs or
reviews an lskel has to disassemble the loader's preamble to convince
themselves that the embedded hash check is present and correct [0][1].
Two acceptable fixes were identified in those threads: Complete the
integrity check before the admission hook fires, or add a second hook
that collects the verification result after the loader ran [2]. Covering
both the loader and its maps directly with the PKCS#7 signature is what
Blaise Boscaccy's patchsets proposed in several forms. Let's implement
the former, without growing the UAPI, and in particular as a single
unified scheme where the signature spans the raw bytes rather than
derived hashes.
A signed loader binds its metadata map(s) through the existing fd_array,
and an exclusive map is already bound to a program digest (excl_prog_hash).
So when a signature is present, collect the exclusive maps from fd_array
and append their frozen contents to the instructions before verification:
The signature now covers insns || metadata_0 || metadata_1 || [...] in
the fd_array order, and verification completes in bpf_check(), once the
fd_array maps are resolved into used_maps, before the LSM admission hook
and the rest of verification. A program is either BPF_SIG_UNSIGNED or
BPF_SIG_VERIFIED, with nothing in between. While folding the fd_array
maps, a non-exclusive map bound to a signed program is rejected, so every
map folded into the signature is exclusive. A signed loader that fails
to cover its metadata thus does not load, and BPF_SIG_VERIFIED always
means the instructions and every exclusive map are authentic. The maps
must be frozen so the hashed bytes cannot change before the loader runs;
the map <-> program digest binding is enforced by the verifier for every
used map. Binding maps through fd_array_cnt makes the verifier resolve
and excl-check them (excl_prog_sha vs prog->digest) before it would
otherwise compute the digest, so compute prog->digest up front in
bpf_check(), over the unmodified instructions the signature covers, for
a load that folds metadata.
Unsigned programs are not affected by the signature path; for them the
LSM admission hook merely moves below fd_array resolution, with minimal
bounded work in between. Note, signed loaders generated by older libbpf/
bpftool versions need to be regenerated; some of the recent fixes we've
had on the signed loader side require the latter already to close gaps.
Finally, some remarks around the security_bpf_prog_load() placement
given there was discussion on whether a new hook is needed or the existing
security_bpf_prog() hook should be reused [3]: For a new hook it would
mean that just for loading a single BPF program it has to pass through
four layers of LSM hooks:
1) security_bpf (cmd=PROG_LOAD): for gating various bpf subcmds
2) security_bpf_prog_load: historical admission hook (CAP/token,
prog_type, attach point), pre-verification
3) security_bpf_prog_verify_signature: newly asked admission hook,
same role as 2), plus the BPF signature verdict
4) security_bpf_prog: gate handing the prog fd back to userspace,
verification done & signature verified
The use-cases of 2) and 3) conflate, thus BPF community prefers to just
keep a total of 3 LSM hooks (as-is today): 3) makes 2) incoherent given
they are the /same class/ of hook, that is, access-control admission on
the load and split only by _what_ they can see. Worse, with the split,
for a signed BPF program security_bpf_prog_load 2) admits a program whose
signature has not been checked, so a policy gating at 2) is structurally
unable to express "admit only verified" and every such policy is forced
onto 3) *anyway*. In other words, one doesn't get two complementary hooks,
but rather, one real admission hook aka 3) plus a now-degraded /legacy/
hook 2) that can't answer the question operators actually want to ask.
Reusing security_bpf_prog() 4) for admission is no alternative either:
it fires only after the entire verifier (and JIT) pipeline ran, so
denying a not-yet-verified program at that point burns exactly the
work a denial is supposed to avoid, and by then the program has an id
assigned and the kallsyms/perf/audit load events fired. Policies are
free to also consume the signature verdict at 4), but admission control
belongs into security_bpf_prog_load(). Hence the latter remains the only
admission hook, merely moved past signature verification; with moving
large allocations further down into the BPF verifier, there is now only
minimal work between the old and new location: The preparation work in
bpf_check() is reordered such that only the minimally necessary setup
happens up front: Allocating the env, initializing the verifier log and
resolving the fd_array that a signed BPF metadata map needs. The worst
case allocation up until security_bpf_prog_load() is ~90K which is the
env itself (~54K) plus the continuous fd_array cache (at most 32K). The
insn_aux_data array is moved into a later stage in the verification.
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Link: https://lore.kernel.org/bpf/2f71d6c03698eb17d51f7247efde777627ee578a.camel@HansenPartnership.com [0]
Link: https://lore.kernel.org/lkml/ecf0521ed302db672672ebfbc670ecfba36a6e00.camel@HansenPartnership.com [1]
Link: https://lore.kernel.org/bpf/88703f00d5b7a779728451008626efa45e42db3d.camel@HansenPartnership.com [2]
Link: https://lore.kernel.org/bpf/DJOFY21DYUI4.19WKQ3NPZ4H5R@gmail.com [3]
Link: https://lore.kernel.org/bpf/20260708075343.358712-3-daniel@iogearbox.net
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
The fd_array passed to BPF_PROG_LOAD carries the map and module BTF file
descriptors a program binds. The verifier reads it more than once during
a load: process_fd_array() walks it to bind the maps and BTFs, and
check_and_resolve_insns() and the kfunc BTF resolver later read it again
to resolve the program's BPF_PSEUDO_MAP_IDX* and module kfunc refs.
For signed BPF, we need these upfront in memory, thus resolve each fd to
its object once and cache it by fd_array index, then bind that cached
object for the rest of the load. env->fd_array becomes a small per-slot
{map, btf} cache rather than a bpfptr_t; every later reference is then
an in-bounds lookup of an already-resolved object, and an index outside
the cache is rejected instead of read from user memory:
- continuous (fd_array_cnt given): the caller declares the length and
every entry is resolved and bound up front (used also by the BPF
signed loader)
- sparse (no fd_array_cnt): left as the legacy path with no fd_array
cache; each reference reads its fd from the caller's fd_array and
resolves it on the spot. Deduplication in used_maps and the kfunc BTF
table keeps this correct, and only unsigned programs use this shape.
Split these into separate helpers to make it easier to follow.
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Acked-by: Anton Protopopov <a.s.protopopov@gmail.com>
Link: https://lore.kernel.org/bpf/20260708075343.358712-2-daniel@iogearbox.net
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Remove unused arch_needs_cpu() hook. No architectures use it after
s390 removed its use case.
Suggested-by: Heiko Carstens <hca@linux.ibm.com>
Signed-off-by: Mete Durlu <meted@linux.ibm.com>
Reviewed-by: Heiko Carstens <hca@linux.ibm.com>
Reviewed-by: Thomas Gleixner <tglx@kernel.org>
Acked-by: Rafael J. Wysocki (Intel) <rafael@kernel.org>
Signed-off-by: Alexander Gordeev <agordeev@linux.ibm.com>
Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
Background:
==========
Currently lockdep does not print out the held locks of non-current
tasks that are running on some other CPU, due to the fact that
the held locks array is in flux and may be unreliable to print.
Syzkaller on the other hand found it that the analysis of locking
bugs is easier if we print this information too, because the
more locking information the merrier. In particular races are
bound to have multiple tasks running on different CPUs, and
the exclusion of their held locks information is unnecessarily
limiting.
So while it's still true that printing out their held locks
array is racy, it's not as bad as it seems.
There's 16 internal callers to lockdep_print_held_locks():
- 14 callers call it with the current task, which should be
safe out of box.
- 1 caller, debug_show_all_locks(), calls it with RCU held,
which should guarantee that 'p' cannot go away under us.
- 1 caller, debug_show_held_locks(), exposes the internal API
with the constraint that it should only be called by drivers
or platform code if the task isn't actively running - we can
assume that if it nevertheless does, it will be Their Problem™.
As for held locks being changed from under debug_show_held_locks(),
while the task cannot go away, so the held-locks array itself is
safe (although potentially non-stable), AFAICS the worst-case race
can be garbage printed out by print_lock(), not any actual crashes.
In particular:
unsigned int class_idx = hlock->class_idx;
may be stale (belong to a lock that already got released on another
CPU), but it should still be a valid class index bound by
MAX_LOCKDEP_KEYS, and thus the lock_classes_in_use bitmap use
should be safe.
The other two accesses are ::acquire_ip and ::instance:
printk(KERN_CONT "%px", hlock->instance);
print_lock_name(hlock, lock);
printk(KERN_CONT ", at: %pS\n", (void *)hlock->acquire_ip);
But both are printed out as pointers, so no risk of dereference
of a dangling pointer. We may print a garbage pointer.
Also note that the check itself doesn't protect debug_show_held_locks()
from printing garbage, as there's nothing that keeps a task from
becoming runnable a nanosecond after we've run the task_is_running()
check. In fact I'd argue that it's better to make this function
*more* racy, for the simple robustness reason that we absolutely
do not want it to crash even in the racy case.
TL;DR: it should be fine to print the held locks of running
tasks too, as long as we print out the information as well
that a task is running, so that users are aware of any
racy output.
Implementation:
==============
Implement that change.
Also re-flow the function and streamline the printout into
a single statement for all cases, which changes
the 'no locks held by' / '%d lock[s] held by' phrasing that had a
dependency on English spelling of plurals, to a uniform:
locks held by bash/1234: %d
Which spells correctly for 0, 1 and higher values, and should also
be easier to parse both for humans and for scripts.
Finally, print out the last CPU a task has ran on. This is very
useful information for races and for locking bugs in particular.
This basically extends the 'on CPU#%d' message we print for
running tasks to all tasks we print.
Reported-by: Tetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp>
Suggested-by: Tetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp>
Tested-by: Tetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp>
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Cc: Boqun Feng <boqun@kernel.org>
Cc: Gary Guo <gary@garyguo.net>
Cc: Mark Brown <broonie@kernel.org>
Cc: Theodore Tso <tytso@mit.edu>
Cc: Miguel Ojeda <ojeda@kernel.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Will Deacon <will@kernel.org>
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Cc: Waiman Long <longman@redhat.com>
Link: https://patch.msgid.link/akoeSIQGwqd9cZwd@gmail.com
Add a matching sysctl to go with CONFIG_SYSCALL_USER_DISPATCH.
kernel.syscall_user_dispatch (default 1 - allow) controls whether
userspace may arm syscall user dispatch (both via prctl and ptrace).
Disarming is always permitted - same semantics as comparable knobs.
Disabling while a task has armed syscall user dispatch does not
cause it to become inactive - instead it remains active until the
user attempts to disable/re-enable via prctl or ptrace. On the
next attempt to re-enable, the prctl/ptrace call fails gracefully.
The alternative would cause programs translating non-linux syscalls
to interpret those syscalls as linux syscalls, resulting in undefined
userland behavior.
Signed-off-by: Gregory Price <gourry@gourry.net>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Link: https://patch.msgid.link/20260706140020.873735-3-gourry@gourry.net
Syscall User Dispatch is presently built under CONFIG_GENERIC_SYSCALL and
cannot be disabled independently.
Add CONFIG_SYSCALL_USER_DISPATCH to make it an optional feature.
Signed-off-by: Gregory Price <gourry@gourry.net>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Link: https://patch.msgid.link/20260706140020.873735-2-gourry@gourry.net
check_ptr_to_btf_access() lets program-type btf_struct_access callbacks
validate writes before the default BTF access path rejects non-read
accesses. That bypasses the read-only policy for untrusted BTF pointers
created by helpers such as bpf_rdonly_cast().
Reject non-read accesses through PTR_UNTRUSTED BTF pointers at the
common entry point, before the callback branch to handle all cases.
Fixes: 282de143ea ("bpf: Introduce allocated objects support")
Signed-off-by: Nicholas Dudar <main.kalliope@gmail.com>
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Reviewed-by: Amery Hung <ameryhung@gmail.com>
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
The verifier loses precision when simulating stack spills for the
following register types:
- PTR_TO_TP_BUFFER
- PTR_TO_INSN
- CONST_PTR_TO_DYNPTR
These types are not allow-listed in the is_spillable_regtype(),
because of that check_stack_write_fixed_off() takes the branch
that marks the slots STACK_MISC.
There are no technical reasons for this limitation.
This commit replaces an explicit list of pointer types in
is_spillable_regtype() with explicit list of non-pointer types.
The function is renamed to is_pointer_regtype() for clarity.
Reported-by: Andrii Nakryiko <andrii.nakryiko@gmail.com>
Suggested-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://lore.kernel.org/bpf/20260707-missing-spillable-types-v1-1-44a92121dc41@gmail.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
timekeeping_apply_adjustment() modifies xtime_nsec to ensure monotonicity
when mult changes:
xtime_nsec -= offset
This ensures that the time reported to userspace does not jump when the
multiplier is adjusted from one tick to the next. However, the ntp_error
accumulator which tracks the difference between intended and actual
clock position was not being updated to reflect this additional
discrepancy.
An earlier attempt at this compensation existed as:
ntp_error -= (interval - offset) << ntp_error_shift
but was removed in commit c2cda2a5bd ("timekeeping/ntp: Don't align
NTP frequency adjustments to ticks") because it was a major source of
NTP error. That's because (interval - offset) was wrong: the subtraction
of "interval" prematurely accounted for the changed xtime_interval of
the next tick, which would be correctly accounted in the next
accumulation anyway — a double subtraction.
What is actually needed is just the "offset" part: ntp_error must be
told that xtime_nsec moved by "offset" without a corresponding change
in the intended position. For the normal ±1 mult dithering this is
negligible (the adjustments cancel over time), but for larger mult
changes — such as when an external reference clock sets a new
frequency — the one-time uncompensated offset is significant.
Fix by adjusting ntp_error by the correct amount:
ntp_error += offset << ntp_error_shift
This keeps ntp_error consistent with the actual xtime_nsec position
after the adjustment, and ensures the discrepancy is correctly smoothed
away over time and the clock returns to where it should have been.
Fixes: c2cda2a5bd ("timekeeping/ntp: Don't align NTP frequency adjustments to ticks")
Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Assisted-by: Kiro:claude-opus-4.6-1m
Acked-by: John Stultz <jstultz@google.com>
Link: https://patch.msgid.link/20260621220051.1030462-3-dwmw2@infradead.org
The fault handler for the timens page does not have access to the target
task and therefore can not be invoked remotely.
Currently the handler relies on the fact that the vvar mapping is marked as
VM_IO and VM_PFNMAP for which the mm core always prevents remote access.
However the VM_IO and VM_PFNMAP flags are going to be removed.
Add an explicit check to prevent remote access to the mapping.
Signed-off-by: Thomas Weißschuh <thomas.weissschuh@linutronix.de>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Tested-by: Nam Cao <namcao@linutronix.de>
Link: https://patch.msgid.link/20260630-vdso-mlockall-v4-4-6c93708ce723@linutronix.de
All architectures are now fully using the generic vDSO infrastructure.
They don't need these declarations anymore to implement the functions
in architecture-specific code.
Move them to the private header.
Signed-off-by: Thomas Weißschuh <thomas.weissschuh@linutronix.de>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Link: https://patch.msgid.link/20260707-timekeeping-header-cleanup-v1-2-e85ad96409a9@linutronix.de
This header uses some definitions from linux/hrtimer_bases.h.
Currently this header is included transitively, which will change.
Include the header explicitly.
Signed-off-by: Thomas Weißschuh (Schneider Electric) <thomas.weissschuh@linutronix.de>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Link: https://patch.msgid.link/20260702-hrtimer-header-dependencies-v1-5-c50b19bda473@linutronix.de
This header uses some definitions from linux/hrtimer_bases.h.
Currently this header is included transitively, which will change.
Include the header explicitly.
Signed-off-by: Thomas Weißschuh (Schneider Electric) <thomas.weissschuh@linutronix.de>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Link: https://patch.msgid.link/20260702-hrtimer-header-dependencies-v1-4-c50b19bda473@linutronix.de
The usage of the hrtimer base forces hrtimer.h to also expose the base
structure definitions.
Move the function to hrtimer.c to avoid this.
Signed-off-by: Thomas Weißschuh (Schneider Electric) <thomas.weissschuh@linutronix.de>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Link: https://patch.msgid.link/20260702-hrtimer-header-dependencies-v1-3-c50b19bda473@linutronix.de
The usage of the hrtimer base introduces a dependency on the timer base
structure definitions from the widely-used hrtimer.h.
Move the helper to hrtimer_bases.h to trim this dependency.
Also adapt the two only callers to now include hrtimer_bases.h.
Signed-off-by: Thomas Weißschuh (Schneider Electric) <thomas.weissschuh@linutronix.de>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Link: https://patch.msgid.link/20260702-hrtimer-header-dependencies-v1-2-c50b19bda473@linutronix.de
hrtimer_get_next_event() runs on every tick-stop decision via
get_next_timer_interrupt() -> cmp_next_hrtimer_event(). When high
resolution timers are active it must return KTIME_MAX -- the caller
documents and depends on this.
The function takes cpu_base->lock, checks hres_active, and returns
KTIME_MAX.
Taking the lock is not required because cpu_base->hres_active is only
written by the local CPU in hrtimer_switch_to_hres() from hard interrupt
context and in hrtimers_cpu_starting() during bring-up.
All callers of hrtimer_get_next_event() reach it from the tick-stop /
cpuidle paths with interrupts disabled on that CPU. No writer can therefore
race with the read, so an unlocked hres_active check is stable and the lock
can be skipped in this case.
On a 176-thread AMD EPYC 9D64 running a production workload, bucketing
callers of native_queued_spin_lock_slowpath(), the slowpath had 199 samples
in total, of which 54 are attributed to hrtimer_get_next_event(), i.e. this
accounts for ~27% of slowpath hits on this specific workload.
Move the hres_active check before the lock guard region to address this.
[ tglx: Massaged change log and comments ]
Signed-off-by: Usama Arif <usama.arif@linux.dev>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Link: https://patch.msgid.link/20260707153800.542394-1-usama.arif@linux.dev
The function validates tv_sec to be within [-KTIME_SEC_MAX, KTIME_SEC_MAX]
but never validates that tv_nsec is within the valid range of
[0, NSEC_PER_SEC-1] before using it in timespec64_add().
timespec64_add() expects both timespec64 structures to have normalized
values with tv_nsec in the range [0, 999999999]. If off->val.tv_nsec
contains invalid values (negative or >= NSEC_PER_SEC), it could lead to
incorrect calculations or unexpected behavior.
Add validation to ensure tv_nsec is within the valid range before
performing the addition.
Fixes: 04a8682a71 ("fs/proc: Introduce /proc/pid/timens_offsets")
Signed-off-by: Malaya Kumar Rout <malayarout91@gmail.com>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Link: https://patch.msgid.link/20260704093429.89350-1-malayarout91@gmail.com
When the WARN_ON_ONCE(i >= tmigr_hierarchy_levels) assertion triggers,
the function returns -EINVAL without freeing the 'stack' memory allocated
via kzalloc_objs() at the beginning of the function.
Add kfree(stack) before returning to prevent the memory leak.
Fixes: 6c181b5667 ("timers/migration: Convert "while" loops to use "for"")
Signed-off-by: Malaya Kumar Rout <malayarout91@gmail.com>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Link: https://patch.msgid.link/20260704085533.87098-1-malayarout91@gmail.com
tk_aux_sysfs_init() creates one child kobject per auxiliary clock. If a
later child or sysfs group creation fails, the current error path only
puts the parent kobjects and leaves earlier children and groups behind.
Store the child kobjects during init and remove the successfully created
groups and kobjects on failure.
Fixes: 7b5ab04f03 ("timekeeping: Fix resource leak in tk_aux_sysfs_init() error paths")
Signed-off-by: Yuho Choi <dbgh9129@gmail.com>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Link: https://patch.msgid.link/20260703165337.168445-1-dbgh9129@gmail.com
init_clocksource_sysfs() registers the clocksource subsystem before
registering the clocksource device. If device_register() fails, the
function returns the error while leaving the subsystem registered.
Unregister the clocksource subsystem on that failure path so the
successful subsystem registration is unwound before returning.
Fixes: d369a5d8fc ("clocksource: convert sysdev_class to a regular subsystem")
Signed-off-by: Yuho Choi <dbgh9129@gmail.com>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Link: https://patch.msgid.link/20260702215733.84588-1-dbgh9129@gmail.com
The kstat_irqs field of struct irq_desc is used to store a per-cpu count
of interrupt events. It is initialized in init_desc(), along with all
the other fields in struct irq_desc that need explicit initialization,
and therefore it's always available (non-NULL) for any valid interrupt
descriptor (with a caveat - see below).
When CONFIG_SPARSE_IRQ is enabled, all interrupt descriptors are always
allocated dynamically via alloc_desc(), which calls init_desc(), so in
that case kstat_irqs is guaranteed to be non-NULL before a valid struct
irq_desc pointer is even returned.
By contrast, when CONFIG_SPARSE_IRQ is disabled, interrupt descriptors
are allocated statically in the irq_desc[] array, and kstat_irqs is
initialized implicitly to NULL. The per-cpu pointer is initialized only
later, for all descriptors, via start_kernel() -> early_irq_init() ->
init_desc().
The kstat_irqs field is used mostly for printing interrupt statistics
(i.e. reading /proc/interrupts), and that cannot happen until much
later, when user-space is fully initialized. So, there is no concern
with that use case.
The list below includes all functions where the NULL check is removed,
along with a list of all possible call chains and/or a brief explanation
of why it's safe to remove the NULL check in that case.
* irq_desc_kstat_cpu() [include/linux/irqdesc.h]
- All direct call sites use it for printing IRQ statistics.
- Indirect call site: per_cpu_count_show() - also used for printing
interruptstatistics.
* kstat_irqs_cpu() [kernel/irq/irqdesc.c]
- Called by sun3_int7() and sun3_int5() [arch/m68k/sun3/sun3ints.c]
These are interrupt handlers and cannot be called until their
corresponding interrupts are initialized in sun3_init_IRQ(). The
call chain leading to that is:
start_kernel() -> init_IRQ() [arch/m68k/kernel/ints.c] ->
mach_init_IRQ = sun3_init_IRQ()
The init_IRQ() call happens right *after* the early_irq_init() call,
which means the descriptors are already fully initialized by the
time the interrupt handlers are even registered.
- Called by show_interrupts() [arch/s390/kernel/irq.c] - used for
printing interrupt statistics.
* kstat_irqs() [kernel/irq/irqdesc.c]
The only possible call chain is via fs/proc/stat.c:
stat_open() -> show_stat() -> show_all_irqs() -> kstat_irqs_usr() ->
kstat_irqs()
It is used for printing interrupt statistics.
* kstat_snapshot_irqs()
The only possible call chain is via kernel/watchdog.c:
watchdog_timer_fn() -> is_softlockup() -> start_counting_irqs() ->
kstat_snapshot_irqs()
The watchdog timer cannot fire early, before early_irq_init().
* kstat_get_irq_since_snapshot()
The only possible call chain is via kernel/watchdog.c:
watchdog_timer_fn() -> report_cpu_status() -> print_irq_counts() ->
kstat_get_irq_since_snapshot()
The watchdog timer cannot fire early, before early_irq_init().
Signed-off-by: Radu Rendec <radu@rendec.net>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Link: https://patch.msgid.link/20260705210951.2717741-2-radu@rendec.net
sock_map_update_common() and __sock_map_delete() hold stab->lock and call
sock_map_unref() -> sock_map_del_link(), which takes sk_callback_lock for
write. That gives the order stab->lock -> sk_callback_lock.
The reverse order comes from the SK_SKB stream parser.
sk_psock_strp_data_ready() holds sk_callback_lock for read, and after the
verdict tcp_bpf_strp_read_sock() acks the consumed data inline via
__tcp_cleanup_rbuf(). The ACK goes out egress, where a sched_cls program
deletes from the sockmap and takes stab->lock:
WARNING: possible circular locking dependency detected
------------------------------------------------------
syz.9.8824 is trying to acquire lock:
(&stab->lock){+.-.}-{3:3}, at: __sock_map_delete net/core/sock_map.c:421
but task is already holding lock:
(clock-AF_INET){++.-}-{3:3}, at: sk_psock_strp_data_ready net/core/skmsg.c:1173
-> #1 (clock-AF_INET){++.-}-{3:3}:
_raw_write_lock_bh
sock_map_del_link net/core/sock_map.c:167
sock_map_unref net/core/sock_map.c:184
sock_map_update_common net/core/sock_map.c:509
sock_map_update_elem_sys net/core/sock_map.c:588
map_update_elem kernel/bpf/syscall.c:1805
-> #0 (&stab->lock){+.-.}-{3:3}:
_raw_spin_lock_bh
__sock_map_delete net/core/sock_map.c:421
sock_map_delete_elem net/core/sock_map.c:452
bpf_prog_06044d24140080b6
tcx_run net/core/dev.c:4451
sch_handle_egress net/core/dev.c:4541
__dev_queue_xmit net/core/dev.c:4808
...
tcp_bpf_strp_read_sock net/ipv4/tcp_bpf.c:701
strp_data_ready net/strparser/strparser.c:402
sk_psock_strp_data_ready net/core/skmsg.c:1174
tcp_data_queue net/ipv4/tcp_input.c:5661
Possible unsafe locking scenario:
CPU0 CPU1
---- ----
rlock(clock-AF_INET);
lock(&stab->lock);
lock(clock-AF_INET);
lock(&stab->lock);
*** DEADLOCK ***
A tc, xdp, socket_filter or flow_dissector program has no reason to
update or delete a sockmap, and redirect does not go through here. Drop
them from may_update_sockmap() so the verifier rejects it. It also
closes the matching sockhash inversion.
Suggested-by: John Fastabend <john.fastabend@gmail.com>
Signed-off-by: Sechang Lim <rhkrqnwk98@gmail.com>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Reviewed-by: John Fastabend <john.fastabend@gmail.com>
Reviewed-by: Emil Tsalapatis <emil@etsalapatis.com>
Link: https://lore.kernel.org/bpf/20260630145410.3648099-2-rhkrqnwk98@gmail.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
parse_synth_field() builds a "__data_loc ..." type string before
assigning it to field->type. If the seq_buf check fails, the common
cleanup cannot free the temporary string. Free it before leaving.
Link: https://patch.msgid.link/20260603062533.1096320-2-pengyu@kylinos.cn
Signed-off-by: Yu Peng <pengyu@kylinos.cn>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
user_event_mm_dup() walks the parent mm's enabler list locklessly under
rcu_read_lock() during fork() (from copy_process()); it does not take
event_mutex:
rcu_read_lock();
list_for_each_entry_rcu(enabler, &old_mm->enablers, mm_enablers_link)
enabler->event = user_event_get(orig->event);
user_event_enabler_destroy() removes an enabler from that list with
list_del_rcu() and then, without waiting for a grace period, drops the
enabler's user_event reference with user_event_put() and frees the enabler
with kfree(). A reader that loaded the enabler before the list_del_rcu()
can still be walking it, which leads to two use-after-frees:
- kfree(enabler) frees the enabler while that reader dereferences
enabler->event.
- user_event_put() may drop the last reference to the user_event, which
is then freed (via delayed_destroy_user_event() on a work queue), while
the same reader does user_event_get(orig->event) on it.
Both are reachable by an unprivileged task that can open user_events_data:
one multithreaded process that registers an enabler and then concurrently
unregisters it and calls fork() triggers the race. KASAN reports a
slab-use-after-free in user_event_mm_dup() during clone(), with a
"refcount_t: addition on 0" warning when the user_event is freed.
The enabler use-after-free was found first; the user_event one was reported
by XIAO WU, and the earlier enabler-only fix did not address it.
Defer both the user_event_put() and the kfree(enabler) to a work item
queued with queue_rcu_work(), so they run only after an RCU grace period,
once all readers walking the enabler list have finished. The put must run
in process context because user_event_put() takes event_mutex on the last
reference, so a work queue is used rather than call_rcu(). The now-unlocked
put lets the locked argument of user_event_enabler_destroy() be removed;
all callers are updated.
Fixes: 7235759084 ("tracing/user_events: Use remote writes for event enablement")
Cc: stable@vger.kernel.org
Link: https://patch.msgid.link/20260707165912.2560537-2-michael.bommarito@gmail.com
Reported-by: XIAO WU <xiaowu.417@qq.com>
Closes: https://lore.kernel.org/all/tencent_89647CE40DC452B891C65C94D1B271DE8E07@qq.com/
Suggested-by: Beau Belgrave <beaub@linux.microsoft.com>
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Michael Bommarito <michael.bommarito@gmail.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
BTF struct walks relax the struct-size check for accesses through a
trailing flexible array. That is valid for ordinary BTF type walking, but
PTR_TO_BTF_ID | MEM_ALLOC values point to objects allocated with the static
BTF type size.
When walking a MEM_ALLOC object, reject the access before applying the
flexible-array relaxation if the access range extends past the struct size.
Apply the same policy to struct ID matching so kfunc and kptr type checks
do not walk past the allocated object bounds either.
Fixes: 958cf2e273 ("bpf: Introduce bpf_obj_new")
Fixes: 36d8bdf75a ("bpf: Add alloc/xchg/direct_access support for local percpu kptr")
Signed-off-by: Yiyang Chen <chenyy23@mails.tsinghua.edu.cn>
Reviewed-by: Amery Hung <ameryhung@gmail.com>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://lore.kernel.org/bpf/4b8c8a81102ba4b595011434c881194f264ddc59.1782807039.git.chenyy23@mails.tsinghua.edu.cn
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Tracepoints require that RCU is watching. To prevent them from being used
in places that RCU is not watching, the trace_##event() macro always
calls rcu_is_watching() even when the event is not enabled and warns if
RCU is not watching. This is to make sure a warning is triggered even if
the tracepoint is never enabled (as it is only a bug when it is).
It was noticed that tracepoints could be hidden within
trace_#event#_enabled() calls, which are used to do extra work for the
tracepoint only if the tracepoint is enabled. But this also can hide the
fact that a tracepoint is placed in a location that can be called when RCU
is not watching.
Commit 9764e731ef ("tracepoint: Add lockdep rcu_is_watching() check to
trace_##name##_enabled()") added a check to the trace_##event##_enabled()
macro to make sure RCU is watching when it is called to make sure not to
hide the bug of a tracepoint being called when RCU is not watching.
There is one case in the irq_disable tracepoint where it is within a
trace_irq_disable_enabled() block, but it checks if RCU is watching, and
if it isn't, it makes a call to ct_irq_enter() that makes RCU watch again.
But because trace_irq_disable_enabled() now checks if RCU is watching and
will trigger if it isn't. This is a false warning as the code within
the block handles this case.
Add a new internal macro __trace_##event##_enabled() that doesn't check if
RCU is watching, and convert the irq_enable/disable tracepoints over to
it.
Link: https://patch.msgid.link/20260701132744.6a7fc68b@robin
Reported-by: Geert Uytterhoeven <geert@linux-m68k.org>
Closes: https://lore.kernel.org/all/CAMuHMdXud_RpWag_hFqa2ByBGRxg6KnxGL1ObCWZrpTsk3TfAw@mail.gmail.com/
Fixes: 9764e731ef ("tracepoint: Add lockdep rcu_is_watching() check to trace_##name##_enabled()")
Tested-by: Geert Uytterhoeven <geert+renesas@glider.be>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
String event fields are not necessarily NUL-terminated, so the filter
predicate functions (filter_pred_string(), filter_pred_strloc() and
filter_pred_strrelloc()) pass the field length to the regex match
callbacks, and the length-aware matchers honour it.
regex_match_glob() was the exception: it ignored the length and called
glob_match(), which scans the string until it hits a NUL byte. Some
string fields are not NUL-terminated. One example is the dynamic char
array of the xfs_* namespace tracepoints, which is copied without a
trailing NUL. For such a field, glob matching reads past the end of
the event field, causing a KASAN slab-out-of-bounds read in
glob_match(), reached via regex_match_glob() and filter_match_preds()
from the xfs_lookup tracepoint.
Add a length-bounded glob_match_len() and use it from regex_match_glob()
so glob matching always stops at the field boundary. The matching loop
is factored into a shared helper so glob_match() keeps its behaviour.
Fixes: 60f1d5e3ba ("ftrace: Support full glob matching")
Cc: stable@vger.kernel.org
Link: https://patch.msgid.link/da1aaf125fc3b63320b0c540fd6afa7c3d5b4f1a.1782836943.git.hhhuang@smu.edu.sg
Reported-by: Yuan Tan <yuantan098@gmail.com>
Reported-by: Yifan Wu <yifanwucs@gmail.com>
Reported-by: Juefei Pu <tomapufckgml@gmail.com>
Reported-by: Zhengchuan Liang <zcliangcn@gmail.com>
Reported-by: Xin Liu <bird@lzu.edu.cn>
Assisted-by: Codex:GPT-5.4
Signed-off-by: Huihui Huang <hhhuang@smu.edu.sg>
Signed-off-by: Ren Wei <n05ec@lzu.edu.cn>
Acked-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
func_set_flag() dereferences tr->current_trace_flags before verifying
that the current tracer is actually the function tracer. When the active
tracer has been switched away from "function" (e.g., to "wakeup_rt"),
tr->current_trace_flags can be NULL, leading to a NULL pointer
dereference and kernel crash.
The call chain that triggers this is:
trace_options_write()
-> __set_tracer_option()
-> trace->set_flag() /* func_set_flag */
In func_set_flag(), the first operation is:
if (!!set == !!(tr->current_trace_flags->val & bit))
This dereferences tr->current_trace_flags unconditionally. The safety
check that guards against a non-function tracer:
if (tr->current_trace != &function_trace)
return 0;
is placed *after* the dereference, which is too late.
This was observed with the following crash dump:
BUG: unable to handle page fault at 0000000000000000
RIP: func_set_flag+0xd
Call Trace:
__set_tracer_option+0x27
trace_options_write+0x75
vfs_write+0x12a
ksys_write+0x66
do_syscall_64+0x5b
RIP: ffffffff914c973d RSP: ff67ec88b01dfdf0 RFLAGS: 00010202
RAX: 0000000000000000 RBX: ff3a826e80354580 RCX: 0000000000000001
RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffffffff93918080
The disassembly confirms the fault:
func_set_flag+0: mov 0x1f08(%rdi), %rax ; RAX = tr->current_trace_flags = NULL
func_set_flag+13: mov (%rax), %eax ; page fault: dereference NULL
At the time of the crash:
tr->current_trace_flags = 0x0 (NULL)
tr->current_trace = wakeup_rt_tracer (not function_trace)
The scenario is that a process opens a function tracer option file (such
as "func_stack_trace"), then the current tracer is switched to another
tracer (e.g., "wakeup_rt"), which sets current_trace_flags to NULL. When
the process subsequently writes to the option file, func_set_flag() is
invoked and crashes on the NULL dereference.
Fix this by moving the current_trace check before the
current_trace_flags dereference, so that func_set_flag() returns early
when the function tracer is not active.
Cc: stable@vger.kernel.org
Link: https://patch.msgid.link/20260624061715.1445655-1-xiangzao@linux.alibaba.com
Fixes: 76680d0d28 ("tracing: Have function tracer define options per instance")
Signed-off-by: Yuanhe Shu <xiangzao@linux.alibaba.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
The tracepoint_printk symbol is not exported, so make it
static to remove the following sparse warning:
kernel/trace/trace.c:90:5: warning: symbol 'tracepoint_printk' was not declared. Should it be static?
Fixes: dd293df639 ("tracing: Move trace sysctls into trace.c")
Link: https://patch.msgid.link/20260617105822.904164-1-ben.dooks@codethink.co.uk
Signed-off-by: Ben Dooks <ben.dooks@codethink.co.uk>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
Commit 8928e4a3be ("ring-buffer: Show persistent buffer dropped events
in trace_pipe file") split the "commit" variable in
ring_buffer_read_page() into "commit" (raw) and "size" (masked page
size), but the inner copy loop's terminator was changed to compare rpos
against "event_size" instead of "size".
rpos is the cumulative read offset within the page; event_size is the
length of the single event just copied. The loop thus breaks after the
first event, so only one event is copied per call. This regresses the
per-event memcpy path (partial reads, the active commit page, and
mapped/remote buffers) used by splice/trace_pipe_raw and mmap consumers
into a one-event-at-a-time read.
Compare rpos against the page size as the original code did.
Link: https://patch.msgid.link/20260616175538.111628-1-devnexen@gmail.com
Fixes: 8928e4a3be ("ring-buffer: Show persistent buffer dropped events in trace_pipe file")
Signed-off-by: David Carlier <devnexen@gmail.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
A NULL old_name in a newly loaded livepatch's function entry causes a
NULL pointer dereference in strcmp():
klp_init_patch()
klp_add_nops()
klp_find_func()
strcmp(old_func->old_name, func->old_name)
Add klp_check_patch() at the beginning of klp_enable_patch() to reject
patches with NULL old_name before they reach this code path.
Reported-by: sashiko-bot <sashiko-bot@kernel.org>
Closes: https://lore.kernel.org/live-patching/20260529040130.95A9C1F00893@smtp.kernel.org/
Suggested-by: Petr Mladek <pmladek@suse.com>
Suggested-by: Miroslav Benes <mbenes@suse.cz>
Signed-off-by: Yafang Shao <laoar.shao@gmail.com>
Acked-by: Miroslav Benes <mbenes@suse.cz>
Reviewed-by: Petr Mladek <pmladek@suse.com>
Tested-by: Petr Mladek <pmladek@suse.com>
Link: https://patch.msgid.link/20260628114635.33572-1-laoar.shao@gmail.com
Signed-off-by: Petr Mladek <pmladek@suse.com>
There are 2 possible scenarios where the cgroup_taskset structure
passed into the cgroup can_attach() and attach() methods can contain
task migration data with multiple source cpusets.
- A multithread application with threads in different cpusets is
fully migrated into a new cpuset.
- Disabling v2 cpuset controller will move all the tasks in child
cpusets to the parent cpuset.
The current cpuset_can_attach() and cpuset_attach() functions still
expect task migration is from one source cpuset to one destination
cpuset.
Fix that by tracking the set of source (old) cpusets in singly linked
lists. The list will be iterated when necessary to properly update
internal data.
To ensure proper DL tasks accounting, the nr_migrate_dl_tasks in both
the source and destination cpusets are decremented/incremented with
their values added to nr_deadline_tasks when the migration is successful.
The setting of the global attach_ctx.cpus_updated and
attach_ctx.mems_updated flags are also moved from cpuset_attach()
to cpuset_can_attach() as the correct source cpuset can no longer be
determined in cpuset_attach() and cpuset states will not be changed
between cpuset_attach() and cpuset_can_attach() with an earlier patch.
Signed-off-by: Waiman Long <longman@redhat.com>
Reviewed-by: Ridong Chen <ridong.chen@linux.dev>
Signed-off-by: Tejun Heo <tj@kernel.org>
The cpuset_attach_task() was introduced in commit 42a11bf5c5
("cgroup/cpuset: Make cpuset_fork() handle CLONE_INTO_CGROUP properly")
to enable the CLONE_INTO_CGROUP flag of clone(2) to behave more like
moving a task from one cpuset into another one. That commits didn't
move the mpol_rebind_mm() and cpuset_migrate_mm() calls for group leader
into cpuset_attach_task().
When the CLONE_INTO_CGROUP flag is used without CLONE_THREAD, the new
task is its own group leader. So it is still not equivalent to moving
task between cpusets in this case. Make CLONE_INTO_CGROUP behaves
more close to cpuset_attach() by moving the mpol_rebind_mm() and
cpuset_migrate_mm() calls inside cpuset_attach_task().
Also move the stack local cpus_updated, mems_updated and queue_task_work
flags into attach_ctx so that these flags can be accessed inside and
outside of cpuset_attach_task(). The cpuset_fork() function is updated
to set up these flags and do memory migration if necessary.
Reviewed-by: Ridong Chen <ridong.chen@linux.dev>
Signed-off-by: Waiman Long <longman@redhat.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
There are two possible ways that migration of tasks from multiple source
cpusets to a target cpuset can happen. Either a multithread application
with threads in different cpusets is wholely migrated to a new cpuset
or disabling of v2 cpuset controller will move all the tasks in child
cpusets to the parent cpuset.
In the former case, it is the mm setting of the group leader that
really matters. So attach_ctx.old_cs should track the oldcs of the
thread leader. In the latter case, effective_mems of child cpusets
must always be a subset of the parent. So no real page migration
will not be necessary no matter which child cpuset is selected as
attach_ctx.old_cs.
IOW, attach_ctx.old_cs should be updated to match the latest task
group leader in cpuset_can_attach(), but fall back to that of the first
task if there is no group leader in the taskset.
Suggested-by: Ridong Chen <ridong.chen@linux.dev>
Signed-off-by: Waiman Long <longman@redhat.com>
Reviewed-by: Ridong Chen <ridong.chen@linux.dev>
Signed-off-by: Tejun Heo <tj@kernel.org>
Expand the scope of cpuset_can_attach_check() by including the setting
of setsched flag inside cpuset_can_attach_check() with the new @oldcs
and @psetsched argument. As cpuset_can_attach_check() is also called
from cpuset_can_fork(), set the new arguments to NULL from that caller.
Signed-off-by: Waiman Long <longman@redhat.com>
Reviewed-by: Ridong Chen <ridong.chen@linux.dev>
Signed-off-by: Michal Koutný <mkoutny@suse.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
Extract the DL bandwidth allocation code in cpuset_attach() to a new
cpuset_reserve_dl_bw() helper to simplify code.
No functional change is expected.
Signed-off-by: Waiman Long <longman@redhat.com>
Reviewed-by: Ridong Chen <ridong.chen@linux.dev>
Reviewed-by: Gregory Price <gourry@gourry.net>
Signed-off-by: Tejun Heo <tj@kernel.org>
Put the task attach related cpuset_attach_old_cs and
cpuset_attach_nodemask_to static variables into the new attach_ctx
structure to improve readability and ease maintanence.
No functional change is expected.
Suggested-by: Ridong Chen <ridong.chen@linux.dev>
Signed-off-by: Waiman Long <longman@redhat.com>
Reviewed-by: Ridong Chen <ridong.chen@linux.dev>
Signed-off-by: Tejun Heo <tj@kernel.org>
Commit e44193d39e ("cpuset: let hotplug propagation work wait for
task attaching") was introduced to let hotplug operation to wait
until the completion of task attach operation. However, it is still
possible that the states of the source or destination cpuset can
be changed between the cpuset_can_attach() call and the subsequent
cpuset_attach()/cpuset_cancel_attach() call.
As a result, data gathered during cpuset_can_attach() cannot be reliably
used in the subsequent cpuset_attach()/cpuset_cancel_attach()
call at all. Make the task attach operation more robust
and allow the sharing of data between cpuset_can_attach() and
cpuset_attach()/cpuset_cancel_attach() by making cpuset_write_resmask()
and cpuset_partition_write() wait for the completion of task attach
as well.
Ideally, an ongoing task attach operation should block any cpuset write
operation that can change its internal state until the operation is
completed. However, the attach_in_progress flag is currently per cpuset
and only the destination cpuset will have this flag set. The flag is not
set in the source cpuset where the tasks will be moved from. Even if we
extend the scope to include the source cpuset, it will not block cpuset
operation that changes the state of one of its ancestor cpuset which may
indirectly impact the state of the source or destination cpuset. It may
be too costly to set the flag for the whole subtree, it is far easier
to just make the flag global and block all the cpuset write operation
whenever a task attach operation is in progress.
Make that change by creating a new cpuset attach context (attach_ctx)
structure to hold the global in_progress flag and use it for blocking
cpuset write operation if a cpuset attach operation is in progress. Also
add a new wait_attach_done_lock() helper to do the waiting for an
ongoing attach operation and acquire the cpuset_mutex.
The comments about validate_change() are no longer valid as it won't
be called at all if an attach operation is in progress. So the comments
can be removed.
The per-cpuset attach_in_progress flag is also currently used in
partition_is_populated() and cpuset_is_populated() to determine if
an empty cpuset will have incoming task. This check will no longer be
needed as this function will not be called when there is a task attach
in progress. So the flag check is now removed.
Reviewed-by: Ridong Chen <ridong.chen@linux.dev>
Signed-off-by: Waiman Long <longman@redhat.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
Whenever memory node mask is changed, there are 4 places where the node
mask has to be updated or used.
1) task's node mask via cpuset_change_task_nodemask()
2) memory policy binding via mpol_rebind_mm()
3) if memory migration is enabled, migrate from old_mems_allowed to
the new node mask via cpuset_migrate_mm().
4) setting old_mems_allowed
These memory actions are done in cpuset_update_tasks_nodemask() and
cpuset_attach(). However there are inconsistencies in what node masks
are being used in these 2 functions.
In cpuset_update_tasks_nodemask(),
- cpuset_change_task_nodemask(): guarantee_online_mems()
- mpol_rebind_mm(): mems_allowed
- cpuset_migrate_mm(): guarantee_online_mems()
- old_mems_allowed: guarantee_online_mems()
In cpuset_attach(),
- cpuset_change_task_nodemask(): guarantee_online_mems()
- mpol_rebind_mm(): effective_mems
- cpuset_migrate_mm(): effective_mems
- old_mems_allowed: effective_mems
These inconsistencies dates back to quite a long time ago and it is
hard to say what should be the correct values.
The guarantee_online_mems() function returns a node mask from current or
an ancestor cpuset that is a subset of node_states[N_MEMORY]. Nodes in
node_states[N_MEMORY] are all online, i.e. in node_states[N_ONLINE].
However, node in node_states[N_ONLINE] may not have memory. So
node_states[N_MEMORY] should be a subset of node_states[N_ONLINE].
The guarantee_online_mems() function should mostly be useful for v1
where mems_allowed is the same as effective_mems. With v2, the memory
nodes in effective_mems should be a subset of node_states[N_MEMORY]
except when a memory hot-unplug operation is in progress and a memory
node is removed from node_states[N_MEMORY] but not yet reflected in
the effective_mems's as cpuset_handle_hotplug() has not been called
from cpuset_track_online_nodes().
Let use the following setup for both of them and make them consistent.
- cpuset_change_task_nodemask(): guarantee_online_mems()
- mpol_rebind_mm(): effective_mems
- cpuset_migrate_mm(): guarantee_online_mems()
- old_mems_allowed: guarantee_online_mems()
So for v2, it is effectively all effective_mems most of the time. For
v1, mpol_rebind_mm() uses mems_allowed which may differ from what
guarantee_online_mems() returns, but it conforms to what the cpuset v1
documentation says with respect to setting memory policy.
Signed-off-by: Waiman Long <longman@redhat.com>
Reviewed-by: Ridong Chen <ridong.chen@linux.dev>
Reviewed-by: Gregory Price <gourry@gourry.net>
Signed-off-by: Tejun Heo <tj@kernel.org>
The nr_deadline_tasks variable in the cpuset structure was introduced by
commit 6c24849f55 ("sched/cpuset: Keep track of SCHED_DEADLINE task
in cpusets"). It is reported by sashiko [1] that nr_deadline_tasks
can currently be modified by inc_dl_tasks_cs() under rq->lock and
by cpuset_attach() under cpuset_mutex. So if both updates happen
simultaneously, the nr_deadline_tasks variable can be corrupted leading
to incorrect operations down the road.
Fix that by changing its type to atomic_t so that nr_deadline_tasks
are always atomically updated. This fix patch is a low hanging fruit.
It can handle some of the races between a concurrent sched_setscheduler()
and cpuset_can_attach()/cpuset_attach() calls, but not all of them like
the other issue raised by sashiko [2]. This will be handled hopefully
in a future follow up patch.
[1] https://sashiko.dev/#/patchset/20260626181923.133658-1-longman%40redhat.com
[2] https://sashiko.dev/#/patchset/20260630033344.352702-1-longman%40redhat.com
Fixes: 6c24849f55 ("sched/cpuset: Keep track of SCHED_DEADLINE task in cpusets")
Reviewed-by: Ridong Chen <ridong.chen@linux.dev>
Signed-off-by: Waiman Long <longman@redhat.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
Creating a child cpuset where cpuset.mems is never set leads to a div/0
when a VMA mempolicy with MPOL_F_RELATIVE_NODES rebinds in response to a
CPU hotplug event.
Reproduction steps:
1) Create a cgroup w/ cpuset controls (do not set cpuset.mems)
2) Move the task into the child cpuset
3) Create a VMA mempolicy for that task with MPOL_F_RELATIVE_NODES
4) unplug and hotplug a cpu
echo 0 > /sys/devices/system/cpu/cpu1/online
echo 1 > /sys/devices/system/cpu/cpu1/online
5) mempolicy rebind does a div/0 in mpol_relative_nodemask on the
call to __nodes_fold()
The cpuset code passes (cs->mems_allowed) which is not guaranteed to have
nodes to the rebind routine. Use cs->effective_mems instead, which is
guaranteed to have a non-empty nodemask once we reach that code path.
Link: https://lore.kernel.org/all/CA+0ovCiEz6SP_sn3kN4Tb+_oC=eHMXy_Ffj=usV3wREdQrUtww@mail.gmail.com/
Fixes: ae1c802382 ("cpuset: apply cs->effective_{cpus,mems}")
Closes: https://lore.kernel.org/linux-mm/CA+0ovCgxbZkXa+OU8w3s84R3KNPNxxRfmsNR-udh+afQBbGNmw@mail.gmail.com/
Suggested-by: Gregory Price <gourry@gourry.net>
Suggested-by: Waiman Long <longman@redhat.com>
Acked-by: Waiman Long <longman@redhat.com>
Signed-off-by: Farhad Alemi <farhad.alemi@berkeley.edu>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Alistair Popple <apopple@nvidia.com>
Cc: Byungchul Park <byungchul@sk.com>
Cc: Gregory Price <gourry@gourry.net>
Cc: "Huang, Ying" <ying.huang@linux.alibaba.com>
Cc: Joshua Hahn <joshua.hahnjy@gmail.com>
Cc: Matthew Brost <matthew.brost@intel.com>
Cc: Rakie Kim <rakie.kim@sk.com>
Cc: Rasmus Villemoes <linux@rasmusvillemoes.dk>
Cc: Zi Yan <ziy@nvidia.com>
Cc: Tejun Heo <tj@kernel.org>
Cc: Ridong Chen <ridong.chen@linux.dev>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: "Michal Koutný" <mkoutny@suse.com>
Cc: <stable@vger.kernel.org>
[ david: add a comment, slightly rephrase description ]
Signed-off-by: David Hildenbrand (Arm) <david@kernel.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
The automatic CPU-intensive work item detection reads the worker task's
se.sum_exec_runtime without a lock in wq_worker_running(),
wq_worker_tick() and process_one_work(). The scheduler updates that
field under the rq lock (from the tick via update_curr(), or cross-CPU
via task_sched_runtime()), raising:
BUG: KCSAN: data-race in wq_worker_running+0xa8/0xe8
race at unknown origin, with read to 0xffff0009a11d1df8 of 8 bytes by task 238535 on cpu 68:
wq_worker_running
schedule
schedule_preempt_disabled
__mutex_lock
mutex_lock_nested
cgroup_bpf_release
process_one_work
worker_thread
kthread
ret_from_fork
value changed: 0x0000000088482ba0 -> 0x00000000884893c0
The value only feeds a heuristic, so the race is benign-ish. Unlike
commit ecf5aad9a4 ("workqueue: annotate racy PWQ_STAT_CPU_TIME update
in wq_worker_tick()") that only needs data_race(), these are plain reads
whose result drives a subtraction and comparison, so use READ_ONCE() for
a single, non-torn load, which also silences KCSAN.
Signed-off-by: Breno Leitao <leitao@debian.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
In tracing_set_tracer(), the assignment 'ret = 0' following the
__tracing_resize_ring_buffer() error check is a dead store. After
this point, all subsequent code paths either return with a constant
value (-EINVAL, 0, -EBUSY) or reassign ret before reading it
(tracing_arm_snapshot_locked, tracer_init).
Remove the unnecessary assignment.
No functional change.
Link: https://patch.msgid.link/6a2a37c4.f0a9eb5a.2fc603.7724@mx.google.com
Signed-off-by: Wayen.Yan <win847@gmail.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
CONFIG_PM_WAKELOCKS_LIMIT is documented as the maximum number of
user-space wakeup sources, but the limit check is performed before
the counter is incremented and only rejects new wakeup sources when the
current number is greater than the limit. This allows one extra wakeup
source to be created.
Reject new wakeup sources once the counter has reached the limit.
Fixes: b86ff9820f ("PM / Sleep: Add user space interface for manipulating wakeup sources, v3")
Signed-off-by: Haowen Tu <tuhaowen@uniontech.com>
[ rjw: Subject edits ]
Link: https://patch.msgid.link/20260624053839.2150567-1-tuhaowen@uniontech.com
Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
This ensures that any RCU readers traversing the instance list
have finished, before releasing the reference on the tracer that
the instance points to.
Cc: stable@vger.kernel.org
Fixes: a6ed2aee54 ("tracing: Switch to kvfree_rcu() API")
Link: https://patch.msgid.link/20260609045430.1589786-1-crwood@redhat.com
Suggested-by: Steven Rostedt <rostedt@goodmis.org>
Signed-off-by: Crystal Wood <crwood@redhat.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
When RB_FORCE_8BYTE_ALIGNMENT is true, rb_calculate_event_length()
reserves the space of event->array[0] for placing the data length and
rb_update_event() stores the data length in event->array[0]
accordingly. As a result the whole event length will add extra 4 bytes
for sizeof(event.array[0]) unconditionally.
But ring_buffer_event_length() only subtracts the
sizeof(event->array[0]) for events larger than RB_MAX_SMALL_DATA +
sizeof(event->array[0]). As a result, small events on architectures
with RB_FORCE_8BYTE_ALIGNMENT=true report a data length that is 4
bytes larger than expected.
To fix it, add the RB_FORCE_8BYTE_ALIGNMENT as a condition to subtract
the size of that length field whenever RB_FORCE_8BYTE_ALIGNMENT is
true.
This issue is observed in a riscv64 kernel with
CONFIG_HAVE_64BIT_ALIGNED_ACCESS set to y, when we run ftrace selftest
trace_marker_raw.tc, we get the weird log: for cases where the id is
1..100, the number of data field is 8*N, but once id exceeds 100, the
number of data field becomes 8*N+4:
# 1 buf: 58 00 00 00 80 5e d1 63 (number of data field is 8*1)
...
# a buf: 58 ... (number of data field is 8*2)
...
# 64 buf: 58 ... (number of data field is 8*13)
# 65 buf: 58 ... (number of data field is 8*13+4)
After applying this change, the number of data field keeps being 8*N+4
consistently.
Link: https://patch.msgid.link/20260607072431.125633-2-hui.wang@canonical.com
Fixes: 2271048d1b ("ring-buffer: Do 8 byte alignment for 64 bit that can not handle 4 byte align")
Signed-off-by: Hui Wang <hui.wang@canonical.com>
Reviewed-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
Some __create_synth_event() error paths run after parse_synth_field()
succeeds but before the field is stored in fields[]. The common cleanup
then misses the field. Free it before freeing argv.
Link: https://patch.msgid.link/20260603062533.1096320-1-pengyu@kylinos.cn
Signed-off-by: Yu Peng <pengyu@kylinos.cn>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
WATCHDOG_INTERVAL_NS has been unused since it was introduced by commit
763aacf86f ("clocksource: Rewrite watchdog code completely"); the
watchdog timer rearming uses WATCHDOG_INTERVAL (in jiffies) directly and
the nanosecond variant has never had a user.
Remove the dead macro. No functional change.
Signed-off-by: Zhan Xusheng <zhanxusheng@xiaomi.com>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Link: https://patch.msgid.link/20260706084557.3845091-3-zhanxusheng@xiaomi.com
__hrtimer_reprogram() takes a @next_timer argument but never references
it; it only stores @expires_next into cpu_base->expires_next and
reprograms the clock event device. The argument has been unused since
commit b14bca97c9 ("hrtimer: Consolidate reprogramming code").
Drop it and update the two callers. No functional change.
Signed-off-by: Zhan Xusheng <zhanxusheng@xiaomi.com>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Link: https://patch.msgid.link/20260706084557.3845091-2-zhanxusheng@xiaomi.com
The comments related to raw monotonic timestamps for the various
snapshot mechanisms in code and struct documentation are ambiguous. They
reference them as CLOCK_MONOTONIC_RAW timestamps, but with the arrival
of AUX clocks that's not longer correct.
The raw monotonic timestamps only represent CLOCK_MONOTONIC_RAW for the
system time clock IDs, i.e. REALTIME, MONOTONIC, BOOTTIME, TAI.
For AUX clocks they refer to the monotonic raw clock which is related to
the individual AUX clocks. These monotonic raw timestamps have the same
conversion factor as CLOCK_MONOTONIC_RAW, but differ from that by an
offset:
MONORAW(AUX$N) = MONORAW(SYSTEM) + OFFSET(AUX$N)
The offset is established when a AUX clock is enabled and stays constant
for the lifetime of the AUX clock.
Update the comments so they reflect reality.
Reported-by: Thomas Weißschuh <thomas.weissschuh@linutronix.de>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Reviewed-by: Thomas Weißschuh <thomas.weissschuh@linutronix.de>
Link: https://patch.msgid.link/87wlv9k3wz.ffs@fw13
- Fix a perf_event_attr::remove_on_exec bug for group
events (Taeyang Lee)
- Fix uprobes CALL emulation interaction with shadow stacks,
and add a testcase for this (David Windsor)
- Fix uprobes unregister bug (Jiri Olsa)
Signed-off-by: Ingo Molnar <mingo@kernel.org>
-----BEGIN PGP SIGNATURE-----
iQJFBAABCgAvFiEEBpT5eoXrXCwVQwEKEnMQ0APhK1gFAmpKCX8RHG1pbmdvQGtl
cm5lbC5vcmcACgkQEnMQ0APhK1hwJRAAne1HniwuV9PV2GQHNKipeKV320tod2iR
WY9yy9ez5WJbPttS7Fy28NobQKxuTanNWnAJqqM3TJF0tPPSrrYqkhAGrpEb2ab0
G7gl4KlrO//5kKTZnnI86t/quSV5BDt00UUdMBFp+hbNYXNul5/AeUxqMhNoGqB6
DpJHrV+7kNQZ4I7tjatYWL11hZHpEhrx4QWLUsnd+nDwddpmdsRNXVZwpGh9+Dh4
+XXcgLD7M8FqNFN3GCVfhJKO8x8HRaBWv3FHeGqbCUL9k5viWcn+N91IbrcPVat1
HBk0JDtpLK286WHLFy7uROafCA59AlYp5DX7mobXi0VF1FdMqPtjaXEsZM7Ng/P5
/tpbV5P4irrUnMSCEDSDzqZJWXbcBSqCJ9p6z5/Tjzo3VegHyrXe29wjVoqvXjLx
og/9OnPZv/2QEsE37rBRwNC889ihFMUDZh3T+uUc1YKUEwYWFXwtUECTg+0Oi+lL
mLTdK05j/6NmVkcY77mFjQfTMFAZeD78g6cPY3yDHiHRFtxPOIhknCKBeLD5BjXD
tz08x3MN0ItVEBXfuKAlkb1KPhHy8x02IrpdfVojoQ7lUBRH/JqLvnSGgvVsRM+0
v9D3N9BAuB3s8DmxEdh0wyRzISl6jzBHJDJ83+uWSfZWLlEW+RVn9NbH2ubNU/jt
FO3U81r1MvE=
=Mb4z
-----END PGP SIGNATURE-----
Merge tag 'perf-urgent-2026-07-05' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
Pull perf events fixes from Ingo Molnar:
- Fix a perf_event_attr::remove_on_exec bug for group events
(Taeyang Lee)
- Fix uprobes CALL emulation interaction with shadow stacks, and
add a testcase for this (David Windsor)
- Fix uprobes unregister bug (Jiri Olsa)
* tag 'perf-urgent-2026-07-05' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip:
uprobes/x86: Use proper mm_struct in __in_uprobe_trampoline
selftests/x86: Add shadow stack uprobe CALL test
x86/uprobes: Keep shadow stack in sync for emulated CALLs
perf/core: Detach event groups during remove_on_exec
The NULL check before kvfree() is unnecessary and triggers a Coccinelle
warning, since kvfree() handles NULL arguments internally.
Reported by ifnullfree.cocci Coccinelle semantic patch script.
Signed-off-by: Vasileios Almpanis <vasilisalmpanis@gmail.com>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Reviewed-by: André Almeida <andrealmeid@igalia.com>
Link: https://patch.msgid.link/20260630133530.2577832-1-vasilisalmpanis@gmail.com
After commit f90fff1e15 ("posix-cpu-timers: fix race between
handle_posix_cpu_timers() and posix_cpu_timer_del()"), tsk->sighand is
stable in handle_posix_cpu_timers(), so it can use the plain
spin_lock_irqsave(&tsk->sighand->siglock).
Signed-off-by: Oleg Nesterov <oleg@redhat.com>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Reviewed-by: Bradley Morgan <include@grrlz.net>
Link: https://patch.msgid.link/akjp8AGpY8eJG5I1@redhat.com
Wongi and Jungwoo decoded and reported a non-leader exec() related race
which can result in an UAF:
sys_timer_delete() exec()
posix_cpu_timer_del()
// Observes old leader
p = pid_task(pid, pid_type); de_thread()
switch_leader();
release_task(old_leader)
__exit_signal(old_leader)
sighand = lock(old_leader, sighand);
posix_cpu_timers*_exit();
sighand = lock_task_sighand(p) unhash_task(old_leader);
sh = lock(p, sighand) old_leader->sighand = NULL;
unlock(sighand);
(p->sighand == NULL)
unlock(sh)
return NULL;
// Returns without action
if(!sighand)
return 0;
free_posix_timer();
This is "harmless" unless the deleted timer was armed and enqueued in
p->signal because on exec() a TGID targeted timer is inherited.
As sys_timer_delete() freed the underlying posix timer object
run_posix_cpu_timers() or any timerqueue related add/delete operations on
other timers will access the freed object's timerqueue node, which results
in an UAF.
There is a similar problem vs. posix_cpu_timer_set(). For regular posix
timers it just transiently returns -ESRCH to user space, but for the use
case in do_cpu_nanosleep() it's the same UAF just that the k_itimer is
allocated on the stack.
Also posix_cpu_timer_rearm() fails to rearm the timer, which means it stops
to expire.
While debating solutions Frederic pointed out another problem:
posix_cpu_timer_del(tmr)
__exit_signal(p)
posix_cpu_timers*_exit(p);
unhash_task(p);
p->sighand = NULL;
sh = lock_task_sighand(p)
sighand = p->sighand;
if (!sighand)
return NULL;
lock(sighand);
if (!sh)
WARN_ON_ONCE(timer_queued(tmr));
On weakly ordered architectures it is not guaranteed that
posix_cpu_timer_del() will observe the stores in posix_cpu_timers*_exit()
when p->sighand is observed as NULL, which means the WARN() can be a false
positive.
Solve these issues by:
1) Changing the store in __exit_signal() to smp_store_release().
2) Adding a smp_acquire__after_ctrl_dep() into the !sighand path
of lock_task_sighand().
3) Creating a helper function for looking up the task and locking sighand
which does not return when sighand == NULL. Instead it retries the
task lookup and only if that fails it gives up.
4) Using that helper in the three affected functions.
#1/#2 ensures that the reader side which observes sighand == NULL also
observes all preceeding stores, i.e. the stores in posix_cpu_timers*_exit()
and the ones in unhash_task().
#3 ensures that the above described non-leader exec() situation is handled
gracefully. When the task lookup returns the old leader, but sighand ==
NULL then it retries. In the non-leader exec() case the subsequent task
lookup will observe the new leader due to #1/#2. In normal exit() scenarios
the subsequent lookup fails.
When the task lookup fails, the function also checks whether the timer is
still enqueued and issues a warning if that's the case. Unfortunately there
is nothing which can be done about it, but as the task is already not
longer visible the timer should not be accessed anymore. This check also
requires memory ordering, which is not provided when the first lookup
fails. To achieve that the check is preceeded by a smp_rmb() which pairs
with the smp_wmb() in write_seqlock() in __exit_signal(). That ensures that
the stores in posix_cpu_timers*_exit() are visible.
The history of the non-leader exec() issue goes back to the early days of
posix CPU timers, which stored a pointer to the group leader task in the
timer. That obviously fails when a non-leader exec() switches the leader.
commit e0a7021710 ("posix-cpu-timers: workaround to suppress the problems
with mt exec") added a temporary workaround for that in 2010 which survived
about 10 years. The fix for the workaround changed the task pointer to a
pid pointer, but failed to see the subtle race described above. So the
Fixes tag picks that commit, which seems to be halfways accurate.
Thanks to Frederic Weissbecker, Oleg Nesterov and Peter Zijlstra for
review, feedback and suggestions and to Wongi and Jungwoo for the excellent
bug report and analysis!
Fixes: 55e8c8eb2c ("posix-cpu-timers: Store a reference to a pid not a task")
Reported-by: Wongi Lee <qw3rtyp0@gmail.com>
Reported-by: Jungwoo Lee <jwlee2217@gmail.com>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Reviewed-by: Oleg Nesterov <oleg@redhat.com>
Cc: stable@vger.kernel.org
Removing struct uprobe_trampoline object and it's tracking code,
because it's not needed. We can do same thing directly on top of
struct vm_area_struct objects.
This makes the code simpler and allows easy propagation of the
trampoline vma object into child process in following change.
Note the original code called destroy_uprobe_trampoline if the
optimiation failed, but it only freed the struct uprobe_trampoline
object, not the vma. The new vma leak is fixed in following change.
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Oleg Nesterov <oleg@redhat.com>
Acked-by: Andrii Nakryiko <andrii@kernel.org>
Link: https://patch.msgid.link/20260703114917.238144-3-jolsa@kernel.org
- Initialize task local storage before fork bails out to
free the task (Jann Horn)
- Fix insn_aux_data leak on verifier error path (KaFai Wan)
- Reject BPF inode storage map creation when BPF LSM is
uninitialized (Matt Bobrowski)
- Mask pseudo pointer values in verifier logs when pointer
leaks are not allowed (Nuoqi Gui)
- Harden BPF JIT against spraying via IBPB flush (Pawan Gupta)
- Reject a skb-modifying SK_SKB stream parser since the latter
is only meant to measure the next message (Sechang Lim)
- Fix bpf_refcount_acquire to reject refcounted allocation
arguments with a non-zero fixed offset (Yiyang Chen)
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
-----BEGIN PGP SIGNATURE-----
iIsEABYKADMWIQTFp0I1jqZrAX+hPRXbK58LschIgwUCaka0NhUcZGFuaWVsQGlv
Z2VhcmJveC5uZXQACgkQ2yufC7HISIMPywD/ePh+eOZjFwXy88kOdNa2qmKLuTQm
nv6jPDOEafVDfbUA/R0qT9t+144K5Ig1NU58uR9vMIevqWRjLrnQtWU7KI0D
=4aKP
-----END PGP SIGNATURE-----
Merge tag 'bpf-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf
Pull BPF fixes from Daniel Borkmann:
- Initialize task local storage before fork bails out to free the task
(Jann Horn)
- Fix insn_aux_data leak on verifier error path (KaFai Wan)
- Reject BPF inode storage map creation when BPF LSM is uninitialized
(Matt Bobrowski)
- Mask pseudo pointer values in verifier logs when pointer leaks are
not allowed (Nuoqi Gui)
- Harden BPF JIT against spraying via IBPB flush (Pawan Gupta)
- Reject a skb-modifying SK_SKB stream parser since the latter is only
meant to measure the next message (Sechang Lim)
- Fix bpf_refcount_acquire to reject refcounted allocation arguments
with a non-zero fixed offset (Yiyang Chen)
* tag 'bpf-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf:
bpf: Prefer dirty packs for eBPF allocations
bpf: Prefer packs that won't trigger an IBPB flush on allocation
bpf: Skip redundant IBPB in pack allocator
bpf: Restrict JIT predictor flush to cBPF
x86/bugs: Enable IBPB flush on BPF JIT allocation
bpf: Support for hardening against JIT spraying
bpf: Reject BPF_MAP_TYPE_INODE_STORAGE creation if BPF LSM is uninitialized
bpf,fork: wipe ->bpf_storage before bailouts that access it
bpf: Fix insn_aux_data leak on verifier err_free_env path
selftests/bpf: Cover pseudo-BTF ksym log masking
bpf: Mask pseudo pointer values in verifier logs
selftests/bpf: Cover refcount acquire node offsets
bpf: Reject offset refcount acquire arguments
selftests/bpf: test rejection of a packet-modifying SK_SKB stream parser
bpf, sockmap: reject a packet-modifying SK_SKB stream parser
selftests/bpf: don't modify the skb in the strparser parser prog
The commit cited below should not have been merged. It attemted to fix an
existing problem ansd thereby introduced new problems by keeping the
pi_state in state Q_REQUEUE_PI_IN_PROGRESS and leaking it.
Based on the commit description the intention was to handle the case
when task_blocks_on_rt_mutex() returns -EDEADLK and the following
remove_waiter() dereferences the NULL pointer in waiter->task.
That is already handled by Davidlohr in commit 40a25d59e8
("locking/rtmutex: Skip remove_waiter() when waiter is not enqueued") and
requires no further acting.
Revert the commit breaking the "waiter == owner" case again.
Fixes: 74e144274a ("futex/requeue: Prevent NULL pointer dereference in remove_waiter() on self-deadlock")
Reported-by: Michael Bommarito <michael.bommarito@gmail.com>
Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Cc: stable@vger.kernel.org
Link: https://patch.msgid.link/20260701131150.0Ijhq4Dw@linutronix.de
Closes: https://lore.kernel.org/all/20260629020049.2082397-1-michael.bommarito@gmail.com
wq_worker_tick() bumps pwq->stats[PWQ_STAT_CPU_TIME] on every scheduler
tick before pool->lock is taken. For unbound workqueues the
pool_workqueue is shared by all workers of the pool across CPUs, so
concurrent ticks on different CPUs perform an unsynchronized 64-bit
read-modify-write on the same counter. KCSAN reports this as a
data-race:
BUG: KCSAN: data-race in wq_worker_tick / wq_worker_tick
read-write to 0xffff0004d6989500 of 8 bytes by interrupt on cpu 29:
wq_worker_tick+0x70/0x418
sched_tick+0x248/0x3a0
update_process_times+0x200/0x260
tick_nohz_handler+0x230/0x2f8
__hrtimer_run_queues+0x1ec/0x6c8
hrtimer_interrupt+0x174/0x4b8
...
read-write to 0xffff0004d6989500 of 8 bytes by interrupt on cpu 24:
wq_worker_tick+0x70/0x418
sched_tick+0x248/0x3a0
...
value changed: 0x000000000010a1d0 -> 0x000000000010a9a0
The counter is purely advisory, so an occasional lost update is
harmless, and every other stats[] update already runs under pool->lock.
Annotate the update with data_race().
Signed-off-by: Breno Leitao <leitao@debian.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
For debugging probe events, it is helpful to verify the compiled
fetch instructions for each probe argument. This introduces a new
kernel config CONFIG_PROBE_EVENTS_DUMP_FETCHARG to decode the
instruction sequence of each argument and display it under a
commented line starting with '#' immediately following the dynamic
event definition (such as in dynamic_events, kprobe_events,
uprobe_events, etc.).
For example:
/sys/kernel/tracing # cat dynamic_events
p:kprobes/p_vfs_read_0 vfs_read arg1=+0(file):ustring arg2=%ax:x16
# arg1: ARG(0) -> ST_USTRING(offset=0,size=4) -> END
# arg2: REG(80) -> ST_RAW(size=2) -> END
Link: https://lore.kernel.org/all/178271359508.1176915.11895926894608440183.stgit@devnote2/
Assisted-by: Antigravity:gemini-3.5-flash
Signed-off-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
We can pass NULL as id pointer to btf_type_skip_modifiers if we do not
need the id of the returned btf_type.
Link: https://lore.kernel.org/all/20260623132937.3494895-1-martin@kaiser.cx/
Signed-off-by: Martin Kaiser <martin@kaiser.cx>
Signed-off-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
The shared probe argument parser rejects file offsets for kernel probes.
This path is used outside the kprobe event parser too, but the diagnostic
currently says "with kprobe" even when emitted from another probe path.
Make the diagnostic probe-agnostic.
Link: https://lore.kernel.org/all/20260622160032.99834-1-pyudistira519@gmail.com/
Signed-off-by: Yudistira Putra <pyudistira519@gmail.com>
Signed-off-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
perf_event_remove_on_exec() removes events by calling
perf_event_exit_event(). For top-level events, this removes the event from
the context with DETACH_EXIT only.
This can leave inconsistent group state when a removed event is a group
leader and the group contains siblings without remove_on_exec. If the group
was active, the surviving siblings can remain active and attached to the
removed leader's sibling list, but are no longer represented by a valid
group leader on the PMU context active lists.
A later close of the removed leader uses DETACH_GROUP and can promote the
still-active siblings from this stale group state. The next schedule-in can
then add an already-linked active_list entry again, corrupting the PMU
context active list.
With DEBUG_LIST enabled, this is caught as a list_add double-add in
merge_sched_in().
Fix this by detaching group relationships when remove_on_exec removes an
event. This preserves the existing task-exit and revoke behavior, while
ensuring surviving siblings are ungrouped before the removed event leaves
the context.
Fixes: 2e498d0a74 ("perf: Add support for event removal on exec")
Signed-off-by: Taeyang Lee <0wn@theori.io>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/ai65GgZcC0LAlWLG@Taeyangs-MacBook-Pro.local
Load balancing can migrate delayed dequeue tasks to even the load between
CPUs.
sched_balance_rq()
-> detach_task()
-> deactivate_task(DEQUEUE_NOCLOCK)
-> set_task_cpu(dst_cpu)
-> attach_task()
-> activate_task(ENQUEUE_NOCLOCK)
-> wakeup_preempt()
A delayed task with shorter slice can be dequeued during pick_next_entity()
but then jump to preempt because eligible.
Always jump to update for delayed dequeue task even with shorter slice.
Fixes: ba0d3bf5f9 ("sched/eevdf: Cancel slice protection if short slice task is eligible")
Reported-by: K Prateek Nayak <kprateek.nayak@amd.com>
Signed-off-by: Vincent Guittot <vincent.guittot@linaro.org>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: K Prateek Nayak <kprateek.nayak@amd.com>
Tested-by: K Prateek Nayak <kprateek.nayak@amd.com>
Link: https://patch.msgid.link/20260701164920.1571352-1-vincent.guittot@linaro.org
The sub-scheduler implementation has grown and will continue to expand. Move
the sub-scheduler functions from ext.c into a new kernel/sched/ext/sub.c.
sub.h holds the prototypes and the !CONFIG_EXT_SUB_SCHED no-op stubs.
scx_dispatch_sched() is shared: balance_one() in ext.c and the
scx_bpf_sub_dispatch() kfunc in sub.c both call it, and the latter re-enters
it as sub-scheduler dispatch nests. It moves into sub.h as a static
__always_inline so both callers keep it inlined and per-level stack stays
bounded across the recursion. The event macros it uses move to internal.h.
No functional change.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
The following trivial helpers in ext.c are called from both ext.c and the
sub-scheduler code. Define them as static inline in internal.h.
- scx_bypass_dsq()
- scx_bypass_dsp_enabled()
- scx_ops_sanitize_err()
- scx_schedule_reenq_local()
No functional change.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
The sub-scheduler implementation is about to move into its own sub.c, from
where it calls a set of ext.c helpers and shares a few ext.c globals. Make
those reachable across the new file boundary ahead of the move.
No functional change.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
A later change moves the sub-scheduler implementation out of ext.c into its
own file, from where it calls a number of file-local ext.c helpers. Give
those helpers the scx_ prefix that cross-file sched_ext symbols carry, ahead
of the move so the mechanical rename stays out of the code-motion patch. No
functional change.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
__bpf_prog_put_rcu() is the call_rcu() callback for non-sleepable programs.
security_bpf_prog_free() called from there fires bpf_prog_free in softirq;
if a sleepable LSM prog is attached to that hook, might_fault() BUGs:
BUG: sleeping function called from invalid context
in_atomic(): 1, irqs_disabled(): 0, non_block: 0, pid: 5038
preempt_count: 101, expected: 0
Call Trace:
<IRQ>
__bpf_prog_enter_sleepable+0x1cd/0x320 kernel/bpf/trampoline.c:1255
bpf_trampoline_6442549705+0x53/0xd7
security_bpf_prog_free+0xde/0x130 security/security.c:5465
__bpf_prog_put_rcu+0xab/0xd0 kernel/bpf/syscall.c:2365
rcu_do_batch kernel/rcu/tree.c:2617 [inline]
handle_softirqs+0x236/0x800 kernel/softirq.c:622
</IRQ>
The call_rcu/call_rcu_tasks_trace split reflects the freed program's
sleepability, not that of any attached observer.
security_bpf_prog_free() also frees prog->aux->security, which has to stay
after the grace period, so drop bpf_prog_free from sleepable_lsm_hooks
rather than move the call. Non-sleepable observers still run there.
Fixes: 1b67772e4e ("bpf,lsm: Refactor bpf_prog_alloc/bpf_prog_free LSM hooks")
Signed-off-by: Sechang Lim <rhkrqnwk98@gmail.com>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Link: https://lore.kernel.org/bpf/20260701080757.1394144-1-rhkrqnwk98@gmail.com
In bpf_do_misc_fixups(), the conversion from a function address to a
BPF immediate value is handled using the BPF_CALL_IMM macro inside the
'patch_map_ops_generic' label block. However, immediately following it
in the 'patch_call_imm' label block, the immediate value is calculated
manually by subtracting __bpf_call_base from fn->func.
Inspired by KaFai Wan's review comments on fixing helper call offsets,
use the BPF_CALL_IMM macro in 'patch_call_imm' as well to clean this up.
This removes the redundant manual pointer arithmetic and ensures coding
style consistency across adjacent label blocks within the same function.
Signed-off-by: Tiezhu Yang <yangtiezhu@loongson.cn>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Link: https://lore.kernel.org/bpf/20260701035107.8069-1-yangtiezhu@loongson.cn
Adding BPF_OBJ_GET_INFO_BY_FD support for tracing_multi links.
We expose following tracing_multi link data:
- attach_type of the program
- number of ids
- array of BTF ids
- array of its related kernel addresses
- array of cookies
The change follows the kprobe_multi and uprobe_multi link-info convention
of optional output arrays with an in/out count,
On top of standard tracing link data we also expose addresses, because they
are useful info for user (especially when the attachment was done via pattern).
This data is hidden when kallsyms does not allow exposing kernel pointer values.
Assisted-by: Codex:GPT-5
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Acked-by: Leon Hwang <leon.hwang@linux.dev>
Acked-by: Andrii Nakryiko <andrii@kernel.org>
Link: https://lore.kernel.org/bpf/20260629212208.895962-2-jolsa@kernel.org
To identify the task most likely responsible for a stall, add
last_woken_worker (L: pool->lock) to worker_pool and record it in
kick_pool() just before wake_up_process(). This captures the idle
worker that was kicked to take over when the last running worker went to
sleep; if the pool is now stuck with no running worker, that task is the
prime suspect and its backtrace is dumped by show_pool_no_running_worker().
Using struct worker * rather than struct task_struct * avoids any
lifetime concern: workers are only destroyed via set_worker_dying()
which requires pool->lock, and set_worker_dying() clears
last_woken_worker when the dying worker matches.
show_cpu_pool_busy_workers() holds pool->lock while calling
sched_show_task(), so last_woken_worker is either NULL or points to a
live worker with a valid task. More precisely, set_worker_dying() clears
last_woken_worker before setting WORKER_DIE, so a non-NULL
last_woken_worker means the kthread has not yet exited and worker->task
is still alive.
Suggested-by: Petr Mladek <pmladek@suse.com>
Reviewed-by: Petr Mladek <pmladek@suse.com>
Signed-off-by: Breno Leitao <leitao@debian.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
Checksum verification is needed
1. for crash kernels. In a crash, we can't be sure the kernel is
intact.
2. if we're worried about relocating the kernel into a region used by
some DMA that wasn't properly cancelled.
If KHO is enabled then relocations will happen to KHO scratch, which
is free from DMA regions.
If we used CMA to allocate segments then relocations are not going to
happen at all.
Therefore, we can safely disable checksum verification in both of those
cases.
Instead of adding a new variable to purgatory, just skip adding regions
and save the default value of SHA256 hash.
Saves ~250ms on my 4.0 GHz CPU. This is an important saving for the
live-update project.
Signed-off-by: Michal Clapinski <mclapinski@google.com>
Reviewed-by: Pratyush Yadav (Google) <pratyush@kernel.org>
Reviewed-by: Pasha Tatashin <pasha.tatashin@soleen.com>
Acked-by: Baoquan He <baoquan.he@linux.dev>
Link: https://patch.msgid.link/20260602123311.1841746-1-mclapinski@google.com
Signed-off-by: Pasha Tatashin <pasha.tatashin@soleen.com>
Signed-off-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
The pack allocator only flushes predictors when reusing a dirty pack for
cBPF, eBPF allocations never trigger a flush. Currently, eBPF picks the
first free pack, which could be a clean pack. As an optimization, leaving
a clean pack for cBPF can avoid flushes.
Prefer dirty packs for eBPF and keep clean packs free for cBPF. This
mirrors the existing cBPF preference for clean packs: each program kind
prefers the pack that avoids an extra flush, and falls back to the other
kind only when no preferred pack has room. eBPF reuse of a dirty pack is
harmless since eBPF being privileged does not flush.
Signed-off-by: Pawan Gupta <pawan.kumar.gupta@linux.intel.com>
Acked-by: Daniel Borkmann <daniel@iogearbox.net>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Currently BPF pack allocator picks the chunks from the first available
pack. While this is okay, it naturally leads to more frequent flushes
when there are multiple packs in the system that weren't used since the
last flush.
As an optimization prefer allocating the new programs from packs that
are unused since last flush. When all packs are dirty, allocation forces
a flush and marks all packs clean.
Below are some future optimizations ideas:
1. Currently, the "dirty" tracking is only done at the pack-level.
Flush frequency can further be reduced with chunk-level tracking.
This requires a new bitmap per-pack to track the dirty state.
2. IBPB flush is done on all CPUs, even if only a single CPU ran the
BPF program. On a system with hundreds of CPUs this could be a
major bottleneck forcing hundreds of IPIs to deliver the flush.
The solution is to track the CPUs where a BPF program ran, and
issue IBPB only on those CPUs.
3. Avoid IBPB when flush is already done at other sources (e.g.
context switch).
Signed-off-by: Pawan Gupta <pawan.kumar.gupta@linux.intel.com>
Acked-by: Daniel Borkmann <daniel@iogearbox.net>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
bpf_prog_pack_alloc() issues IBPB on all CPUs on every cBPF allocation,
even when reusing chunks from an existing pack where no new memory was
touched since the last IBPB.
Since IBPB on all CPUs is heavy, Dave Hansen suggested to track allocation
since last IBPB, and only issue IBPB at reuse for the chunks that have not
seen an IBPB since they were last freed.
Track per-pack whether an IBPB is needed via arch_flush_needed. Set it when
allocating a chunk, reset on IBPB flush. On reuse, conditionally issue the
flush. Since IBPB invalidates all BTB entries, clear the flag on all packs
after flushing.
Signed-off-by: Pawan Gupta <pawan.kumar.gupta@linux.intel.com>
Acked-by: Daniel Borkmann <daniel@iogearbox.net>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Currently predictor flush on memory reuse is done for all BPF JIT
allocations, but only cBPF programs can be loaded by an unprivileged user.
eBPF is privileged by default, and flushing predictors for all CPUs on
every eBPF reuse penalizes the common case for no security benefit.
eBPF allocations can be frequent on busy systems, only flush predictors
for cBPF programs. Trampoline and dispatcher allocations also skip the
flush as they are eBPF-only.
Signed-off-by: Pawan Gupta <pawan.kumar.gupta@linux.intel.com>
Acked-by: Daniel Borkmann <daniel@iogearbox.net>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
The BPF JIT allocator packs many small programs into larger executable
allocations and reuses space within those allocations as programs are
loaded and freed. When fresh code is written into space that a previous
program occupied, an indirect jump into the new program can reuse a branch
prediction left behind by the old one.
Flush the indirect branch predictors before reusing JIT memory so that
indirect jumps into a newly written program don't reuse predictions from an
old program that occupied the same space.
Introduce bpf_arch_pred_flush_enabled static key and bpf_arch_pred_flush
static call for flushing the branch predictors on JIT memory reuse.
Architectures that need a flush, can update it to a predictor flush
function. By default, its a NOP and does not emit any CALL.
Allocations larger than a pack are not covered by this flush. That is safe
because cBPF programs (the unprivileged attack surface) are bounded well
below a pack size. Issue a warning if this assumption is ever violated
while the flush is active.
Signed-off-by: Pawan Gupta <pawan.kumar.gupta@linux.intel.com>
Acked-by: Daniel Borkmann <daniel@iogearbox.net>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
- fprobe: Fix stability and spelling typos
. Fix NULL pointer dereference in fprobe_fgraph_entry(): Prevent general
protection faults by checking shadow-stack reservation bounds. Skip
mid-flight registered fprobes that were not counted during sizing.
- eprobe: Fix string pointer extraction
. Correct the casting of string pointers read from the ringbuffer to
prevent truncation of base event pointer variables when dereferencing
FILTER_PTR_STRING fields.
- tracing/probes: Clean up argument parsing and BTF helper logic
. Make the $ prefix mandatory for comm access: Require the $ prefix for
special fetcharg variables like $comm and $COMM, preventing naming
conflicts with regular BTF-based event fields.
. Fix double addition of offset for @+FOFFSET: Clear the temporary offset
variable after setting the FETCH_OP_FOFFS instruction to avoid applying
the offset multiple times.
. Remove WARN_ON_ONCE from parse_btf_arg: Prevent triggering a kernel warning
via user-space input when creating a kprobe event on a raw address.
. Fix typo in a log message: Correct a spelling error ("$-valiable") in
trace probe log messages.
- samples/trace_events: Improve error checking
. Validate the thread pointer returned from kthread_run() in the trace
events sample code to properly handle thread creation failures.
-----BEGIN PGP SIGNATURE-----
iQFPBAABCgA5FiEEh7BulGwFlgAOi5DV2/sHvwUrPxsFAmpD3BEbHG1hc2FtaS5o
aXJhbWF0c3VAZ21haWwuY29tAAoJENv7B78FKz8beGkIAMbMZ7zoCIlKEfZaRIHS
ms2BTUfh5mkyoIJ+dWXel9UmsvL2kj4FzKWt+gYKsYuxqu3o62BGIkaNffXJEKnc
qRvgVxsxOeJqXjmnSDjVhukBDGb8X1SaRd4DZMuEO77d2nYztYO61ejkF94S6WaQ
VE46pFu0FWGZvFGyyfrCEYFIkFF+4scsDB4ncHLR8u5tNqzg7NaUdvK+tc88/Exk
k7+qqo8aCwMjd8wkfsyCH6DvQmmN5sKcCEvOL8zzE6D4xfX9kRd0//FBZPDmGUC4
lQ2jNhMYhdLdOP+I7dIi+FBaY0Lye8mwHrnrqI2DSvYO4cb4LQvn2hexKLbN2dZE
dt8=
=3Y70
-----END PGP SIGNATURE-----
Merge tag 'probes-fixes-v7.2-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace
Pull probes fixes from Masami Hiramatsu:
"fprobe fixes and spelling typos:
- Fix NULL pointer dereference in fprobe_fgraph_entry(). Prevent
general protection faults by checking shadow-stack reservation
bounds. Skip mid-flight registered fprobes that were not counted
during sizing.
eprobe: fix string pointer extraction
- Correct the casting of string pointers read from the ringbuffer to
prevent truncation of base event pointer variables when
dereferencing FILTER_PTR_STRING fields.
tracing/probes: clean up argument parsing and BTF helper logic
- Make the $ prefix mandatory for comm access: Require the $ prefix
for special fetcharg variables like $comm and $COMM, preventing
naming conflicts with regular BTF-based event fields.
- Fix double addition of offset for @+FOFFSET: Clear the temporary
offset variable after setting the FETCH_OP_FOFFS instruction to
avoid applying the offset multiple times.
- Remove WARN_ON_ONCE from parse_btf_arg: Prevent triggering a kernel
warning via user-space input when creating a kprobe event on a raw
address.
- Fix typo in a log message: Correct a spelling error ("$-valiable")
in trace probe log messages.
samples/trace_events: improve error checking
- Validate the thread pointer returned from kthread_run() in the
trace events sample code to properly handle thread creation
failures"
* tag 'probes-fixes-v7.2-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace:
tracing/probes: Make the $ prefix mandatory for comm access
tracing/fprobe: Fix NULL pointer dereference in fprobe_fgraph_entry()
tracing/probes: Fix double addition of offset for @+FOFFSET
tracing: eprobe: read the complete FILTER_PTR_STRING pointer
tracing/events: Fix to check the simple_tsk_fn creation
tracing/probes: Remove WARN_ON_ONCE from parse_btf_arg
tracing: probes: fix typo in a log message
Export irq_can_set_affinity() for loadable drivers that need a runtime
check for IRQ affinity capability.
In hierarchical IRQ setups where the effective irqchip path lacks
.irq_set_affinity(), drivers may need to switch to a fallback policy.
Without this export, module drivers cannot use the core helper and have
to open-code equivalent checks.
Signed-off-by: Hangtian Zhu <hangtian.zhu@oss.qualcomm.com>
Acked-by: Thomas Gleixner <tglx@kernel.org>
Link: https://patch.msgid.link/20260519011627.713068-2-hangtian.zhu@oss.qualcomm.com
Signed-off-by: Jeff Johnson <jeff.johnson@oss.qualcomm.com>
Multiple readers access audit_queue.qlen via skb_queue_len() without
holding the queue lock or using READ_ONCE(), while kauditd writes to
this field via the skb_dequeue() → __skb_unlink() path with WRITE_ONCE()
protected by a spinlock. This constitutes data races.
All affected skb_queue_len(&audit_queue) call sites:
- kauditd_thread() wait_event_freezable() condition
- audit_receive_msg() AUDIT_GET handler (s.backlog assignment)
- audit_receive() backlog check
- audit_log_start() backlog check and pr_warn()
KCSAN reports the following conflicting access pattern (one example):
==================================================================
BUG: KCSAN: data-race in audit_log_start / skb_dequeue
write (marked) to 0xffffffff8512ee20 of 4 bytes by task 661 on cpu 57:
skb_dequeue+0x70/0xf0
kauditd_send_queue+0x71/0x220
kauditd_thread+0x1cb/0x430
kthread+0x1c2/0x210
ret_from_fork+0x162/0x1a0
ret_from_fork_asm+0x1a/0x30
read to 0xffffffff8512ee20 of 4 bytes by task 36586 on cpu 1:
audit_log_start+0x2a0/0x6b0
audit_core_dumps+0x64/0xa0
do_coredump+0x14b/0x1260
get_signal+0xeb2/0xf70
arch_do_signal_or_restart+0x41/0x170
exit_to_user_mode_loop+0xa2/0x1c0
do_syscall_64+0x1a3/0x1c0
entry_SYSCALL_64_after_hwframe+0x76/0xe0
value changed: 0x00000001 -> 0x00000000
==================================================================
Resolve the race by switching to lockless helper skb_queue_len_lockless(),
which internally uses READ_ONCE() and properly pairs with the WRITE_ONCE()
write accesses already present on the writer side.
Cc: stable@vger.kernel.org
Fixes: 3197542482 ("audit: rework audit_log_start()")
Signed-off-by: Chi Wang <wangchi@kylinos.cn>
Reviewed-by: Ricardo Robaina <rrobaina@redhat.com>
[PM: line length tweak]
Signed-off-by: Paul Moore <paul@paul-moore.com>
Fix checkpatch "Macros with complex values should be enclosed in
parentheses" error.
Suggested-by: Andrea Righi <arighi@nvidia.com>
Signed-off-by: Cheng-Yang Chou <yphbchou0911@gmail.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
clock_base_next_timer_safe() was added by commit a64ad57e41 ("hrtimer:
Simplify run_hrtimer_queues()") but has never had a caller; the queue
iteration in __hrtimer_run_queues() uses clock_base_next_timer() instead.
The two are functionally equivalent: struct hrtimer embeds the timerqueue
node at offset 0, so container_of() of a NULL node yields NULL. Thus
clock_base_next_timer() already returns NULL on an empty queue and the
explicit NULL check in the _safe variant is redundant.
Being a static __always_inline function it does not trigger
-Wunused-function, so the dead code has gone unnoticed. Remove it.
No functional change.
Signed-off-by: Zhan Xusheng <zhanxusheng@xiaomi.com>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Link: https://patch.msgid.link/20260625140901.929554-1-zhanxusheng@xiaomi.com
When a CPU pool is stalled with no running worker, the task occupying the
CPU may not be a workqueue worker at all. Trigger a single-CPU backtrace
for the stalled CPU to capture what it is currently executing.
The CPU is snapshotted under pool->lock and the backtrace is triggered
after releasing the lock to avoid any potential issues with NMI delivery.
Skip the backtrace when the CPU is offline. A pool disassociated by CPU
hotplug keeps its pool->cpu, and an NMI to an offline CPU is never acked,
so nmi_trigger_cpumask_backtrace() would busy-wait for its full timeout
in the watchdog's timer context.
Suggested-by: Petr Mladek <pmladek@suse.com>
Reviewed-by: Petr Mladek <pmladek@suse.com>
Signed-off-by: Breno Leitao <leitao@debian.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
show_cpu_pool_busy_workers() dumps every in-flight worker in the pool's
busy_hash, including workers that are not currently running on the CPU.
Restore the task_is_running() filter so only running workers are dumped.
When no running worker is found the pool may be stuck, unable to wake an
idle worker to process pending work, and the watchdog would otherwise
give no feedback. Add show_pool_no_running_worker() to report the pool
id, CPU, idle state, and worker counts in that case.
The pool info message is printed inside pool->lock using
printk_deferred_enter/exit, the same pattern used by the existing
busy-worker loop, to avoid deadlocks with console drivers that queue
work while holding locks also taken in their write paths.
This has been running on the Meta fleet for a while and caught some real
issues, for instance EFI stalls stalling the workqueue [1].
Link: https://lore.kernel.org/all/20260616-efi_timeout-v3-0-76dd1d26657b@debian.org/ [1]
Suggested-by: Petr Mladek <pmladek@suse.com>
Fixes: 8823eaef45 ("workqueue: Show all busy workers in stall diagnostics")
Reviewed-by: Petr Mladek <pmladek@suse.com>
Signed-off-by: Breno Leitao <leitao@debian.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
The crash memory exclude of crashk_res and crashk_cma memory on powerpc
are almost identical to the generic crash_exclude_core_ranges().
By introducing the architecture-specific arch_crash_exclude_mem_range()
function with a default implementation of crash_exclude_mem_range(),
and using crash_exclude_mem_range_guarded as powerpc's separate
implementation, the generic crash_exclude_core_ranges() helper function
can be reused.
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Hari Bathini <hbathini@linux.ibm.com>
Cc: Madhavan Srinivasan <maddy@linux.ibm.com>
Cc: Mahesh Salgaonkar <mahesh@linux.ibm.com>
Cc: Michael Ellerman <mpe@ellerman.id.au>
Cc: Ritesh Harjani (IBM) <ritesh.list@gmail.com>
Cc: Shivang Upadhyay <shivangu@linux.ibm.com>
Acked-by: Breno leitao <leitao@debian.org>
Acked-by: Baoquan He <bhe@redhat.com>
Reviewed-by: Sourabh Jain <sourabhjain@linux.ibm.com>
Acked-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Signed-off-by: Jinjie Ruan <ruanjinjie@huawei.com>
Link: https://patch.msgid.link/20260629094746.191843-9-ruanjinjie@huawei.com
Signed-off-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
The crash memory alloc, and the exclude of crashk_res, crashk_low_res
and crashk_cma memory are almost identical across different architectures,
handling them in the crash core would eliminate a lot of duplication, so
add crash_prepare_headers() helper to handle them in the common code.
To achieve the above goal, three architecture-specific functions are
introduced:
- arch_get_system_nr_ranges(). Pre-counts the max number of memory ranges.
- arch_crash_populate_cmem(). Collects the memory ranges and fills them
into cmem.
- arch_crash_exclude_ranges(). Architecture's additional crash memory
ranges exclusion, defaulting to empty.
Reviewed-by: Sourabh Jain <sourabhjain@linux.ibm.com>
Acked-by: Baoquan He <bhe@redhat.com>
Acked-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Signed-off-by: Jinjie Ruan <ruanjinjie@huawei.com>
Link: https://patch.msgid.link/20260629094746.191843-4-ruanjinjie@huawei.com
Signed-off-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Since $comm or $COMM are not event field but special fetcharg
variables to access current->comm, It should not be accessed
without '$' prefix even with typecast.
Link: https://lore.kernel.org/all/178231209724.732967.12049805699091810641.stgit@devnote2/
Fixes: 69efd863a7 ("tracing/eprobes: Allow use of BTF names to dereference pointers")
Signed-off-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
fprobe_fgraph_entry() sizes a shadow-stack reservation in one walk of
the per-ip fprobe list and fills it in a second walk, both under
rcu_read_lock() only. A fprobe registered on an already-live ip can
become visible between the two walks, so the fill walk processes an
exit_handler the sizing walk did not count and used runs past
reserved_words. If the sizing walk counted nothing, fgraph_data is NULL
and the first write_fprobe_header() faults:
Oops: general protection fault, probably for non-canonical address ...
KASAN: null-ptr-deref in range [0x0000000000000000-0x0000000000000007]
RIP: 0010:fprobe_fgraph_entry+0xa38/0xf10 kernel/trace/fprobe.c:167
Call Trace:
<TASK>
function_graph_enter_regs+0x44c/0xa10 kernel/trace/fgraph.c:677
ftrace_graph_func+0xc5/0x140 arch/x86/kernel/ftrace.c:671
__kernel_text_address+0x9/0x40 kernel/extable.c:78
arch_stack_walk+0x117/0x170 arch/x86/kernel/stacktrace.c:26
kmem_cache_free+0x188/0x580 mm/slub.c:6378
tcp_data_queue+0x18d/0x6550 net/ipv4/tcp_input.c:5590
[...]
</TASK>
The list cannot be frozen across the two walks, so skip a node that does
not fit the reservation and count it as missed.
Link: https://lore.kernel.org/all/20260619184425.3824774-1-rhkrqnwk98@gmail.com/
Fixes: 4346ba1604 ("fprobe: Rewrite fprobe on function-graph tracer")
Signed-off-by: Sechang Lim <rhkrqnwk98@gmail.com>
Signed-off-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
Since commit 533059281e ("tracing: probeevent: Introduce new argument
fetching code") wrongly use @offset local variable during the parsing,
the offset value is added twice when dereferencing.
Reset the @offset after setting it in FETCH_OP_FOFFS.
Link: https://lore.kernel.org/all/178217905962.643090.1978577464942171332.stgit@devnote2/
Fixes: 533059281e ("tracing: probeevent: Introduce new argument fetching code")
Signed-off-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
Cc: stable@vger.kernel.org
For a char * element in an event, the FILTER_PTR_STRING filter type is
used. When the event occurs, a pointer is stored in the ringbuffer.
If an eprobe references such a char * element of a "base event", the
stored pointer is truncated when it's read from the ringbuffer.
$ cd /sys/kernel/tracing
$ echo 'e rcu.rcu_utilization $s:x64 $s:string' > dynamic_events
$ echo 1 > tracing_on
$ echo 1 > events/eprobes/enable
$ sleep 1
$ echo 0 > events/eprobes/enable
$ cat trace
<idle>-0 ...: (rcu.rcu_utilization) arg1=0x4f arg2=(fault)
<idle>-0 ...: (rcu.rcu_utilization) arg1=0x2 arg2=(fault)
The problem is in get_event_field
val = (unsigned long)(*(char *)addr);
addr points to the position in the ringbuffer where the pointer was
stored. The assignment reads only the lowest byte of the pointer.
Fix the cast to read the whole pointer. The output of the test above
is now
<idle>-0 ... arg1=0xffffffff81c7d3f3 arg2="Start scheduler-tick"
<idle>-0 ... arg1=0xffffffff81c57340 arg2="End scheduler-tick"
Link: https://lore.kernel.org/all/20260620145339.3234726-1-martin@kaiser.cx/
Fixes: f04dec9346 ("tracing/eprobes: Fix reading of string fields")
Signed-off-by: Martin Kaiser <martin@kaiser.cx>
Signed-off-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
When CONFIG_BPF_LSM=y is set, BPF inode storage maps
(BPF_MAP_TYPE_INODE_STORAGE) are compiled into the kernel. However,
if the BPF LSM is not explicitly enabled at boot time (e.g. omitted
from the "lsm=" boot parameter), lsm_prepare() is never executed for
the BPF LSM.
Consequently, the BPF inode security blob offset
(bpf_lsm_blob_sizes.lbs_inode) is never initialized and remains at
its default compiled size of 8 bytes instead of being updated to a
valid offset past the reserved struct rcu_head (typically 16 bytes
or more).
When a privileged user creates and updates a BPF_MAP_TYPE_INODE_STORAGE
map, bpf_inode() evaluates inode->i_security + 8. This erroneously
aliases the struct rcu_head.func callback pointer at the beginning
of the inode->i_security blob. During subsequent map element cleanup
or inode destruction, writing NULL to owner_storage clears the queued
RCU callback pointer. When rcu_do_batch() later executes the queued
callback, it attempts an instruction fetch at address 0x0, triggering
an immediate kernel panic.
Fix this by introducing a global bpf_lsm_initialized boolean flag
marked with __ro_after_init. Set this flag to true inside bpf_lsm_init()
when the LSM framework successfully registers the BPF LSM. Gate map
allocation in inode_storage_map_alloc() on this flag, returning
-EOPNOTSUPP if the BPF LSM is in turn uninitialized.
This fail-fast approach prevents userspace from allocating inode
storage maps when the supporting BPF LSM infrastructure is absent,
avoiding zombie map states.
Fixes: 8ea636848a ("bpf: Implement bpf_local_storage for inodes")
Reported-by: oxsignal <awo@kakao.com>
Signed-off-by: Matt Bobrowski <mattbobrowski@google.com>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Reviewed-by: Emil Tsalapatis <emil@etsalapatis.com>
Reviewed-by: Amery Hung <ameryhung@gmail.com>
Link: https://lore.kernel.org/bpf/20260628201103.3624525-1-mattbobrowski@google.com
Three sites enumerate every scx_event_stats field by name:
scx_read_events(), scx_attr_events_show(), and scx_dump_state().
Adding a new SCX_EV_* event requires three separate manual edits.
Missing any one silently drops counts or omits the field from
diagnostic output with no compile-time indication.
SCX_EVENTS_LIST(SCX_EVENT) in internal.h becomes the single
authoritative list of event names, so all three sites stay in sync.
No functional changes.
Signed-off-by: Cheng-Yang Chou <yphbchou0911@gmail.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
put_prev_task_scx() warns when a runnable task drops to a lower sched_class
without SCX_OPS_ENQ_LAST, on the assumption that balance_one() would have
kept it running. Core scheduling breaks that: a forced-idle SMT sibling
reschedules through the core_pick fast path in pick_next_task(), which skips
pick_task_scx() and thus balance_one(), so a runnable task can drop to idle
with ENQ_LAST unset.
Gate the warning on sched_cpu_cookie_match(): a cookie mismatch means core
scheduling forced the idle, while a match (or core scheduling off) still
catches a genuine missing-ENQ_LAST drop.
Fixes: 7c65ae81ea ("sched_ext: Don't call put_prev_task_scx() before picking the next task")
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
A few comments still describe the pre-EEVDF CFS world:
- sysctl_sched_base_slice is documented as "Minimal preemption
granularity for CPU-bound tasks". That was the wording of the old
sysctl_sched_min_granularity, renamed in commit e4ec3318a1
("sched/debug: Rename sysctl_sched_min_granularity to
sysctl_sched_base_slice"). Under EEVDF it is the default base time
slice / request size (r_i) used to compute the virtual deadline, as
documented in update_deadline().
- Two comments still mention sched_slice(), which was removed when the
fair class committed to EEVDF in commit 5e963f2bd4 ("sched/fair:
Commit to EEVDF"). The dequeue-path comment should simply refer to
the task's slice (se->slice); the forced-idle comment describes the
slice accounting now performed by __entity_slice_used(), which is the
function actually used right below it.
No functional changes.
[ mingo: Ported to a more recent scheduler base ]
Signed-off-by: Zhan Xusheng <zhanxusheng@xiaomi.com>
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Link: https://patch.msgid.link/20260629030200.3165589-1-zhanxusheng@xiaomi.com
perf_event_open() currently gates perf_allow_kernel() only on
!attr.exclude_kernel.
However, users can still request kernel callchain collection with
attr.exclude_callchain_kernel == 0 even when attr.exclude_kernel == 1.
That still requires kernel profiling privilege, but the existing check
does not enforce it.
Update the permission check to call perf_allow_kernel() when either
kernel sampling is requested or kernel callchains are requested.
This keeps permission checks aligned with requested data and prevents
unprivileged use of kernel callchain capture.
Signed-off-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/20260616044654.3468742-9-dapeng1.mi@linux.intel.com
An unprivileged hardware perf event using exclude_kernel=1 can leak kernel
register data to user space via PERF_SAMPLE_REGS_INTR or PERF_SAMPLE_IP.
Due to hardware skid, a PMI may trigger after the CPU has already entered
kernel space (Ring 0), bypassing the perf_allow_kernel() privilege
barrier.
This security vulnerability is severely exacerbated by upcoming support
for SIMD register sampling via XSAVES, which could expose sensitive kernel
FPU states (such as active cryptographic keys).
Fix this by ensuring that sampled register data is dropped if the event's
exclude_kernel attribute is set but the PMI catches the CPU in kernel mode.
Signed-off-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://lore.kernel.org/all/20260529085613.CCAFB1F00893@smtp.kernel.org/
Link: https://patch.msgid.link/20260616044654.3468742-8-dapeng1.mi@linux.intel.com
nr_migrations_cold, nr_wakeups_passive and nr_wakeups_idle are not
being updated anywhere. So remove them.
These are per process stats. So updating sched stats version isn't
necessary.
Signed-off-by: Shrikanth Hegde <sshegde@linux.ibm.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: K Prateek Nayak <kprateek.nayak@amd.com>
Tested-by: K Prateek Nayak <kprateek.nayak@amd.com>
Link: https://patch.msgid.link/20260625124648.802832-2-sshegde@linux.ibm.com
psi_account_irqtime() reads irq_time_read() into a per-rq cumulative
counter and only bails out when the delta vs. the previously accounted
amount is negative. A delta of exactly zero is treated as "do the
work": psi_write_begin() is taken, cpu_clock(cpu) is read (which on
x86 ends up in native_sched_clock() / rdtsc) and the cgroup ancestor
chain is walked to add zero to every group's PSI_IRQ_FULL bucket.
The zero-delta case is common in practice -- it fires every time a
context switch crosses a PSI group boundary on a CPU that hasn't
serviced an interrupt between the two switches.
Measured on a 176-thread AMD EPYC 9D64 server running a compute
intensive production workload, instrumented with bpftrace over a 30s
window (irq_time_read() read directly from the per-CPU cpu_irqtime so
that delta == 0 and delta < 0 could be separated):
@total 17,229,311 (100.0%)
@ret_curr_swapper 7,864,195 ( 45.6%) curr->pid == 0
@ret_samegrp 323,299 ( 1.9%) same cgroup as prev
@reached_delta 9,041,817 ( 52.5%)
@delta_positive 6,358,192 ( 36.9%) real work
@delta_zero 2,683,625 ( 15.6%) work wasted (this patch)
@delta_negative (0) ( 0.0%) monotonic clock
So 15.6 % of all psi_account_irqtime() calls - and 29.7 % of the
calls that get past the early returns - hit the delta == 0 case;
delta < 0 did not occur once in the 30 s window. Under the current
code each of those ~89 k calls per second performs the full seqcount
write + cpu_clock() read + cgroup-chain walk just to add 0 to every
group's PSI_IRQ_FULL counter.
Extend the early-return to also cover delta == 0. rq->psi_irq_time
does not need updating in that case (it would store the same value
back) and no PSI bucket would change. The existing behaviour for
delta > 0 is untouched.
Signed-off-by: Usama Arif <usama.arif@linux.dev>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Shakeel Butt <shakeel.butt@linux.dev>
Acked-by: Johannes Weiner <hannes@cmpxchg.org>
Link: https://patch.msgid.link/20260617175219.2494857-2-usama.arif@linux.dev
Because active_balance is initialized to 0, and need_active_balance()
is a pre-condition for setting it to 1, the condition '!active_balance
|| need_active_balance()' is a truism and can be removed.
Signed-off-by: Xin Zhao <jackzxcui1989@163.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/20260617072151.1173416-3-jackzxcui1989@163.com
Active load balancing relies on migration threads, which temporarily preempt
tasks on the source runqueue (src_rq). This preemption can negatively impact
overall system performance. The active balancing logic includes a check to
verify whether the current task (curr) on src_rq can actually run on the
destination runqueue (dst_rq). We have observed that when curr is a CFS task
and its on_rq flag is 0, the active balancing failure rate is exceptionally
high. The following table summarizes test data collected over 300 seconds on an
18-CPU platform under a specific fillback task scenario:
fair: busiest->curr->sched_class == &fair_sched_class
on_rq: busiest->curr->on_rq
total: active balance count triggered of correspondent type
fail: fail to migrate one task in active_load_balance_cpu_stop()
fair && !on_rq !fair && !on_rq
domain total fail total fail
cpu0 0x00003 0 0 0 0
cpu0 0x3ffff 33 33 1 1
cpu1 0x00003 0 0 0 0
cpu1 0x3ffff 42 42 0 0
cpu2 0x0003c 4 4 0 0
cpu2 0x3ffff 12 12 0 0
cpu3 0x0003c 3 3 0 0
cpu3 0x3ffff 8 7 0 0
cpu4 0x0003c 2 2 0 0
cpu4 0x3ffff 5 4 0 0
cpu5 0x0003c 4 4 0 0
cpu5 0x3ffff 8 8 0 0
cpu6 0x003c0 60 60 0 0
cpu6 0x3ffff 28 27 0 0
cpu7 0x003c0 194 184 0 0
cpu7 0x3ffff 35 35 1 1
cpu8 0x003c0 240 228 0 0
cpu8 0x3ffff 28 28 0 0
cpu9 0x003c0 0 0 0 0
cpu9 0x3ffff 10 10 0 0
cpu10 0x03c00 52 50 0 0
cpu10 0x3ffff 0 0 0 0
cpu11 0x03c00 70 68 0 0
cpu11 0x3ffff 1 1 0 0
cpu12 0x03c00 73 72 0 0
cpu12 0x3ffff 0 0 0 0
cpu13 0x03c00 79 76 0 0
cpu13 0x3ffff 0 0 0 0
cpu14 0x3c000 0 0 0 0
cpu14 0x3ffff 57 55 1 0
cpu15 0x3c000 53 52 1 0
cpu15 0x3ffff 30 29 0 0
cpu16 0x3c000 344 341 10 6
cpu16 0x3ffff 103 100 2 1
cpu17 0x3c000 183 179 2 2
cpu17 0x3ffff 78 77 0 0
sum 1839 1791 18 11
In __schedule(), before curr is updated to next, pick_next_task() invokes
sched_balance_rq(). This function temporarily unlocks and relocks the runqueue,
creating a window where other CPUs may observe rq->curr->on_rq as 0.
We can safely skip active balancing when src_rq->curr->on_rq == 0, as other
eligible tasks have likely already been evaluated.
We retain the affinity check on dst_rq to trigger active balancing, since such
tasks are often woken by (or wake up) tasks on src_rq that share similar
affinity constraints. Furthermore, detach_tasks() releases the runqueue lock;
any tasks awakened during this window may preempt the previous CFS task. My
testing (data not shown) indicates that active balancing succeeds in 98.4% of
cases where !fair && on_rq.
This scenario does not require a stop-work callback, but would necessitate an
additional detach/attach path. As Valentin and Vincent have already discussed,
this addition does not appear justified at this time (see [1]).
Since can_migrate_task() already checks on_cpu during the cfs_tasks traversal,
adding an on_rq check will have negligible performance overhead due to cache
locality.
There are two reasons for not combining the on_rq check with the
cpumask_test_cpu() check:
- Avoiding new scenarios that would skip the logic for resetting
balance_interval to min_interval.
- The existing check for whether the busiest CPU recently triggered active
load balancing already filters more cases than the on_rq check.
[1]: https://lore.kernel.org/lkml/20190815145107.5318-5-valentin.schneider@arm.com/
Signed-off-by: Xin Zhao <jackzxcui1989@163.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Valentin Schneider <vschneid@redhat.com>
Link: https://patch.msgid.link/20260617072151.1173416-2-jackzxcui1989@163.com
When a task with a shorter slice is enqueued, we protect the running
task which has a longer slice until it becomes ineligible instead of a
full slice in order to speedup the switch to other tasks until the task
with the shortest slice is scheduled. This helps to the task to not wait
too many full slices before running.
Signed-off-by: Vincent Guittot <vincent.guittot@linaro.org>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Tested-by: K Prateek Nayak <kprateek.nayak@amd.com>
Link: https://patch.msgid.link/20260624151229.1710703-7-vincent.guittot@linaro.org
Even if p will not preempt current, it modifies the avg_vruntime and
possibly the min slice. Make sure to update the slice protection with the
updated figures. As an example, Batch and Sched Idle tasks can otherwise
get a larger lag than their slice and finaly delay the scheduling of a
normal task, which deadline will be a later.
Signed-off-by: Vincent Guittot <vincent.guittot@linaro.org>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Tested-by: K Prateek Nayak <kprateek.nayak@amd.com>
Link: https://patch.msgid.link/20260624151229.1710703-6-vincent.guittot@linaro.org
If a short slice task will not be the next to be picked but is eligible,
we cancel the slice protection to speedup the time when the short slice
task will be the next to run.
Signed-off-by: Vincent Guittot <vincent.guittot@linaro.org>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Tested-by: K Prateek Nayak <kprateek.nayak@amd.com>
Link: https://patch.msgid.link/20260624151229.1710703-5-vincent.guittot@linaro.org
Even if resched is already set, we might want to update or even cancel
the slice protection and ensure that the newly waking task will be the
next one to run.
Signed-off-by: Vincent Guittot <vincent.guittot@linaro.org>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Tested-by: K Prateek Nayak <kprateek.nayak@amd.com>
Link: https://patch.msgid.link/20260624151229.1710703-4-vincent.guittot@linaro.org
Take into account the lag of current task when updating the slice
protection in order to ensure that the absolute value of lags will remain
in the range [0 : slice+tick]
A task that already has a negative lag will see its protection reduced
whereas a task with positive lag will keep a full slice protection.
Signed-off-by: Vincent Guittot <vincent.guittot@linaro.org>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Tested-by: K Prateek Nayak <kprateek.nayak@amd.com>
Link: https://patch.msgid.link/20260624151229.1710703-3-vincent.guittot@linaro.org
If a shorter slice task can preempt current at wakeup, we make sure that
the decision will not be overwritten in between by setting the task as the
next buddy. This still implies that the waking task remains eligible when
the scheduler will actually pick the next task to run.
Signed-off-by: Vincent Guittot <vincent.guittot@linaro.org>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Tested-by: K Prateek Nayak <kprateek.nayak@amd.com>
Link: https://patch.msgid.link/20260624151229.1710703-2-vincent.guittot@linaro.org
On a tangential note, I just noticed set_preempt_buddy() has two unused
parameters. Seems to have been like that since it was introduced in
commit e837456fdc ("sched/fair: Reimplement NEXT_BUDDY to align with
EEVDF goals").
Signed-off-by: K Prateek Nayak <kprateek.nayak@amd.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Change fair/cgroup to a single runqueue.
Infamously fair/cgroup isn't working for a number of people; typically
the complaint is latencies and/or overhead. The latency issue is due
to the intermediate entries that represent a combination of tasks and
thereby obfuscate the runnability of tasks.
The approach here is to leave the cgroup hierarchy as is; including
the intermediate enqueue/dequeue but move the actual EEVDF runqueue
outside. This means things like the shares_weight approximation are
fully preserved.
That is, given a hierarchy like:
R
|
se--G1
/ \
G2--se se--G3
/ \ |
T1--se se--T2 se--T3
This is fully maintained for load tracking, however the EEVDF parts of
cfs_rq/se go unused for the intermediates and are instead connected
like:
_R_
/ | \
T1 T2 T3
Since the effective weight of the entities is determined by the
hierarchy, this gets recomputed on enqueue,set_next_task and tick.
Notably, the effective weight (se->h_load) is computed from the
hierarchical fraction: se->load / cfs_rq->load.
Since EEVDF is now exclusively operating on rq->cfs, it needs to
consider cfs_rq->h_nr_queued rather than cfs_rq->nr_queued. Similarly,
only tasks can get delayed, simplifying some of the cgroup cleanup.
One place where additional information was required was
set_next_task() / put_prev_task(), where we need to track 'current'
both in the hierarchical sense (cfs_rq->h_curr) and in the flat sense
(cfs_rq->curr).
As a result of only having a single level to pick from, much of the
complications in pick_next_task() and preemption go away.
Since many of the hierarchical operations are still there, this won't
immediately fix the performance issues, but hopefully it will fix some
of the latency issues.
TODO: split struct cfs_rq / struct sched_entity
TODO: try and get rid of h_curr
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/20260605124052.227463677%40infradead.org
For all the reasons described in the preceding patches, the way cgroup
weight is computed is problematic. However, changing it is bound to
also lead to trouble. Esp. since people might have taken to inflating
the weight value where they can.
Since things are configurable, change the default and hope this serves
more people than it hurts, esp. in the longer run.
Specifically, this prepares for a flattened runqueue, where the hierarchical
weight becomes far more important (F_g^d terms), so getting rid of small F_g is
imperative.
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/20260605124052.080482755%40infradead.org
Since we are exploring this space; include a scheme that scales by total number
of runnable tasks. This results in:
F_g_n' = M * F_g_n
This will obviously have: avg(F_g_n') > 1, (it will be ~M/N in fact).
And while that sounds odd, it actually has a fairly straight foward meaning for
"cpu.weight": average weight per member task.
This is an entirely valid and workable option, it is however wildly different
from the traditional meaning.
Included for completeness (and curiosity).
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/20260605124051.921991975%40infradead.org
Improve upon the previous scheme ("max") by no longer assuming maximal
concurrency. Instead scale by: 'min(nr_tasks, nr_cpus)'. This handles
the low concurrency cases more gracefully:
F_g_n' = min(M, N) * F_g_n
Notably this is the first mode where:
avg(F_g_n) = 1
In the single task case it reduces to ("smp") and then it nicely scales up
until it hits N, where it behaves like ("max").
This is no longer clipped at nice -20. Strictly speaking it isn't different
from the normal SMP scenario where all tasks are extremely unbalanced. There
are no unnatural inflations in this scheme.
The meaning of "cpu.weight" would be: weight per active CPU.
NOTE: Compute the group wide number of tasks by extending the tg->load_avg
computation with tg->runnable_avg, since cfs_rq->runnable_avg is based on
cfs_rq->h_nr_running.
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/20260605124051.740585993%40infradead.org
In order to avoid the average CPU fraction avg(F_g_n) becoming tiny '1/N',
assume each cgroup is maximally concurrent and distrubute 'N*weight', such
that:
F_g_n' = N * F_g_n
Giving:
avg(F_g_n') = N*avg(F_g_n) ~ N * 1/N = 1
And while this sounds like it solves things, remember what that ~ meant. There
is the corner case when a cgroup is minimally loaded, eg a single runnable
task, therefore limit the CPU fraction to that of a nice -20 task to avoid
getting too much load.
This last bit is what makes it different from a previous proposal to allow
raising cpu.weight to '100 * N', that would not limit the mininal concurrency
case and results in a very large F_g_n. And just like F_g_n << 1 is
problematic, so is F_g_n >> 1 for the exact same reasons (it would drown the
kthreads, but it also risks overflowing the load values).
So while this might appear to be a better scheme than the current default
scheme, it doesn't really handle less than maximal concurrency nicely -- it
clips and introduces artificially large weights. So where the traditional SMP
mode works well when nr_tasks << nr_cpus, MAX doesn't work well in that regime
and vice-versa.
The meaning of "cpu.weight" would be: weight per allowed CPU.
Included for completeness (and infrastructure).
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/20260605124051.589618504%40infradead.org
Instead of calculating the proportional fraction of the group weight for each
CPU, just give each CPU the full measure, ignoring these pesky SMP problems.
This makes the SMP cgroup fraction (F_g_n) equal to 1, and ensures a single
task in a cgroup competes on equal footing to a task in a level above.
However, as already explored, this is not a very good policy because it gets
the SMP weight distribution wrong. Included for completeness.
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/20260605124051.450303977%40infradead.org
The effective task weight (W_t') for a task in cgroup g on CPU n is given by:
W_t
W_t' = W_g * F_g_n * ----------
\Sum W_t_n
Where W_g is the group's weight (cpu.weight), F_g_n is the fraction of the
group weight for CPU n and W_t/W is the relative weight of this task against
all other tasks in the same group on the same CPU.
Furthermore, this makes:
\Sum W_t_n
F_g_n = ----------
\Sum W_t
The fraction of weight inside the group of CPU n against the whole group.
The problem is with F_g_n, the primary goal of this fraction is to make sure
that the relative weight of tasks, when distributed over CPUs is maintained.
For example, consider 4 (equal weight) tasks and 2 CPUs with a 1:3
distribution, then if F_g_n would simply be 1 (no weight re-distribution) the
effective relative weights (W_t') of the tasks in our group would be:
CPU0 CPU1
W_g W_g/3
W_g/3
W_g/3
IOW, the lucky task on CPU0 would get an equal amount of weight as all 3 tasks
on CPU1 combined. However, with the weight redistribution, this becomes:
CPU0 CPU1
W_g/4 W_g/4
W_g/4
W_g/4
All tasks are equal weight (as intended). However, as is already evident from
this example, the more CPUs you add, the smaller F_g_n becomes, which creates a
disparity against tasks not in our group.
Specifically:
avg(F_g_n) ~ 1/N
This leads to a weight mismatch in the hierarchy. IOW tasks cannot compete
fairly across hierarchy levels.
*Notably*, what is meant by avg(F_g_n) being proportional to 1/N is that when
there are at least N runnable tasks, the average of this fraction tends to 1/N.
For a hierarchy of depth d, this gets even worse, since that gets terms on the
order of:
avg(F_g_n)^d ~ 1/(N^d)
Given fixed point arithmetic, this also leads to numerical trouble.
However, the meaning of "cpu.weight" is simple and intiutive: the total weight
of the cgroup. But as explored above, there is deception in this simplicity.
Prepare to add a few alternative methods for distributing weight.
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/20260605124051.338602724%40infradead.org
A divide-by-zero crash is observed when running hackbench:
[14697.488452] CPU: 112 UID: 0 PID: 124791 Comm: hackbench Not tainted 7.1.0-rc2+
[14697.492627] RIP: 0010:propagate_entity_load_avg+0x35f/0x3e0
[14697.506799] <TASK>
[14697.507411] __dequeue_task+0x2b4/0xc70
[14697.508677] dequeue_task_fair+0x36/0x370
[14697.509047] dequeue_task+0x101/0x2f0
[14697.509426] __schedule+0x1b1/0x1a00
[14697.510868] anon_pipe_read+0x3da/0x450
[14697.511400] vfs_read+0x361/0x390
[14697.512053] __x64_sys_read+0x19/0x30
The divide-by-zero happens here:
if (scale_load_down(gcfs_rq->load.weight)) {
load_sum = div_u64(gcfs_rq->avg.load_sum,
scale_load_down(gcfs_rq->load.weight));
}
gcfs_rq->load.weight is an insane large value and is truncated
to the lower 32 bits by div_u64, which happen to be 0.
Using AI for investigation, the cause is a u32 overflow in
update_tg_cfs_runnable(), and flat pickup became a victim when using
tg_tasks():
u32 new_sum, divider;
...
new_sum = se->avg.runnable_avg * divider; <-- boom
The following sequence shows how this triggers the crash:
propagate_entity_load_avg()
update_tg_cfs_runnable() # u32 overflow corrupts runnable_sum
__update_load_avg_cfs_rq()
___update_load_avg() # computes insane runnable_avg
update_tg_load_avg() # propagates to tg->runnable_avg
update_cfs_group()
calc_concur_shares()
tg_tasks() # long-to-int truncation, negative nr
reweight_entity() # corrupted se->load.weight
update_load_add() # corrupted cfs_rq->load.weight
propagate_entity_load_avg()
update_tg_cfs_load()
div_u64() # divide-by-zero
Fix by widening new_sum from u32 to u64 (no need to force tg_tasks()
to return unsigned long after this fix)
Fixes: 95246d1ec8 ("sched/pelt: Relax the sync of runnable_sum with runnable_avg")
Assisted-by: Claude:claude-opus-4.6
Signed-off-by: Chen Yu <yu.c.chen@intel.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/a22eea2b-4c4a-4623-9a44-d7b18c0c91c8@intel.com
The way wakeup_preempt() works since commit 704069649b ("sched/core: Rework
sched_class::wakeup_preempt() and rq_modified_*()") is that it will call
rq->next_class->wakeup_preempt(rq, p) when p is of an equal or higher class,
and raise ->next_class when higher.
This means that:
running idle task
wakeup fair-A
(next_class == idle)
if (sched_class_above(fair, idle)) {
wakeup_preempt_idle(fair-A);
resched_curr(rq);
next_class = fair;
}
wakeup fair-B
(next_class == fair)
if (fair == fair)
wakeup_preempt_fair(fair-B);
(but current is idle)
All wakeup_preempt_$class() methods, except for wakeup_preempt_scx() (for whoem
this was build) ignore cross-class wakeups by testing if @p is of the right
class, but per the above case, it also should check current.
This is mostly harmless in the current form, but will lead to trouble with
later patches.
Fixes: 704069649b ("sched/core: Rework sched_class::wakeup_preempt() and rq_modified_*()")
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/20260626074605.GB2568396%40noisy.programming.kicks-ass.net
Currently, copy_process() can bail out to free_task() before p->bpf_storage
has been initialized, with this call graph (shown here for the
!CONFIG_MEMCG case):
copy_process
dup_task_struct
arch_dup_task_struct
[copies the entire task_struct, including ->bpf_storage member]
[RLIMIT_NPROC check fails]
delayed_free_task
free_task
bpf_task_storage_free
rcu_dereference(task->bpf_storage)
bpf_local_storage_destroy
In this case, the nascent task's ->bpf_storage member that
bpf_local_storage_destroy() operates on is a plain copy of the parent's
->bpf_storage pointer, not a real initialized pointer.
This leads to badness (kernel hangs, UAF).
This is reachable as long as the process calling fork() has been inserted
into a task storage map.
Cc: stable@kernel.org
Fixes: a10787e6d5 ("bpf: Enable task local storage for tracing programs")
Signed-off-by: Jann Horn <jannh@google.com>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
A child sub-scheduler dereferences its parent scx_sched throughout its life,
e.g., in scx_sub_disable() which reparents the child's tasks and calls
parent->ops.sub_detach() after unlinking from the parent. However, the
parent is pinned only through parent->sub_kset, which is dropped during
disable. The parent scx_sched can be RCU-freed while a child is still
disabling.
Take a direct reference on the parent in scx_alloc_and_add_sched(), dropped
in scx_sched_free_rcu_work(), so a parent always outlives its descendants.
Signed-off-by: Tejun Heo <tj@kernel.org>
Use kick_pool_pick() to select and claim the worker under pool->lock and
issue the wakeup with wake_up_process() after the lock is dropped.
Unlike __queue_work(), this path has no surrounding RCU section, so take
rcu_read_lock() before dropping pool->lock to keep the picked worker's
task_struct valid across the wakeup.
Signed-off-by: Breno Leitao <leitao@debian.org>
Tested-by: Krishna Magar <kmagar@redhat.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
__queue_work() is the enqueue hot path: it inserts the work item and
calls kick_pool() while holding pool->lock. kick_pool() ends in a
wakeup, which takes the target task's rq->lock, so rq->lock nests under
pool->lock on every enqueue that wakes a worker on a contended unbound
pool.
Use kick_pool_pick() to select and claim the worker under pool->lock and
issue the wakeup with wake_up_process() right after dropping the lock.
Signed-off-by: Breno Leitao <leitao@debian.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
Factor the worker selection out of kick_pool() into kick_pool_pick(),
which picks and claims the worker under pool->lock but, instead of waking
it, returns the worker's task via an out-param so the caller can issue
the wakeup after dropping pool->lock. BH kicks and wake_cpu setup still
happen under the lock.
kick_pool() becomes a thin wrapper that wakes the returned task, so all
existing callers keep waking under pool->lock.
Pure refactor, no functional change.
Signed-off-by: Breno Leitao <leitao@debian.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
__csd_lock_record() publishes per-CPU diagnostic state (cur_csd,
cur_csd_func, cur_csd_info) that is consumed from a remote CPU by
csd_lock_wait_toolong() via smp_load_acquire(&cur_csd). To order the
matching cur_csd_func/cur_csd_info stores before the cur_csd
publication, the producer issues smp_wmb() before writing cur_csd;
to order the publication before the subsequent callback execution or
CSD unlock, it issues smp_mb() after the write. The clear path
mirrors this with smp_mb() before storing NULL into cur_csd so the
preceding callback/unlock is observed first.
The smp_mb() pair is heavier than what the consumer actually
requires (on x86 each emits a locked full barrier). The consumer
only needs to observe the matching cur_csd_func/cur_csd_info when it
sees a non-NULL cur_csd, and to observe the preceding callback/unlock
when it sees NULL -- both of which a release/acquire pair provides.
The extra two-way ordering enforced by smp_mb() -- that cur_csd
publication be observed before callback execution or unlock becomes
visible -- would only matter if cur_csd were an exact live-state
marker. csd_lock_wait_toolong() does not treat it that way: it
snapshots cur_csd via smp_load_acquire() and then prints / dumps /
re-IPIs without an RCU-style stall-ended recheck, so the diagnostic
already tolerates the remote CPU completing its work between snapshot
and report. cur_csd is best-effort context, not a precise stall
boundary.
Replace the smp_wmb() + plain store + smp_mb() in the publish path,
and the smp_mb() + plain store in the clear path, with
smp_store_release(). This pairs with the smp_load_acquire() in
csd_lock_wait_toolong(): preceding cur_csd_func/cur_csd_info stores
become visible before a remote reader observes the non-NULL
publication, and any preceding callback/unlock becomes visible before
a reader observes the NULL clear.
Signed-off-by: Usama Arif <usama.arif@linux.dev>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Reviewed-by: Dmitry Ilvokhin <d@ilvokhin.com>
Link: https://patch.msgid.link/20260629105745.1696683-1-usama.arif@linux.dev
The callers already have a valid irq_data pointer, so the NULL checks in
__irq_domain_[de]activate_irq() is redundant.
Remove it.
[ tglx: Massage changelog, fix subject prefix ]
Signed-off-by: Kemeng Shi <shikemeng@huaweicloud.com>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Link: https://patch.msgid.link/20260623075134.4432-4-shikemeng@huaweicloud.com
When irq_domain_trim_hierarchy() fails in irq_domain_alloc_irqs_locked(),
the error handling path fails to undo the hierarchy allocations.
Add the missing cleanup.
[ tglx: Massage change log ]
Signed-off-by: Kemeng Shi <shikemeng@huaweicloud.com>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Link: https://patch.msgid.link/20260623075134.4432-2-shikemeng@huaweicloud.com
Add a umh field to struct kernel_clone_args. When set, copy_fs() copies
from pid 1's fs_struct instead of the kthread's fs_struct. This ensures
usermodehelper threads always get init's filesystem state regardless of
their parent's (kthreadd's) fs.
Usermodehelper threads are not allowed to create mount namespaces
(CLONE_NEWNS), share filesystem state (CLONE_FS), or be started from
a non-initial mount namespace. No usermodehelper currently does this so
we don't need to worry about this restriction.
Set .umh = 1 in user_mode_thread(). At this stage pid 1's fs points to
rootfs which is the same as kthreadd's fs, so this is functionally
equivalent.
Link: https://patch.msgid.link/20260601-work-kthread-nullfs-v4-20-77ee053060e0@kernel.org
Signed-off-by: Christian Brauner (Amutable) <brauner@kernel.org>
Add a real_fs field to task_struct that always mirrors the fs field.
This lays the groundwork for distinguishing between a task's permanent
fs_struct and one that is temporarily overridden via scoped_with_init_fs().
When a kthread temporarily overrides current->fs for path lookup, we
need to know the original fs_struct for operations like exit_fs() and
unshare_fs_struct() that must operate on the real, permanent fs.
For now real_fs is always equal to fs. It is maintained alongside fs in
all the relevant paths: exit_fs(), unshare_fs_struct(),
switch_fs_struct(), and copy_fs().
Link: https://patch.msgid.link/20260601-work-kthread-nullfs-v4-4-77ee053060e0@kernel.org
Signed-off-by: Christian Brauner (Amutable) <brauner@kernel.org>
Add spaces around bitwise AND and shift operators in sys_exit()
to comply with the Linux kernel coding style.
Signed-off-by: mingzhu wang <mingzhu.wang@transsion.com>
Link: https://patch.msgid.link/20260609021436.1739-1-mingzhu.wang@transsion.com
Signed-off-by: Christian Brauner (Amutable) <brauner@kernel.org>
- Remove a misguided warning along with an inaccurate comment next to
it from the cpuidle core (Rafael Wysocki)
- Clear need_freq_update as appropriate in the .adjust_perf()
path of the schedutil cpufreq governor to avoid calling
cpufreq_driver_adjust_perf() unnecessarily on every scheduler
utilization update (Zhongqiu Han)
-----BEGIN PGP SIGNATURE-----
iQFGBAABCAAwFiEEcM8Aw/RY0dgsiRUR7l+9nS/U47UFAmo+ujsSHHJqd0Byand5
c29ja2kubmV0AAoJEO5fvZ0v1OO1o5UH/0PUjz0I+HrrPi6O6Nt2vaA/o+fI8DRA
d+M2KbZqh1e/Jk8wMB9IwErFzolY6rrJ4HeBkzBo7VNIofgvljH7EwrrUhYhv1c1
xWViKUS6JUpAOwA+44tGKhiDU5PrNHyVDXqjJ5id6pfBiVMxtRcCCd1xBDQRNXg2
r3hRnUz9F3vP93AB0lujsSQeMj2g6XmTX+AyBmDcrYwrO/vZOao1OWNOnxfbt4Yd
TPokkyPCBuLAQYM7eDF1jMsbRqDIo4/zHbVMmVdUYzOd944S1Ruira89V5A7Txzg
Ih7DCct2UDwAMKubMYEVlCdSkXZl7IRXep+qDeOkLo3xhs0UORgEXwg=
=V/er
-----END PGP SIGNATURE-----
Merge tag 'pm-7.2-rc1-2' of git://git.kernel.org/pub/scm/linux/kernel/git/rafael/linux-pm
Pull power management fixes from Rafael Wysocki:
"These fix the schedutil cpufreq governor and drop a bogus warning
from the cpuidle core:
- Remove a misguided warning along with an inaccurate comment
next to it from the cpuidle core (Rafael Wysocki)
- Clear need_freq_update as appropriate in the .adjust_perf()
path of the schedutil cpufreq governor to avoid calling
cpufreq_driver_adjust_perf() unnecessarily on every scheduler
utilization update (Zhongqiu Han)"
* tag 'pm-7.2-rc1-2' of git://git.kernel.org/pub/scm/linux/kernel/git/rafael/linux-pm:
cpuidle: Allow exit latency to exceed target residency
cpufreq: schedutil: Fix uncleared need_freq_update on the .adjust_perf() path
When bpf_check() allocates env->insn_aux_data successfully but later
fails to allocate env->succ, it jumps directly to err_free_env.
The existing vfree(env->insn_aux_data) sits before the err_free_env
label, so that direct jump bypasses it and leaks insn_aux_data.
Move vfree(env->insn_aux_data) into err_free_env so all early and late
exit paths release it consistently.
Fixes: 2f69c56854 ("bpf: make bpf_insn_successors to return a pointer")
Signed-off-by: KaFai Wan <kafai.wan@linux.dev>
Reviewed-by: Anton Protopopov <a.s.protopopov@gmail.com>
Link: https://lore.kernel.org/r/20260624123536.114757-1-kafai.wan@linux.dev
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
print_bpf_insn() masks ldimm64 immediates for pointer-bearing pseudo
sources when pointer leaks are not allowed, but the mask only covers
BPF_PSEUDO_MAP_FD and BPF_PSEUDO_MAP_VALUE.
BPF_PSEUDO_MAP_IDX, BPF_PSEUDO_MAP_IDX_VALUE, and BPF_PSEUDO_BTF_ID can
also be resolved to kernel pointer values before the verifier log prints
the instruction. Include them in the existing pointer classification so
the log prints 0x0 instead of the rewritten address.
Fixes: 4976b718c3 ("bpf: Introduce pseudo_btf_id")
Fixes: 387544bfa2 ("bpf: Introduce fd_idx")
Signed-off-by: Nuoqi Gui <gnq25@mails.tsinghua.edu.cn>
Link: https://lore.kernel.org/r/20260623-f01-13-pseudo-btf-id-cap-bpf-v2-1-a190ebb8f3e2@mails.tsinghua.edu.cn
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Stack reads can read back bytes that belong to a previously spilled scalar
constant zero. Today mark_reg_stack_read() only treats STACK_ZERO bytes as
known zero bytes, so the destination register can become unknown even
though every byte in the read range is known to be zero. This can lead to
rejecting otherwise valid programs once the loaded byte is used as a
pointer offset.
The original reproducer uses a variable-offset stack byte read emitted by
clang 22.1.6 at -O2/-O3 from a small helper-based BPF C program. Fixed
offset reads have a related mixed case as well: pure scalar-zero spill
reads are already handled, but a fixed read spanning both STACK_ZERO and
scalar const-zero STACK_SPILL bytes still falls back to unknown.
Teach mark_reg_stack_read() to also consider STACK_SPILL bytes backed by a
spilled scalar constant zero as zero bytes, and use that path for both
variable-offset stack reads and fixed-offset mixed reads. Keep the existing
pure register-fill behavior unchanged.
When a zero result depends on such a spill, mark the contributing stack
slots precise before accepting the const-zero result so pruning cannot
reuse a zero-spill state for a later non-zero spill state.
No deployed-program regression is currently known, so target bpf-next.
Assisted-by: opencode:gpt-5.5
Signed-off-by: Woojin Ji <random6.xyz@gmail.com>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://lore.kernel.org/r/20260625-bpf-stack-var-off-zero-v1-v3-1-a068210a761b@gmail.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
bpf_refcount_acquire() increments the refcount at the caller-supplied
pointer plus the refcount field offset, then returns the caller-supplied
pointer unchanged.
The verifier records the return value as a base pointer to the refcounted
object.
bpf_list_pop_front() and bpf_rbtree_remove() can return embedded
graph-node pointers as PTR_TO_BTF_ID | MEM_ALLOC with a fixed offset equal
to the node field offset. Passing such a pointer directly to
bpf_refcount_acquire() currently passes the refcounted-kptr type check.
That makes the runtime operation start from base + node_off while the
verifier models the returned pointer as the object base.
Require refcount-acquire arguments to have zero fixed offset by carrying
the requirement through check_func_arg_reg_off() to __check_ptr_off_reg().
Programs can still acquire a refcount from a graph-node-derived pointer
after normalizing it with container_of().
Fixes: 7c50b1cb76 ("bpf: Add bpf_refcount_acquire kfunc")
Signed-off-by: Yiyang Chen <chenyy23@mails.tsinghua.edu.cn>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Acked-by: Yonghong Song <yonghong.song@linux.dev>
Link: https://lore.kernel.org/r/2f894647f56f71838fdddeb97a3e057ed35ea92e.1782192383.git.chenyy23@mails.tsinghua.edu.cn
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
scx_arena_grow() was left open-coded when 2e05f2fd0d ("sched_ext: Add
scx_arena_to_kaddr() / scx_kaddr_to_arena()") introduced the helper.
Replace the manual bpf_arena_map_kern_vm_start() fetch and #ifdef guard
with scx_arena_to_kaddr(sch, p).
Signed-off-by: Cheng-Yang Chou <yphbchou0911@gmail.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
-----BEGIN PGP SIGNATURE-----
iQJEBAABCAAuFiEEwPw5LcreJtl1+l5K99NY+ylx4KYFAmo9F/IQHGF4Ym9lQGtl
cm5lbC5kawAKCRD301j7KXHgpn0fEADBdgWwlq9P+GLD0rGnWSe7AubvUWfjT+QG
ZrzTLuob57dwbavHJr5TcV3y3VrvuGjRnnBdiSgjTpZR/SC3Q8w+8bsxsqt3hS7i
n55FcvU0WVUy7QPFYK4EUgzsEuy4xz/SWgoYELU7Jhbuff8xE6u8RqXs9r/ZQ/DZ
oY0/+JGc2425cc6RgN5HxvotR7eOtQvt5YoyyGFXNeP1dh4qzTndIbBmyiC2Bfr4
H68gNuGG6R+b6G2TjQsLUwrDE8zfEi8LIIyieaqGaLyR0//MMckJ5pTL/pOpIGiF
QrTXiIPoLQxOJvS0QbTXoHyXuDvlvK4tjChrpe8KJnY6i1DsTmU61Y5sbH/znXyR
p8cv6s/jciBuYVJ0UMDcw/BwmAYKueEmLUv1tp9bzMxx3UYgLHkMlyN3UWUQDMuZ
yu5KvtNNLut15t88tWvAFqdZ+1tWWgle228dXwgdnE2dS/njd+vPqnk+SnzmHacW
ZBNmXgczLm/LzZIJ6wnOMJoxTrOwVFLnI6uNcZwsqPNulZq8piEx1/88Lhvmkix/
+r6XokeNJWfKS8okMPaV4BmBXwtXv610NAUsIAZHo7lc9aTtG3qNGCxHbjJFye4B
VHdenPsx13iwVQdNwqcwK0L91W90eVTkyeLVxM/ej+TdH77JX1s1zM1uypnVo9mX
NxVwLaR4eA==
=ywFi
-----END PGP SIGNATURE-----
Merge tag 'block-7.2-20260625' of git://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux
Pull block fixes from Jens Axboe:
- blk-cgroup locking rework and fixes:
- fix a use-after-free in __blkcg_rstat_flush()
- defer freeing policy data until after an RCU grace period
- defer the blkcg css_put until the blkg is unlinked from
the queue
- unwind the queue_lock nesting under RCU / blkcg->lock
across the lookup, create, associate and destroy paths
- NVMe fixes via Keith:
- Fix a crash and memory leak during invalid cdev teardown,
and related cdev cleanups (Maurizio, John)
- nvmet fixes: handle TCP_CLOSING in the tcp state_change
handler, reject short AUTH_RECEIVE buffers, handle inline
data with a nonzero offset in rdma, fix an sq refcount leak,
and allocate ana_state with the port (Maurizio, Michael,
Bryam, Wentao, Rosen)
- nvme-fc fix to not cancel requests on an IO target before it
is initialized (Mohamed)
- nvme-apple fix to prevent shared tags across queues on Apple
A11 (Nick)
- Various smaller fixes and cleanups (John)
- MD fixes via Yu Kuai:
- raid1/raid10 fixes for writes_pending and barrier reference
leaks on write and discard failures, plus REQ_NOWAIT handling
fixes (Abd-Alrhman)
- raid5 discard accounting and validation, and a batch of fixes
for stripe batch races (Yu Kuai, Chen)
- Protect raid1 head_position during read balancing (Chen)
- block bio-integrity fixes: correct an error injection static key
decrement, fix GFP flag confusion in bio_integrity_alloc_buf(), and
handle REQ_OP_ZONE_APPEND in __bio_integrity_action() (Christoph)
- Fixes for bio_iov_iter_bounce_write(): revert the iov_iter after a
short copy, and respect the iov_iter nofault flag (Qu)
- Invalidate the cached plug timestamp after a task switch, and clear
PF_BLOCK_TS in copy_process() (Usama)
- Fix the IORING_URING_CMD_REISSUE flags check in blkdev_uring_cmd()
(Yitang)
- Remove a redundant plug in __submit_bio() (Wen)
- Don't warn when reclassifying a busy socket lock in nbd (Deepanshu)
* tag 'block-7.2-20260625' of git://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux: (45 commits)
block: handle REQ_OP_ZONE_APPEND in __bio_integrity_action
block: fix GFP_ flags confusion in bio_integrity_alloc_buf
block, bfq: don't grab queue_lock to initialize bfq
mm/page_io: don't nest queue_lock under rcu in bio_associate_blkg_from_page()
blk-cgroup: don't nest queue_lock under blkcg->lock in blkcg_destroy_blkgs()
blk-cgroup: don't nest queue_lock under rcu in bio_associate_blkg()
blk-cgroup: don't nest queue_lock under rcu in blkg_lookup_create()
blk-cgroup: don't nest queue_lock under rcu in blkcg_print_blkgs()
blk-cgroup: delay freeing policy data after rcu grace period
blk-cgroup: protect iterating blkgs with blkcg->lock in blkcg_print_stat()
md/raid5: avoid R5_Overlap races while breaking stripe batches
md/raid5: use stripe state snapshot in break_stripe_batch_list()
blk-cgroup: defer blkcg css_put until blkg is unlinked from queue
blk-cgroup: fix UAF in __blkcg_rstat_flush()
block, bfq: protect async queue reset with blkcg locks
nbd: don't warn when reclassifying a busy socket lock
block: fix incorrect error injection static key decrement
md/raid5: let stripe batch bm_seq comparison wrap-safe
md/raid1: protect head_position for read balance
md/raid1: free r1_bio when REQ_NOWAIT is set and read would block on retry
...
scx_rcu_cpu_stall() previously recorded the detector CPU rather than the
stalled one, and the expedited grace period path had no stalled CPU to
report at all.
Thread a cpumask through panic_on_rcu_stall() and scx_rcu_cpu_stall()
to capture all stalled CPUs. Report cpumask_first() as exit_cpu and the
full CPU list in the exit message. Task-only stalls yield exit_cpu = -1.
Store the stall mask in scx_sched rather than scx_exit_info, keeping the
BPF-visible struct unchanged. scx_dump_state() reads sch->stall_cpus
directly and dumps all stalled CPUs first to avoid losing them to
truncation.
Signed-off-by: Cheng-Yang Chou <yphbchou0911@gmail.com>
Reviewed-by: Paul E. McKenney <paulmck@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
handle_lockup() uses raw_smp_processor_id() for exit_cpu, which is wrong
for two paths:
- scx_hardlockup_irq_workfn() has the hung CPU in a local variable but
irq_work may run elsewhere. Pass the local cpu explicitly.
- scx_rcu_cpu_stall() records the detector CPU rather than the stalled
one. Pass -1 for now. The next patch fixes this properly.
Signed-off-by: Cheng-Yang Chou <yphbchou0911@gmail.com>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
scx_bpf_cpu_rq() exposes rq pointers without requiring the rq lock and
has emitted a deprecation warning since commit
5c48d88fe0 ("sched_ext: deprecation warn for scx_bpf_cpu_rq()").
The supported replacements cover the intended uses:
scx_bpf_locked_rq() for locked rq access and
scx_bpf_cpu_curr() for remote curr lookup.
Remove the kfunc, its BTF registrations, the deprecation warning
state, and the BPF-side prototype and compat fallback.
Signed-off-by: Christian Loehle <christian.loehle@arm.com>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
Reviewed-by: Hongyan Xia <hongyan.xia@transsion.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
scx_kick_syncs is a per-CPU __rcu pointer, so per_cpu_ptr() returns struct
scx_kick_syncs __rcu **. alloc_kick_syncs() and free_kick_syncs() stored it
in a plain struct scx_kick_syncs **ksyncs, which sparse flags as an __rcu
address-space mismatch. Annotate ksyncs to match. Its accesses already go
through rcu_*_pointer().
Fixes: 987e00035c ("sched_ext: Rename pnt_seq to kick_sync")
Reported-by: kernel test robot <lkp@intel.com>
Closes: https://lore.kernel.org/oe-kbuild-all/202606122315.pbnDHP0n-lkp@intel.com/
Signed-off-by: Tejun Heo <tj@kernel.org>
task_can_run_on_remote_rq() operates under the assumption that
p->migration_disabled is stable, i.e. if the kernel observed
is_migration_disabled(p) == true, then the BPF scheduler must have also
been able to see this when dispatching the task, and it's the BPF
scheduler's fault that it tried to dispatch a task with migration
disabled to a CPU other than the task's current CPU.
This assumption does not always hold. It's possible that the BPF
scheduler saw is_migration_disabled(p) == false, while the kernel
observes is_migration_disabled(p) == true in dispatch_to_local_dsq()
-> task_can_run_on_remote_rq().
The crucial thing here is that with CONFIG_PREEMPT_RCU, migration is
disabled while a task is executing a BPF program. So, if there's a
situation where the BPF scheduler checks a task while it's not executing
a BPF program, while the kernel checks it while it is executing one,
the BPF scheduler will be killed through no fault of its own.
Consider the following scenario:
1. SCX task @p is executing on CPU A and CPU A gets preempted by a
higher-priority scheduling class. On entry to __schedule(),
p->migration_disabled == 0.
2. In put_prev_task_scx() @p is enqueued on the BPF scheduler's internal
data structures, making it available for other CPUs to dispatch.
3. CPU B enters ops.dispatch(), pops @p from the BPF scheduler's data
structures, checks is_migration_disabled(p) which returns false,
and dispatches @p to CPU B's local DSQ.
4. On CPU A, @p hasn't been switched out yet. Execution reaches
trace_sched_switch() which enters a BPF program, as the BPF scheduler
hooks into the sched_switch tracepoint to detect idle->fair
transitions. On entry into the BPF program, @p disables migration.
5. CPU B enters finish_dispatch() -> dispatch_to_local_dsq() ->
task_can_run_on_remote_rq() which observes
is_migration_disabled(p) == true, triggering scx_error().
This all happens while holding CPU B's rq lock, so it's not
synchronized with @p switching out.
This patch fixes this by moving the call to task_can_run_on_remote_rq()
after @p's rq lock is acquired in dispatch_to_local_dsq(). This way, we
synchronize with @p switching out, since @p holds its rq lock all
the way until it's switched out. Thus, any BPF programs that are called
between put_prev_task_scx() and the end of the context switch are
guaranteed to have finished and cannot influence p->migration_disabled.
Also add a lockdep assertion in task_can_run_on_remote_rq() which
ensures the task rq lock is held if enforce == true.
Signed-off-by: Kuba Piecuch <jpiecuch@google.com>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
When a task leaves the BPF scheduler's control, p->scx.dsq_vtime and
p->scx.slice keep whatever values they last held. The slice value is
core-managed and is refilled on the next enqueue, but dsq_vtime is owned
by the BPF scheduler and is never cleared by the core, so a task that
leaves SCX and later returns carries a stale dsq_vtime across the
round-trip.
The stale values are also visible to other SCX schedulers that inspect
the scx fields of non-SCX tasks.
Fix this by resetting both dsq_vtime and slice in scx_disable_task(),
after ops.disable(), so the BPF scheduler can still observe the task's
final values and non-SCX tasks do not retain stale SCX state.
Signed-off-by: Andrea Righi <arighi@nvidia.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
-----BEGIN PGP SIGNATURE-----
iQJIBAABCgAyFiEEgMe7l+5h9hnxdsnuWYigwDrT+vwFAmo7CrQUHGJoZWxnYWFz
QGdvb2dsZS5jb20ACgkQWYigwDrT+vxsvw/7B4rHdVs1/LERE4H0bBKKNz7mr6g/
6JOBDYc9ERgZpDgAwBKm+uuFzg8KLK0lf5PU7p2zNxjMVia+c4AMTHw/icN91yfL
NZLP013GQmZ6s1r290AMnVtUqd1hNm2R+r4nzXTm85y9B25tZIiJj3gssIXPKDTE
74rrRiNVO9+zj8JTED6kbSoAq40iv94Ux9VnumHy6kF1izsFAht7eJmkICgeOLRX
okBq86Tqivwv85XpqOQZTHudBwvS91HHsAdClW9UcXyRQo3RJAcba2tkglfZ2Ozo
RbfqxxKNuW/3TTziOflfzm/oImZuMQdlvu98udMQ+LacOmLN+SWPdyVFly9X87i+
ZGH4bKxtqcba8EYFZ+VRrG3JNuOWwcfmAPC93WhuwhywJ6evRg9Wufffss6GLUmH
g9id/KbMI0RRkYnne7w9nmmCwLge0/3klhf5dwyZZ/q9IyKPUqjyagdbjFA4Esp5
wLK1H9645pXtZtj5h8qMlZQ+rrDt/IYWqsCo15pB1Z4y3Lpb+JaYsHOS+Ye/3PDA
0MDeCoTvK4scZGtM6l10jn3fdGVy6EL+3thteQz/+FHzmMEXp53TnrQTBhKtdGc0
RfPLZEMuFCjbr+6u6bSu3thD1eH4TgQHIWY1Fl4FdczXzc6+A6blBY4BSZAqX8kU
xjNfpptAieR4BtM=
=h+xD
-----END PGP SIGNATURE-----
Merge tag 'pci-v7.2-changes' of git://git.kernel.org/pub/scm/linux/kernel/git/pci/pci
Pull pci updates from Bjorn Helgaas:
"Enumeration:
- Remove MPS/MRRS Kconfig settings (CONFIG_PCIE_BUS_*) that worked
around a WiFi device defect; use a quirk or boot-time
"pci=pcie_bus_tune_*" kernel parameter instead (Bjorn Helgaas)
- Always lift 2.5GT/s restriction in PCIe failed link retraining to
avoid clamping a link to 2.5GT/s after hot-plug changes the device
(Maciej W. Rozycki)
- Request bus reassignment when not probe-only to fix an enumeration
regression on Marvell CN106XX and possibly other DT-based systems
(Ratheesh Kannoth)
- Fix procfs race between pci_proc_init() and pci_bus_add_device()
that resulted in 'proc_dir_entry ... already registered' warnings
and pointer corruption (Krzysztof Wilczyński)
- Fix sysfs race that causes 'duplicate filename' warnings and boot
panics by converting PCI resource files to static attributes
(Krzysztof Wilczyński)
- Expose sysfs 'resourceN_resize' attributes only on platforms with
PCI mmap (Krzysztof Wilczyński)
- Require CAP_SYS_ADMIN to write to sysfs 'resourceN_resize'
attributes (Krzysztof Wilczyński)
- Add security_locked_down(LOCKDOWN_PCI_ACCESS) to alpha PCI resource
mmap path to match the generic path (Krzysztof Wilczyński)
- Use kstrtobool() to parse the 'rom' attribute input to avoid the
unexpected behavior of enabling the ROM when writing '0' with no
trailing newline (Krzysztof Wilczyński)
Resource management:
- Improve resource claim logging for debuggability (Ilpo Järvinen)
- Clean up several uses of const parameters (Ilpo Järvinen)
- Check option ROM header signatures and lengths before accessing to
avoid page faults and alignment faults (Guixin Liu)
ASPM:
- Don't reconfigure ASPM when entering low-power D-state; only do it
when returning back to D0 (Carlos Bilbao)
Power management:
- During suspend, set power state to 'unknown' for all devices, not
just those with drivers (Lukas Wunner)
- Skip restoring Resizable BARs and VF Resizable BARs if device
doesn't respond to config reads, to avoid invalid array accesses
(Marco Nenciarini)
- Add pci_suspend_retains_context() so drivers can tell whether
devices retain internal state across suspend/resume, since some
platforms reset devices on suspend; use this in nvme to avoid
issues on Qcom RCs (Manivannan Sadhasivam)
Power control:
- Only to power on/off devices that actually support power control to
avoid poking at incompatible devices mentioned in DT (Manivannan
Sadhasivam)
Virtualization and resets:
- Log device readiness timeouts as errors, not warnings, because the
device is likely unusable in this case (Bjorn Helgaas)
- Wait for device readiness after soft reset (D3hot ->
D0uninitialized transition), when the device may respond with
Request Retry Status (RRS) if it needs more time to initialize
(Bjorn Helgaas)
- Drop unnecessary retries when restoring BARs because resets should
now already include all required delays (Lukas Wunner)
- Avoid FLR for MediaTek MT7925 WiFi, where FLR fails after a VM
terminates uncleanly (Jose Ignacio Tornos Martinez)
- Avoid SBR for Qualcomm WCN6855/WCN7850 WiFi, SDX62/SDX65 modems,
which seem not to support it correctly (Jose Ignacio Tornos
Martinez)
Peer-to-peer DMA:
- Prevent P2PDMA as well as CPU access to non-mappable BARs, e.g.,
s390 ISM BARs (Matt Evans)
- Add Intel QAT, DSA, IAA devices to whitelist (Lukas Wunner)
Endpoint framework:
- Add endpoint controller APIs for use by function drivers to
discover auxiliary blocks like DMA engines (Koichiro Den)
- Remember DesignWare eDMA engine base/size and expose them via the
EPC aux-resource API (Koichiro Den)
- Add endpoint embedded doorbell fallback, used if MSI allocation
fails (Koichiro Den)
- Validate BAR index and remove dead BAR read in endpoint doorbell
test (Carlos Bilbao)
- Unwind MSI/MSI-X vectors if NTB initialization fails part-way
through (Koichiro Den)
- Cache sleepable pci_irq_vector() value at ISR setup to avoid
calling it from hardirq context (Koichiro Den)
- Call sleepable pci_epc_raise_irq() from a work item instead of
atomic context, e.g., when setting bits in NTB peer doorbells in
the ntb_peer_db_set() path (Koichiro Den)
- Report 0-based vNTB doorbell vector to account for link event 0 and
historically skipped slot 1 (Koichiro Den)
- Prevent configfs writes to vNTB db_count and other values that are
already in use after EPC attach (Koichiro Den)
- Account for vNTB db_valid reserved slots (link event 0 and
historically skipped slot 1) so they don't appear as valid
doorbells (Koichiro Den)
- Implement vNTB .db_vector_count()/mask() for doorbells so clients
can use multiple vectors and avoid thundering herds (Koichiro Den)
- Report 0-based NTB doorbell vector to account for link event 0 and
historically skipped slot 1 (Koichiro Den)
- Fix doorbell bitmask and IRQ vector handling to clear only
specified bits, use the correct vector for non-contiguous Linux IRQ
numbers, and validate incoming vectors (Koichiro Den)
- Implement NTB .db_vector_count()/mask() for doorbells so clients
can use multiple vectors (Koichiro Den)
Native PCIe controller infrastructure:
- Add pci_host_common_link_train_delay() for the mandatory delay
after > 5GT/s Link training completes and use it for cadence HPA,
j721e, LGA; dwc; aardvark, mediatek-gen3, rzg3s (Hans Zhang)
- Protect root bus removal with rescan lock in altera, brcmstb,
cadence, dwc, iproc, mediatek, plda, rockchip to prevent
use-after-free or crashes when racing with sysfs rescan or hotplug
(Hans Zhang)
- Add pci_host_common_parse_ports() for use by any native driver to
parse Root Port properties (per-Link features like width, speed,
PHY, power and reset control, etc should be described in Root Port
stanzas, not the host bridge; currently only reset GPIOs
implemented) (Sherry Sun)
New native PCIe controller drivers:
- Add DT binding and driver for UltraRISC DP1000 PCIe controller
(Xincheng Zhang, Jia Wang)
Altera PCIe controller driver:
- Do not dispose of the parent IRQ mapping, which belongs to the
parent interrupt controller (Mahesh Vaidya)
- Fix chained IRQ handler ordering issue and resource leaks on probe
failure (Mahesh Vaidya)
AMD MDB PCIe controller driver:
- Assert PERST# on shutdown so any connected Endpoints are held in
reset during shutdown (Sai Krishna Musham)
Amlogic Meson PCIe controller driver:
- Propagate devm_add_action_or_reset() failure to fix probe error
path (Shuvam Pandey)
- Add .remove() callback to deinitialize the host bridge and power
off the PHY (Shuvam Pandey)
Broadcom iProc PCIe controller driver:
- Restore .map_irq() assignment; its removal broke INTx on the iproc
platform bus driver (Mark Tomlinson)
Broadcom STB PCIe controller driver:
- No change, but products using certain WiFi devices may be affected
by removal of CONFIG_PCIE_BUS_* (see above)
Freescale i.MX6 PCIe controller driver:
- Move IMX6SX_GPR12_PCIE_TEST_POWERDOWN handling into the core reset
functions (Richard Zhu)
- Assert PERST# before enabling regulators to ensure that even if
power is enabled, endpoint stays inactive until REFCLK is stable
(Sherry Sun)
- Parse reset properties in Root Port nodes (falling back to host
bridge) to help support Key E connectors and the pwrctrl framework
(Sherry Sun)
- Configure i.MX95 REF_USE_PAD before PHY reset (Richard Zhu)
- Assert i.MX95 ref_clk_en after reference clock stabilizes (Richard
Zhu)
- Integrate new pwrctrl API for DTs with Root Port-level power
supplies (Sherry Sun)
Intel Gateway PCIe controller driver:
- Enable clock before PHY init for correct ordering (Florian Eckert)
- Add .start_link() callback so the driver works again (Florian
Eckert)
- Stop overwriting the ATU base address discovered by
dw_pcie_get_resources() (Florian Eckert)
- Add DT 'atu' region since this is hardware-specific, and fall back
to driver default if lacking (Florian Eckert)
Loongson PCIe controller driver:
- Ignore downstream devices only on internal bridges to avoid
Loongson hardware issue (Rong Zhang)
- Quirk old Loongson-3C6000 bridges that advertise incorrect
supported link speeds (Ziyao Li)
Marvell MVEBU PCIe controller driver:
- Use fixed-width interrupt masks to avoid truncation in 64-bit
builds (Rosen Penev)
MediaTek PCIe controller driver:
- Use FIELD_PREP() to fix incorrect operator precedence in
PCIE_FTS_NUM_L0 (Li RongQing)
- Fix IRQ domain leak when port fails to enable (Manivannan
Sadhasivam)
- Use actual physical address for MSI message address instead of
virt_to_phys() (Manivannan Sadhasivam)
- Add EcoNet EN7528 to DT binding (Caleb James DeLisle)
MediaTek PCIe Gen3 controller driver:
- Deassert PCIE_PHY_RSTB so REFCLK is stable for at least 100ms
(PCIE_T_PVPERL_MS) before deasserting PERST# (Jian Yang)
- Add .shutdown() to assert PERST# before powering down device (Jian
Yang)
- Do full device power down on removal, including asserting PERST#,
when removing driver (Chen-Yu Tsai)
- Fix a 'failed to create pwrctrl devices' error message that was
inadvertently skipped (Chen-Yu Tsai)
NVIDIA Tegra194 PCIe controller driver:
- Program the DesignWare PORT_AFR L1 entrance latency based on the
'aspm-l1-entry-delay-ns' DT property (Manikanta Maddireddy)
Qualcomm PCIe controller driver:
- Add Eliza SoC compatible in DT binding (Krishna Chaitanya Chundru)
- Set max OPP during resume so DBI register accesses don't fail with
NoC errors (Qiang Yu)
- Add pci_host_common_d3cold_possible() to determine whether
downstream devices are already in D3hot and wakeup-enabled devices
are capable of generating PME from D3cold (Krishna Chaitanya
Chundru)
- Add .get_ltssm() callback to get the LTSSM status without DBI,
since DBI may be inaccessible after PME_Turn_Off (Krishna Chaitanya
Chundru)
- Power down PHY via PARF_PHY_CTRL before disabling rails/clocks to
avoid power leakage (Krishna Chaitanya Chundru)
- Decide whether suspend should put the link in L2 and power down
using pci_host_common_d3cold_possible() instead of checking whether
ASPM L1 is enabled (Krishna Chaitanya Chundru)
- Add qcom D3cold support to tear down interconnect bandwidth and OPP
votes (Krishna Chaitanya Chundru)
- Handle unsupported mixed PERST#/PHY DT configurations, e.g., PHY in
RP node while PERST# is in the RC node, but warn about the DT issue
(Qiang Yu)
- Program T_POWER_ON based on DT 't-power-on-us' property in case
hardware advertises incorrect values (Krishna Chaitanya Chundru)
- Disable ASPM L0s for SA8775P (Shawn Guo)
- Initialize DWC MSI lock for firmware-managed ECAM hosts, which
don't use the dw_pcie_host_init() path that initializes the lock
(Yadu M G)
Renesas RZ/G3S PCIe controller driver:
- Add RZ/V2N DT support (Lad Prabhakar)
SOPHGO PCIe controller driver:
- Add 'dma-coherent' DT property for sg2042-pcie driver (Han Gao)
Synopsys DesignWare PCIe controller driver:
- Apply ECRC TLP Digest workaround for all DesignWare cores prior to
5.10a, not just 4.90a and 5.00a (Manikanta Maddireddy)
- Use common struct dw_pcie 'mode' rather than duplicating it in
artpec6, dra7xx, dwc-pcie, and keembay driver structs (Hans Zhang)
- Use DEFINE_SHOW_ATTRIBUTE for ltssm_status debugfs to reduce
boilerplate and fix a seq_file memory leak by including a
.release() callback (Hans Zhang)
- Fix a signedness bug in fault injection test code (Dan Carpenter)
- Avoid NULL pointer dereference when tearing down debugfs for
controller that lacks RAS DES capability (Shuvam Pandey)
MicroSemi Switchtec management driver:
- Add Gen6 Device IDs (Ben Reed)
Miscellaneous:
- Remove unused gpio.h include from amd-mdb, designware-plat, fu740,
visconti drivers (Andy Shevchenko)
- Fix typos in documentation (josh ziegler)
- Use FIELD_MODIFY() instead of open-coding it (Hans Zhang)"
* tag 'pci-v7.2-changes' of git://git.kernel.org/pub/scm/linux/kernel/git/pci/pci: (168 commits)
PCI/sysfs: Use kstrtobool() to parse the ROM attribute input
PCI/sysfs: Limit BAR resize attribute scope to platforms with PCI mmap
PCI/sysfs: Remove pci_create_legacy_files() and pci_sysfs_init()
PCI/sysfs: Convert legacy I/O and memory attributes to static definitions
PCI/sysfs: Add __weak pci_legacy_has_sparse() helper
alpha/PCI: Compute legacy size in pci_mmap_legacy_page_range()
PCI: Add macros for legacy I/O and memory address space sizes
PCI/sysfs: Remove pci_{create,remove}_sysfs_dev_files()
alpha/PCI: Convert resource files to static attributes
alpha/PCI: Add static PCI resource attribute macros
alpha/PCI: Remove WARN from __pci_mmap_fits() and __legacy_mmap_fits()
alpha/PCI: Fix __pci_mmap_fits() overflow for zero-length BARs
alpha/PCI: Use PCI resource accessor macros
alpha/PCI: Use BAR index in sysfs attr->private instead of resource pointer
alpha/PCI: Add security_locked_down() check to pci_mmap_resource()
PCI/sysfs: Limit pci_sysfs_init() late_initcall compile scope
PCI/sysfs: Add stubs for pci_{create,remove}_sysfs_dev_files()
PCI/sysfs: Warn about BAR resize failure in __resource_resize_store()
PCI/sysfs: Convert PCI resource files to static attributes
PCI/proc: Fix race between pci_proc_init() and pci_bus_add_device()
...
As reported by sashiko [1], cpuset_update_tasks_nodemask() will do
mpol_rebind_mm() and possibly cpuset_migrate_mm() for all threads of
a multithreaded process. Since commit 3df9ca0a2b ("cpuset: migrate
memory only for threadgroup leaders"), cpuset_attach() had been updated
to rebind and migrate memory only for threadgroup leaders to mark the
group leader as the owner of the mm_struct.
To be consistent and avoid unnecessary performance overhead for heavily
multithreaded processes, follow the cpuset_attach() example and perform
memory rebind and migration only for threadgroup leaders.
Also add a paragraph in cgroup-v2.rst under cpuset.mems that the
threadgroup leader is the memory owner of that threadgroup. Therefore
the non-leading threads shouldn't be in other cgroups whose "cpuset.mems"
doesn't fully overlap that of the group leader.
[1] https://sashiko.dev/#/patchset/20260621032816.1806773-1-longman%40redhat.com
Signed-off-by: Waiman Long <longman@redhat.com>
Reviewed-by: Ridong Chen <ridong.chen@linux.dev>
Signed-off-by: Tejun Heo <tj@kernel.org>
As reported by sashiko [1], cpuset_hotplug_update_tasks() may perform
unnecessary task iteration and updating of tasks' CPU and node masks
when mems_allowed and/or cpus_allowed are not set in cpuset v2. It is
due to the fact that the temporary new_cpus and new_mems masks do not
inherit parent's effective_cpus/mems when they are empty which is the
expected behavior for cpuset v2 since commit 4ec22e9c5a ("cpuset:
Enable cpuset controller in default hierarchy").
Fix that and avoid unnecessary work by enhancing
compute_effective_cpumask() to add the empty cpumask check
and inheriting the parent's versions if empty when in v2. A new
compute_effective_nodemask() helper is also added to perform a similar
function for new effective_mems.
Add new test_cpuset_prs.sh test cases to confirm that effective_cpus
will inherit the parent's version if cpuset.cpus is empty.
[1] https://sashiko.dev/#/patchset/20260621032816.1806773-1-longman%40redhat.com
Suggested-by: Ridong Chen <ridong.chen@linux.dev>
Fixes: 4ec22e9c5a ("cpuset: Enable cpuset controller in default hierarchy")
Signed-off-by: Waiman Long <longman@redhat.com>
Reviewed-by: Ridong Chen <ridong.chen@linux.dev>
Signed-off-by: Tejun Heo <tj@kernel.org>
The deprecation notices for direct p->scx.slice/dsq_vtime writes and for
ops->cpu_acquire/release() use plain pr_warn(), so they repeat on every
scheduler (re)load and can flood the kernel log.
The slice/dsq_vtime notice is the worst offender: it is emitted from the
BPF verifier's btf_struct_access callback, which is re-evaluated as the
verifier explores program paths, so a single scheduler load can print it
many times -- hundreds of lines on some hosts, dozens within the same
second.
Switch both notices to pr_warn_ratelimited() so each deprecation is still
reported but bursts no longer spam the log, and add the missing newline
to the slice/dsq_vtime message.
Signed-off-by: Breno Leitao <leitao@debian.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
- Fix an MM-CID race that can cause an OOB write
(Rik van Riel)
- Fix a debugobjects OOM handling race (Thomas Gleixner)
Signed-off-by: Ingo Molnar <mingo@kernel.org>
-----BEGIN PGP SIGNATURE-----
iQJFBAABCgAvFiEEBpT5eoXrXCwVQwEKEnMQ0APhK1gFAmo6t3URHG1pbmdvQGtl
cm5lbC5vcmcACgkQEnMQ0APhK1iXwg/8CPIGhMJBMG97dl+m3Iq6RlzSNCTU6+Ho
AClN/+jtduU/sEedvFC150Yvjhe1p78hHkVuFyu4hvZaevFg4v70vz0TpsfOWS4p
wvKxEZpuhmDJTPBx8ZWq8oD+ZgKr0VVr0PQcHHiHggDKbb5UEY4H8SDHSx8bcJqg
2iq4NuOYZX+PRAmsPg0MAhpnfHiYkz2HPoLwDXkol0G53WMAzsKxroLO9bXni7WJ
PI+ru3WmecjqTaDDUgQO0LabLnl7vuEjsIQEK4tfQqlMWwepsrURcxNuL/61S38d
cmn4b2vnJ8G2pjLiKKv4imVdfrUh1TsoG7JakezTad+OQeKlThbkmdIST/FW5ey7
roI8Ux1+/S1G9orgbI+buNC1G5/Bi4RH7Y20DzjvlWRXc89OVsybBJn6owIgZgSS
ksGnsfBxbds9ErtKdp5aGhtJ9IKfIAO9OsoJh9gNxIyuqV0F03FDI5Tq7zuw/zPz
faEFHH9uTlBjEQbJAlpJ2Z4M4nNyMj8OMhZgw/kxq6ogJDa49kr0hnGdhd2b18v6
kNXkXERPoreKdkjeRC4/qwuhIxMSLISHDZEEt2gk2Rrl8ly62rMhOfn7BV4ffHnO
6/YasPW1iBLH4h6l8yoi3k8YEmjO0s+mEoMINxkdChJWot7EplsxUyWcziRMxkpi
Be5441cyO9o=
=bywo
-----END PGP SIGNATURE-----
Merge tag 'core-urgent-2026-06-23' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
Pull misc core fixes from Ingo Molnar:
- Fix an MM-CID race that can cause an OOB write (Rik van Riel)
- Fix a debugobjects OOM handling race (Thomas Gleixner)
* tag 'core-urgent-2026-06-23' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip:
debugobjects: Plug race against a concurrent OOM disable
sched/mmcid: Fix OOB clear_bit when CID is MM_CID_UNSET in fixup path
Follow-up to the v7.2 sched_ext pull. Pure source reorganization with no
functional change: the kernel/sched/ext* files move into a new
kernel/sched/ext/ subdirectory, and the headers and sources are made
self-contained so editor tooling can parse each file on its own.
-----BEGIN PGP SIGNATURE-----
iIQEABYKACwWIQTfIjM1kS57o3GsC/uxYfJx3gVYGQUCajmn3w4cdGpAa2VybmVs
Lm9yZwAKCRCxYfJx3gVYGYRyAP4qs7Xlkva8ppSiJX/nja4OpQixpCZ1zi1yPDGK
iaJ+tQEA+bnkb2QI0Cwd9l+kY12/kl8gHUfqG0UPVyyEOvAeXgs=
=MvGz
-----END PGP SIGNATURE-----
Merge tag 'sched_ext-for-7.2-1' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/sched_ext
Pull sched_ext tree reorg from Tejun Heo:
"Pure source reorganization with no functional change:
- the kernel/sched/ext* files move into a new kernel/sched/ext/
subdirectory
- the headers and sources are made self-contained so editor tooling
can parse each file on its own"
* tag 'sched_ext-for-7.2-1' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/sched_ext:
sched_ext: Move shared helpers from ext.c into internal.h and cid.h
sched_ext: Make kernel/sched/ext/ sources self-contained for clangd
sched_ext: Move sources under kernel/sched/ext/
BPF LSM programs can currently attach to xfrm_decode_session(). That
hook may return an error, but security_skb_classify_flow() calls it
from a void path and triggers BUG_ON() if an error is returned.
Disable BPF attachment to the hook to prevent a BPF LSM program from
turning packet classification into a full panic.
Fixes: 9e4e01dfd3 ("bpf: lsm: Implement attach, detach and execution")
Signed-off-by: Bradley Morgan <include@grrlz.net>
Link: https://lore.kernel.org/r/20260619130305.27779-1-include@grrlz.net
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
When the BPF verifier processes a context load of an LSM hook return
value, it calls __mark_reg_s32_range() to narrow the register to the
hook's valid range. However, __mark_reg_s32_range() intersects the new
range with the register's existing bounds using max_t()/min_t() rather
than replacing them.
If the destination register carries stale bounds from a prior instruction
(e.g. BPF_MOV64_IMM), the intersection can produce a range narrower than
reality. The verifier then believes it knows the register's exact value,
while at runtime the actual hook return value is loaded, creating a
verifier/runtime mismatch that can be used to bypass BPF memory safety
checks.
The else branch already calls mark_reg_unknown() to reset register state
before any narrowing. Apply the same reset in the is_retval path so
stale bounds are cleared before __mark_reg_s32_range() intersects.
Fixes: 5d99e198be ("bpf, lsm: Add check for BPF LSM return value")
Cc: stable@vger.kernel.org
Signed-off-by: Tristan Madani <tristan@talencesecurity.com>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://lore.kernel.org/r/20260622230123.3695446-2-tristmd@gmail.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
idle.c and cid.c are included into build_policy.c together with ext.c and
use helpers that ext.c defines. Because the helpers live in ext.c, the two
files can not parse as standalone units and clangd reports errors in them.
Move the helpers to the headers they belong to. The op-dispatch macros and
helpers plus scx_parent() to internal.h, and scx_cpu_arg()/scx_cpu_ret() to
cid.h. No functional change. idle.c and cid.c now parse clean standalone.
Suggested-by: Peter Zijlstra <peterz@infradead.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
The sources under kernel/sched/ext/ build as a single translation unit:
build_policy.c includes the source files and headers. An LSP/clangd editor
parses each as a standalone unit, sees no types, and reports a flood of
errors.
Give each header its dependencies and include guard, and have each source
include the headers it uses.
ext.c, arena.c and the ext headers now parse clean standalone. idle.c and
cid.c still reference a few macros and helpers defined in ext.c. The next
patch moves those to shared headers.
Suggested-by: Peter Zijlstra <peterz@infradead.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
__clean_func_state() cleans dead stack slots in 4-byte halves. When the
high half of a STACK_SPILL slot is dead and the low half remains live,
cleanup converts the live low half to STACK_MISC or STACK_ZERO and clears
the saved spilled_ptr metadata.
That conversion is safe only for scalar spills. For a pointer spill, this
metadata clear lets a later 32-bit fill from the still-live half avoid the
normal non-scalar register-fill check and be treated as an ordinary scalar
stack read.
Leave non-scalar spill slots intact in this half-live shape. This is
conservative for pruning and preserves the existing
check_stack_read_fixed_off() rejection path for partial fills from pointer
spills.
Fixes: be23266b4a ("bpf: 4-byte precise clean_verifier_state")
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Signed-off-by: Nuoqi Gui <gnq25@mails.tsinghua.edu.cn>
Link: https://lore.kernel.org/r/20260617-f01-06-half-slot-pointer-spill-v2-1-42b9cdc3cf64@mails.tsinghua.edu.cn
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
A comment in kernel/irq/msi.c incorrectly refers to
CONFIG_PCI_MSI_ARCH_FALLBACK instead of CONFIG_PCI_MSI_ARCH_FALLBACKS.
Correct it.
Discovered while searching for CONFIG_* symbols referenced in code but
not defined in any Kconfig file.
Signed-off-by: Ethan Nelson-Moore <enelsonmoore@gmail.com>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Link: https://patch.msgid.link/20260613213544.90613-1-enelsonmoore@gmail.com
The sched_ext sources had grown to ten ext* files directly under
kernel/sched/. Move them into a new kernel/sched/ext/ subdirectory and drop
the now-redundant ext_ prefix. ext.c/h keep their names.
kernel/sched/ext.{c,h} -> kernel/sched/ext/ext.{c,h}
kernel/sched/ext_internal.h -> kernel/sched/ext/internal.h
kernel/sched/ext_types.h -> kernel/sched/ext/types.h
kernel/sched/ext_idle.{c,h} -> kernel/sched/ext/idle.{c,h}
kernel/sched/ext_cid.{c,h} -> kernel/sched/ext/cid.{c,h}
kernel/sched/ext_arena.{c,h} -> kernel/sched/ext/arena.{c,h}
The include paths in build_policy.c and sched.h, the MAINTAINERS glob, and a
few documentation and comment references are updated to match. No code or
symbol changes.
Suggested-by: Linus Torvalds <torvalds@linux-foundation.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
- consolidate s390 idle time accounting by moving all CPU time tracking
to the architecture backend and eliminate the mix of architecture-
specific and common code accounting
- Add missing EXPORT_SYMBOL_GPL() to kcpustat_field_idle() and
kcpustat_field_iowait() functions
- Finalize ptep_get() conversion by replacing direct page table entry
dereferencing with proper accessors (ptep_get(), pmdp_get(), etc.)
- Explicitly check the buffer length in PKEY_VERIFYPROTK ioctl and
pkey_pckmo implementations and fail if the length is exceeded
-----BEGIN PGP SIGNATURE-----
iI0EABYKADUWIQQrtrZiYVkVzKQcYivNdxKlNrRb8AUCajlD5xccYWdvcmRlZXZA
bGludXguaWJtLmNvbQAKCRDNdxKlNrRb8K6cAP48oh4X6XLdK3Eohkm9WjkI0mxZ
Qh/tkI1o1GcckwU/zAD+O0LKfJIcb4mo0cH7FgTOJTUCbmPBX711zjom2t9Bvww=
=BTXg
-----END PGP SIGNATURE-----
Merge tag 's390-7.2-2' of git://git.kernel.org/pub/scm/linux/kernel/git/s390/linux
Pull more s390 updates from Alexander Gordeev:
- consolidate s390 idle time accounting by moving all CPU time tracking
to the architecture backend and eliminate the mix of architecture-
specific and common code accounting
- Add missing EXPORT_SYMBOL_GPL() to kcpustat_field_idle() and
kcpustat_field_iowait() functions
- Finalize ptep_get() conversion by replacing direct page table entry
dereferencing with proper accessors (ptep_get(), pmdp_get(), etc.)
- Explicitly check the buffer length in PKEY_VERIFYPROTK ioctl and
pkey_pckmo implementations and fail if the length is exceeded
* tag 's390-7.2-2' of git://git.kernel.org/pub/scm/linux/kernel/git/s390/linux:
s390/pkey: Check length in pkey_pckmo handler implementation
s390/pkey: Check length in PKEY_VERIFYPROTK ioctl
s390/idle: Add missing EXPORT_SYMBOL_GPL()
s390/mm: Complete ptep_get() conversion
s390/idle: Remove idle time and count sysfs files
s390/idle: Provide arch specific kcpustat_field_idle()/kcpustat_field_iowait()
s390/irq/idle: Use stcke instead of stckf for time stamps
s390/timex: Move union tod_clock type to separate header
The compat version of settimeofday() uses '>' instead of '>=' when
validating tv_usec against USEC_PER_SEC, allowing the value 1000000 to pass
the check. After the subsequent conversion to nanoseconds (tv_nsec *=
NSEC_PER_USEC), this results in tv_nsec == NSEC_PER_SEC, which violates the
timespec invariant that tv_nsec must be strictly less than NSEC_PER_SEC.
The native settimeofday() was already fixed in commit ce4abda5e1 ("time:
Fix off-by-one in settimeofday() usec validation"), but the compat
counterpart was missed.
Fix it by using '>=' to reject tv_usec values outside the valid range [0,
USEC_PER_SEC - 1].
Fixes: 5e0fb1b57b ("y2038: time: avoid timespec usage in settimeofday()")
Signed-off-by: Wang Yan <wangyan01@kylinos.cn>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Acked-by: Arnd Bergmann <arnd@arndb.de>
Cc: stable@vger.kernel.org
Link: https://patch.msgid.link/20260622103348.120255-1-wangyan01@kylinos.cn
replace_effective_prog() and purge_effective_progs() located the slot in
the effective array by walking the program hlist and counting entries
linearly. That count does not match the array layout: compute_effective_
progs() places BPF_F_PREORDER programs at the front (ancestor cgroup
first, attach order within a cgroup) and the rest after them (descendant
cgroup first). So when a preorder program is present, the linear hlist
position no longer equals the program's index in the effective array.
For replace_effective_prog() (bpf_link_update()) this overwrote the
wrong slot, corrupting the effective order. For purge_effective_progs(),
it could dummy out a slot belonging to a different program and leave the
detached program in the array while bpf_prog_put() drops its reference,
i.e. a use-after-free.
Fix both by replaying compute_effective_progs()'s placement (including
the per-cgroup preorder reversal) in a shared effective_prog_pos()
helper. Identify the entry by its struct bpf_prog_list pointer rather
than by (prog, link) value, so the lookup resolves to exactly the
attachment the syscall selected even when the same bpf_prog is attached
to several cgroups in the hierarchy.
Fixes: 4b82b181a2 ("bpf: Allow pre-ordering for bpf cgroup progs")
Signed-off-by: Amery Hung <ameryhung@gmail.com>
Acked-by: Yonghong Song <yonghong.song@linux.dev>
Link: https://lore.kernel.org/r/20260619063520.2690547-2-ameryhung@gmail.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
When struct prog_assoc_struct_ops was added,
BPF_PROG_ASSOC_STRUCT_OPS_LAST_FIELD referenced prog_fd instead of the
actual last field, flags.
Fixes: b5709f6d26 ("bpf: Support associating BPF program with struct_ops")
Signed-off-by: Thiébaud Weksteen <tweek@google.com>
Reviewed-by: Jakub Sitnicki <jakub@cloudflare.com>
Acked-by: Yonghong Song <yonghong.song@linux.dev>
Link: https://lore.kernel.org/r/20260618040934.4113938-1-tweek@google.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
llvm commit [1] allowed attaching type tag records to modifier BTF
records. This is useful for using typedefs that encompass a base type
and a type tag, e.g.:
typedef struct rbtree __arena rbtree_t;
Modify btf_check_type_tags() so that it allows this sequence of records.
The function now only checks for record loops in BTF modifier record
chains. Rename to btf_check_modifier_chain_length to reflect this.
Also expand the BTF modifier traversal code to take into account that
type record can be interleaved with other modifier records. In effect
this means traversing all modifiers to collect the type tags.
Also modify existing selftests to now accept modifier records (const,
typedef) that point to type tag records.
[1] https://github.com/llvm/llvm-project/pull/203089
Signed-off-by: Emil Tsalapatis <emil@etsalapatis.com>
Link: https://lore.kernel.org/r/20260616061454.7869-1-emil@etsalapatis.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
When BPF_WRITE goes through a PTR_TO_BTF_ID register, check_ptr_to_btf_access()
delegates to env->ops->btf_struct_access(). Most implementations
(bpf_scx_btf_struct_access, tc_cls_act_btf_struct_access, etc.) return
-EACCES for disallowed fields without logging anything, so the verifier
rejects the program with an empty message. For example a scx program doing
1: R1=trusted_ptr_task_struct()
...
4: (7b) *(u64 *)(r1 +0) = r2
verification time 83 usec
the program is rejected
leaves the user guessing which field is off-limits.
Emit verbose message.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Reviewed-by: Emil Tsalapatis <emil@etsalapatis.com>
Acked-by: Yonghong Song <yonghong.song@linux.dev>
Link: https://lore.kernel.org/r/20260615232146.5491-1-alexei.starovoitov@gmail.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
This patch is a follow up to recent implementation of
stack_map_get_build_id_offset_sleepable() [1].
stack_map_get_build_id_offset() and its sleepable variant each cached
only the last successfully resolved VMA, with separate bookkeeping in
each function. A run of IPs in a VMA with no usable build ID will
repeat the lookup for every frame: find_vma() in the non-sleepable
path, a VMA lock and a blocking build_id_parse_file() in the sleepable.
Factor the per-call cache into a shared struct stack_map_build_id_cache
with two independent slots [2][3], used by both functions:
* resolved - last VMA that produced a build ID (file, build_id and
range), reused to skip the lookup and the parse;
* unresolved - last VMA with no usable build ID (range only), reused to
emit a raw IP without another lookup or parse.
Keeping the slots independent means a build-ID-less VMA no longer evicts
the last resolved build ID, so a trace alternating between a binary and a
region without one stops re-resolving the binary on every return.
The shared lookup tests [vm_start, vm_end), matching the sleepable path;
the non-sleepable path previously reused a build ID for ip == vm_end
(range_in_vma() is inclusive) and now re-resolves it correctly.
[1] https://lore.kernel.org/bpf/20260525223948.1920986-1-ihor.solodrai@linux.dev/
[2] https://lore.kernel.org/bpf/CAEf4Bza2fRDGhLQoPE-EzM7F34xaEJfi5Exmxb-iWVUN3F06=g@mail.gmail.com/
[3] https://lore.kernel.org/bpf/CAEf4BzZXJFr=1iiVx937ht=4PYQkQHg=eFk810zhMDzXQG3ihw@mail.gmail.com/
Suggested-by: Andrii Nakryiko <andrii@kernel.org>
Signed-off-by: Ihor Solodrai <ihor.solodrai@linux.dev>
Link: https://lore.kernel.org/r/20260615195536.1065107-1-ihor.solodrai@linux.dev
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
check_stack_write_fixed_off() computes the byte slot for a fixed-offset
stack write as -off - 1, and records each written byte in slot_type[] with
(slot - i) % BPF_REG_SIZE.
The Spectre v4 sanitization pre-check uses slot_type[i] instead. For a
4-byte write at fp-8 after the lower half of fp-8 has been zeroed, the
pre-check scans bytes 0..3 and sees STACK_ZERO while the actual write updates
bytes 7..4. That can leave the second half-slot write without nospec_result
even though the bytes being overwritten still require sanitization.
Use the same slot index in the sanitization pre-check that the write path uses
when updating slot_type[].
Fixes: 2039f26f3a ("bpf: Fix leakage due to insufficient speculative store bypass mitigation")
Acked-by: Luis Gerhorst <luis.gerhorst@fau.de>
Reviewed-by: Jiayuan Chen <jiayuan.chen@linux.dev>
Reviewed-by: Emil Tsalapatis <emil@etsalapatis.com>
Signed-off-by: Nuoqi Gui <gnq25@mails.tsinghua.edu.cn>
Link: https://lore.kernel.org/r/20260618-f01-11-stack-nospec-slot-index-v3-1-780297041721@mails.tsinghua.edu.cn
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Everything:
Total patches: 108
Reviews/patch: 0.84
Reviewed rate: 75%
Patch series in this merge:
- The 2 patch series "taskstats: fix TGID dead-thread stat retention"
from Yiyang Chen fixes a taskstats TGID aggregation bug where fields
added in the TGID query path were not preserved after thread exit, and
adds a kselftest covering the regression.
- The 2 patch series "lib/tests: string_helpers: Slight improvements"
from Andy Shevchenko improves lib/tests/string_helpers_kunit.c a little.
- The 2 patch series "lib/base64: decode fixes" from Josh Law addreesses
minor issues in lib/base64.c.
- The 3 patch series "selftests/filelock: Make output more kselftestish"
from Mark Brown makes the output from the ofdlocks test a bit easier for
tooling to work with, and also ignores the generated file.
- The 3 patch series "uaccess: unify inline vs outline
copy_{from,to}_user() selection" from Yury Norov simplifies the usercopy
code by removing the selectability of inlining copy_{from,to}_user().
- The 5 patch series "ocfs2: validate inline xattr header consumers"
from ZhengYuan Huang fixes a number of possible issues in the ocfs2
xattr code.
- The 8 patch series "lib and lib/cmdline enhancements" from Dmitry
Antipov provides additional robustness checking in the cmdline handling
code and its in-kernel testing and selftests.
- The 18 patch series "cleanup the RAID6 P/Q library" from Christoph
Hellwig cleans up the RAID6 P/Q library to match the recent updates to
the RAID 5 XOR library and other CRC/crypto libraries.
- The 3 patch series "ocfs2: harden inode validators against forged
metadata" from Michael Bommarito adds three structural checks to OCFS2
dinode validation so malformed on-disk fields are rejected before
ocfs2_populate_inode() copies them into the in-core inode.
- The 2 patch series "lib/raid: replace __get_free_pages() call with
kmalloc()" from Mike Rapoport cleans up the lib/raid code by using
kmalloc() in more places.
-----BEGIN PGP SIGNATURE-----
iHUEABYKAB0WIQTTMBEPP41GrTpTJgfdBJ7gKXxAjgUCajgeIQAKCRDdBJ7gKXxA
jmlWAQCLJVDZNJMFaXy4a+YHdu3tfemLpSy83A0Le61tOZUdBQD/Sf/7rhgeaM74
32yp53TZLA8xHImCGEin/1ddPJ8DbgY=
=GW2I
-----END PGP SIGNATURE-----
Merge tag 'mm-nonmm-stable-2026-06-21-10-22' of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm
Pull non-MM updates from Andrew Morton:
- "taskstats: fix TGID dead-thread stat retention" (Yiyang Chen)
Fix a taskstats TGID aggregation bug where fields added in the TGID
query path were not preserved after thread exit, and adds a kselftest
covering the regression.
- "lib/tests: string_helpers: Slight improvements" (Andy Shevchenko)
Improve lib/tests/string_helpers_kunit.c a little
- "lib/base64: decode fixes" (Josh Law)
Address minor issues in lib/base64.c
- "selftests/filelock: Make output more kselftestish" (Mark Brown)
Make the output from the ofdlocks test a bit easier for tooling to
work with. Also ignore the generated file
- "uaccess: unify inline vs outline copy_{from,to}_user() selection"
(Yury Norov)
Simplify the usercopy code by removing the selectability of inlining
copy_{from,to}_user().
- "ocfs2: validate inline xattr header consumers" (ZhengYuan Huang)
Fix a number of possible issues in the ocfs2 xattr code
- "lib and lib/cmdline enhancements" (Dmitry Antipov)
Provide additional robustness checking in the cmdline handling code
and its in-kernel testing and selftests
- "cleanup the RAID6 P/Q library" (Christoph Hellwig)
Clean up the RAID6 P/Q library to match the recent updates to the
RAID 5 XOR library and other CRC/crypto libraries
- "ocfs2: harden inode validators against forged metadata" (Michael
Bommarito)
Add three structural checks to OCFS2 dinode validation so malformed
on-disk fields are rejected before ocfs2_populate_inode() copies them
into the in-core inode
- "lib/raid: replace __get_free_pages() call with kmalloc()" (Mike
Rapoport)
Clean up the lib/raid code by using kmalloc() in more places
* tag 'mm-nonmm-stable-2026-06-21-10-22' of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm: (108 commits)
ocfs2: fix circular locking dependency in ocfs2_dio_end_io_write
ocfs2: fix NULL h_transaction deref in ocfs2_assure_trans_credits
lib: interval_tree_test: validate benchmark parameters
ocfs2: avoid moving extents to occupied clusters
treewide: fix transposed "sign" typos and update spelling.txt
ocfs2: fix UBSAN array-index-out-of-bounds in ocfs2_sum_rightmost_rec
fat: reject BPB volumes whose data area starts beyond total sectors
selftests/uevent: increase __UEVENT_BUFFER_SIZE to avoid ENOBUFS on busy systems
lib/test_firmware: allocate the configured into_buf size
fs: efs: remove unneeded debug prints
checkpatch: cuppress warnings when Reported-by: is followed by Link:
MAINTAINERS: add Alexander as a kcov reviewer
mailmap: update Alexander Sverdlin's Email addresses
fs: fat: inode: replace sprintf() with scnprintf()
ocfs2: fix out-of-bounds write in ocfs2_remove_refcount_extent
ocfs2: fix race between ocfs2_control_install_private() and ocfs2_control_release()
ocfs2/dlm: require a ref for locking_state debugfs open
ocfs2: reject FITRIM ranges shorter than a cluster
ocfs2: validate fast symlink target during inode read
ocfs2: add journal NULL check in ocfs2_checkpoint_inode()
...
states_show() adds CPU hotplug state names into a single sysfs buffer
using sprintf(). With enough registered states, this can write past the
end of the PAGE_SIZE buffer.
Use sysfs_emit_at() so output is bounded.
Fixes: 98f8cdce1d ("cpu/hotplug: Add sysfs state interface")
Signed-off-by: Bradley Morgan <include@grrlz.net>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Cc: stable@vger.kernel.org
Link: https://patch.msgid.link/20260619163719.12103-2-include@grrlz.net
cpuhp_invoke_callback() unwinds earlier callbacks for the same
hotplug state when one instance fails. The rollback path currently
reuses ret, so a successful rollback can hide the original error and
make the failed transition look successful.
Keep the rollback result separate from the original error.
Fixes: 724a86881d ("smp/hotplug: Callback vs state-machine consistency")
Signed-off-by: Bradley Morgan <include@grrlz.net>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Cc: stable@vger.kernel.org
Link: https://patch.msgid.link/20260619163719.12103-1-include@grrlz.net
- kho: make memory preservation compatible with deferred initialization of
the memory map
- luo: add LIVEUPDATE_SESSION_GET_NAME ioctl and parameter verification for
LIVEUPDATE_IOCTL_CREATE_SESSION ioctl
- luo: documentation updates for liveupdate=on command line option, systemd
support and the current compatibility status
- luo: remove the fixed limits on the number of files that can be preserved
within a single session, and the total number of sessions managed by the
Live Update Orchestrator
- fixes that didn't cut as urgent and were delayed until the merge window:
* reference count incoming File-Lifecycle-Bound (FLB) data so it cannot
be freed while a subsystem is still using it
* fixes for a TOCTOU race in luo_session_retrieve(), a use-after-free in
the file finish and unpreserve paths, concurrent session mutations
during reboot and serialization on preserve_context kexec
* make sure ioctls for incoming LUO sessions are blocked for outgoing
sessions and vice versa
* make sure KHO scratch size is always aligned by CMA_MIN_ALIGNMENT_BYTES
* fix memblock tests build issue introduced by KHO changes
-----BEGIN PGP SIGNATURE-----
iQEzBAABCgAdFiEEeOVYVaWZL5900a/pOQOGJssO/ZEFAmo315YACgkQOQOGJssO
/ZE1SQf/e+ETXoC/oAhg7aIR8HM8NRWxgp1Ira6fpLJ0IvUlPQSE3yumLaVUymg3
T3mK+b0BbIEjb6QT4/+UQEFnh01C+opxceghjCc2z6HIsp2Y/VCXnZNj31FWrspZ
yWAVkhppHwEpA6z45jDM6ua8KI5V3TwyZ15xxCUewVYSKqw7GrvijHlAowAa5H3n
elwqfjjiooGnDKTrcjpVJj7Qix9LRf0OJhGqlip5np8TW9S/pU3gT7g304iQEgSj
GiDs/3vmOkV82HV5ifAgv21Tjs4bCQi1hk9f8SpgF10onnhokSp2/uzVmfqYqcxx
hvANOdCBUoO4FzIg/9uBriSynGzYbw==
=urD+
-----END PGP SIGNATURE-----
Merge tag 'liveupdate-v7.2-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/liveupdate/linux
Pull liveupdate updates from Mike Rapoport:
"Kexec Handover (KHO):
- make memory preservation compatible with deferred initialization
of the memory map
Live Update Orchestrator (LUO):
- add LIVEUPDATE_SESSION_GET_NAME ioctl and parameter verification
for LIVEUPDATE_IOCTL_CREATE_SESSION ioctl
- documentation updates for liveupdate=on command line option,
systemd support and the current compatibility status
- remove the fixed limits on the number of files that can be
preserved within a single session, and the total number of
sessions managed by the LUO
Misc fixes:
- reference count incoming File-Lifecycle-Bound (FLB) data so
it cannot be freed while a subsystem is still using it
- fixes for a TOCTOU race in luo_session_retrieve(), a use-
after-free in the file finish and unpreserve paths, concurrent
session mutations during reboot and serialization on
preserve_context kexec
- make sure ioctls for incoming LUO sessions are blocked for
outgoing sessions and vice versa
- make sure KHO scratch size is always aligned by
CMA_MIN_ALIGNMENT_BYTES
- fix memblock tests build issue introduced by KHO changes"
* tag 'liveupdate-v7.2-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/liveupdate/linux: (36 commits)
liveupdate: Document that retrieve failure is permanent
docs: memfd_preservation: fix rendering of ABI documentation
selftests/liveupdate: Add stress-files kexec test
selftests/liveupdate: Add stress-sessions kexec test
selftests/liveupdate: Test session and file limit removal
liveupdate: Remove limit on the number of files per session
liveupdate: Remove limit on the number of sessions
liveupdate: defer session block allocation and physical address setting
kho: add support for linked-block serialization
liveupdate: Extract luo_session_deserialize_one helper
liveupdate: Extract luo_file_deserialize_one helper
liveupdate: register luo_ser as KHO subtree
liveupdate: centralize state management into struct luo_ser
liveupdate: avoid mixing cleanup guards with goto in luo_session_retrieve_fd
liveupdate: change file_set->count type to u64 for type safety
liveupdate: Remove unused ser field from struct luo_session
liveupdate: fix u-a-f in luo_file_unpreserve_files() and luo_file_finish()
liveupdate: block session mutations during reboot
liveupdate: fix TOCTOU race in luo_session_retrieve()
liveupdate: skip serialization for context-preserving kexec
...
rt_spin_unlock() releases the RCU protection before unlocking the
lock. That opens the door for the following UAF scenario:
T1 T2
spin_lock(&p->lock); rcu_read_lock();
invalidate(p); p = rcu_dereference(ptr);
rcu_assign_pointer(ptr, NULL); if (!p) return;
spin_unlock(&p->lock); spin_lock(&p->lock)
lock(&lock->lock);
rcu_read_lock();
kfree_rcu(p); rcu_read_unlock();
....
spin_unlock(&p->lock)
rcu_read_unlock(); // Ends grace period
rcu_do_batch()
kfree(p);
UAF -> rt_mutex_cmpxchg_release(&lock->lock...)
Regular spinlocks keep preemption disabled accross the unlock operation,
which provides full RCU protection, but the RT substitution fails to
resemble that. Same applies for the rwlock substitution.
Move the rcu_read_unlock() invocation past the unlock operations to match
the non-RT semantics. This makes it asymmetric vs. rt_xxx_lock(), but
that's harmless as the caller needs to hold RCU read lock across the lock
operation. The migrate_enable() call stays before the unlock operation
because there is no per CPU operation in the unlock path which would
require migration to be kept disabled.
Fixes: 0f383b6dc9 ("locking/spinlock: Provide RT variant")
Reported-by: syzbot+000c800a02097aaa10ed@syzkaller.appspotmail.com
Decoded-by: Jann Horn <jannh@google.com>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Reviewed-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Acked-by: Al Viro <viro@zeniv.linux.org.uk>
Cc: stable@vger.kernel.org
Link: https://patch.msgid.link/87jyrud75z.ffs@fw13
In mm_cid_fixup_cpus_to_tasks(), when rq->curr has the target mm and
mm_cid.active is set, the CID is checked with cid_in_transit() before
setting the transition bit. In per-CPU mode a newly forked or exec'd
task can be running with mm_cid.cid == MM_CID_UNSET because CIDs are
assigned lazily on schedule-in. With cid_in_transit() the guard passes
for MM_CID_UNSET (no transit bit), converts it to MM_CID_UNSET |
MM_CID_TRANSIT and stores it back; later mm_cid_schedout() feeds this
to clear_bit() with MM_CID_UNSET as the bit number, triggering an
out-of-bounds write.
Symptoms: this is genuine memory corruption, but a bounded out-of-bounds
write, not an arbitrary one. MM_CID_UNSET is the fixed sentinel BIT(31),
so once the bad value reaches mm_cid_schedout() the cid_from_transit_cid()
strip leaves MM_CID_UNSET, which fails the "cid < max_cids" convergence
test and falls into mm_drop_cid() -> clear_bit(MM_CID_UNSET,
mm_cidmask(mm)). The cid bitmap is embedded in the mm_struct slab object
(after cpu_bitmap and mm_cpus_allowed) and is only num_possible_cpus()
bits wide, so clearing bit 31 is a deterministic OOB bit-clear at a
fixed offset of 2^31 / 8 == 256 MiB past the bitmap base. The address is
not attacker-influenced (fixed sentinel -> fixed offset) and the op only
clears a single bit; what sits 256 MiB further along the direct map is
whatever kernel object happens to live there, so this corrupts one bit of
unpredictable kernel memory -- it is not an arbitrary-address or
arbitrary-value write.
It triggers only in per-CPU CID mode, when a CPU is running an active
task of the target mm whose cid is still MM_CID_UNSET -- the
fork()/execve() window before that task's next schedule-in assigns it a
real CID -- and a per-CPU -> per-task fixup walks over it (the mode
fallback driven by a thread exit, sched_mm_cid_exit(), or by the deferred
max_cids recompute in mm_cid_work_fn()).
In practice syzkaller surfaced it as a KASAN use-after-free reported in
__schedule -> mm_cid_switch_to, where the offending clear_bit() is inlined
via mm_cid_schedout() -> mm_drop_cid().
Guard the transition-bit assignment against MM_CID_UNSET, in addition to
the existing cid_in_transit() check, so the bit is only set on a genuine
task-owned CID. A CPU-owned (MM_CID_ONCPU) CID of a running active task
is handled by the cid_on_cpu(pcp->cid) branch above and never reaches
this path, so excluding MM_CID_UNSET (and the already-transitioning case)
is sufficient.
Fixes: fbd0e71dc3 ("sched/mmcid: Provide CID ownership mode fixup functions")
Signed-off-by: Rik van Riel <riel@surriel.com>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Assisted-by: Claude:claude-opus-4-8 syzkaller
Reviewed-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: stable@vger.kernel.org
Link: https://patch.msgid.link/20260616203818.1516263-1-riel@surriel.com
Everything:
Total patches: 321
Reviews/patch: 1.52
Reviewed rate: 68%
Excluding DAMON:
Total patches: 227
Reviews/patch: 2.07
Reviewed rate: 90%
Excluding DAMON and selftests:
Total patches: 203
Reviews/patch: 2.14
Reviewed rate: 91%
Patchsets in this merge:
- The 2 patch series "selftests/mm: clean up build output and verbosity"
from Li Wang removes some noise from the MM selftests build out.
- The 3 patch series "mm: Free contiguous order-0 pages efficiently"
from Ryan Roberts speeds up the freeing of a batch of 0-order pages by
first scanning them for coalescing opportunities. This is applicable to
vfree() and to the releasing of frozen pages.
- The 11 patch series "mm/damon: introduce DAMOS failed region quota
charge ratio" from SeongJae Park addresses a DAMOS usability issue: The
DAMOS quota often exhausts prematurely because it charges for all memory
attempted, causing slow and inconsistent performance when actions fail
on unreclaimable memory.
To fix this, a new feature lets users set a smaller, flexible quota
charge ratio (via a numerator and denominator) for failed regions.
Since failed actions cause less overhead, reducing their quota cost
ensures more predictable and efficient DAMOS processing
- The 8 patch series "selftests/cgroup: improve zswap tests robustness
and support large page sizes" from Li Wang fixes various spurious
failures and improves the overall robustness of the cgroup zswap
selftests.
- The 3 patch series "fix MAP_DROPPABLE not supported errno" from
Anthony Yznaga fixes an issue in the mlock selftests on arm32.
- The 2 patch series "mm: huge_memory: clean up defrag sysfs with
shared" from Breno Leitao does some maintenance work in the huge_memory
code.
- The 3 patch series "treewide: fixup gfp_t printks" from Brendan
Jackman uses the special vprintf() gfp_t conversion in various places.
- The 6 patch series "mm: Fix vmemmap optimization accounting and
initialization" from Muchun Song fixes several bugs in the vmemmap
optimization, mainly around incorrect page accounting and memmap
initialization in the DAX and memory hotplug paths. It also fixes
pageblock migratetype initialization and struct page initialization for
ZONE_DEVICE compound pages.
- The 4 patch series "mm/damon: repost non-hotfix reviewed patches in
damon/next tree" from is a sprinkle of unrelated minor bugfixes for
DAMON.
- The 3 patch series "mm: remove page_mapped()" from David Hildenbrand
remove this function from the tree, replacing it with folio_mapped().
- The 10 patch series "mm/damon: let DAMON be paused and resumed" from
SeongJae Park permits DAMON to be paused and resumed without losing its
current state.
- The 3 patch series "kasan: hw_tags: Disable tagging for stack and
page-tables" from Muhammad Usama Anjum simplifies and speeds up kasan by
removing its ineffective tagging of stacks and page tables.
- The 7 patch series "mm/damon/reclaim,lru_sort: monitor all system rams
by default" from SeongJae Park simplifies deployment on diverse hardware
like NUMA systems by updating DAMON_RECLAIM and DAMON_LRU_SORT to
automatically monitor the physical address range covering all System RAM
areas by default, replacing the overly restrictive behavior that only
targeted the single largest memory block to save on negligible overhead.
- The 2 patch series "mm/damon/sysfs: document filters/ directory as
deprecated" from SeongJae Park updates some DAMON docs.
- The 8 patch series "mm: use spinlock guards for zone lock" from Dmitry
Ilvokhin switches zone->lock handling over to using the guard()
mechanisms.
- The 2 patch series "mm/filemap: tighten mmap_miss hit accounting" from
fujunjie fixes a flaw where the mmap_miss counter over-credited page
cache hits during fault-arounds and page-fault retries. This results in
significant reduction of redundant synchronous mmap readahead I/O,
drastically cutting down execution time and gigabytes read for sparse
random or strided memory access workloads.
- The 2 patch series "selftests/cgroup: Fix false positive failures in
test_percpu_basic" from Li Wang fixes a couple of false-positives in the
cgroup kmem selftests.
- The 2 patch series "mm/damon/reclaim: support monitoring intervals
auto-tuning" from SeongJae Park adds a new parameter to DAMON permitting
DAMON_RECLAIM to automatically tune DAMON's sampling and aggregation
intervals.
- The 2 patch series "mm/damon/stat: add kdamond_pid parameter" from
SeongJae Park chnges DAMON_STAT to provide the pid of its kdamond.
- The 2 patch series "mm/kmemleak: dedupe verbose scan output" from
Breno Leitao removes large amounts of duplicated backtraces from the
verbose-mode kmemleak output.
- The 8 patch series "mm: remove CONFIG_HAVE_BOOTMEM_INFO_NODE (Part 1)"
from David Hildenbrand reduces our use of CONFIG_HAVE_BOOTMEM_INFO_NODE,
with a view to removing it entirely in a later series.
- The 2 patch series "mm/damon: validate min_region_size to be power of
2" from Liew Rui Yan prevents users from passing a non-power-of-2 value
of `addr_unit', as this later results in undesirable behavior.
- The 2 patch series "mm: document read_pages and simplify usage" from
Frederick Mayle does as claimed.
- The 3 patch series "tools/mm/page-types: Fix misc bugs" from Ye Liu
fixes three issues in tools/mm/page-types.c.
- The 4 patch series "mm: misc cleanups from __GFP_UNMAPPED series" from
Brendan Jackman implements several cleanups in the page allocator and
related code.
- The 12 patch series "mm, swap: swap table phase IV: unify allocation"
from Kairui Song unifies the allocation and charging of anon and shmem
swap in folios, provides better synchronization, consolidates the
metadata management, hence dropping the static array and map, and
improves performance.
- The 28 patch series "mm/damon: introduce data attributes monitoring"
from SeongJae Park extends DAMON for monitoring general data attributes
other than accesses.
- The 5 patch series "mm/vmalloc: free unused pages on vrealloc()
shrink" from Shivam Kalra implements the TODO in vrealloc() to unmap and
free unused pages when shrinking across a page boundary.
- The 3 patch series "mm/damon: documentation and comment fixes" from
niecheng does as advertised.
- The 3 patch series "remove mmap_action success, error hooks" from
Lorenzo Stoakes eliminates custom hooks from mmap_action by removing the
problematic success_hook which allowed drivers to improperly access
uninitialized VMAs. It replaces the error_hook with a simple error-code
field and updates the memory char driver accordingly.
- The 14 patch series "mm/damon: minor improvements for code readability
and tests" from SeongJae Park implements minor improvements in code
readability and tests for DAMON.
- The 2 patch series "mm/damon: fix macro arguments and clarify quota
goals doc" from Maksym Shcherba does those things.
- The 2 patch series "userfaultfd: merge fs/userfaultfd.c into
mm/userfaultfd.c" from Mike Rapoport performs that code movement.
- The 15 patch series "mm/mglru: improve reclaim loop and dirty folio"
from Kairui Song and others cleans up and slightly improves MGLRU's
reclaim loop and dirty writeback handling. Large performance
improvements are measured.
- The 3 patch series "use vma locks for proc/pid/{smaps|numa_maps}
reads" from Suren Baghdasaryan uses per-vma locks when reading
/proc/pid/smaps and /proc/pid/numa_maps similar to reduce contention on
central mmap_lock.
- The 2 patch series "refactors thpsize_shmem_enabled_store() and
thpsize_shmem_enabled_show()" from Ran Xiaokai provides some cleanup
work in the THP code.
- The 2 patch series "selftests/memfd: fix compilation warnings" from
Konstantin Khorenko fixes a few build glitches in the memfd selftest
code.
- The 4 patch series "memcg: shrink obj_stock_pcp and cache multiple
objcgs" from Shakeel Butt resolves a 68% performance regression caused
by NUMA-node cache thrashing around struct obj_stock_pcp by shrinking
its existing fields and expanding it into a multi-slot array that caches
up to five obj_cgroup pointers per CPU, allowing per-node variants of
the same memcg to coexist within a single 64-byte cache line.
- The 2 patch series "zram: writeback fixes" from Sergey Senozhatsky
addresses a couple of unrelated zram writeback issues.
- The 9 patch series "mm: switch THP shrinker to list_lru" from Johannes
Weiner resolves NUMA-awareness issues and streamlines callsite
interaction by refactoring and extending the list_lru API to completely
replace the complex, open-coded deferred split queue for Transparent
Huge Pages (THPs).
- The 2 patch series "mm: improve large folio readahead for exec memory"
from Usama Arif improves large-folio readahead on systems like 64K-page
arm64 by preventing the mmap_miss check from permanently disabling
target-oriented VM_EXEC readahead, and by generalizing the
force_thp_readahead gate to support mappings with any usefully large
maximum folio order under the cache cap.
- The 6 patch series "userfaultfd/pagemap: pre-existing fixes" from
Kiryl Shutsemau fixes a bunch of minor issues in the
userfaultfd/pagemap, all of which were flagged by Sashiko review of
proposed new material.
- The 5 patch series "mm/sparse-vmemmap: Provide generic
vmemmap_set_pmd() and vmemmap_check_pmd()" from Muchun Song provides
generic versions of these two functions so the four arch-specific
implementations can be removed.
- The 2 patch series "mm/swap, PM: hibernate: fix swapoff race in
uswsusp by pinning swap device" from Youngjun Park addresses a
uswsusp-vs-swapoff race and reduces the swap device reference
taking/releasing frequency.
- The 2 patch series "mm/hmm: A fix and a selftest" from Dev Jain does
as claimed.
-----BEGIN PGP SIGNATURE-----
iHUEABYKAB0WIQTTMBEPP41GrTpTJgfdBJ7gKXxAjgUCajQcWQAKCRDdBJ7gKXxA
jkjFAQD14ZCAS7c/lsNHPRpOARR2zrCJiIp+vhZDEx5V+d8u0QD+I4ca11CnQ+OA
dKCeabWWYuTR1GDtTHbLVCRfpGexBwM=
=cWPO
-----END PGP SIGNATURE-----
Merge tag 'mm-stable-2026-06-18-09-26' of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm
Pull MM updates from Andrew Morton:
- "selftests/mm: clean up build output and verbosity" (Li Wang)
Remove some noise from the MM selftests build
- "mm: Free contiguous order-0 pages efficiently" (Ryan Roberts)
Speed up the freeing of a batch of 0-order pages by first scanning
them for coalescing opportunities. This is applicable to vfree() and
to the releasing of frozen pages
- "mm/damon: introduce DAMOS failed region quota charge ratio"
(SeongJae Park)
Address a DAMOS usability issue: The DAMOS quota often exhausts
prematurely because it charges for all memory attempted, causing slow
and inconsistent performance when actions fail on unreclaimable
memory.
To fix this, a new feature lets users set a smaller, flexible quota
charge ratio (via a numerator and denominator) for failed regions.
Since failed actions cause less overhead, reducing their quota cost
ensures more predictable and efficient DAMOS processing
- "selftests/cgroup: improve zswap tests robustness and support large
page sizes" (Li Wang)
Fix various spurious failures and improves the overall robustness of
the cgroup zswap selftests
- "fix MAP_DROPPABLE not supported errno" (Anthony Yznaga)
Fix an issue in the mlock selftests on arm32
- "mm: huge_memory: clean up defrag sysfs with shared" (Breno Leitao)
Some maintenance work in the huge_memory code
- "treewide: fixup gfp_t printks" (Brendan Jackman)
Use the special vprintf() gfp_t conversion in various places
- "mm: Fix vmemmap optimization accounting and initialization" (Muchun
Song)
Fix several bugs in the vmemmap optimization, mainly around incorrect
page accounting and memmap initialization in the DAX and memory
hotplug paths. It also fixes pageblock migratetype initialization and
struct page initialization for ZONE_DEVICE compound pages
- "mm/damon: repost non-hotfix reviewed patches in damon/next tree"
A sprinkle of unrelated minor bugfixes for DAMON
- "mm: remove page_mapped()" (David Hildenbrand)
Remove this function from the tree, replacing it with folio_mapped()
- "mm/damon: let DAMON be paused and resumed" (SeongJae Park)
Allow DAMON to be paused and resumed without losing its current state
- "kasan: hw_tags: Disable tagging for stack and page-tables" (Muhammad
Usama Anjum)
Simplify and speed up kasan by removing its ineffective tagging of
stacks and page tables
- "mm/damon/reclaim,lru_sort: monitor all system rams by default"
(SeongJae Park)
Simplify deployment on diverse hardware like NUMA systems by updating
DAMON_RECLAIM and DAMON_LRU_SORT to automatically monitor the
physical address range covering all System RAM areas by default,
replacing the overly restrictive behavior that only targeted the
single largest memory block to save on negligible overhead
- "mm/damon/sysfs: document filters/ directory as deprecated" (SeongJae
Park)
Update some DAMON docs
- "mm: use spinlock guards for zone lock" (Dmitry Ilvokhin)
Switch zone->lock handling over to using the guard() mechanisms
- "mm/filemap: tighten mmap_miss hit accounting" (fujunjie)
Fix a flaw where the mmap_miss counter over-credited page cache hits
during fault-arounds and page-fault retries. This results in
significant reduction of redundant synchronous mmap readahead I/O,
drastically cutting down execution time and gigabytes read for sparse
random or strided memory access workloads
- "selftests/cgroup: Fix false positive failures in test_percpu_basic"
(Li Wang)
Fix a couple of false-positives in the cgroup kmem selftests
- "mm/damon/reclaim: support monitoring intervals auto-tuning"
(SeongJae Park)
Add a new parameter to DAMON permitting DAMON_RECLAIM to
automatically tune DAMON's sampling and aggregation intervals
- "mm/damon/stat: add kdamond_pid parameter" (SeongJae Park)
Change DAMON_STAT to provide the pid of its kdamond
- "mm/kmemleak: dedupe verbose scan output" (Breno Leitao)
Remove large amounts of duplicated backtraces from the verbose-mode
kmemleak output
- "mm: remove CONFIG_HAVE_BOOTMEM_INFO_NODE (Part 1)" (David
Hildenbrand)
Reduce our use of CONFIG_HAVE_BOOTMEM_INFO_NODE, with a view to
removing it entirely in a later series
- "mm/damon: validate min_region_size to be power of 2" (Liew Rui Yan)
Prevent users from passing a non-power-of-2 value of `addr_unit', as
this later results in undesirable behavior
- "mm: document read_pages and simplify usage" (Frederick Mayle)
- "tools/mm/page-types: Fix misc bugs" (Ye Liu)
Fix three issues in tools/mm/page-types.c
- "mm: misc cleanups from __GFP_UNMAPPED series" (Brendan Jackman)
Implement several cleanups in the page allocator and related code
- "mm, swap: swap table phase IV: unify allocation" (Kairui Song)
Unify the allocation and charging of anon and shmem swap in folios,
provides better synchronization, consolidates the metadata
management, hence dropping the static array and map, and improves
performance
- "mm/damon: introduce data attributes monitoring" (SeongJae Park(
Extend DAMON to monitor general data attributes other than accesses
- "mm/vmalloc: free unused pages on vrealloc() shrink" (Shivam Kalra)
Implement the TODO in vrealloc() to unmap and free unused pages when
shrinking across a page boundary
- "mm/damon: documentation and comment fixes" (niecheng)
- "remove mmap_action success, error hooks" (Lorenzo Stoakes)
Eliminate custom hooks from mmap_action by removing the problematic
success_hook which allowed drivers to improperly access uninitialized
VMAs. It replaces the error_hook with a simple error-code field and
updates the memory char driver accordingly
- "mm/damon: minor improvements for code readability and tests"
(SeongJae Park)
- "mm/damon: fix macro arguments and clarify quota goals doc" (Maksym
Shcherba)
- "userfaultfd: merge fs/userfaultfd.c into mm/userfaultfd.c" (Mike
Rapoport)
- "mm/mglru: improve reclaim loop and dirty folio" (Kairui Song and
others)
Clean up and slightly improves MGLRU's reclaim loop and dirty
writeback handling. Large performance improvements are measured
- "use vma locks for proc/pid/{smaps|numa_maps} reads" (Suren
Baghdasaryan)
Use per-vma locks when reading /proc/pid/smaps and numa_maps similar
to reduce contention on central mmap_lock
- "refactors thpsize_shmem_enabled_store() and thpsize_shmem_enabled_show()"
(Ran Xiaokai)
Some cleanup work in the THP code
- "selftests/memfd: fix compilation warnings" (Konstantin Khorenko)
Fix a few build glitches in the memfd selftest code.
- "memcg: shrink obj_stock_pcp and cache multiple objcgs" (Shakeel
Butt)
Resolve a 68% performance regression caused by NUMA-node cache
thrashing around struct obj_stock_pcp by shrinking its existing
fields and expanding it into a multi-slot array that caches up to
five obj_cgroup pointers per CPU, allowing per-node variants of the
same memcg to coexist within a single 64-byte cache line.
- "zram: writeback fixes" (Sergey Senozhatsky)
address a couple of unrelated zram writeback issues
- "mm: switch THP shrinker to list_lru" (Johannes Weiner)
Resolve NUMA-awareness issues and streamlines callsite interaction by
refactoring and extending the list_lru API to completely replace the
complex, open-coded deferred split queue for Transparent Huge Pages
- "mm: improve large folio readahead for exec memory" (Usama Arif)
Improve large-folio readahead on systems like 64K-page arm64 by
preventing the mmap_miss check from permanently disabling
target-oriented VM_EXEC readahead, and by generalizing the
force_thp_readahead gate to support mappings with any usefully large
maximum folio order under the cache cap.
- "userfaultfd/pagemap: pre-existing fixes" (Kiryl Shutsemau)
Fix a bunch of minor issues in the userfaultfd/pagemap, all of which
were flagged by Sashiko review of proposed new material
- "mm/sparse-vmemmap: Provide generic vmemmap_set_pmd() and
vmemmap_check_pmd()" (Muchun Song)
Provide generic versions of these two functions so the four
arch-specific implementations can be removed.
- "mm/swap, PM: hibernate: fix swapoff race in uswsusp by pinning swap
device" (Youngjun Park)
Address a uswsusp-vs-swapoff race and reduces the swap device
reference taking/releasing frequency.
- "mm/hmm: A fix and a selftest" (Dev Jain)
* tag 'mm-stable-2026-06-18-09-26' of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm: (321 commits)
selftests/mm/hmm-tests: test pagemap reads of PMD device-private entries
fs/proc/task_mmu: do not warn on seeing non-migration pmd entry
lib/test_hmm: check alloc_page_vma() return value and handle OOM
mm/compaction: cap compact_gap() at COMPACT_CLUSTER_MAX
mm/swap: remove redundant swap device reference in alloc/free
mm/swap, PM: hibernate: fix swapoff race in uswsusp by pinning swap device
mm/filemap: use folio_next_index() for start
vmalloc: fix NULL pointer dereference in is_vm_area_hugepages()
sparc/mm: drop vmemmap_check_pmd helper and use generic code
loongarch/mm: drop vmemmap_check_pmd helper and use generic code
riscv/mm: drop vmemmap_pmd helpers and use generic code
arm64/mm: drop vmemmap_pmd helpers and use generic code
mm/sparse-vmemmap: provide generic vmemmap_set_pmd() and vmemmap_check_pmd()
rust: page: mark Page::nid as inline
userfaultfd: build __VMA_UFFD_FLAGS from config-gated masks
userfaultfd: gate must_wait writability check on pte_present()
mm/huge_memory: preserve pmd_swp_uffd_wp on device-private PMD downgrade
fs/proc/task_mmu: fix hugetlb self-deadlock in pagemap_scan_pte_hole()
fs/proc/task_mmu: use huge_page_size() in pagemap_scan_hugetlb_entry()
fs/proc/task_mmu: fix make_uffd_wp_huge_pte() prot-update race
...
Lee Jia Jie reported that since event::addr_filter_ranges is used
under RCU, it should be RCU freed.
Reported-by: Lee Jia Jie <jiajie.lee@starlabs.sg>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
- Do not invalidate entire buffer for invalid sub-buffers
For the persistent ring buffer, if one sub-buffer is found to be invalid,
it invalidates the entire per CPU ring buffer. This can lose a lot of
valuable data if there's some corruption with the writes to the buffer not
syncing properly on a hard crash. Instead, if a sub-buffer is found to be
invalid, simply zero it out and mark it for "missed events".
When the persistent ring buffer is read and a sub-buffer that was cleared
due to being invalid on boot up is discovered, the output will show
"[LOST EVENTS]" to let the user know that events were missing at that
location. Displaying the events from valid buffers can still be useful.
- Add a test to be able to test corrupted sub-buffers
If a persistent ring buffer is created as "ptraingtest" and the new config
that adds the test is enabled, when a panic happens, the kernel will
randomly corrupt one of the per CPU ring buffers. On boot up, the
sub-buffers with the corruption should be cleared and flagged. When
reading this buffer, the missed events should should [LOST EVENTS].
- Add commit number in the sub-buffer meta debug info
The commit is used to know the content of a meta page. Add it to the
buffer_meta file that is shown for each per CPU buffer.
- Clean up the persistent ring buffer validation code
Add some helper functions and make variable names more consistent.
-----BEGIN PGP SIGNATURE-----
iIoEABYKADIWIQRRSw7ePDh/lE+zeZMp5XQQmuv6qgUCajMDZBQccm9zdGVkdEBn
b29kbWlzLm9yZwAKCRAp5XQQmuv6qhDLAP4i0b8HP159LOEy/IiQKZ7hfAaUhTK8
imMVS6V1bVKTjgD8Dy/f9m8259ZIcgN9mjSKXlT30qAeGmhjjDCdc0TnagM=
=mokF
-----END PGP SIGNATURE-----
Merge tag 'trace-ring-buffer-v7.2' of git://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace
Pull ring-buffer updates from Steven Rostedt
- Do not invalidate entire buffer for invalid sub-buffers
For the persistent ring buffer, if one sub-buffer is found to be
invalid, it invalidates the entire per CPU ring buffer. This can lose
a lot of valuable data if there's some corruption with the writes to
the buffer not syncing properly on a hard crash. Instead, if a
sub-buffer is found to be invalid, simply zero it out and mark it for
"missed events".
When the persistent ring buffer is read and a sub-buffer that was
cleared due to being invalid on boot up is discovered, the output
will show "[LOST EVENTS]" to let the user know that events were
missing at that location. Displaying the events from valid buffers
can still be useful.
- Add a test to be able to test corrupted sub-buffers
If a persistent ring buffer is created as "ptraingtest" and the new
config that adds the test is enabled, when a panic happens, the
kernel will randomly corrupt one of the per CPU ring buffers. On boot
up, the sub-buffers with the corruption should be cleared and
flagged. When reading this buffer, the missed events should should
[LOST EVENTS].
- Add commit number in the sub-buffer meta debug info
The commit is used to know the content of a meta page. Add it to the
buffer_meta file that is shown for each per CPU buffer.
- Clean up the persistent ring buffer validation code
Add some helper functions and make variable names more consistent.
* tag 'trace-ring-buffer-v7.2' of git://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace:
ring-buffer: Better comment the use of RB_MISSED_EVENTS
ring-buffer: Show persistent buffer dropped events in trace_pipe file
ring-buffer: Show persistent buffer dropped events in trace file
ring-buffer: Have dropped subbuffers be persistent across reboots
ring-buffer: Cleanup buffer_data_page related code
ring-buffer: Cleanup persistent ring buffer validation
ring-buffer: Show commit numbers in buffer_meta file
ring-buffer: Add persistent ring buffer invalid-page inject test
ring-buffer: Skip invalid sub-buffers when rewinding persistent ring buffer
ring-buffer: Skip invalid sub-buffers when validating persistent ring buffer
- Remove a redundant IS_ERR() check
trace_pipe_open() already checks for IS_ERR() and does it again in the
return path. Remove the return check.
- Export seq_buf_putmem_hex() to allow kunit tests against them
To add Kunit tests on seq_buf_putmem_hex(), it needs to be exported.
- Replace strcat() and strcpy() with seq_buf() logic
The code for synthetic events uses a series of strcat() and strcpy() which
can be error prone. Replace them with seq_buf() logic that does all the
necessary bound checking.
- Add a lockdep rcu_is_watching() to trace_##event##_enabled() call
The trace_##event##_enabled() is a static branch that is true if the
"event" is enabled. But this can hide bugs if this logic is in a location
where RCU is disabled and not "watching". It would only trigger if lockdep
is enabled and the event is enabled.
Add a "rcu_is_watching()" warning if lockdep is enabled in that helper
function to trigger regardless if the event is enabled or not.
- Remove the local variable in the trace_printk() macro
For name space integrity, remove the _______STR variable in the
trace_printk() macro for using the sizeof() macro directly.
- Use guard()s for the trace_recursion_record.c file
- Fix typo in a comment of eventfs_callback() kerneldoc
- Use trace_call__##event() in events called within trace_##event##_enabled()
A couple of events are called within an if block guarded by
trace_##event##_enabled(). That is a static key that is only enabled when
the event is enabled. The trace_call_##event() calls the tracepoint code
directly without adding a redundant static key for that check.
- Allow perf to read synthetic events
Currently, perf does not have the ability to enable a synthetic event. If
it does, it will either cause a kernel warning or error with "No such
device". Synthetic events are not much different than kprobes and perf can
handle fine with a few modifications.
- Replace printk(KERN_WARNING ...) with pr_warn()
- Replace krealloc() on an array with krealloc_array()
- Fix README file path name for synthetic events
- Change tracing_map tracing_map_array to use a flexible array
Instead of allocating a separate pointer to hold the pages field of
tracing_map_array, allocate the pages field as a flexible array when
allocating the structure.
- Fold trace_iterator_increment() into trace_find_next_entry_inc()
The function trace_iterator_increment() was only used by
trace_find_next_entry_inc(). It's not big enough to be a helper function
for one user. Fold it into its caller.
- Make field_var_str field a flexible array of hist_elt_data
Instead of allocating a separate pointer for the field_var_str array of
the hist_elt_data structure, allocate it as a flexible array when
allocating the structure.
- Disable KCOV for trace_irqsoff.c
Like trace_preemptirq.c, trace_irqsoff.c has code that will crash when
KCOV is enabled on ARM. The irqsoff tracing can be called on ARM because
the irqsoff tracing code can be run from early interrupt code and produce
coverage unrelated to syscall inputs.
- Fix warning in __unregister_ftrace_function() called by perf
Perf calls unregister_ftrace_function() without checking if its ftrace_ops
has already been unregistered. There's an error path where on clean up it
will unregister the ftrace_ops even if it wasn't registered and causes a
warning.
-----BEGIN PGP SIGNATURE-----
iIoEABYKADIWIQRRSw7ePDh/lE+zeZMp5XQQmuv6qgUCajHGbRQccm9zdGVkdEBn
b29kbWlzLm9yZwAKCRAp5XQQmuv6qqP/AP4nv/Tmv6T7vxMzL7dY/WJ0xT2xkW7Z
tlejy1AM/RBaTAEAkkgdDNhIAJwxCxsmY6/Zr1mRWzaF7O4kX94KYFcBHww=
=7nyZ
-----END PGP SIGNATURE-----
Merge tag 'trace-v7.2' of git://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace
Pull tracing updates from Steven Rostedt:
- Remove a redundant IS_ERR() check
trace_pipe_open() already checks for IS_ERR() and does it again in
the return path. Remove the return check.
- Export seq_buf_putmem_hex() to allow kunit tests against them
To add Kunit tests on seq_buf_putmem_hex(), it needs to be exported.
- Replace strcat() and strcpy() with seq_buf() logic
The code for synthetic events uses a series of strcat() and strcpy()
which can be error prone. Replace them with seq_buf() logic that does
all the necessary bound checking.
- Add a lockdep rcu_is_watching() to trace_##event##_enabled() call
The trace_##event##_enabled() is a static branch that is true if the
"event" is enabled. But this can hide bugs if this logic is in a
location where RCU is disabled and not "watching". It would only
trigger if lockdep is enabled and the event is enabled.
Add a "rcu_is_watching()" warning if lockdep is enabled in that
helper function to trigger regardless if the event is enabled or not.
- Remove the local variable in the trace_printk() macro
For name space integrity, remove the _______STR variable in the
trace_printk() macro for using the sizeof() macro directly.
- Use guard()s for the trace_recursion_record.c file
- Fix typo in a comment of eventfs_callback() kerneldoc
- Use trace_call__##event() in events within trace_##event##_enabled()
A couple of events are called within an if block guarded by
trace_##event##_enabled(). That is a static key that is only enabled
when the event is enabled. The trace_call_##event() calls the
tracepoint code directly without adding a redundant static key for
that check.
- Allow perf to read synthetic events
Currently, perf does not have the ability to enable a synthetic
event. If it does, it will either cause a kernel warning or error
with "No such device". Synthetic events are not much different than
kprobes and perf can handle fine with a few modifications.
- Replace printk(KERN_WARNING ...) with pr_warn()
- Replace krealloc() on an array with krealloc_array()
- Fix README file path name for synthetic events
- Change tracing_map tracing_map_array to use a flexible array
Instead of allocating a separate pointer to hold the pages field of
tracing_map_array, allocate the pages field as a flexible array when
allocating the structure.
- Fold trace_iterator_increment() into trace_find_next_entry_inc()
The function trace_iterator_increment() was only used by
trace_find_next_entry_inc(). It's not big enough to be a helper
function for one user. Fold it into its caller.
- Make field_var_str field a flexible array of hist_elt_data
Instead of allocating a separate pointer for the field_var_str array
of the hist_elt_data structure, allocate it as a flexible array when
allocating the structure.
- Disable KCOV for trace_irqsoff.c
Like trace_preemptirq.c, trace_irqsoff.c has code that will crash
when KCOV is enabled on ARM. The irqsoff tracing can be called on ARM
because the irqsoff tracing code can be run from early interrupt code
and produce coverage unrelated to syscall inputs.
- Fix warning in __unregister_ftrace_function() called by perf
Perf calls unregister_ftrace_function() without checking if its
ftrace_ops has already been unregistered. There's an error path where
on clean up it will unregister the ftrace_ops even if it wasn't
registered and causes a warning.
* tag 'trace-v7.2' of git://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace:
perf/ftrace: Fix WARNING in __unregister_ftrace_function
tracing: Disable KCOV instrumentation for trace_irqsoff.o
tracing: Turn hist_elt_data field_var_str into a flexible array
tracing: Move trace_iterator_increment() into trace_find_next_entry_inc()
tracing: Simplify pages allocation for tracing_map logic
tracing: Fix README path for synthetic_events
tracing: Use krealloc_array() for trace option array growth
tracing/branch: Use pr_warn() instead of printk(KERN_WARNING)
tracing: Allow perf to read synthetic events
HID: Use trace_call__##name() at guarded tracepoint call sites
cpufreq: amd-pstate: Use trace_call__##name() at guarded tracepoint call site
tracefs: Fix typo in a comment of eventfs_callback() kerneldoc
tracing: Switch trace_recursion_record.c code over to use guard()
tracing: Remove local variable for argument detection from trace_printk()
tracepoint: Add lockdep rcu_is_watching() check to trace_##name##_enabled()
tracing: Bound synthetic-field strings with seq_buf
seq_buf: Export seq_buf_putmem_hex() and add KUnit tests
tracing: Remove redundant IS_ERR() check in trace_pipe_open()
Several comments transpose the letters in "assigned" and "unsigned",
spelling them with "sing" instead of "sign". Correct all of them.
Of these, the misspelling of "assigned" is not yet flagged by checkpatch,
so also add it to scripts/spelling.txt.
The remaining matches of `grep -ri singed` are RISINGEDGE register and
enum names, not typos.
Link: https://lore.kernel.org/20260612181633.734458-1-iamsharduld@gmail.com
Signed-off-by: Shardul Deshpande <iamsharduld@gmail.com>
Suggested-by: Andrew Morton <akpm@linux-foundation.org>
Reviewed-by: SeongJae Park <sj@kernel.org>
Cc: Joe Perches <joe@perches.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
- added more checks for DMA attributes in the debug code, especially
to ensure that mappings are created and released with matching
attributes (Leon Romanovsky)
- better default configuration for CMA on NUMA machines (Feng Tang)
- little code cleanup in dma benchmark tool (Rosen Penev)
-----BEGIN PGP SIGNATURE-----
iHUEABYIAB0WIQSrngzkoBtlA8uaaJ+Jp1EFxbsSRAUCajESmAAKCRCJp1EFxbsS
RKEuAQCpRF+ifclSitwzKYEUX0OY+ZqBcP+L2wqQRkjqbicv7gD+OcE5BhoctdP1
sXUaoZRHHeQ+Pz5kjS2ppSmrG2tT8AQ=
=t3Sk
-----END PGP SIGNATURE-----
Merge tag 'dma-mapping-7.2-2026-06-16' of git://git.kernel.org/pub/scm/linux/kernel/git/mszyprowski/linux
Pull dma-mapping updates from Marek Szyprowski:
- added checks for DMA attributes in the debug code, especially to
ensure that mappings are created and released with matching
attributes (Leon Romanovsky)
- better default configuration for CMA on NUMA machines (Feng Tang)
- code cleanup in dma benchmark tool (Rosen Penev)
* tag 'dma-mapping-7.2-2026-06-16' of git://git.kernel.org/pub/scm/linux/kernel/git/mszyprowski/linux:
dma: map_benchmark: turn dma_sg_map_param buf into a flexible array
dma-contiguous: simplify numa cma area handling
dma-contiguous: add kconfig option to setup numa cma area if not configured explicitly
dma-debug: Ensure mappings are created and released with matching attributes
dma-debug: Feed DMA attribute for unmapping flows too
dma-debug: Record DMA attributes in debug entry
dma-debug: Remove unused DMA attribute parameter
ntb: Use consistent DMA attributes when freeing DMA mappings
ntb: Store original DMA address for future release
-----BEGIN PGP SIGNATURE-----
iQJPBAABCAA5FiEESH4wyp42V4tXvYsjUqAMR0iAlPIFAmoyRfwbFIAAAAAABAAO
bWFudTIsMi41KzEuMTIsMiwyAAoJEFKgDEdIgJTycTcP/iiyiE+mkR5ruuPNwPsb
3JH6xKVmwcdC+EeyoNLDcR5aNAiY3GSuwXWo9oVX9HAbA9fC68od7ITuPmQn2j6l
Rbl9ITtx+mmfhDFrYNIhUXjLlhddojRBPPzfNG68lOzlG7JG75DiIAt30BRuXlbJ
njn+eeOuRsL2Y5JJn8Vvv9GnV7NyGb5lvrJzAeOv5zc7qngcu3RSUjt6NryfauRn
ZDBqrhQcDpFJ8cZoLq1D9p83tDStzjHBn9VkpXX7Y4dl5fTowI85CWCgFiXLg1KQ
KGPq0XXDI1Yw4UNABCxffhxa8nyQyaA35DAllpUNB6hY9QlMZLOGxZjmTtqxeQtP
SA01c3RpxQAvyiuja1zzvso+Km3o9doHk7gLDq6z++YfO0u0wjh2wS4IVOKfvyIk
LpK6vNQb0gjNqQmn1EYvBPpinnypSeffGGdkmkEZosr4ACAwEE+wg84422NS4gji
1erRecvHnlYIF5wIjCkzLeqGwQLSn4TblHf9/4ImaZrjf8uWaxSsBZam5OfjJpsU
LP3//oeIbEnwBNgPE8bIRw5s4L+cenTeAmS8vl7qR5InshSo1FmtYWFbrmw3/Gfy
ajV/f67J807XN03p220aRFm1x48aA5eye3VGI7wYnWVwRLIPjEh8nISEa/pV3jxW
4b/N7+xze8zYV9zTPPRGh1u5
=yXoT
-----END PGP SIGNATURE-----
Merge tag 'printk-for-7.2' of git://git.kernel.org/pub/scm/linux/kernel/git/printk/linux
Pull printk updates from Petr Mladek:
- Add upper case flavor for printing MAC addresses (%p[mM][U]) and use
it in the nintendo driver
- Fix matching of hash_pointers= parameter modes
- Fix size check of vsprintf() field_width and precision values
- Add check of size returned by vsprintf()
- Add KUnit test for restricted pointer printing (%pK)
- Some code cleanup
* tag 'printk-for-7.2' of git://git.kernel.org/pub/scm/linux/kernel/git/printk/linux:
HID: nintendo: Use %pM format specifier for MAC addresses
vsprintf: Add upper case flavour to %p[mM]
lib/vsprintf: replace min_t/max_t with min/max
printk: fix typos in comments
lib/vsprintf: Require exact hash_pointers mode matches
vsprintf: Add test for restricted kernel pointers
vsprintf: Only export no_hash_pointers to test module
lib/vsprintf: Limit the returning size to INT_MAX
lib/vsprintf: Fix to check field_width and precision
- new virtio CAN driver
- support for LoongArch architecture in fw_cfg
- support for firmware notifications in vdpa/octeon_ep
- support for VFs in virtio core
- fixes, cleanups all over the place, notably
- vhost: fix vhost_get_avail_idx for a non empty ring
fixing an significant old perf regression
- plus READ_ONCE annotations mean virtio ring is now
free of KCSAN warnings
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
-----BEGIN PGP SIGNATURE-----
iQFDBAABCgAtFiEEXQn9CHHI+FuUyooNKB8NuNKNVGkFAmoye9IPHG1zdEByZWRo
YXQuY29tAAoJECgfDbjSjVRpwHYIAMqFAn8OKVub093fOPYsa3dnsCI+6Y70Q7BK
XWXmEUs41Zbrye7GCVVKBdGog/v8WYcjA5yzyZDeCpAwbn8Vwiw6VbgD+t5IJFG+
iLtUlbotvod+esyKbSZdTSSIO5Lxxw0QqDdOt3qhz5aWwMX4LYZSO0BvNzFGaNT5
TiyKRpPtsbfox09YYrDYfm1vFhlR0mE3fMFsAy8dvauliy3rAh9bWg54oGeFdACD
PGWGvsVycl7Ly++yflqFNp8nPE+azL4jbFm3/SrjMhFcTpIEV692nhrSmiIKK7Cv
K5au5nvAmtt1sRyDHucFB9+pbUOYkNLkc3DrH/NqGFwUwxF3+pk=
=H7B4
-----END PGP SIGNATURE-----
Merge tag 'for_linus' of git://git.kernel.org/pub/scm/linux/kernel/git/mst/vhost
Pull virtio updates from Michael Tsirkin:
- new virtio CAN driver
- support for LoongArch architecture in fw_cfg
- support for firmware notifications in vdpa/octeon_ep
- support for VFs in virtio core
- fixes, cleanups all over the place, notably:
- vhost: fix vhost_get_avail_idx for a non empty ring
fixing an significant old perf regression
- READ_ONCE() annotations mean virtio ring is now
free of KCSAN warnings
* tag 'for_linus' of git://git.kernel.org/pub/scm/linux/kernel/git/mst/vhost: (37 commits)
can: virtio: Fix comment in UAPI header
can: virtio: Add virtio CAN driver
virtio: add num_vf callback to virtio_bus
fw_cfg: Add support for LoongArch architecture
vdpa/octeon_ep: fix IRQ-to-ring mapping in interrupt handler
vdpa/octeon_ep: Add vDPA device event handling for firmware notifications
vdpa/octeon_ep: Use 4 bytes for mailbox signature
vdpa/octeon_ep: Fix PF->VF mailbox data address calculation
vhost_task_create: kill unnecessary .exit_signal initialization
vhost: remove unnecessary module_init/exit functions
vdpa/mlx5: Use kvzalloc_flex() for MTT command memory
vdpa_sim_net: switch to dynamic root device
vdpa_sim_blk: switch to dynamic root device
virtio-mem: Destroy mutex before freeing virtio_mem
virtio-balloon: Destroy mutex before freeing virtio_balloon
tools/virtio: fix build for kmalloc_obj API and missing stubs
virtio_ring: Add READ_ONCE annotations for device-writable fields
vduse: fix compat handling for VDUSE_IOTLB_GET_FD/VDUSE_VQ_GET_INFO
tools/virtio: check mmap return value in vringh_test
vhost/net: complete zerocopy ubufs only once
...
The need_freq_update flag makes sugov_should_update_freq() return true
regardless of the rate_limit_us throttling, and is cleared in
sugov_update_next_freq(). sugov_update_single_freq() and
sugov_update_shared() go through that helper, so the flag does not
persist there.
However, sugov_update_single_perf(), used by drivers implementing the
.adjust_perf() callback (e.g. intel_pstate or amd-pstate in passive
mode) calls cpufreq_driver_adjust_perf() directly and never goes through
sugov_update_next_freq(), so the need_freq_update flag is not cleared in
that path.
Before commit 75da043d8f ("cpufreq/sched: Set need_freq_update in
ignore_dl_rate_limit()"), this was effectively harmless because
sugov_should_update_freq() still honored the rate limit even when
need_freq_update was set. After that change, the flag forces
sugov_should_update_freq() to always return true, so once set, it
stays effective indefinitely on the .adjust_perf() path.
As a result, cpufreq_driver_adjust_perf() gets called on every scheduler
utilization update (with the runqueue lock held) rather than being
throttled by rate_limit_us, even if the driver itself may skip redundant
hardware updates.
Clear need_freq_update at the end of the adjust_perf path as well.
Fixes: 75da043d8f ("cpufreq/sched: Set need_freq_update in ignore_dl_rate_limit()")
Signed-off-by: Zhongqiu Han <zhongqiu.han@oss.qualcomm.com>
Reviewed-by: Hongyan Xia <hongyan.xia@transsion.com>
Reviewed-by: Christian Loehle <christian.loehle@arm.com>
Cc: All applicable <stable@vger.kernel.org>
[ rjw: Subject and changelog edits ]
Link: https://patch.msgid.link/20260616154733.2405236-1-zhongqiu.han@oss.qualcomm.com
Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
A comment in kernel/time/hrtimer.c incorrectly refers to
CONFIG_NOHZ_COMMON instead of CONFIG_NO_HZ_COMMON. Correct it.
Discovered while searching for CONFIG_* symbols referenced in code but
not defined in any Kconfig file.
Signed-off-by: Ethan Nelson-Moore <enelsonmoore@gmail.com>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Link: https://patch.msgid.link/20260609034314.25029-1-enelsonmoore@gmail.com
update_rlimit_cpu() converts the RLIMIT_CPU value to nanoseconds with
u64 nsecs = rlim_new * NSEC_PER_SEC;
On 32-bit kernels both rlim_new (unsigned long) and NSEC_PER_SEC
(1000000000L) are 32-bit, so the multiplication is performed in unsigned
long and truncated for rlim_new > 4 seconds before being widened to u64.
The same file already casts to u64 for the matching computation in
check_process_timers():
u64 softns = (u64)soft * NSEC_PER_SEC;
As a result, the truncated value is installed into the CPUCLOCK_PROF
expiry cache (nextevt), causing the process CPU timer to be programmed
to fire prematurely for any RLIMIT_CPU soft limit >= 5 seconds. The
actual SIGXCPU/SIGKILL decision in check_process_timers() already casts
to u64 and is therefore correct, so limit enforcement is not broken;
only the expiry-cache programming is wrong. Apply the same cast here so
both paths convert rlim_cur identically.
64-bit kernels are unaffected.
Fixes: 858cf3a8c5 ("timers/itimer: Convert internal cputime_t units to nsec")
Signed-off-by: Zhan Xusheng <zhanxusheng@xiaomi.com>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Cc: stable@vger.kernel.org
Link: https://patch.msgid.link/20260616112017.1681372-1-zhanxusheng@xiaomi.com
Commit f24df84cbe ("time/jiffies: Register jiffies clocksource before
usage") moved the jiffies clocksource registration into
clocksource_default_clock(), so that it is registered lazily on the first
call. __clocksource_register() acquires clocksource_mutex, but the first
caller is timekeeping_init(), which invokes clocksource_default_clock()
while holding tk_core.lock, a raw spinlock.
Acquiring a sleeping mutex while holding a raw spinlock is invalid.
The default clocksource only has to be registered before
tk_setup_internals() consumes its mult/shift/maxadj. Neither
clocksource_default_clock(), the ->enable() callback, nor the registration
itself need tk_core.lock, so fetch and enable the clock before acquiring
the lock. This preserves the "register before usage" ordering while
keeping clocksource_mutex out of the raw spinlock section.
clocksource_default_clock() has a second caller,
clocksource_done_booting(), which invokes it with clocksource_mutex already
held. That path avoids a recursive lock because timekeeping_init() has
already run and set cs_jiffies_registered, so the registration is skipped
there. This change does not alter that; it only fixes the invalid wait
context in timekeeping_init().
Fixes: f24df84cbe ("time/jiffies: Register jiffies clocksource before usage")
Signed-off-by: Mikhail Gavrilov <mikhail.v.gavrilov@gmail.com>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Reported-by: Breno Leitao <leitao@debian.org>
Reported-by: Oleg Nesterov <oleg@redhat.com>
Reviewed-by: Breno Leitao <leitao@debian.org>
Cc: stable@vger.kernel.org
Link: https://patch.msgid.link/20260616070914.65818-1-mikhail.v.gavrilov@gmail.com
-----BEGIN PGP SIGNATURE-----
iQJIBAABCgAyFiEES0KozwfymdVUl37v6iDy2pc3iXMFAmowqboUHHBhdWxAcGF1
bC1tb29yZS5jb20ACgkQ6iDy2pc3iXOVWhAAurksqggNN38womlfvPEVsKshcmYL
2AyOkw8v7SzUUugAmDPFgcEv38MY8/7BJm0J2h39FAu2UDXVwBt1v3xBVMr95AMU
NrS6/z9UWwsJabGPs8L8rqISKMJBqbhZKye7zMSSAqNDZhmqGsGORoTz03t0vhyc
/6fS3TsEbdiFHQ/VKR3mItlIdEU6NO5n3AsTzuvP5xc8vZ1xpQcsXkitKR/86HMW
v9KZqSGpdauZ4QP/NlLLRz99nFaxSUkzrFGj22g1hukwsyKwRe31sbUkLU5c4f9a
EBFI18i2SQ0VTPkpJlo/tIrdJrUDDNTLE69zSXnvdHRZJ4/ZZ08Vw1AifUwA8r+R
oNNzkTDcZn7DtRD8Usk7keFOaQmboFI3rx+NdvA6Sw26076ao2Xx3MTrsEm4lnC8
Q9l406M93sFM1DREnPyFSETdohsAVzAnwwCe/R62xpfnZgIiYTjbWFvYRoMjjrkA
1fC0YgAXcYia8p53tBeKGypC4z3LK3PMgB9O5bxKOI0q8EViISdKfqcj9B+Dlm7P
grCe/RETTAznm+MP11MucO4eh1E4ckBH+tPLNhDMJrC7cVMbh2I1uVoUjXkPiaG2
PILKWCMuc1mO4hXlo5ro+yEvYzmd9qdq/VFXOzCNXw+4KAsmQPrktpz+YMo/Lb8E
pAKr83vdLfw+bVU=
=d85R
-----END PGP SIGNATURE-----
Merge tag 'audit-pr-20260615' of git://git.kernel.org/pub/scm/linux/kernel/git/pcmoore/audit
Pull audit updates from Paul Moore:
- Fix a recursive deadlock when duplicating executable file rules
Avoid multiple lookups and attempted I_MUTEX_PARENT locks when moving
watched files by passing the already resolved inodes through the
audit code.
- Fix removal of executable watch rules after the file is deleted
Prior to this fix we were unable to remove an executable file watch
where the file had been previously deleted due to a negative dentry
check in the code that performs the lookup on the file watches.
- Convert our basic "unsigned" type usage to "unsigned int".
* tag 'audit-pr-20260615' of git://git.kernel.org/pub/scm/linux/kernel/git/pcmoore/audit:
audit: fix recursive locking deadlock in audit_dupe_exe()
audit: fix removal of dangling executable rules
audit: use 'unsigned int' instead of 'unsigned'
This depends on the BPF arena work in bpf-next and should be pulled after
it. for-7.1-fixes was merged in a few times to avoid conflicts with the
feature changes.
- Most of this continues the in-development sub-scheduler support, which
lets a root BPF scheduler delegate to nested sub-schedulers. The
dispatch-path building blocks landed in 7.1. A follow-up patchset in
development will complete enqueue-path support for hierarchical
scheduling. This cycle adds most of that infrastructure:
- Topological CPU IDs (cids): a dense, topology-ordered CPU numbering
where the CPUs of a core, LLC, or NUMA node form contiguous ranges,
so a topology unit becomes a (start, length) slice. Raw CPU numbers
are sparse and don't track topological closeness, which makes them
clumsy for sharding work across sub-schedulers and awkward in BPF.
- cmask: bitmaps windowed over a slice of cid space, so a sub-scheduler
can track, for example, the idle cids of its shard without a full
NR_CPUS cpumask.
- A struct_ops variant that cid-form sub-schedulers register with,
along with the cid-form kfuncs they call.
- BPF arena integration, which sub-scheduler support is built on. The
bpf-next additions let the kernel read and write the BPF scheduler's
arena directly, turning it into a real kernel/BPF shared-memory
channel. Shared state like the per-CPU cmask now lives there.
- scx_qmap is reworked to exercise the new arena and cid interfaces.
- Exit-dump improvements: dump the faulting CPU first, expose the exit
CPU to BPF and userspace, and normalize the dump header.
- Misc kfuncs and cleanups: a task-ID lookup kfunc, __printf checking on
the error and dump formatters, header reorganization, and assorted
fixes.
-----BEGIN PGP SIGNATURE-----
iIQEABYKACwWIQTfIjM1kS57o3GsC/uxYfJx3gVYGQUCajCCYw4cdGpAa2VybmVs
Lm9yZwAKCRCxYfJx3gVYGSnKAP4/v5Y5VCaB/C6a4RaWuhCKI5tR8hLd9zMwJY1h
0KrhQAEAtVVe8FoTQFFZhP12HEuebvNARqJ8E8bDQCb41B/FfQ4=
=Xrkh
-----END PGP SIGNATURE-----
Merge tag 'sched_ext-for-7.2' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/sched_ext
Pull sched_ext updates from Tejun Heo:
"Most of this continues the in-development sub-scheduler support, which
lets a root BPF scheduler delegate to nested sub-schedulers. The
dispatch-path building blocks landed in 7.1. A follow-up patchset in
development will complete enqueue-path support for hierarchical
scheduling. This cycle adds most of that infrastructure:
- Topological CPU IDs (cids): a dense, topology-ordered CPU numbering
where the CPUs of a core, LLC, or NUMA node form contiguous ranges,
so a topology unit becomes a (start, length) slice. Raw CPU numbers
are sparse and don't track topological closeness, which makes them
clumsy for sharding work across sub-schedulers and awkward in BPF.
- cmask: bitmaps windowed over a slice of cid space, so a
sub-scheduler can track, for example, the idle cids of its shard
without a full NR_CPUS cpumask.
- A struct_ops variant that cid-form sub-schedulers register with,
along with the cid-form kfuncs they call.
- BPF arena integration, which sub-scheduler support is built on. The
bpf-next additions let the kernel read and write the BPF
scheduler's arena directly, turning it into a real kernel/BPF
shared-memory channel. Shared state like the per-CPU cmask now
lives there.
- scx_qmap is reworked to exercise the new arena and cid interfaces.
Additionally:
- Exit-dump improvements: dump the faulting CPU first, expose the
exit CPU to BPF and userspace, and normalize the dump header.
- Misc kfuncs and cleanups: a task-ID lookup kfunc, __printf checking
on the error and dump formatters, header reorganization, and
assorted fixes"
* tag 'sched_ext-for-7.2' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/sched_ext: (59 commits)
sched_ext: Add scx_arena_to_kaddr() / scx_kaddr_to_arena()
sched_ext: Make scx_bpf_kick_cid() return s32
sched_ext: Add scx_cmask_test() and scx_cmask_for_each_cid()
tools/sched_ext: Order single-cid cmask helpers as (cid, mask)
sched_ext: Order single-cid cmask helpers as (cid, mask)
selftests/sched_ext: Fix dsq_move_to_local check
sched_ext: Guard BPF arena helper calls to fix 32-bit build
sched_ext: idle: Fix errno loss in scx_idle_init()
sched_ext: Convert ops.set_cmask() to arena-resident cmask
sched_ext: Sub-allocator over kernel-claimed BPF arena pages
sched_ext: Require an arena for cid-form schedulers
sched_ext: Add cmask mask ops
sched_ext: Track bits[] storage size in struct scx_cmask
sched_ext: Rename scx_cmask.nr_bits to nr_cids
tools/sched_ext: scx_qmap: Fix qa arena placement
sched_ext: Mark !CONFIG_EXT_SUB_SCHED dummy stubs static inline
sched_ext: Replace tryget_task_struct() with get_task_struct()
sched_ext: Add scx_task_iter_relock() and use it in scx_root_enable_workfn()
sched_ext: Fix ops_cid layout assert
sched_ext: Use offsetofend on both sides of the ops_cid layout assert
...
- Last cycle deferred css teardown on cgroup removal until the cgroup
depopulated, so a css is not taken offline while tasks can still
reference it. Disabling a controller through cgroup.subtree_control
still had the same problem. This reworks the deferral from per-cgroup
to per-css so that path is covered too.
- New RDMA controller monitoring files: rdma.peak for per-device peak
usage and rdma.events / rdma.events.local for resource-limit
exhaustion. The max-limit parser was rewritten, fixing two input
parsing bugs.
- cpuset: fix a sched-domain leak on the domain-rebuild failure path and
skip a redundant hardwall ancestor scan on v2.
- Misc: pair the remaining lockless cgroup.max.* reads with WRITE_ONCE,
assorted selftest robustness fixes, and doc path corrections.
-----BEGIN PGP SIGNATURE-----
iIQEABYKACwWIQTfIjM1kS57o3GsC/uxYfJx3gVYGQUCajB0sw4cdGpAa2VybmVs
Lm9yZwAKCRCxYfJx3gVYGTZQAPwJy730XnUGvLqSiNZT05xzoq6G16u0cC61/U0g
t+yU3wEA4mvObdARiJufo+f3Nsg1NY0NzF82xrCS60EsY+rXCgQ=
=H56K
-----END PGP SIGNATURE-----
Merge tag 'cgroup-for-7.2' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/cgroup
Pull cgroup updates from Tejun Heo:
- Last cycle deferred css teardown on cgroup removal until the cgroup
depopulated, so a css is not taken offline while tasks can still
reference it. Disabling a controller through cgroup.subtree_control
still had the same problem. This reworks the deferral from per-cgroup
to per-css so that path is covered too.
- New RDMA controller monitoring files: rdma.peak for per-device peak
usage and rdma.events / rdma.events.local for resource-limit
exhaustion. The max-limit parser was rewritten, fixing two input
parsing bugs.
- cpuset: fix a sched-domain leak on the domain-rebuild failure path
and skip a redundant hardwall ancestor scan on v2.
- Misc: pair the remaining lockless cgroup.max.* reads with WRITE_ONCE,
assorted selftest robustness fixes, and doc path corrections.
* tag 'cgroup-for-7.2' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/cgroup: (22 commits)
cgroup: Migrate tasks to the root css when a controller is rebound
docs: cgroup: Fix stale source file paths
cgroup/cpuset: Free sched domains on rebuild guard failure
cgroup: pair max limit READ_ONCE() with WRITE_ONCE()
selftests/cgroup: enable memory controller in hugetlb memcg test
cgroup/rdma: Drop unnecessary READ_ONCE() on event counters
cgroup: Defer kill_css_finish() in cgroup_apply_control_disable()
cgroup: Add per-subsys-css kill_css_finish deferral
cgroup: Move populated counters to cgroup_subsys_state
cgroup: Annotate unlocked nr_populated_* accesses with READ_ONCE/WRITE_ONCE
cgroup: Inline cgroup_has_tasks() in cgroup.h
cgroup/rdma: document rdma.peak, rdma.events and rdma.events.local
cgroup/rdma: add rdma.events.local for per-cgroup allocation failure attribution
cgroup/rdma: add rdma.events to track resource limit exhaustion
cgroup/rdma: add rdma.peak for per-device peak usage tracking
selftests/cgroup: check malloc return value in alloc_anon functions
cgroup/cpuset: Skip hardwall ancestor scan in cpuset v2 in cpuset_current_node_allowed()
selftests/cgroup: fix misleading debug message in test_cgfreezer_time_child
selftests/cgroup: fix child process escaping to parent cleanup in test_cpucg_nice
selftests/cgroup: Add NULL check after malloc in cgroup_util.c
...
- Continued progress toward making alloc_workqueue() unbound by default:
more callers converted to WQ_PERCPU / system_percpu_wq / system_dfl_wq,
and new warnings for queues that use neither WQ_PERCPU nor WQ_UNBOUND
or the legacy system_wq / system_unbound_wq.
- Misc: drop the now-trivial apply_wqattrs_lock()/unlock() wrappers,
forbid the TEST_WORKQUEUE benchmark from being built-in, and fix a
spurious pointer level in the worker debug-dump path.
-----BEGIN PGP SIGNATURE-----
iIQEABYKACwWIQTfIjM1kS57o3GsC/uxYfJx3gVYGQUCajBxOQ4cdGpAa2VybmVs
Lm9yZwAKCRCxYfJx3gVYGeRdAP9/c9aAoUqVwvCjkPTnRQQ85XCzpc5T/aR7sjed
k7844AEAuxGMVPaMD9CNSN8AGOG8oseXjUUh500CRkQJj1Y65wQ=
=WkYL
-----END PGP SIGNATURE-----
Merge tag 'wq-for-7.2' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/wq
Pull workqueue updates from Tejun Heo:
- Continued progress toward making alloc_workqueue() unbound by
default: more callers converted to WQ_PERCPU / system_percpu_wq /
system_dfl_wq, and new warnings for queues that use neither WQ_PERCPU
nor WQ_UNBOUND or the legacy system_wq / system_unbound_wq.
- Misc: drop the now-trivial apply_wqattrs_lock()/unlock() wrappers,
forbid the TEST_WORKQUEUE benchmark from being built-in, and fix a
spurious pointer level in the worker debug-dump path.
* tag 'wq-for-7.2' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/wq:
drm/bridge: anx7625: Add WQ_PERCPU add to alloc_workqueue
wifi: ath6kl: fix invalid workqueue flags in ath6kl_usb_create()
btrfs: Drop WQ_PERCPU from ordered_flags in btrfs_init_workqueues()
workqueue: Add warnings and ensure one among WQ_PERCPU or WQ_UNBOUND is present
workqueue: Add warnings and fallback if system_{unbound}_wq is used
workqueue: drop spurious '*' from print_worker_info() fn declaration
workqueue: forbid TEST_WORKQUEUE from being built-in
workqueue: drop apply_wqattrs_lock()/unlock() wrappers
umh: replace use of system_unbound_wq with system_dfl_wq
rapidio: rio: add WQ_PERCPU to alloc_workqueue users
media: ddbridge: add WQ_PERCPU to alloc_workqueue users
platform: cznic: turris-omnia-mcu: replace use of system_wq with system_percpu_wq
media: synopsys: hdmirx: replace use of system_unbound_wq with system_dfl_wq
virt: acrn: Add WQ_PERCPU to alloc_workqueue users
- Add a missing return value check for module_extend_max_pages()
to prevent a kernel oops on memory allocation failure.
- Force sh_addr to 0 for architecture-specific module sections on
arm, arm64, m68k, and riscv. This prevents non-zero section
addresses when linking modules with ld.bfd -r, which may cause
tools to misbehave and result in worse compressibility.
- Replace pr_warn! with pr_warn_once! for set_param null pointer
warnings in Rust abstractions, now that the _once variant is
available.
Note that as part of our maintainer rotation, Petr Pavlu will be
taking over module subsystem pull requests for the next six months.
Signed-off-by: Sami Tolvanen <samitolvanen@google.com>
-----BEGIN PGP SIGNATURE-----
iHUEABYKAB0WIQSE9au1u/dCZerzchhaByWrOaGnegUCaiycUAAKCRBaByWrOaGn
ekFVAP9dELwbhP0QjWFhewtSMDYXsvqyEMZ91XAc0DX84/8LVgD+MGIPkaH2rO+Q
saUUE+X6zAYpug5pop/GtDpufPRrLwI=
=0DZj
-----END PGP SIGNATURE-----
Merge tag 'modules-7.2-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/modules/linux
Pull modules updates from Sami Tolvanen:
- Add a missing return value check for module_extend_max_pages() to
prevent a kernel oops on memory allocation failure.
- Force sh_addr to 0 for architecture-specific module sections on arm,
arm64, m68k, and riscv. This prevents non-zero section addresses when
linking modules with ld.bfd -r, which may cause tools to misbehave
and result in worse compressibility.
- Replace pr_warn! with pr_warn_once! for set_param null pointer
warnings in Rust abstractions, now that the _once variant is
available.
* tag 'modules-7.2-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/modules/linux:
rust: module_param: add missing newline to pr_warn_once
module: decompress: check return value of module_extend_max_pages()
rust: module_param: use `pr_warn_once!` for null pointer warning
module, riscv: force sh_addr=0 for arch-specific sections
module, m68k: force sh_addr=0 for arch-specific sections
module, arm64: force sh_addr=0 for arch-specific sections
module, arm: force sh_addr=0 for arch-specific sections
-----BEGIN PGP SIGNATURE-----
iQIzBAABCAAdFiEE+soXsSLHKoYyzcli6rmadz2vbToFAmowIHwACgkQ6rmadz2v
bTqRog/+KIO6wpvuizt8N+vzgKeEYR/KlEPEyV5dGAopIk1Ct5Ahzti1/V6zg7hk
oddWhPewF7QncVLdnL6zRCRahXCHOi30148acG8pj3Fwj/iYw/zVihVkppVt9qRU
REAvFj0/hYJOGzttSt9UfQ78ZftrZNhXFCRIK/zUWj72IVMJD04Vbawo8iIixVtu
I+fCB3A1kv+RMlZ/3TWk11z3Fko4lK/jKpujegIzv6s+n+0lAraF0aw0cWMwNpUj
mtJx2PktPHYbjA5CwE1Fyxf6EhV4OhV5gwyET7gzqITG5zjUo45pMNcA7EAxDyiC
NvTGIZMsAC+9iSSLNSfdjBWExXFYHHuCkeoxkw9ei05Qrye/7lYzwwtgImuyhBZG
xNXrnSNCG5BPl+DCLy09LzzwNdRoP7zYZV3QH/ZU0uMTS6foegYoSM8jSa2skmM+
mR8a4qrMeZTv+XWvScduOqtFMBOv3mSPkZMkoyqReLaMIPse5NuUOTxOGn5REu4X
tQLW0GLa6ad/7udv7DYrSfmwK1dihyOPmZl/GMuGroXm4rW4jNojljNBdTFEWjCQ
Rd9qArq1Z1z/g8mWjTdiRRGa/QzF5rmJRLgjjUKIzUlQ5/77t96ls0jLexsRZQx5
hCdu/HoK+pIRIUy20NtdqA8ykowsfUkyEBjYR28GV4RHvtmgW5k=
=rtQD
-----END PGP SIGNATURE-----
Merge tag 'bpf-next-7.2' of git://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next
Pull bpf updates from Alexei Starovoitov:
"Major changes:
- Recover from BPF arena page faults using a scratch page and add
ptep_try_set() for lockless empty-slot installs on x86 and arm64.
This allows BPF kfuncs to access arena pointers directly.
The 'arena_direct_access' stable branch was created for this work
and was pulled into sched-ext and bpf-next trees (Tejun Heo, Kumar
Kartikeya Dwivedi)
- Lift old restriction and support 6+ arguments in BPF programs and
kfuncs on x86 and arm64 (Yonghong Song, Puranjay Mohan)
Other features and fixes:
- Add 24-bit BTF vlen and reclaim unused bits in the BTF UAPI to ease
addition of new BTF kinds (Alan Maguire)
- Raise the maximum BPF call chain depth from 8 to 16 frames (Alexei
Starovoitov)
- Refactor object relationship tracking in the verifier and fix a
dynptr use-after-free bug (Amery Hung)
- Harden the signed program loader and reject exclusive maps as inner
maps (Daniel Borkmann)
- Replace the verifier min/max bounds fields with a circular number
(cnum) representation and improve 32->64 bit range refinements
(Eduard Zingerman)
- Introduce the arena library and runtime (libarena) with a buddy
allocator, rbtree and SPMC queue data structures, ASAN support and
a parallel test harness. Allow subprograms to return arena pointers
and switch to a BTF type-tag based __arena annotation (Emil
Tsalapatis)
- Cache build IDs in the sleepable stackmap path and avoid faultable
build ID reads under mm locks (Ihor Solodrai)
- Introduce the tracing_multi link to attach a single BPF program to
many kernel functions at once. Allow specifying the uprobe_multi
target via FD (Jiri Olsa)
- Extend the bpf_list family of kfuncs with bpf_list_add/del(), and
bpf_list_is_first/is_last/empty() (Kaitao Cheng)
- Extend the BPF syscall with common attributes support for
prog_load, btf_load and map_create (Leon Hwang)
- Wrap rhashtable as BPF map (Mykyta Yatsenko, Herbert Xu)
- Add sleepable support for tracepoint programs and fix deadlocks in
LRU map due to NMI reentry (Mykyta Yatsenko)
- Fix OOB access in bpf_flow_keys, fix nullness analysis of inner
arrays, enforce write checks for global subprograms (Nuoqi Gui)
- Report the maximum combined stack depth and print a breakdown of
instructions processed per subprogram (Paul Chaignon)
- Add an XDP load-balancer benchmark and arm64 JIT support for stack
arguments (Puranjay Mohan)
- Add kfuncs to traverse over wakeup_sources (Samuel Wu)
- Allow sleepable BPF programs to use LPM trie maps directly (Vlad
Poenaru)
- Many more fixes and cleanups across the verifier, BTF, sockmap,
devmap, bpffs, security hooks, s390/riscv/loongarch JITs,
rqspinlock, libbpf, bpftool, selftests"
* tag 'bpf-next-7.2' of git://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next: (336 commits)
selftests/bpf: Work around llvm stack overflow in crypto progs
selftests/bpf: add test for bpf_msg_pop_data() overflow
bpf, sockmap: fix integer overflow in bpf_msg_pop_data() bounds check
sockmap: Fix use-after-free in udp_bpf_recvmsg()
bpf, sockmap: keep sk_msg copy state in sync
bpf, sockmap: Fix wrong rsge offset in bpf_msg_push_data()
bpf, sockmap: reject overflowing copy + len in bpf_msg_push_data()
selftsets/bpf: Retry map update on helper_fill_hashmap()
selftests/bpf: Add test for sleepable lsm_cgroup rejection
selftests/bpf: Add test to verify the fix for bpf_setsockopt() helper
bpf: Fix bpf_get/setsockopt to tos for ipv4-mapped ipv6 socket
selftests/bpf: Avoid static LLVM linking for cross builds
selftests/bpf: Use common CFLAGS for urandom_read
selftests/bpf: Initialize operation name before use
tools/bpf: build: Append extra cflags
libbpf: Initialize CFLAGS before including Makefile.include
bpftool: Append extra host flags
bpftool: Avoid adding EXTRA_CFLAGS to HOST_CFLAGS
bpftool: Pass host flags to bootstrap libbpf
selftests/bpf: correct CONFIG_PPC64 macro name in comment
...
blk_time_get_ns() caches ktime_get_ns() in current->plug->cur_ktime
and marks the task with PF_BLOCK_TS. That cache is only valid while the
task keeps running; if the task is switched out, wall-clock time
advances and the cached value must not be reused when the task runs again.
The existing invalidation covers explicit plug flushes through
__blk_flush_plug(), and the schedule() / rtmutex paths through
sched_update_worker(). It does not cover in-kernel preemption paths such
as preempt_schedule(), preempt_schedule_notrace(), and
preempt_schedule_irq(), which enter __schedule(SM_PREEMPT) directly and
return without calling sched_update_worker().
As a result, a task preempted while holding a plug with PF_BLOCK_TS set
can reuse a stale plug->cur_ktime after it is scheduled back in. blk-iocost
then consumes that stale timestamp through ioc_now(), producing stale vnow
values for throttle decisions, and through ioc_rqos_done(), inflating
on-queue time and feeding false missed-QoS samples into vrate
adjustment.
Move the schedule-side invalidation to finish_task_switch(), which runs
for the scheduled-in task after every actual context switch regardless
of which schedule entry point was used. Keep __blk_flush_plug() as the
explicit flush/finish-plug invalidation path, and remove only the
PF_BLOCK_TS handling from sched_update_worker().
Fixes: 06b23f92af ("block: update cached timestamp post schedule/preemption")
Cc: stable@vger.kernel.org
Signed-off-by: Usama Arif <usama.arif@linux.dev>
Link: https://patch.msgid.link/20260616141604.328820-3-usama.arif@linux.dev
Signed-off-by: Jens Axboe <axboe@kernel.dk>
PF_BLOCK_TS is only set in blk_time_get_ns() when current->plug is
non-NULL, and blk_finish_plug() clears it via __blk_flush_plug()
before NULLing the plug pointer. copy_process() breaks the
invariant by inheriting PF_BLOCK_TS from the parent while resetting
the child's plug to NULL.
Clear PF_BLOCK_TS alongside that assignment so callers can rely on
"PF_BLOCK_TS set implies current->plug != NULL" and dereference
current->plug unguarded.
Fixes: 06b23f92af ("block: update cached timestamp post schedule/preemption")
Cc: stable@vger.kernel.org
Signed-off-by: Usama Arif <usama.arif@linux.dev>
Link: https://patch.msgid.link/20260616141604.328820-2-usama.arif@linux.dev
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Heiko Carstens says:
===================
This is supposed to improve s390 idle time accounting, and brings it
back to the state it was before arch_cpu_idle_time() was removed from
s390 [3].
In result all cpu time accounting is done by the s390 architecture backend
again, instead of having a mix of architecure specific and common code
accounting (common code: idle, s390 architecture: everything else).
===================
Signed-off-by: Alexander Gordeev <agordeev@linux.ibm.com>
- Dump the stack to the buffer on timerlat uret threashold event
Record the stack trace in the buffer for THREAD_URET as well as
THREAD_CONTEXT when the threshold is hit. Otherwise, if the threshold was
not hit at task wakeup, but was at task return, it will not produce a
stack trace making it harder to debug.
- Have osnoise trace prints print to all buffers
The osnoise tracer is allowed to print to the main buffer. Add a
osnoise_print() helper function and use trace_array_vprintk() to print
osnoise output.
-----BEGIN PGP SIGNATURE-----
iIoEABYKADIWIQRRSw7ePDh/lE+zeZMp5XQQmuv6qgUCajDrBBQccm9zdGVkdEBn
b29kbWlzLm9yZwAKCRAp5XQQmuv6qgpwAQDmFBFP3dU6LiGp6U3UTYauAApah401
zXfmNxbGFO5rmwEAxIHCzoG3OZgrjrYhrkW9qBZDho7+1Frt7vphK71fYAE=
=0ayh
-----END PGP SIGNATURE-----
Merge tag 'trace-latency-v7.2' of git://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace
Pull tracing latency updates from Steven Rostedt:
- Dump the stack to the buffer on timerlat uret threashold event
Record the stack trace in the buffer for THREAD_URET as well as
THREAD_CONTEXT when the threshold is hit. Otherwise, if the threshold
was not hit at task wakeup, but was at task return, it will not
produce a stack trace making it harder to debug.
- Have osnoise trace prints print to all buffers
The osnoise tracer is allowed to print to the main buffer. Add a
osnoise_print() helper function and use trace_array_vprintk() to
print osnoise output.
* tag 'trace-latency-v7.2' of git://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace:
tracing/osnoise: Array printk init and cleanup
tracing/osnoise: Dump stack on timerlat uret threshold event
- eprobes: BTF support for dereferencing pointers
. tracing/eprobes: Allow use of BTF names to dereference pointers. Add syntax
to the parsing of eprobes to typecast structure pointer trace event fields,
enabling BTF-based dereferencing instead of relying on manual offsets.
- probes: Improvements and robustness enhancements
. tracing: Use flexible array for entry fetch code. Store probe entry fetch
instructions in the probe_entry_arg allocation via a flexible array
member to simplify memory allocation and lifetime management.
. tracing: Replace BUG_ON with lockdep_assert_held in uprobe_buffer functions.
Replace BUG_ON() calls with lockdep_assert_held() in uprobe buffer
enable/disable paths to prevent kernel crashes and better verify lock ownership.
. tracing/probes: Ensure the uprobe buffer size is bigger than event size.
Add a BUILD_BUG_ON() assertion to guarantee that the per-CPU uprobe
working buffer size is always larger than the maximum probe event size.
-----BEGIN PGP SIGNATURE-----
iQFPBAABCgA5FiEEh7BulGwFlgAOi5DV2/sHvwUrPxsFAmowny4bHG1hc2FtaS5o
aXJhbWF0c3VAZ21haWwuY29tAAoJENv7B78FKz8bLCAH/R4476TaJKhS6iAaMILz
6RaWmbbsrf4BjgMX3PnM3ROJo5oT3lZG3HQZIbrIdpAO8baB0uRLM8LXv3fteSkC
5C+tfAnTg7vGP1XbdGiDLKPcJUgVQo+SOjhT9f7VpTB/nhsNWC3aFVIxeLcbx89U
g3S9cMZ5ErBDnWsDiQ6v2mhNKJa+0f6egutSV7drN1yUkToPAnCfi9UrbMh/Tesh
zYhWcBCCcj3/JFUlPMcqamCWn12162jNp2dqZXTcsMYMOQpFUygeqIVdXlaH5wAp
niBpZrslP46hF84OBIdF8g8yP0xUnTToRMQmyFbeqTU1gmvBNeCV0VLoFB+02/OW
ujA=
=et3e
-----END PGP SIGNATURE-----
Merge tag 'probes-v7.2' of git://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace
Pull probes updates from Masami Hiramatsu:
- BTF support for dereferencing pointers
Add syntax to the parsing of eprobes to typecast structure pointer
trace event fields, enabling BTF-based dereferencing instead of
relying on manual offsets.
- Improvements and robustness enhancements
- Use flexible array for entry fetch code.
Store probe entry fetch instructions in the probe_entry_arg
allocation via a flexible array member to simplify memory
allocation and lifetime management.
- Replace BUG_ON with lockdep_assert_held in uprobe_buffer functions
Replace BUG_ON() calls with lockdep_assert_held() in uprobe buffer
enable/disable paths to prevent kernel crashes and better verify
lock ownership.
- Ensure the uprobe buffer size is bigger than event size.
Add a BUILD_BUG_ON() assertion to guarantee that the per-CPU
uprobe working buffer size is always larger than the maximum probe
event size.
* tag 'probes-v7.2' of git://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace:
tracing/eprobes: Allow use of BTF names to dereference pointers
tracing: Replace BUG_ON with lockdep_assert_held in uprobe_buffer functions
tracing: Use flexible array for entry fetch code
tracing/probes: Ensure the uprobe buffer size is bigger than event size
Fixes to tool and kunit core and new features to both to support
JUnit XML (primitive) and backtrace suppression API:
- bug/kunit: Core support for suppressing warning backtraces
- kunit: tool: Parse and print the reason tests are skipped
- kunit: tool: Add (primitive) support for outputting JUnit XML
- kunit:tool: Don't write to stdout when it should be disabled
- kunit: Add backtrace suppression self-tests
- drm: Suppress intentional warning backtraces in scaling unit tests
- kunit: Add documentation for warning backtrace suppression API
- kunit: Fix spelling mistakes in comments and messages
- gen_compile_commands: Ignore libgcc.a
- kunit: qemu_configs: Add or1k / openrisc configuration
-----BEGIN PGP SIGNATURE-----
iQIzBAABCgAdFiEEPZKym/RZuOCGeA/kCwJExA0NQxwFAmowHNQACgkQCwJExA0N
QxxjdQ/+MibsF3MON3LIOPC2/p23zoAv+jDkw4gxYWC3L9Wh3gqsZLUuYwyGc8Xn
hO9xN7PEXressO4kukveWYnNtt//aXL3gnZ6Zysbt/BCO73LPxy1IGITDFL58ZVT
F12ym07VsbqQIHMXR+ANJfxCE8RuaPzdkSW635sLQAhpB5W1Fb51XXfStbLswrVN
iX8XNnOYGjBES8nDiwFKPCtgwvp19XN4B51iSo3XWU8nhcrrBxrk0ToWfyQVwEA0
T87hRL81cHLH4WREw1lwEVQ+lRY5WGBwQsKB2JFmI3HaTwcHbNY8p1sa/YARspmY
LFCRLZu/KLio0a4ZxaVq690JCeQ2osgBhCvF9B/sxtc90/kYM5QycpnmpdgyPBZq
v0vEnFBG7nAj1epHvmt/S2z11oTpdDbkuTZIEDvW/wsI82wbB6kAkzMY30BRitSJ
E+RLZxTtSIovI44BgFVFypVWRa2XMaCKL2+GGZoIJiGUwCeJZSTs1nKtnkhYuhel
daHosU85uB6GFnKE1L8LVJ+Vii+iOZUXN9F4B+2vCJ+eXm/0t85yIm5zEmTpKvNu
sS8c3KRIkK5kSRitKJtUpiuxUQtPaJCA/w2B9FdPx5+yHmHa+Em3JUn8ZuM1ZRem
lNIy1b0gM+kJvGT8ZN1eNDSwrHrC+cJ3RI4AeBYPcJ2ECnFvTYI=
=nc4g
-----END PGP SIGNATURE-----
Merge tag 'linux_kselftest-kunit-7.2-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/shuah/linux-kselftest
Pull kunit updates from Shuah Khan:
"Fixes to tool and kunit core and new features to both to support JUnit
XML (primitive) and backtrace suppression API:
- Core support for suppressing warning backtraces
- Parse and print the reason tests are skipped
- Add (primitive) support for outputting JUnit XML
- Don't write to stdout when it should be disabled
- Add backtrace suppression self-tests
- Suppress intentional warning backtraces in scaling unit tests
- Add documentation for warning backtrace suppression API
- Fix spelling mistakes in comments and messages
- gen_compile_commands: Ignore libgcc.a
- qemu_configs: Add or1k / openrisc configuration"
* tag 'linux_kselftest-kunit-7.2-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/shuah/linux-kselftest:
kunit:tool: Don't write to stdout when it should be disabled
kunit: tool: Add (primitive) support for outputting JUnit XML
kunit: tool: Parse and print the reason tests are skipped
kunit: Add documentation for warning backtrace suppression API
drm: Suppress intentional warning backtraces in scaling unit tests
kunit: Add backtrace suppression self-tests
bug/kunit: Core support for suppressing warning backtraces
kunit: Fix spelling mistakes in comments and messages
kunit: qemu_configs: Add or1k / openrisc configuration
gen_compile_commands: Ignore libgcc.a
-----BEGIN PGP SIGNATURE-----
iQFPBAABCAA5FiEEe7vIQRWZI0iWSE3xu+CwddJFiJoFAmovtcQbFIAAAAAABAAO
bWFudTIsMi41KzEuMTIsMiwyAAoJELvgsHXSRYia17cH/ROle2R7pTZbk60nupo3
JHWvvOB6j2xh5s4bxbcoBIb8ZdS5L3YeAE8u/1NUGFHckIJ/7/rB3njxeDtoJyIn
K1ER5LD2wd0jf0Z6P2WOB4MmoF0LSi+KcT0xRbMlmcd9KM9xQIFy7aLxYoWXlqqM
T+kDmLuMtUawZGlN+foSFMp2ewO9Iz31kiQACreLa246zTIkCAhqzTBBCFRPkCMX
cEES6J8Xb3YcfO66PUXqUC63pS0ikovP8P3yt92jWgaXTrntGDQYG/zxm2dkdwQl
UklmXA0KT1mkaTA8vUDPdBk9LFWVCdhK2rkMWyeirQZqIsHYGfj8vB1HoR839m6a
vTk=
=YSns
-----END PGP SIGNATURE-----
Merge tag 'slab-for-7.2' of git://git.kernel.org/pub/scm/linux/kernel/git/vbabka/slab
Pull slab updates from Vlastimil Babka:
- Support for "allocation tokens" (currently available in Clang 22+)
for smarter partitioning of kmalloc caches based on the allocated
object type, which can be enabled instead of the "random"
per-caller-address-hash partitioning.
It should be able to deterministically separate types containing a
pointer from those that do not (Marco Elver)
- Improvements and simplification of the kmem_cache_alloc_bulk() and
mempool_alloc_bulk() API. This includes adaptation of callers
(Christoph Hellwig)
- Performance improvements and cleanups related mostly to sheaves
refill (Hao Li, Shengming Hu, Vlastimil Babka)
- Several fixups for the slabinfo tool (Xuewen Wang)
* tag 'slab-for-7.2' of git://git.kernel.org/pub/scm/linux/kernel/git/vbabka/slab:
mm/slab: do not limit zeroing to orig_size when only red zoning is enabled
mm/slub: preserve original size in _kmalloc_nolock_noprof retry path
mm: simplify the mempool_alloc_bulk API
mm/slab: improve kmem_cache_alloc_bulk
mm/slub: detach and reattach partial slabs in batch
mm/slub: introduce helpers for node partial slab state
mm/slub: use empty sheaf helpers for oversized sheaves
tools/mm/slabinfo: remove redundant slab->partial assignment
tools/mm/slabinfo: remove dead assignment in get_obj_and_str()
tools/mm/slabinfo: Fix trace disable logic inversion
MAINTAINERS: add slab-related scripts and tools to SLAB ALLOCATOR
mm/slub: fix typo in sheaves comment
mm, slab: simplify returning slab in __refill_objects_node()
mm, slab: add an optimistic __slab_try_return_freelist()
slab: fix kernel-docs for mm-api
slab: improve KMALLOC_PARTITION_RANDOM randomness
slab: support for compiler-assisted type-based slab cache partitioning
mm/slub: defer freelist construction until after bulk allocation from a new slab
The former s390 specific arch_cpu_idle_time() implementation was
removed, since its implementation was racy and reported idle time
could go backwards [1].
However this removal was not necessary, since independently of the s390
architecture specific races there exists the iowait counter update race,
which can also lead to reported idle time going backwards [2].
With Frederic Weisbecker's recent cpu idle time accounting refactoring
kernel_cpustat got a sequence counter. Use this to implement s390 specific
variants of kcpustat_field_idle() and kcpustat_field_iowait(). This is
logically a revert of [1] and moves cpu idle time accounting back into s390
architecture code, which is also more precise than the dyntick idle time
accounting by nohz/scheduler.
For comparing cross cpu time stamps it is necessary to use the stcke
instead of the stckf instruction in irq entry path. Furthermore this
open-codes a sequence lock in assembler and C code, which is required to
update the irq entry time stamp to the per cpu idle_data structure in a
race free manner.
[1] commit be76ea6144 ("s390/idle: remove arch_cpu_idle_time() and corresponding code")
[2] commit ead70b7523 ("timers/nohz: Add a comment about broken iowait counter update race")
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
Acked-by: Frederic Weisbecker <frederic@kernel.org>
Signed-off-by: Alexander Gordeev <agordeev@linux.ibm.com>
SMP load-balancing updates:
- A large series to introduce infrastructure for cache-aware load
balancing, with the goal of co-locating tasks that share data
within the same Last Level Cache (LLC) domain. By improving cache
locality, the scheduler can reduce cache bouncing and cache misses,
ultimately improving data access efficiency.
Implemented by Chen Yu and Tim Chen, based on early prototype work
by Peter Zijlstra, with fixes by Jianyong Wu, Peter Zijlstra and
Shrikanth Hegde.
- A series to simplify CONFIG_SCHED_SMT ifdef usage (Shrikanth Hegde)
Fair scheduler updates:
- A series to improve SD_ASYM_CPUCAPACITY scheduling by introducing
SMT awareness (Andrea Righi, K Prateek Nayak)
- A series to optimize cfs_rq and sched_entity allocation for
better data locality (Zecheng Li)
- A preparatory series to change fair/cgroup scheduling to a single
runqueue, without the final change (Peter Zijlstra)
- Auto-manage ext/fair dl_server bandwidth (Andrea Righi)
- Fix cpu_util runnable_avg arithmetic (Hongyan Xia)
- Optimize update_tg_load_avg()'s rate-limiting code
(Rik van Riel)
- Allow account_cfs_rq_runtime() to throttle current hierarchy
(K Prateek Nayak)
- Update util_est after updating util_avg during dequeue, to fix
the util signal update logic, which reduces signal noise
(Vincent Guittot)
Scheduler topology updates:
- Allow multiple domains to claim sched_domain_shared (K Prateek Nayak)
- Add parameter to split LLC (Peter Zijlstra)
Core scheduler updates:
- Use trace_call__<tp>() to save a static branch (Gabriele Monaco)
Scheduler statistics updates:
- Drop now-stale mul_u64_u64_div_u64() cputime over-approximation guard
(Nicolas Pitre)
Deadline scheduler updates:
- Reject debugfs dl_server writes for offline CPUs (Andrea Righi)
- Fix replenishment logic for non-deferred servers
(Yuri Andriaccio)
RT scheduling updates:
- Turn RT_PUSH_IPI default off for non PREEMPT_RT (Steven Rostedt)
- Update default bandwidth for real-time tasks to 1.0
(Yuri Andriaccio)
Proxy scheduling updates:
- A series to implement Optimized Donor Migration for Proxy Execution
(John Stultz, Peter Zijlstra)
- Various proxy scheduling cleanups and fixes
(Peter Zijlstra, K Prateek Nayak)
Misc fixes, improvements and cleanups by Aaron Lu, Andrea Righi,
Zenghui Yu, Chen Yu, Guanyou.Chen, John Stultz, Shrikanth Hegde,
Peter Zijlstra, Liang Luo and Yiyang Chen.
Signed-off-by: Ingo Molnar <mingo@kernel.org>
-----BEGIN PGP SIGNATURE-----
iQJFBAABCgAvFiEEBpT5eoXrXCwVQwEKEnMQ0APhK1gFAmouy9ERHG1pbmdvQGtl
cm5lbC5vcmcACgkQEnMQ0APhK1iLrxAApGz9fhzT8k8slnaSe9kzX5K3OtBLivi1
ZHPeyNHggonASdZ7+apwJq+sdtmAgEw/phMN69fHhteIIa24WCHEpFtLndDHAkAm
eUKRgBI/eAMrOBECldZ69QO1pfYmMV/2aZH/IyTDtgFvXYZS2+UtsE9fPCvx84V2
Uxa5r7a8NTrpbhsZ2YhncpLkrJx0SfaBQBtvckDVWQBbh5stvIZncCatYjX0M33x
yddLVkM7e5f6nenb5+rwW3TMayfzhkeaR+r9vUuZE1mt9ItwIqKfsi9PBFRFXNyr
zzTxGpN6iQCgtFtySJQpIYamjxJhMZYpWcirYXRkXbmjPgc3PIgA9fRcPcj9bZ0A
Z1sM9yBDFjvZ7Eds9TW7iEsmxSpbXDAXDs6SjQa34QU8bh7qJEOG9AQYDcULXZQM
5NK7+b6GEMvcS7P4Y66CKwvxYFFGVGzX3CrFEgngkOWNJeoWhz5EpgmYIocgl/uS
1TPIWASdFLyvvtnGxMx9iWcdb4mGbXvMLbDa3luSBDWAtYxQe0w9iLi87eqypSvU
irZBSbKX737OlrOzNy/d4DKk9sbNsk/In1IK3jfHCt+7iRpVB5aWBfekrg0DIREw
Jo83ln1nApdusK11DLQpbZcUerLHsvmGXRlGJa3IRGAXUS7MAnJ+XpYkKgmjW8Vm
rw4QLfkMbP4=
=V3Wc
-----END PGP SIGNATURE-----
Merge tag 'sched-core-2026-06-14' of gitolite.kernel.org:pub/scm/linux/kernel/git/tip/tip
Pull scheduler updates from Ingo Molnar:
"SMP load-balancing updates:
- A large series to introduce infrastructure for cache-aware load
balancing, with the goal of co-locating tasks that share data
within the same Last Level Cache (LLC) domain. By improving cache
locality, the scheduler can reduce cache bouncing and cache misses,
ultimately improving data access efficiency.
Implemented by Chen Yu and Tim Chen, based on early prototype work
by Peter Zijlstra, with fixes by Jianyong Wu, Peter Zijlstra and
Shrikanth Hegde.
- A series to simplify CONFIG_SCHED_SMT ifdef usage (Shrikanth Hegde)
Fair scheduler updates:
- A series to improve SD_ASYM_CPUCAPACITY scheduling by introducing
SMT awareness (Andrea Righi, K Prateek Nayak)
- A series to optimize cfs_rq and sched_entity allocation for better
data locality (Zecheng Li)
- A preparatory series to change fair/cgroup scheduling to a single
runqueue, without the final change (Peter Zijlstra)
- Auto-manage ext/fair dl_server bandwidth (Andrea Righi)
- Fix cpu_util runnable_avg arithmetic (Hongyan Xia)
- Optimize update_tg_load_avg()'s rate-limiting code (Rik van Riel)
- Allow account_cfs_rq_runtime() to throttle current hierarchy
(K Prateek Nayak)
- Update util_est after updating util_avg during dequeue, to fix the
util signal update logic, which reduces signal noise (Vincent
Guittot)
Scheduler topology updates:
- Allow multiple domains to claim sched_domain_shared (K Prateek
Nayak)
- Add parameter to split LLC (Peter Zijlstra)
Core scheduler updates:
- Use trace_call__<tp>() to save a static branch (Gabriele Monaco)
Scheduler statistics updates:
- Drop now-stale mul_u64_u64_div_u64() cputime over-approximation
guard (Nicolas Pitre)
Deadline scheduler updates:
- Reject debugfs dl_server writes for offline CPUs (Andrea Righi)
- Fix replenishment logic for non-deferred servers (Yuri Andriaccio)
RT scheduling updates:
- Turn RT_PUSH_IPI default off for non PREEMPT_RT (Steven Rostedt)
- Update default bandwidth for real-time tasks to 1.0 (Yuri
Andriaccio)
Proxy scheduling updates:
- A series to implement Optimized Donor Migration for Proxy Execution
(John Stultz, Peter Zijlstra)
- Various proxy scheduling cleanups and fixes (Peter Zijlstra,
K Prateek Nayak)
Misc fixes, improvements and cleanups by Aaron Lu, Andrea Righi,
Zenghui Yu, Chen Yu, Guanyou.Chen, John Stultz, Shrikanth Hegde,
Peter Zijlstra, Liang Luo and Yiyang Chen"
* tag 'sched-core-2026-06-14' of gitolite.kernel.org:pub/scm/linux/kernel/git/tip/tip: (91 commits)
sched/fair: Fix newidle vs core-sched
sched/deadline: Use task_on_rq_migrating() helper
sched/core: Combine separate 'else' and 'if' statements
sched/fair: Fix cpu_util runnable_avg arithmetic
sched/fair: Unify cfs_rq throttling via account_cfs_rq_runtime()
sched/fair: Move the throttled tasks to a local list in tg_unthrottle_up()
sched/fair: Call update_curr() before unthrottling the hierarchy
sched/fair: Use throttled_csd_list for local unthrottle
sched/fair: Convert cfs bandwidth throttling to use guards
sched/fair: Allocate cfs_tg_state with percpu allocator
sched/fair: Remove task_group->se pointer array
sched/fair: Co-locate cfs_rq and sched_entity in cfs_tg_state
sched: restore timer_slack_ns when resetting RT policy on fork
MAINTAINERS: Fix spelling mistake in Peter's name
sched: Simplify ttwu_runnable()
sched/proxy: Remove superfluous clear_task_blocked_in()
sched/proxy: Remove PROXY_WAKING
sched/proxy: Switch proxy to use p->is_blocked
sched/proxy: Only return migrate when needed
sched: Be more strict about p->is_blocked
...
Core perf code updates:
- Reveal PMU type in fdinfo (Chun-Tse Shao)
Intel CPU PMU driver updates:
- Fix various inaccurate hard-coded event configurations
(Dapeng Mi)
Intel uncore PMU driver updates (Zide Chen):
- Fix discovery unit lookup bug for multi-die systems
- Guard against invalid box control address
- Fix PCI device refcount leak in UPI discovery
- Defer ADL global PMON enable to enable_box() to save power
- Fix uncore_die_to_cpu() for offline dies
- Implement global init callback for GNR uncore
AMD CPU PMU driver updates:
- Always use the NMI latency mitigation (Sandipan Das)
AMD uncore PMU driver updates:
- Use Node ID to identify DF and UMC domains (Sandipan Das)
Signed-off-by: Ingo Molnar <mingo@kernel.org>
-----BEGIN PGP SIGNATURE-----
iQJFBAABCgAvFiEEBpT5eoXrXCwVQwEKEnMQ0APhK1gFAmouvyARHG1pbmdvQGtl
cm5lbC5vcmcACgkQEnMQ0APhK1hVSA//RmBeZ9R7QiCJn770b+9ZPvk90PKN8TYQ
TXcPMTB1gZRtw7q4CF8/CFxUvxGySD+/RaekXkQzfATPurwgHkwKv3xpOZuSGcfW
dTsuAV3k+Azj3on3dGjmGZoZeWsdG12c0uF7YNylW/hfPQ/KAW1THVKv7ufX6Mcx
VrEh7oh/5N9paAqtZz5TmesB2wfC05vToj27jE0HL6tnDovF4Q8IsZkfC3P139Kd
iCwvOnc/jpfke26UP5csPnQIF5zTIy2TZrvDQVlqryPZ4NrmMnA6XC117u8VAIm2
29JBP72wbMvRT4hXSs8thfvPj1zF/y3YMCN0D+wkxpYD6gqlsjlHhoY9fJ+Ann7t
gMi0ccSGNC6COVMq1RhzrGdf6JscGcIxXzS1abNJw6cLmfv45UmlbF1ePq9nPV3n
8ZbotrBovS9TonLPmmXNpR9dKKLSH2URCF/xD92KS5Wbvod7sum8N/rpV3Ncmxxf
6g8qTsOxpJjEq9pnTndycqvtzZnyf0/68db5Y+5ag1zKNVPhbsV7AU/tVytZt1Hk
1bTNcc7lCp2GRBPAmDCk5KI235C0UpLS/llG2jEEwXmTzLxk7pzwoJMcl5sABHNA
VL6AhLP8rmHY+r0e8otdI10xg8Pt/WWvKNuOY4i4PiIf34YG1ARoY0EwzmlwcSmF
WsKdagUmIqQ=
=Di4Y
-----END PGP SIGNATURE-----
Merge tag 'perf-core-2026-06-14' of gitolite.kernel.org:pub/scm/linux/kernel/git/tip/tip
Pull performance events updates from Ingo Molnar:
"Core perf code updates:
- Reveal PMU type in fdinfo (Chun-Tse Shao)
Intel CPU PMU driver updates:
- Fix various inaccurate hard-coded event configurations (Dapeng Mi)
Intel uncore PMU driver updates (Zide Chen):
- Fix discovery unit lookup bug for multi-die systems
- Guard against invalid box control address
- Fix PCI device refcount leak in UPI discovery
- Defer ADL global PMON enable to enable_box() to save power
- Fix uncore_die_to_cpu() for offline dies
- Implement global init callback for GNR uncore
AMD CPU PMU driver updates:
- Always use the NMI latency mitigation (Sandipan Das)
AMD uncore PMU driver updates:
- Use Node ID to identify DF and UMC domains (Sandipan Das)"
* tag 'perf-core-2026-06-14' of gitolite.kernel.org:pub/scm/linux/kernel/git/tip/tip: (22 commits)
perf/x86/amd/uncore: Use Node ID to identify DF and UMC domains
perf: Reveal PMU type in fdinfo
perf/x86/intel/uncore: Implement global init callback for GNR uncore
perf/x86/intel/uncore: Fix uncore_die_to_cpu() for offline dies
perf/x86/intel/uncore: Move die_to_cpu() to uncore.c
perf/x86/intel/uncore: Defer ADL global PMON enable to enable_box()
perf/x86/intel/uncore: Fix PCI device refcount leak in UPI discovery
perf/x86/intel/uncore: Guard against invalid box control address
perf/x86/intel/uncore: Fix discovery unit lookup for multi-die systems
perf/x86/amd/core: Always use the NMI latency mitigation
perf/x86/intel: Update event constraints and cache_extra_regsfor CWF
perf/x86/intel: Update event constraints and cache_extra_regsfor SRF
perf/x86/intel: Update event constraints and cache_extra_regsfor NVL
perf/x86/intel: Update event constraints for PTL
perf/x86/intel: Update event constraints and cache_extra_regsfor ARL
perf/x86/intel: Update event constraints and cache_extra_regsfor LNL
perf/x86/intel: Update event constraints and cache_extra_regsfor MTL
perf/x86/intel: Update event constraints and cache_extra_regsfor ADL
perf/x86/intel: Update event constraints for DMR
perf/x86/intel: Update event constraints and cache_extra_regsfor SPR
...
Futex updates:
- Optimize futex hash bucket access patterns (Peter Zijlstra)
- Large series to address the robust futex unlock race for real,
by Thomas Gleixner:
"The robust futex unlock mechanism is racy in respect
to the clearing of the robust_list_head::list_op_pending
pointer because unlock and clearing the pointer are not
atomic. The race window is between the unlock and clearing the
pending op pointer. If the task is forced to exit in this
window, exit will access a potentially invalid pending op
pointer when cleaning up the robust list. That happens if
another task manages to unmap the object containing the lock
before the cleanup, which results in an UAF. In the worst case
this UAF can lead to memory corruption when unrelated content
has been mapped to the same address by the time the access
happens.
User space can't solve this problem without help from the kernel.
This series provides the kernel side infrastructure to help it
along:
1) Combined unlock, pointer clearing, wake-up for the
contended case
2) VDSO based unlock and pointer clearing helpers with a
fix-up function in the kernel when user space was interrupted
within the critical section.
... with help by André Almeida:
- Add a note about robust list race condition (André Almeida)
- Add self-tests for robust release operations (André Almeida)
Context analysis updates:
- Implement context analysis for 'struct rt_mutex'.
(Bart Van Assche)
- Bump required Clang version to 23 (Marco Elver)
Guard infrastructure updates:
- Series to remove NULL check from unconditional guards
(Dmitry Ilvokhin)
Lockdep updates:
- Restore self-test migrate_disable() and sched_rt_mutex
state on PREEMPT_RT (Karl Mehltretter)
Membarriers updates:
- Use per-CPU mutexes for targeted commands (Aniket Gattani)
- Modernize membarrier_global_expedited with cleanup guards
(Aniket Gattani)
- Add rseq stress test for CFS throttle interactions (Aniket Gattani)
percpu-rwsems updates:
- Extract __percpu_up_read() to optimize inlining overhead
(Dmitry Ilvokhin)
Seqlocks updates:
- Allow UBSAN_ALIGNMENT to fail optimizing (Heiko Carstens)
Lock tracing:
- Add contended_release tracepoint to sleepable locks
such as mutexes, percpu-rwsems, rtmutexes, rwsems
and semaphores. (Dmitry Ilvokhin)
MAINTAINERS updates:
- MAINTAINERS: Add RUST [SYNC] entry (Boqun Feng)
Misc updates and fixes by Randy Dunlap, YE WEI-HONG,
Fabricio Parra, Dmitry Ilvokhin and Peter Zijlstra.
Signed-off-by: Ingo Molnar <mingo@kernel.org>
-----BEGIN PGP SIGNATURE-----
iQJFBAABCgAvFiEEBpT5eoXrXCwVQwEKEnMQ0APhK1gFAmoujYURHG1pbmdvQGtl
cm5lbC5vcmcACgkQEnMQ0APhK1jQFw//SNo1zxqGJuVudIWTpk5hUbWO6La9dc2C
2RW4lMNM551AaGFK6V01u1hD7dBZkWBqER+f3+CMtkqi734pAevTfkYJOitnGOOX
YZicGqTufV+Q+hcEiHAY3ADF6384SiyvAGNHL5wk3eLqC+Raug0f0cNt4opzFvr7
I6vRcW02JQMRKBe50+Aoa5zg7ymmbqaH0KkiPvfY5UFl38MKABhkY1izA9o+/Q/S
1/joNo5iQ3Ciqf6tn76vTg4Xiz7napikHB5LfWxorgmQo7FEdUY54GGMaa8aBOPi
EfgIV9p+qOJhmWn4H1HgVSYlnUn3tJICYnVVCruXWXD+pzITS+F5qSQ3p25F3VIt
c3jh3NaVW71pR/Sbj4zGvf8QguJL340RGJs/e5qcj91pjOSCH5B75ruNgFgYK9Lv
iRi6dnHBBHxaN57YHTOrhphvjyfc0L7i6cGyJgHJWwyi0y3XBYlNuP/VLPuwquKl
anK86hlkl9W4kO0OOukD1xhfuRv7066EdJc3iWWlp9yz+yI5xDGuV4FhvEHRoy38
R9VY2bwzgZrW/NkodHIb95+W30qEmeq2z1W4wiBu+mf1JnyW2g/n8jKkc6AINMpI
VGcom3MdinxjUtmybZT9Fx0SzBYCc+aRPkh7lPV7eXdZAWufxUO/6eVKiHBQ2HUO
RFDkOZU0r+c=
=gpHw
-----END PGP SIGNATURE-----
Merge tag 'locking-core-2026-06-14' of gitolite.kernel.org:pub/scm/linux/kernel/git/tip/tip
Pull locking updates from Ingo Molnar:
"Futex updates:
- Optimize futex hash bucket access patterns (Peter Zijlstra)
- Large series to address the robust futex unlock race for real, by
Thomas Gleixner:
"The robust futex unlock mechanism is racy in respect to the
clearing of the robust_list_head::list_op_pending pointer because
unlock and clearing the pointer are not atomic.
The race window is between the unlock and clearing the pending op
pointer. If the task is forced to exit in this window, exit will
access a potentially invalid pending op pointer when cleaning up
the robust list.
That happens if another task manages to unmap the object
containing the lock before the cleanup, which results in an UAF.
In the worst case this UAF can lead to memory corruption when
unrelated content has been mapped to the same address by the time
the access happens.
User space can't solve this problem without help from the kernel.
This series provides the kernel side infrastructure to help it
along:
1) Combined unlock, pointer clearing, wake-up for the
contended case
2) VDSO based unlock and pointer clearing helpers with a
fix-up function in the kernel when user space was interrupted
within the critical section.
... with help by André Almeida:
- Add a note about robust list race condition (André Almeida)
- Add self-tests for robust release operations (André Almeida)
Context analysis updates:
- Implement context analysis for 'struct rt_mutex'. (Bart Van Assche)
- Bump required Clang version to 23 (Marco Elver)
Guard infrastructure updates:
- Series to remove NULL check from unconditional guards (Dmitry
Ilvokhin)
Lockdep updates:
- Restore self-test migrate_disable() and sched_rt_mutex state on
PREEMPT_RT (Karl Mehltretter)
Membarriers updates:
- Use per-CPU mutexes for targeted commands (Aniket Gattani)
- Modernize membarrier_global_expedited with cleanup guards (Aniket
Gattani)
- Add rseq stress test for CFS throttle interactions (Aniket Gattani)
percpu-rwsems updates:
- Extract __percpu_up_read() to optimize inlining overhead (Dmitry
Ilvokhin)
Seqlocks updates:
- Allow UBSAN_ALIGNMENT to fail optimizing (Heiko Carstens)
Lock tracing:
- Add contended_release tracepoint to sleepable locks such as
mutexes, percpu-rwsems, rtmutexes, rwsems and semaphores (Dmitry
Ilvokhin)
MAINTAINERS updates:
- MAINTAINERS: Add RUST [SYNC] entry (Boqun Feng)
Misc updates and fixes by Randy Dunlap, YE WEI-HONG, Fabricio Parra,
Dmitry Ilvokhin and Peter Zijlstra"
* tag 'locking-core-2026-06-14' of gitolite.kernel.org:pub/scm/linux/kernel/git/tip/tip: (36 commits)
locking: Add contended_release tracepoint to sleepable locks
locking/percpu-rwsem: Extract __percpu_up_read()
tracing/lock: Remove unnecessary linux/sched.h include
futex: Optimize futex hash bucket access patterns
rust: sync: completion: Mark inline complete_all and wait_for_completion
MAINTAINERS: Add RUST [SYNC] entry
cleanup: Specify nonnull argument index
selftests: futex: Add tests for robust release operations
Documentation: futex: Add a note about robust list race condition
x86/vdso: Implement __vdso_futex_robust_try_unlock()
x86/vdso: Prepare for robust futex unlock support
futex: Provide infrastructure to plug the non contended robust futex unlock race
futex: Add robust futex unlock IP range
futex: Add support for unlocking robust futexes
futex: Cleanup UAPI defines
x86: Select ARCH_MEMORY_ORDER_TSO
uaccess: Provide unsafe_atomic_store_release_user()
futex: Provide UABI defines for robust list entry modifiers
futex: Move futex related mm_struct data into a struct
futex: Make futex_mm_init() void
...
- Remove the redundant CONFIG_GENERIC_TIME_VSYSCALL after converting the
remaining users over.
- Rework and sanitize the MIPS VDSO handling, so it does not handle the
time related VDSO if there is no VDSO capable clocksource available.
Also stop mapping VDSO data pages unconditionally even if there is no
usage possible.
-----BEGIN PGP SIGNATURE-----
iQJEBAABCgAuFiEEQp8+kY+LLUocC4bMphj1TA10mKEFAmotog4QHHRnbHhAa2Vy
bmVsLm9yZwAKCRCmGPVMDXSYoYD5D/41UOOboDxkv3Ei0gS12RfFL/pHWPotWMvA
eUS4Ohko2pYubZB+iv87QQD5HC6TBN9yXbUjpvS0wN9jVSlXgQOdJ0JkNYYbPfjH
nsBt7g/ksCA0Tv0KxnUUtECCTu6frtXkSi5iDU03DVNvxLMDmRaXg+D10TKGiCfb
q1g2IoenA6YWz8pUkwOSI5wInepN8Dt9s5VXRsBQS9J9Uin/ZWAIwghIEhfFPNZu
braNUSyJ+A7xISipOjWD1kEfBjcUzr24AuMZ9XHpxnykdWoa57qNEoTnjm+bVWpU
hK0ED4nXSM8qxioV1Kye0h+hUi0T7amNtq6OSfAL+oXWHINvUe2PybqK+sZ1pPFb
0MSP/J4mV9CIvriPA33fTXw/VxigI/puCMtGuSLLXHwh2xW8hadz7wGU9lqF4E7E
f+X9NYqZSmx/82z0KKPFtIQyfrozquRBzy2ZiUloh7ek7GKF5tvuN8lds3Ghuf/6
9Hqr/YquG1Zuu+j95ozVIm7fhDjmMktxAwyAbg5WkNGqOEw7MyhoUGXjKKG29uLa
PRitT9DklL5vndzjizkLc1C2DuC++OhpnJCqRWOjJ9/TbvBC6Rn6aKwYyVTFX9xM
4VauNLQg0q9uG0TyW6o3aakaFk9OIKT+vl7bWO0G4g2YZaKDOs7s+b9sdNNNQq5R
lTL5F0KUCQ==
=Z90m
-----END PGP SIGNATURE-----
Merge tag 'timers-vdso-2026-06-13' of gitolite.kernel.org:pub/scm/linux/kernel/git/tip/tip
Pull vdso updates from Thomas Gleixner:
- Remove the redundant CONFIG_GENERIC_TIME_VSYSCALL after converting
the remaining users over.
- Rework and sanitize the MIPS VDSO handling, so it does not handle the
time related VDSO if there is no VDSO capable clocksource available.
Also stop mapping VDSO data pages unconditionally even if there is no
usage possible.
* tag 'timers-vdso-2026-06-13' of gitolite.kernel.org:pub/scm/linux/kernel/git/tip/tip:
MIPS: VDSO: Fold MIPS_CLOCK_VSYSCALL into MIPS_GENERIC_GETTIMEOFDAY
MIPS: VDSO: Gate microMIPS restriction on GCC version
MIPS: VDSO: Fold MIPS_DISABLE_VDSO into MIPS_GENERIC_GETTIMEOFDAY
clocksource/drivers/mips-gic-timer: Only use VDSO_CLOCKMODE_GIC when it is a available
MIPS: csrc-r4k: Only use VDSO_CLOCKMODE_R4K when it is a available
MIPS: VDSO: Only map the data pages when the vDSO is used
MIPS: Introduce Kconfig MIPS_GENERIC_GETTIMEOFDAY
vdso/datastore: Always provide symbol declarations
MAINTAINERS: Add include/linux/vdso_datastore.h to vDSO block
vdso/gettimeofday: Rename __arch_get_vdso_u_timens_data()
vdso/treewide: Drop GENERIC_TIME_VSYSCALL
vdso/vsyscall: Gate update_vsyscall() behind CONFIG_GENERIC_GETTIMEOFDAY
riscv: vdso: Drop CONFIG_GENERIC_TIME_VSYSCALL guard around syscall fallbacks
- Expand timekeeping snapshot mechanisms
The various snapshot functions are mostly used for PTP to collect
"atomic" snapshots of various involved clocks.
They lack support for the recently introduced AUX clocks and do not
provide the underlying counter value (e.g. TSC) to user space. Exposing
the counter value snapshot allows for better control and steering.
Convert the hard wired ktime_get_snapshot() to take a clock ID, which
allows the caller to select the clock ID to be captured along with
CLOCK_MONONOTONIC_RAW. Additionally capture the underlying hardware
counter value and the clock source ID of the counter.
Expand the hardware based snapshot capture where devices provide a
mechanism to snapshot the hardware PTP clock and the system counter
(usually via PCI/PTM) to support AUX clocks and also provide the
captured counter value back to the caller and not only the clock
timestamps derived from it.
- Add a new optional read_snapshot() callback to clocksources
That is required to capture atomic snapshots from clocksources which
are derived from TSC with a scaling mechanism (e.g. Hyper-V, KVMclock).
The value pair is handed back in the snapshot structure to the callers,
so they can do the necessary correlations in a more precise way.
This touches usage sites of the affected functions and data structure all
over the tree, but stays fully backwards compatible for the existing user
space exposed interfaces. New PTP IOCTLs will provide access to the
extended functionality in later kernel versions.
-----BEGIN PGP SIGNATURE-----
iQJEBAABCgAuFiEEQp8+kY+LLUocC4bMphj1TA10mKEFAmotoSAQHHRnbHhAa2Vy
bmVsLm9yZwAKCRCmGPVMDXSYoYWbEACv/g3pGDxWxfzOI2h6vQgGxDvD3LwmdhPE
bzXRRaxp3/J0rZTQmCghknVGDPVjepNQgKkUMXfaFG2UZmiPHG5qVTXO6DddguS4
cQc0SUO3e422lUPCoBmTULZ+vlctb4LJsWXPQHYNKC73KqMJtWte7T2HBiFDK5RB
O0S34DZtkvOW4tHIu0RwlwCXZ0gcO+zsjxKA8K/P6sMtKBQU1/rRkZZCx2KCvq0F
Rx3NTGoY4if/C83YBq1cEn8BvXrcQQH4ZOOWuySsLGJGRPZ1dXGP+JtfRWutk/f9
HZztlaXcEz71dJXlhBxc0Eb/86uC3POEq7ZYvQdzLbsSZ/3AbalksL9CLyxgdHtc
U964SuwOVPcYfEytd4TWb1nu7JgOR0olYK+l4AbCt4EdKst5TADCJ7rtlZV3Idp+
Yg1GN3TwJcKItUNX9Szk+7MbvB8EWOEl7Obahfm48qDK1pqFe08qhOzSCeRXu+Bb
QiupC3ndzUB1Yjf3DPV6wQl4Fl/TscrAVrPlnGCOJEKXtUKFxvcKquy/W29UD//w
NuxKO2zK05UDsbBEwnZiCrdSGGNiLBYUbHfx2UvA7M0rfrrbjmG4rCFPotxhNb54
UuqgdM8G45MkyBV3qSSh3VC0XeD7UqzQtMYgUjjhvtapLlsri69vzL2DnQUcajSG
dgjzIg9O3g==
=kBrz
-----END PGP SIGNATURE-----
Merge tag 'timers-ptp-2026-06-13' of gitolite.kernel.org:pub/scm/linux/kernel/git/tip/tip
Pull timekeeping updates from Thomas Gleixner:
"Updates for NTP/timekeeping and PTP:
- Expand timekeeping snapshot mechanisms
The various snapshot functions are mostly used for PTP to collect
"atomic" snapshots of various involved clocks.
They lack support for the recently introduced AUX clocks and do not
provide the underlying counter value (e.g. TSC) to user space.
Exposing the counter value snapshot allows for better control and
steering.
Convert the hard wired ktime_get_snapshot() to take a clock ID,
which allows the caller to select the clock ID to be captured along
with CLOCK_MONONOTONIC_RAW. Additionally capture the underlying
hardware counter value and the clock source ID of the counter.
Expand the hardware based snapshot capture where devices provide a
mechanism to snapshot the hardware PTP clock and the system counter
(usually via PCI/PTM) to support AUX clocks and also provide the
captured counter value back to the caller and not only the clock
timestamps derived from it.
- Add a new optional read_snapshot() callback to clocksources
That is required to capture atomic snapshots from clocksources
which are derived from TSC with a scaling mechanism (e.g. Hyper-V,
KVMclock).
The value pair is handed back in the snapshot structure to the
callers, so they can do the necessary correlations in a more
precise way.
This touches usage sites of the affected functions and data structure
all over the tree, but stays fully backwards compatible for the
existing user space exposed interfaces. New PTP IOCTLs will provide
access to the extended functionality in later kernel versions"
* tag 'timers-ptp-2026-06-13' of gitolite.kernel.org:pub/scm/linux/kernel/git/tip/tip: (28 commits)
ptp: vmclock: Use hw_cycles from snapshot for precise TSC pairing
x86/kvmclock: Implement read_snapshot() for kvmclock clocksource
clocksource/hyperv: Implement read_snapshot() for TSC page clocksource
timekeeping: Add clocksource read_snapshot() method and hw_cycles to snapshot
ptp: Switch to ktime_get_snapshot_id() for pre/post timestamps
timekeeping: Add support for AUX clock cross timestamping
timekeeping: Remove system_device_crosststamp::sys_realtime
ALSA: hda/common: Use system_device_crosststamp::sys_systime
wifi: iwlwifi: Use system_device_crosststamp::sys_systime
ptp: Use system_device_crosststamp::sys_systime
timekeeping: Prepare for cross timestamps on arbitrary clock IDs
timekeeping: Remove ktime_get_snapshot()
virtio_rtc: Use provided clock ID for history snapshot
net/mlx5: Use provided clock ID for history snapshot
igc: Use provided clock ID for history snapshot
ice/ptp: Use provided clock ID for history snapshot
wifi: iwlwifi: Adopt PTP cross timestamps to core changes
timekeeping: Add CLOCK ID to system_device_crosststamp
timekeeping: Add system_counterval_t to struct system_device_crosststamp
timekeeping: Add CLOCK_AUX support for ktime_get_snapshot_id()
...
- Fix a long standing TOCTOU in get_cpu_sleep_time_us()
- Make the CPU offline NOHZ handling more robust by disabling NOHZ on the
outgoing CPU early instead of creating unneeded state which needs to be
undone.
- Unify idle CPU time accounting instead of having two different
accounting mechanisms. These two different mechanisms are not really
independent, but the different properties can in the worst case cause
that gloabl idle time can be observed going backwards.
- Consolidate the idle/iowait time retrieval interfaces instead of
converting back and forth between them.
- Make idle interrupt time accounting more robust. The original code
assumes that interrupt time accouting is enabled and therefore stops
elapsing idle time while an interrupt is handled in NOHZ dyntick
state. That assumption is not correct as interrupt time accounting can
be disabled at compile and runtime.
- Fix an accounting error between dyntick idle time and dyntick idle
steal time. The stolen time is not accounted and therefore idle time
becomes inaccurate. The stolen time is now accounted after the fact as
there is no way to predict the steal time upfront.
-----BEGIN PGP SIGNATURE-----
iQJEBAABCgAuFiEEQp8+kY+LLUocC4bMphj1TA10mKEFAmotm7YQHHRnbHhAa2Vy
bmVsLm9yZwAKCRCmGPVMDXSYoWEOEADG/RzW9l7fwUcsnubRyHER7GoIEZGP/VO/
nLQnI+M1LwdVyK8Oq8WntSnqOGevVdUKQlLFYMXdwovS8TWCi7PWcQeMIsfJnwT/
6ugwo4E3mSBcseMjN8eHkEYH+1YmN6JSYQ+5eXT0JUXUxJlMFgtv3ZnTbQOF6Y3k
Nkz1THojzdgTMn6WU/01AOXDR8Nhb2gOOQLDF/1ItZWnsDhbrDE2l99OaAELfeIo
8BZFQBPYRfuR8HaGgd8m2OPGnbw+cXMHUrTscMMQvmYv2wJVcNMB4AmVaUmGMRTR
c3xs8QG2nUmIm72ENfe2pPDPyxy9JUJ54ro6/rLtcNQZ2wVCznhNahm6wRUa8f/0
7gnQE5nS2SNtdpL5StDVzk2+AZl6SrU8+ss51f1owPcNwKs+tt9XE+GVyL5wZpC8
IeD2SzVQLIIc0+ZvIkt9n76lLJkiLZkmt6UAnvhiz5NNn8XQlgC/4Uadk9CwC5vx
t6swpE06D7vx7lm+ycfYUl6VPje5g/YvkkDFJqJulIlEDNpvb85+W6lMTyO7rwVl
ln5TzOGPp7lJDwu6dAfg0QtHcUmUIFq05FuOIRNC4ZNxkxL2XDMzoe5IHxcWzU9O
CHFGBUNZf+30vxvKeh+rkjZalt7/vgQaxb5somGmqv8g/6SJtJyB2qlq+gqH/+lQ
MeG8tFGtVQ==
=+cVw
-----END PGP SIGNATURE-----
Merge tag 'timers-nohz-2026-06-13' of gitolite.kernel.org:pub/scm/linux/kernel/git/tip/tip
Pull NOHZ updates from Thomas Gleixner:
- Fix a long standing TOCTOU in get_cpu_sleep_time_us()
- Make the CPU offline NOHZ handling more robust by disabling NOHZ on
the outgoing CPU early instead of creating unneeded state which needs
to be undone.
- Unify idle CPU time accounting instead of having two different
accounting mechanisms. These two different mechanisms are not really
independent, but the different properties can in the worst case cause
that gloabl idle time can be observed going backwards.
- Consolidate the idle/iowait time retrieval interfaces instead of
converting back and forth between them.
- Make idle interrupt time accounting more robust. The original code
assumes that interrupt time accouting is enabled and therefore stops
elapsing idle time while an interrupt is handled in NOHZ dyntick
state. That assumption is not correct as interrupt time accounting
can be disabled at compile and runtime.
- Fix an accounting error between dyntick idle time and dyntick idle
steal time. The stolen time is not accounted and therefore idle time
becomes inaccurate. The stolen time is now accounted after the fact
as there is no way to predict the steal time upfront.
* tag 'timers-nohz-2026-06-13' of gitolite.kernel.org:pub/scm/linux/kernel/git/tip/tip:
sched/cputime: Handle dyntick-idle steal time correctly
sched/cputime: Handle idle irqtime gracefully
sched/cputime: Provide get_cpu_[idle|iowait]_time_us() off-case
tick/sched: Consolidate idle time fetching APIs
tick/sched: Account tickless idle cputime only when tick is stopped
tick/sched: Remove unused fields
tick/sched: Move dyntick-idle cputime accounting to cputime code
tick/sched: Remove nohz disabled special case in cputime fetch
tick/sched: Unify idle cputime accounting
s390/time: Prepare to stop elapsing in dynticks-idle
powerpc/time: Prepare to stop elapsing in dynticks-idle
sched/cputime: Correctly support generic vtime idle time
sched/cputime: Remove superfluous and error prone kcpustat_field() parameter
sched/idle: Handle offlining first in idle loop
tick/sched: Fix TOCTOU in nohz idle time fetch
- Harden the user space controllable hrtimer interfaces further to
protect against unpriviledged DoS attempts by arming timers in the past.
- Add per-capacity hierarchies to the timer migration code to prevent
timer migration accross different capacity domains. This code has been
disabled last minute as there is a pathological problem with SoCs which
advertise a larger number of capacity domains. The problem is under
investigation and the code won't be active before v7.3, but that turned
out to be less intrusive than a full revert as it preserves the
preparatory steps and allows people to work on the final resolution
- Export time namespace functionality as a recent user can be built as a
module.
- Initialize the jiffies clocksource before using it. The recent
hardening against time moving backward requires that the related
members of struct clocksource have been initialized, otherwise it
clamps the readout to 0, which makes time stand sill and causes boot
delays.
- Fix a more than twenty year old PID reference count leak in an error
path of the POSIX CPU timer code.
- The usual small fixes, improvements and cleanups all over the place.
-----BEGIN PGP SIGNATURE-----
iQJEBAABCgAuFiEEQp8+kY+LLUocC4bMphj1TA10mKEFAmotj9MQHHRnbHhAa2Vy
bmVsLm9yZwAKCRCmGPVMDXSYoZgMEADHAbRUevdga3L6BkPy5X/4HwKp5YpwIzAO
NXrIugAWjObQnhiPoVV9IYrBSj7MKX1hXTOIt/4G8vj/LK1Q7f/YtLIhp5mi7Qhe
EJXS5gRvjbHwlOLko+IRyRNJIy0PcL5VRCSEqmS/wbAblfmuGsVXP6Z4mlbWdG3i
hux6w3sbbVy1rYo/hYEM+sA4IATH6cepzSNbT3jStyqZGWYDRuGlimcMGlIKe0hG
mczpQx+MoBTXL3PsZJrTu8cLACSqZRNkiNaX1yh3xKGLQdrchbydSmL55ucUvQQb
HWWotjt0PWHJgWNQVZpwFA+V2otV+XT7nVoxQlOQHwQxe1qroa781Sg6ZPXNH6Av
KH88cvUuu7f6vVRtijUmiar9IVBy4MGZ7CpZLqCEBeAxMhwATidHOzxYZIGKOxJF
k9Im1nIWkem3eMqGdwrmAuDHNQA9KCDgencDpT+6jJ/6rMuezIteVJjlJFsR4KGa
1bXwX6NqlLvs4dWfiihP8mLsn73t1CeN0HJSJkdjib6tHS0AQpVCV35z8QgRmeMs
bSiFfVFutqAwu1KUC3ESKtzGDVirIjI6FAhbUOzE5SUq0APu1gXBih8142jPuQGz
9hyWBdVVsUIfQdEqY9xhSnXhUXcekPOe5rnDGECNP8MHxP9bNASTBHeOY3ELeKxp
7+Bjk34eqg==
=KRcs
-----END PGP SIGNATURE-----
Merge tag 'timers-core-2026-06-13' of gitolite.kernel.org:pub/scm/linux/kernel/git/tip/tip
Pull timer core updates from Thomas Gleixner:
"Updates for the time/timer core subsystem:
- Harden the user space controllable hrtimer interfaces further to
protect against unpriviledged DoS attempts by arming timers in the
past.
- Add per-capacity hierarchies to the timer migration code to prevent
timer migration accross different capacity domains. This code has
been disabled last minute as there is a pathological problem with
SoCs which advertise a larger number of capacity domains. The
problem is under investigation and the code won't be active before
v7.3, but that turned out to be less intrusive than a full revert
as it preserves the preparatory steps and allows people to work on
the final resolution
- Export time namespace functionality as a recent user can be built
as a module.
- Initialize the jiffies clocksource before using it. The recent
hardening against time moving backward requires that the related
members of struct clocksource have been initialized, otherwise it
clamps the readout to 0, which makes time stand sill and causes
boot delays.
- Fix a more than twenty year old PID reference count leak in an
error path of the POSIX CPU timer code.
- The usual small fixes, improvements and cleanups all over the
place"
* tag 'timers-core-2026-06-13' of gitolite.kernel.org:pub/scm/linux/kernel/git/tip/tip: (31 commits)
posix-cpu-timers: Fix pid refcount leak in do_cpu_nanosleep() error path
time/jiffies: Register jiffies clocksource before usage
timers/migration: Temporarily disable per capacity hierarchies
timers/migration: Turn tmigr_hierarchy level_list into a flexible array
timers/migration: Deactivate per-capacity hierarchies under nohz_full
timers/migration: Fix hotplug migrator selection target on asymetric capacity machines
ntsync: Honour caller's time namespace for absolute MONOTONIC timeouts
time/namespace: Export init_time_ns and do_timens_ktime_to_host()
timers/migration: Update stale @online doc to @available
timers: Fix flseep() typo in kernel-doc comment
hrtimer: Fix the bogus return type of __hrtimer_start_range_ns()
hrtimer: Return ktime_t from hrtimer_get_next_event()/hrtimer_next_event_without()
clocksource: Clean up clocksource_update_freq() functions
alarmtimer: Remove stale return description from alarm_handle_timer()
selftests/posix_timers: Use CLOCK_THREAD_CPUTIME_ID for ITIMER_PROF measurements
scripts/timers: Add timer_migration_tree.py
timers/migration: Handle capacity in connect tracepoints
timers/migration: Split per-capacity hierarchies
timers/migration: Track CPUs in a hierarchy
timers/migration: Abstract out hierarchy to prepare for CPU capacity awareness
...
- Add devm helpers for clocksources, which allows to simplify driver
teardown and probe failure handling.
- More module conversion work
- Update the support for the ARM EL2 virtual timer including the required
ACPI changes.
- Add clockevent and clocksource support for the TI Dual Mode Timer
- Fix the support for multiple watchdog instances in the TEGRA186 driver
- Add D1 timer support to the SUN5I driver
- The usual devicetree updates, cleanups and small fixes all over the place
-----BEGIN PGP SIGNATURE-----
iQJEBAABCgAuFiEEQp8+kY+LLUocC4bMphj1TA10mKEFAmotjBUQHHRnbHhAa2Vy
bmVsLm9yZwAKCRCmGPVMDXSYoaXND/0Z+wx27W4g4n9OTc//dfG+iV4eg05On9Fg
LhlnyanjvIiZwWEBR0AdLDFHgxCqQV0fHpgldsz/Qlk/5uXcpsrGIhqwAiWynppU
SLcZyW5XrZp4syTQfIxaAwAqMzGfW3nHWEkaRYCo/Q8oTDfvrXzY0AWYoK+bDkL3
NmnIuNGdLD+VoD4E3tn0/Q/MTtA4+DI9Cq98S5lOOchkO01ZOxx9OTalhAGFEvOo
zk6YSW2qeUtvc3stahPSvAOaC6EcAalp7wNjWDeHV/c2kUmoKlGpsgvb8Mt+0vPD
exNdctuTFGU2zrS1s0IB22n8RkSCzloxiTBs4Ec88ecn+cLIwq3oeuqchNiH/Quw
YzQWuYhG1ML5i5/g3sAnYu+Hj5DOvB/0IBiYlCxKH4rf49RtBn1AQWt4OzdKzuQS
/fnYqw+4N5XCodHzQbFv7Kbs+21lXXYubt9RLvSl6QYc4xfJRfWCS7rJdXUExw3D
5visGs+tUozi8C23t+BCWJ7jJsd0uNk51gmvOPUI3ybgL5YFwz1YdY4OwAAQT57i
MGiDbC4nMz7nlyWkl40ZVjOvl8bNvCGozq30tRux+CPmpYUi/87ToraKZy8ulSra
+26/ekATKXSfcyUh+TinA10AnnmCdxUyfA1WpZ6z/bCVdZqqHXK6HBf0OY8+u3um
zhrXH2BJBQ==
=4vQE
-----END PGP SIGNATURE-----
Merge tag 'timers-clocksource-2026-06-13' of gitolite.kernel.org:pub/scm/linux/kernel/git/tip/tip
Pull clocksource updates from Thomas Gleixner:
"Updates for clocksource/clockevent drivers:
- Add devm helpers for clocksources, which allows to simplify driver
teardown and probe failure handling.
- More module conversion work
- Update the support for the ARM EL2 virtual timer including the
required ACPI changes.
- Add clockevent and clocksource support for the TI Dual Mode Timer
- Fix the support for multiple watchdog instances in the TEGRA186
driver
- Add D1 timer support to the SUN5I driver
- The usual devicetree updates, cleanups and small fixes all over the
place"
* tag 'timers-clocksource-2026-06-13' of gitolite.kernel.org:pub/scm/linux/kernel/git/tip/tip: (24 commits)
clocksource: move NXP timer selection to drivers/clocksource
clocksource/drivers/timer-tegra186: Reserve and service a kernel watchdog
clocksource/drivers/timer-tegra186: Register all accessible watchdog timers
clocksource/drivers/timer-tegra186: Correct num_wdts for Tegra186 and Tegra234
clocksource/drivers/timer-tegra186: Fix support for multiple watchdog instances
clocksource/drivers/timer-ti-dm: Add clockevent support
clocksource/drivers/timer-ti-dm: Add clocksource support
clocksource/drivers/timer-ti-dm: Fix property name in comment
dt-bindings: timer: arm,arch_timer: Fix requirements for interrupt description
clocksource/drivers/arm_arch_timer: Default to EL2 virtual timer when running VHE
ACPI: GTDT: Parse information related to the EL2 virtual timer
ACPI: GTDT: Account for GTDTv3 size when walking the platform timer descriptors
clocksource: Add devm_clocksource_register_*() helpers
clocksource/drivers/sun5i: Add D1 hstimer support
dt-bindings: timer: allwinner,sun5i-a13-hstimer: add H616 and D1
dt-bindings: timer: Add StarFive JHB100 clint
dt-bindings: timer: renesas,rz-mtu3: document RZ/{T2H,N2H}
dt-bindings: timer: renesas,rz-mtu3: Remove TCIU8 interrupt
dt-bindings: timer: Remove sifive,fine-ctr-bits property
clocksource/drivers/timer-of: Make the code compatible with modules
...
- Rework of /proc/interrupt handling:
/proc/interrupts was subject to micro optimizations for a long time,
but most of the low hanging fruit was left on the table. This rework
addresses the major time consuming issues:
- Printing a long series of zeros one by one via a format string
instead of counting subsequent zeros and emitting a string
constant.
- Simplify and cache the conditions whether interrupts should be printed
- Use a proper iteration over the interrupt descriptor xarray
instead of walking and testing one by one.
- Provide helper functions for the architecture code to emit the
architecture specific counters
- Convert the counter structure in x86 to an array, which
simplifies the output and add mechanisms to suppress unused
architecture interrupts, which just occupy space for
nothing. Adopt the new core mechanisms.
This adjusts the gdb scripts related to interrupt counter statitics
to work with the new mechanisms.
- Prevent a string overflow in the /proc/irq/$N/ directory name
creation code.
-----BEGIN PGP SIGNATURE-----
iQJEBAABCgAuFiEEQp8+kY+LLUocC4bMphj1TA10mKEFAmoths8QHHRnbHhAa2Vy
bmVsLm9yZwAKCRCmGPVMDXSYoSoMEACRODwHjNfULjgD2heHbiPKsmPMRZvwO1Ud
xu5XAoNT1gwxnLo4D+KrGCZeyxka+byRpby6eNg7HdRJuu3DUf8umwt/Q472I9a9
ck8OGFp8ntbxnueISKfzxY/O2eXHYxSKmmfZMv3wdOKbvn5OUlFT6eHPjb8PzVUM
7DiXsBL8s3MNHwdJ3grG5lBh60pt5fujzURwYAqvh/i8jlDHxsFRTMGuhR710knr
YZrgZ4/7ffnEbDsn98xezPewRomIbhhEijgfjkkbnYYUub6Y2RHJqOzZhlp6zNgi
vTsU/suW3ryVuzG34rL2uHvsxOcJY1HNA+ING7fkRmPuKxRGKOMBQfPmLQcWqP69
GxwGIlBvNbAEYievgTCS7GNHTy3t0JbxTGhHcBvX3oMtnnOSTttqH9XzvrTwGxjj
fMUykfvB+40Fp47D+t0JDhgyNNEkixSBjW8/gogZFQ0OdMFX6BQZNT/DLhMMC0LR
JbqMpfsffp5+gYam/wixv3sPlxajMpQ2w8ocgyUHVAeFMo1LOY1spUuO3+Tq7nSj
xt95xVg6HQDr+L+8QmZmnRq27uG276CxPpLotbPMsrn0Ax5PL+fymfmVsFmJFjAR
ZHKK3tSD6M94GtklfKlB/yBJGNRafH4MVZbMa0iUxGI6UyAFr/Yror3mfDK9NsIA
WTwwaqI8qw==
=z6vj
-----END PGP SIGNATURE-----
Merge tag 'irq-core-2026-06-13' of gitolite.kernel.org:pub/scm/linux/kernel/git/tip/tip
Pull interrupt core updates from Thomas Gleixner:
- Rework of /proc/interrupt handling:
/proc/interrupts was subject to micro optimizations for a long time,
but most of the low hanging fruit was left on the table. This rework
addresses the major time consuming issues:
- Printing a long series of zeros one by one via a format string
instead of counting subsequent zeros and emitting a string
constant.
- Simplify and cache the conditions whether interrupts should be
printed
- Use a proper iteration over the interrupt descriptor xarray
instead of walking and testing one by one.
- Provide helper functions for the architecture code to emit the
architecture specific counters
- Convert the counter structure in x86 to an array, which
simplifies the output and add mechanisms to suppress unused
architecture interrupts, which just occupy space for nothing.
Adopt the new core mechanisms.
This adjusts the gdb scripts related to interrupt counter statistics
to work with the new mechanisms.
- Prevent a string overflow in the /proc/irq/$N/ directory name
creation code.
* tag 'irq-core-2026-06-13' of gitolite.kernel.org:pub/scm/linux/kernel/git/tip/tip:
x86/irq: Add missing 's' back to thermal event printout
genirq/proc: Speed up /proc/interrupts iteration
genirq/proc: Runtime size the chip name
genirq: Expose irq_find_desc_at_or_after() in core code
genirq: Add rcuref count to struct irq_desc
genirq/proc: Increase default interrupt number precision to four
genirq: Calculate precision only when required
genirq: Cache the condition for /proc/interrupts exposure
genirq/manage: Make NMI cleanup RT safe
genirq: Expose nr_irqs in core code
scripts/gdb: Update x86 interrupts to the array based storage
x86/irq: Move IOAPIC misrouted and PIC/APIC error counts into irq_stats
x86/irq: Suppress unlikely interrupt stats by default
x86/irq: Make irqstats array based
genirq/proc: Utilize irq_desc::tot_count to avoid evaluation
genirq/proc: Avoid formatting zero counts in /proc/interrupts
x86/irq: Optimize interrupts decimals printing
genirq/proc: Size interrupt directory names for 10-digit interrupt numbers
- deferred probe:
- Fix race where deferred probe timeout work could be permanently
canceled by using mod_delayed_work()
- Fix missing jiffies conversion in deferred_probe_extend_timeout()
- Guard timeout extension with delayed_work_pending() to prevent
premature firing
- Use system_percpu_wq instead of the deprecated system_wq
- Update deferred_probe_timeout documentation
- device:
- Replace direct struct device bitfield access (can_match, dma_iommu,
dma_skip_sync, dma_ops_bypass, state_synced, dma_coherent,
of_node_reused, offline, offline_disabled) with flag-based
accessors using bit operations
- Reject devices with unregistered buses
- Delete unused DEVICE_ATTR_PREALLOC()
- Add low-level device attribute macros with const show/store
callbacks, allowing device attributes to reside in read-only memory
- Move core device attributes to read-only memory
- Constify group array pointers in driver_add_groups() /
driver_remove_groups(), struct bus_type, and struct device_driver
- device property:
- Fix fwnode reference leak in fwnode_graph_get_endpoint_by_id()
- Initialize all fields of fwnode_handle in fwnode_init()
- Provide swnode_get()/swnode_put() wrappers around kobject_get/put()
- Allow passing struct software_node_ref_args pointers directly to
PROPERTY_ENTRY_REF()
- driver_override:
- Migrate amba, cdx, vmbus, and rpmsg to the generic driver_override
infrastructure, fixing a UAF from unsynchronized access to
driver_override in bus match() callbacks
- Remove the now-unused driver_set_override()
- firmware loader:
- Fix recursive lock deadlock in device_cache_fw_images() when async
work falls back to synchronous execution
- Fix device reference leak in firmware_upload_register()
- platform:
- Pass KBUILD_MODNAME through the platform driver registration macro
to create module symlinks in sysfs for built-in drivers; move
module_kset initialization to a pure_initcall and tegra cbb
registration to core_initcall to ensure correct ordering
- Pass THIS_MODULE implicitly through a coresight_init_driver() macro
- sysfs:
- Upgrade OOB write detection in sysfs_kf_seq_show() from printk to
WARN
- Add return value clamping to sysfs_kf_read()
- Rust:
- ACPI:
- Fix missing match data for PRP0001 by exporting
acpi_of_match_device()
- Auxiliary:
- Replace drvdata() with dedicated registration data on
auxiliary_device. drvdata() exposed the driver's bus device
private data beyond the driver's own scope, creating ordering
constraints and forcing the data to outlive all registrations
that access it. Registration data is instead scoped structurally
to the Registration object, making lifecycle ordering enforced
by construction rather than convention.
- Rust-native device driver lifetimes (HRT):
- Allow Rust device drivers to carry a lifetime parameter on their
bus device private data, tied to the device binding scope -- the
interval during which a bus device is bound to a driver. Device
resources like pci::Bar<'a> and IoMem<'a> can be stored directly
in the driver's bus device private data with a lifetime bounded
by the binding scope, so the compiler enforces at build time that
they do not outlive the binding. This removes Devres indirection
from every access site and eliminates try_access() failure paths
in destructors.
Bus driver traits use a Generic Associated Type (GAT)
Data<'bound> to introduce the lifetime on the private data,
rather than parameterizing the Driver trait itself. Auxiliary
registration data, where the lifetime is not introduced by a
trait callback but must be threaded through Registration, uses
the ForLt trait (a type-level abstraction for types generic over
a lifetime).
- Misc:
- Fix DT overlayed devices not probing by reverting the broken
treewide overlay fix and re-running fw_devlink consumer pickup when
an overlay is applied to a bound device
- Use root_device_register() for faux bus root device; add sanity
check for failed bus init
- Fix dev_has_sync_state() data race with READ_ONCE() and move it to
base.h
- Avoid spurious device_links warning when removing a device while
its supplier is unbinding
- Switch ISA bus to dynamic root device
- Fix suspicious RCU usage in kernfs_put()
- Remove devcoredump exit callback
- Constify devfreq_event_class
-----BEGIN PGP SIGNATURE-----
iHUEABYKAB0WIQS2q/xV6QjXAdC7k+1FlHeO1qrKLgUCai1lNQAKCRBFlHeO1qrK
LjX1AP4kEb6SJdvBFlG/IJcG9vyqA7+Cig8OIlx9vEX7e7GvNwEAxL2HltAlAd5Q
cOIJOlu0uM+j1mn0JyuM9b0WJrxDLAA=
=E5mG
-----END PGP SIGNATURE-----
Merge tag 'driver-core-7.2-rc1' of gitolite.kernel.org:pub/scm/linux/kernel/git/driver-core/driver-core
Pull driver core updates from Danilo Krummrich:
"Deferred probe:
- Fix race where deferred probe timeout work could be permanently
canceled by using mod_delayed_work()
- Fix missing jiffies conversion in deferred_probe_extend_timeout()
- Guard timeout extension with delayed_work_pending() to prevent
premature firing
- Use system_percpu_wq instead of the deprecated system_wq
- Update deferred_probe_timeout documentation
device:
- Replace direct struct device bitfield access (can_match, dma_iommu,
dma_skip_sync, dma_ops_bypass, state_synced, dma_coherent,
of_node_reused, offline, offline_disabled) with flag-based
accessors using bit operations
- Reject devices with unregistered buses
- Delete unused DEVICE_ATTR_PREALLOC()
- Add low-level device attribute macros with const show/store
callbacks, allowing device attributes to reside in read-only memory
- Move core device attributes to read-only memory
- Constify group array pointers in driver_add_groups() /
driver_remove_groups(), struct bus_type, and struct device_driver
device property:
- Fix fwnode reference leak in fwnode_graph_get_endpoint_by_id()
- Initialize all fields of fwnode_handle in fwnode_init()
- Provide swnode_get()/swnode_put() wrappers around kobject_get/put()
- Allow passing struct software_node_ref_args pointers directly to
PROPERTY_ENTRY_REF()
driver_override:
- Migrate amba, cdx, vmbus, and rpmsg to the generic driver_override
infrastructure, fixing a UAF from unsynchronized access to
driver_override in bus match() callbacks
- Remove the now-unused driver_set_override()
firmware loader:
- Fix recursive lock deadlock in device_cache_fw_images() when async
work falls back to synchronous execution
- Fix device reference leak in firmware_upload_register()
platform:
- Pass KBUILD_MODNAME through the platform driver registration macro
to create module symlinks in sysfs for built-in drivers; move
module_kset initialization to a pure_initcall and tegra cbb
registration to core_initcall to ensure correct ordering
- Pass THIS_MODULE implicitly through a coresight_init_driver() macro
sysfs:
- Upgrade OOB write detection in sysfs_kf_seq_show() from printk to
WARN
- Add return value clamping to sysfs_kf_read()
Rust:
- ACPI:
Fix missing match data for PRP0001 by exporting
acpi_of_match_device()
- Auxiliary:
Replace drvdata() with dedicated registration data on
auxiliary_device. drvdata() exposed the driver's bus device private
data beyond the driver's own scope, creating ordering constraints
and forcing the data to outlive all registrations that access it.
Registration data is instead scoped structurally to the
Registration object, making lifecycle ordering enforced by
construction rather than convention.
- Rust-native device driver lifetimes (HRT):
Allow Rust device drivers to carry a lifetime parameter on their
bus device private data, tied to the device binding scope -- the
interval during which a bus device is bound to a driver. Device
resources like pci::Bar<'a> and IoMem<'a> can be stored directly in
the driver's bus device private data with a lifetime bounded by the
binding scope, so the compiler enforces at build time that they do
not outlive the binding. This removes Devres indirection from every
access site and eliminates try_access() failure paths in
destructors.
Bus driver traits use a Generic Associated Type (GAT) Data<'bound>
to introduce the lifetime on the private data, rather than
parameterizing the Driver trait itself. Auxiliary registration
data, where the lifetime is not introduced by a trait callback but
must be threaded through Registration, uses the ForLt trait (a
type-level abstraction for types generic over a lifetime).
Misc:
- Fix DT overlayed devices not probing by reverting the broken
treewide overlay fix and re-running fw_devlink consumer pickup when
an overlay is applied to a bound device
- Use root_device_register() for faux bus root device; add sanity
check for failed bus init
- Fix dev_has_sync_state() data race with READ_ONCE() and move it to
base.h
- Avoid spurious device_links warning when removing a device while
its supplier is unbinding
- Switch ISA bus to dynamic root device
- Fix suspicious RCU usage in kernfs_put()
- Remove devcoredump exit callback
- Constify devfreq_event_class"
* tag 'driver-core-7.2-rc1' of gitolite.kernel.org:pub/scm/linux/kernel/git/driver-core/driver-core: (81 commits)
software node: allow passing reference args to PROPERTY_ENTRY_REF()
driver core: platform: set mod_name in driver registration
coresight: pass THIS_MODULE implicitly through a macro
kernel: param: initialize module_kset in a pure_initcall
soc/tegra: cbb: Move driver registration from pure_initcall to core_initcall
firmware_loader: Fix recursive lock in device_cache_fw_images()
driver core: Use system_percpu_wq instead of system_wq
driver core: remove driver_set_override()
rpmsg: use generic driver_override infrastructure
Drivers: hv: vmbus: use generic driver_override infrastructure
cdx: use generic driver_override infrastructure
amba: use generic driver_override infrastructure
rust: devres: add 'static bound to Devres<T>
samples: rust: rust_driver_auxiliary: showcase lifetime-bound registration data
rust: auxiliary: generalize Registration over ForLt
rust: types: add `ForLt` trait for higher-ranked lifetime support
gpu: nova-core: separate driver type from driver data
samples: rust: rust_driver_pci: use HRT lifetime for Bar
rust: io: make IoMem and ExclusiveIoMem lifetime-parameterized
rust: pci: make Bar lifetime-parameterized
...
- Fix a race between cpufreq suspend and CPU hotplug during system
shutdown (Tianxiang Chen)
- Avoid redundant target() calls for unchanged limits and fix a typo
in a comment in the cpufreq core (Viresh Kumar)
- Fix concurrency issues related to sysfs attributes access that affect
cpufreq governors using the common governor code (Zhongqiu Han)
- Simplify frequency limit handling in the conservative cpufreq
governor (Lifeng Zheng)
- Fix descriptions of the conservative governor freq_step tunable and
the ondemand governor sampling_down_factor tunable in the cpufreq
documentation (Pengjie Zhang)
- Fix use-after-free and double free during _OSC evaluation in the PCC
cpufreq driver (Yuho Choi)
- Rework the handling of policy min and max frequency values in the
cpufreq core to allow drivers to specify special initial values for
the scaling_min_freq and scaling_max_freq sysfs attributes (Pierre
Gondois)
- Add cpufreq scaling support for Qualcomm Shikra SoC (Taniya Das,
Imran Shaik).
- Improve the warning message on HWP-disabled hybrid processors printed
by the intel_pstate driver and sync policy->cur during CPU offline in
it (Yohei Kojima, Fushuai Wang)
- Drop cpufreq support for AMD Elan SC4* (Sean Young)
- Minor fixes for cpufreq drivers (Krzysztof Kozlowski, Akashdeep Kaur,
Hans Zhang, Guangshuo Li, Xueqin Luo)
- Clean up dead dependencies on X86 in the cpufreq Kconfig (Julian
Braha)
- Allow the intel_idle driver to avoid exposing C-states that are
redundant when PC6 is disabled (Artem Bityutskiy)
- Fix memory leak and a potential race in the OPP core (Abdun Nihaal,
Di Shen)
- Mark Rust OPP methods as inline (Nicolás Antinori)
- Fix misc device registration failure path in the PM QoS core (Yuho
Choi)
- Add sysctl interface for DPM watchdog timeouts (Tzung-Bi Shih)
- Use complete() instead of complete_all() in device_pm_sleep_init() to
avoid a false-positive warning from lockdep_assert_RT_in_threaded_ctx()
when CONFIG_PROVE_RAW_LOCK_NESTING is enabled (Jiakai Xu)
- Use a flexible array for CRC uncompressed buffers during hibernation
image saving (Rosen Penev)
- Make the LZ4 algorithm available for hibernation compression (l1rox3)
- Move the preallocate_image() call during hibernation after the
"prepare" phase of the "freeze" transition (Matthew Leach)
- Fix a memory leak in rapl_add_package_cpuslocked() in the intel_rapl
power capping driver and use sysfs_emit() in cpumask_show() in that
driver (Sumeet Pawnikar, Yury Norov)
- Fix ValueError when parsing incomplete device properties in the
pm-graph utility (Gongwei Li)
-----BEGIN PGP SIGNATURE-----
iQFGBAABCAAwFiEEcM8Aw/RY0dgsiRUR7l+9nS/U47UFAmor9R8SHHJqd0Byand5
c29ja2kubmV0AAoJEO5fvZ0v1OO1ODoIAInJsQeeWGXfUq4hAFnoA/gtS2oBw7Kw
QVjoGG/DUbM/+ZtlbUee27pqvaYRY+q/Tv9RWKkl5vckASPFIAgCjImbm0g3P7Rz
9OgoBwppqv0Yn5whFgc7nt9jOa7FQKV+jHiJgzRqwMxDFLkk0LldYVzsYpB6m+Lx
Kca0LBeSv8o16EZX7RChC+6W7E5UhGxekSg9AzMPJTCe0gKciwARYiBhbSU5nAlI
WdL5j+ycR3QTApbHA2tueaPKa1qnomLwJscmsE6UxlbikfcCBFZ9EvIsBnwl1WM4
EcPA57IAYcr29N9Cdh/cs6MFkpWdzDBOs/ECZsgdI6fUWniQrPINThI=
=oafL
-----END PGP SIGNATURE-----
Merge tag 'pm-7.2-rc1' of gitolite.kernel.org:pub/scm/linux/kernel/git/rafael/linux-pm
Pull power management updates from Rafael Wysocki:
"Over a half of the changes here are cpufreq updates that include core
modifications, fixes of the old-style governors, new hardware support
in drivers, assorded driver fixes and cleanups, and the removal of one
driver (AMD Elan SC4*).
Apart from that, the intel_idle driver will now be able to avoid
exposing redundant C-states if PC6 is disabled and there are new
sysctl knobs for device suspend/resume watchdog timeouts, hibernation
gets built-in LZ4 support for image compression and there is the usual
collection of assorted fixes and cleanups.
Specifics:
- Fix a race between cpufreq suspend and CPU hotplug during system
shutdown (Tianxiang Chen)
- Avoid redundant target() calls for unchanged limits and fix a typo
in a comment in the cpufreq core (Viresh Kumar)
- Fix concurrency issues related to sysfs attributes access that
affect cpufreq governors using the common governor code (Zhongqiu
Han)
- Simplify frequency limit handling in the conservative cpufreq
governor (Lifeng Zheng)
- Fix descriptions of the conservative governor freq_step tunable and
the ondemand governor sampling_down_factor tunable in the cpufreq
documentation (Pengjie Zhang)
- Fix use-after-free and double free during _OSC evaluation in the
PCC cpufreq driver (Yuho Choi)
- Rework the handling of policy min and max frequency values in the
cpufreq core to allow drivers to specify special initial values for
the scaling_min_freq and scaling_max_freq sysfs attributes (Pierre
Gondois)
- Add cpufreq scaling support for Qualcomm Shikra SoC (Taniya Das,
Imran Shaik).
- Improve the warning message on HWP-disabled hybrid processors
printed by the intel_pstate driver and sync policy->cur during CPU
offline in it (Yohei Kojima, Fushuai Wang)
- Drop cpufreq support for AMD Elan SC4* (Sean Young)
- Minor fixes for cpufreq drivers (Krzysztof Kozlowski, Akashdeep
Kaur, Hans Zhang, Guangshuo Li, Xueqin Luo)
- Clean up dead dependencies on X86 in the cpufreq Kconfig (Julian
Braha)
- Allow the intel_idle driver to avoid exposing C-states that are
redundant when PC6 is disabled (Artem Bityutskiy)
- Fix memory leak and a potential race in the OPP core (Abdun Nihaal,
Di Shen)
- Mark Rust OPP methods as inline (Nicolás Antinori)
- Fix misc device registration failure path in the PM QoS core (Yuho
Choi)
- Add sysctl interface for DPM watchdog timeouts (Tzung-Bi Shih)
- Use complete() instead of complete_all() in device_pm_sleep_init()
to avoid a false-positive warning from lockdep_assert_RT_in_threaded_ctx()
when CONFIG_PROVE_RAW_LOCK_NESTING is enabled (Jiakai Xu)
- Use a flexible array for CRC uncompressed buffers during
hibernation image saving (Rosen Penev)
- Make the LZ4 algorithm available for hibernation compression
(l1rox3)
- Move the preallocate_image() call during hibernation after the
"prepare" phase of the "freeze" transition (Matthew Leach)
- Fix a memory leak in rapl_add_package_cpuslocked() in the
intel_rapl power capping driver and use sysfs_emit() in
cpumask_show() in that driver (Sumeet Pawnikar, Yury Norov)
- Fix ValueError when parsing incomplete device properties in the
pm-graph utility (Gongwei Li)"
* tag 'pm-7.2-rc1' of gitolite.kernel.org:pub/scm/linux/kernel/git/rafael/linux-pm: (40 commits)
PM: dpm_watchdog: Add sysctl interface for DPM watchdog timeouts
PM: QoS: Fix misc device registration unwind
cpufreq: Use policy->min/max init as QoS request
cpufreq: Remove driver default policy->min/max init
cpufreq: Set default policy->min/max values for all drivers
cpufreq: Extract cpufreq_policy_init_qos() function
cpufreq: Documentation: fix conservative governor freq_step description
cpufreq: ti: Add EPROBE_DEFER for K3 SoCs
cpufreq: qcom: Add cpufreq scaling support for Qualcomm Shikra SoC
dt-bindings: cpufreq: Document Qualcomm Shikra SoC EPSS
powercap: intel_rapl: Use sysfs_emit() in cpumask_show()
cpufreq: governor: Fix stale prev_cpu_nice spike when enabling ignore_nice_load
cpufreq: governor: Fix data races on per-CPU idle/nice baselines
PM: hibernate: Use flexible array for CRC uncompressed buffers
powercap: intel_rapl: Fix memory leak in rapl_add_package_cpuslocked()
PM: hibernate: make LZ4 available for hibernation compression
PM: sleep: Use complete() in device_pm_sleep_init()
opp: rust: mark OPP methods as inline
cpufreq: intel_pstate: Improve warning message on HWP-disabled hybrid CPUs
cpufreq: elanfreq: Drop support for AMD Elan SC4*
...
-----BEGIN PGP SIGNATURE-----
iQGzBAABCgAdFiEEu6QRe/mAUYNn5U0PBYqkjnKWLM8FAmofGTIACgkQBYqkjnKW
LM9Glgv/cb27AeuChYy+nTJdsMX6V28mvacJP0nTDl4V//OvWYyehk1IKaQIvicY
aNB0+zFQzMknnDvXL3eK4eyYKsxKhjbBiWSizrLxRLTXD37sgN9nZm+SdHklZ338
f2V5yZjON0+zX1OA5pYyJ15Wc0QtIMd173fYaZdfRHSJwtat0s3J2Ceshltq8qCH
egHWvpCXtI8RlqC93zsU89+aU43A+yb1H306VlCvdPsfTl/An/zQW39NfQttS0qe
9qBD/3o0g2iA4A91qGda+NKlcVHenglbds7muuoCdCUR15n4u9ijVaAXsqLDFJhW
mxLyQr77r6xBwOD9yiPD0pEWsWDByBkJQybW6gWQ2tBnEFPrDlYp65GBCLqvdU31
NSH+jNSgoxNNtNdmxQNUc+LwomPH4tJ33CiQCdwmoErGyfeDOcfNKokydEtKwzjO
3b06vWW5Ae9r/yA/PqgkCg9FckpOGC9nVtFWauWGBjhXkwSUigalgOATFb/rw0wi
0uypCILs
=Hsv1
-----END PGP SIGNATURE-----
Merge tag 'rcu.release.v7.2' of gitolite.kernel.org:pub/scm/linux/kernel/git/rcu/linux
Pull RCU updates from Uladzislau Rezki:
"Torture test updates:
- Improve kvm-series.sh script by adding examples in its header
comment
- Lazy RCU is more fully tested now by replacing call_rcu_hurry()
with call_rcu() and doing rcu_barrier() to motivate lazy callbacks
during a stutter pause
- Add more synonyms for the "--do-normal" group of torture.sh
command-line arguments
Misc changes:
- Reduce stack usage of nocb_gp_wait() to address frame size warning
when built with CONFIG_UBSAN_ALIGNMENT
- The synchronize_rcu() call can detect the flood and latches a
normal/default path temporary switching to wait_rcu_gp() path
- Document using rcu_access_pointer() to fetch the old pointer for
lockless cmpxchg() updates
- Simplify some RCU code using clamp_val()
- Fix a kerneldoc header comment typo in srcu_down_read_fast()"
* tag 'rcu.release.v7.2' of gitolite.kernel.org:pub/scm/linux/kernel/git/rcu/linux:
rcu/nocb: reduce stack usage in nocb_gp_wait()
rcu-tasks: Fix possible boot-time tests failed for the call_rcu_tasks()
rcu: Latch normal synchronize_rcu() path on flood
rcu: Document rcu_access_pointer() feeding into cmpxchg()
rcu: Simplify param_set_next_fqs_jiffies() by applying clamp_val()
rcu: Simplify rcu_do_batch() by applying clamp()
checkpatch: Undeprecate rcu_read_lock_trace() and rcu_read_unlock_trace()
srcu: Fix kerneldoc header comment typo in srcu_down_read_fast()
torture: Allow "norm" abbreviation for "normal"
torture: Improve kvm-series.sh header comment
torture: Add torture_sched_set_normal() for user-specified nice values
rcutorture: Fully test lazy RCU
Allow uprobe_multi link to identify the target binary by an already
opened file descriptor.
Adding new BPF_F_UPROBE_MULTI_PATH_FD flag and the path_fd field for
the attr.link_create.uprobe_multi struct.
When the flag is set, we resolve the target from path_fd, without the
flag, we keep the existing string path behavior.
I don't see a use case for supporting O_PATH file descriptors, because
we need to read the binary first to get probes offsets, so I'm using
the CLASS(fd, f), which fails for O_PATH fds.
Assisted-by: Codex:GPT-5.4
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
Link: https://lore.kernel.org/r/20260611114230.950379-4-jolsa@kernel.org
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Resolve the uprobe_multi user path with user_path_at() instead of copying
the string with strndup_user() and passing it to kern_path(). This removes
the temporary allocation and keeps the lookup logic in one helper.
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
Link: https://lore.kernel.org/r/20260611114230.950379-3-jolsa@kernel.org
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Please consider pulling these changes from the signed vfs-7.2-rc1.procfs tag.
Thanks!
Christian
-----BEGIN PGP SIGNATURE-----
iHUEABYKAB0WIQRAhzRXHqcMeLMyaSiRxhvAZXjcogUCaiwLKgAKCRCRxhvAZXjc
otp1AQD6ONFiqQZJhnHbVg1MJlA8iP80x0qlil0OKGUMOby4NAD+MZymcLjtP+4O
T1b9Uca2/qz+L1F5o7vJAD3OAd4BzQA=
=/KGF
-----END PGP SIGNATURE-----
Merge tag 'vfs-7.2-rc1.procfs' of git://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs
Pull procfs updates from Christian Brauner:
- Revamp fs/filesystems.c
The file was a mess with a hand-rolled linked list in desperate need
of a cleanup. The filesystems list is now RCU-ified, /proc files can
be marked permanent from outside fs/proc/, and the string emitted
when reading /proc/filesystems is pre-generated and cached instead of
pointer-chasing and printfing entry by entry on every read.
The file is read frequently because libselinux reads it and is linked
into numerous frequently used programs (even ones you would not
suspect, like sed!). Scalability also improves since reference
maintenance on open/close is bypassed.
open+read+close cycle single-threaded (ops/s):
before: 442732
after: 1063462 (+140%)
open+read+close cycle with 20 processes (ops/s):
before: 606177
after: 3300576 (+444%)
A follow-up patch adds missing unlocks in some corner cases and
tidies things up.
- Relax the mount visibility check for subset=pid mounts
When procfs is mounted with subset=pid, all static files become
unavailable and only the dynamic pid information is accessible. In
that case there is no point in imposing the full mount visibility
restrictions on the mounter - everything that can be hidden in procfs
is already inaccessible. These restrictions prevented procfs from
being mounted inside rootless containers since almost all container
implementations overmount parts of procfs to hide certain
directories.
As part of this /proc/self/net is only shown in subset=pid mounts for
CAP_NET_ADMIN, reconfiguring subset=pid is rejected, the
SB_I_USERNS_VISIBLE superblock flag is replaced with an
FS_USERNS_MOUNT_RESTRICTED filesystem flag, fully visible mounts are
recorded in a list, and the mount restrictions are finally
documented.
- Protect ptrace_may_access() with exec_update_lock in procfs
Most uses of ptrace_may_access() in procfs should hold
exec_update_lock to avoid TOCTOU issues with concurrent privileged
execve() (like setuid binary execution).
This fixes the easy cases - the owner and visibility checks and the
FD link permission checks - with the gnarlier ones to follow later.
* tag 'vfs-7.2-rc1.procfs' of git://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs:
fs: fix ups and tidy ups to /proc/filesystems caching
proc: protect ptrace_may_access() with exec_update_lock (FD links)
proc: protect ptrace_may_access() with exec_update_lock (part 1)
docs: proc: add documentation about mount restrictions
proc: handle subset=pid separately in userns visibility checks
proc: prevent reconfiguring subset=pid
proc: subset=pid: Show /proc/self/net only for CAP_NET_ADMIN
fs: cache the string generated by reading /proc/filesystems
sysfs: remove trivial sysfs_get_tree() wrapper
fs: RCU-ify filesystems list
fs: move SB_I_USERNS_VISIBLE to FS_USERNS_MOUNT_RESTRICTED
proc: allow to mark /proc files permanent outside of fs/proc/
namespace: record fully visible mounts in list
Please consider pulling these changes from the signed vfs-7.2-rc1.xattr tag.
Thanks!
Christian
-----BEGIN PGP SIGNATURE-----
iHUEABYKAB0WIQRAhzRXHqcMeLMyaSiRxhvAZXjcogUCaiwLKgAKCRCRxhvAZXjc
oiWnAP9ij5MDXEBgslxe/8YzPbyUjIO9qnEJtkGEzL1Z7cyFyAEA8+iAcs8bZmr7
Pzq0A5x3aK17V+34QkgfRiDTjkv0BgY=
=xACw
-----END PGP SIGNATURE-----
Merge tag 'vfs-7.2-rc1.xattr' of git://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs
Pull simple_xattr updates from Christian Brauner:
"This reworks the simple xattr api to make it more efficient and easier
to use for all consumers.
The simple_xattr hash table moves from the inode into a per-superblock
cache, removing the per-inode overhead for the common case of few or
no xattrs. The interface now passes struct simple_xattrs ** so lazy
allocation is handled internally instead of by every caller, kernfs
xattr operations on kernfs nodes shared between multiple superblocks
are properly serialized, and tmpfs constructs "security.foo" xattr
names with kasprintf() instead of kmalloc() plus two memcpy()s.
A follow-up fix links kernfs nodes to their parent before the LSM init
hook runs: with the per-sb cache kernfs_xattr_set() computes the cache
via kernfs_root(kn), which faulted on a freshly allocated node when
selinux_kernfs_init_security() called into it - reproducible as a NULL
pointer dereference on the first cgroup mkdir on SELinux-enabled
systems.
On top of this bpffs gains support for trusted.* and security.* xattrs
so that user space and BPF LSM programs can attach metadata - for
example a content hash or a security label - to pinned objects and
directories and inspect it uniformly like on other filesystems. The
store is in-memory and non-persistent, living only for the lifetime of
the mount like everything else in bpffs"
* tag 'vfs-7.2-rc1.xattr' of git://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs:
bpf: Add simple xattr support to bpffs
kernfs: link kn to its parent before the LSM init hook
simpe_xattr: use per-sb cache
simple_xattr: change interface to pass struct simple_xattrs **
tmpfs: simplify constructing "security.foo" xattr names
kernfs: fix xattr race condition with multiple superblocks
Please consider pulling these changes from the signed kernel-7.2-rc1.task_exec_state tag.
Thanks!
Christian
-----BEGIN PGP SIGNATURE-----
iHUEABYKAB0WIQRAhzRXHqcMeLMyaSiRxhvAZXjcogUCaiwLKgAKCRCRxhvAZXjc
okqlAQDfm+CCTRDZu5EOqYod98xwbmQ/TBJo4rXv/O0E2pRRJwEAqhCNhfTd+MGY
RtkLnY+bdMEI4Ii+JGLAZdWMc3Hu9Q0=
=qF04
-----END PGP SIGNATURE-----
Merge tag 'kernel-7.2-rc1.task_exec_state' of git://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs
Pull task_exec_state updates from Christian Brauner:
"This introduces a new per-task task_exec_state structure and relocates
the dumpable mode and the user namespace captured at execve() from
mm_struct onto it. It stays attached to the task for its full
lifetime.
__ptrace_may_access() and several /proc owner and visibility checks
need to consult two pieces of state for any observable task, including
zombies that have already gone through exit_mm(): the dumpable mode
and the user namespace captured at execve(). Both live on mm_struct
today, which exit_mm() clears from the task long before the task is
reaped. A reader that races with do_exit() observes task->mm == NULL
and either fails the check or falls back to init_user_ns - which
denies legitimate access to non-dumpable zombies that were running in
a nested user namespace.
mm_struct loses ->user_ns and the dumpability bits in ->flags.
MMF_DUMPABLE_BITS is reserved so the MMF_DUMP_FILTER_* layout exposed
via /proc/<pid>/coredump_filter stays stable. task->user_dumpable and
its exit_mm() snapshot are removed.
task_exec_state is the privilege domain established by an execve().
Within a thread group it is shared via refcount; across thread groups
each task has its own:
- CLONE_VM siblings (thread-group members, io_uring workers)
refcount-share the parent's exec_state.
- Non-CLONE_VM clones (fork(), vfork() without CLONE_VM) allocate a
fresh exec_state inheriting the parent's dumpable mode and user_ns.
- execve() in the child allocates a fresh instance and installs it
under task_lock + exec_update_lock via task_exec_state_replace().
- Credential changes (setresuid, capset, ...) and
prctl(PR_SET_DUMPABLE) update dumpability on the current task's
exec_state, i.e., on the thread group's shared instance.
On top of this exec_mmap() no longer tears down the old mm while
holding exec_update_lock for writing and cred_guard_mutex. Neither
lock is needed for that: exec_update_lock only exists to make the mm
swap atomic with the later commit_creds() and all its readers operate
on the new mm; none looks at the detached old mm.
The cost was real: __mmput() runs exit_mmap() over the entire old
address space and can block in exit_aio() waiting for in-flight AIO,
so execve() of a large process blocked ptrace_attach() and every
exec_update_lock reader for the duration of the teardown.
The old mm is now stashed in bprm->old_mm and released from
setup_new_exec() after both locks are dropped, with a backstop in
free_bprm() for the error paths"
* tag 'kernel-7.2-rc1.task_exec_state' of git://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs:
exec: free the old mm outside the exec locks
exec_state: relocate dumpable information
ptrace: add ptracer_access_allowed()
exec: introduce struct task_exec_state
sched/coredump: introduce enum task_dumpable
In do_cpu_nanosleep(), posix_cpu_timer_create() takes a pid reference
via get_pid() and stores it in timer.it.cpu.pid. If the subsequent
posix_cpu_timer_set() call fails, the function returns immediately
without calling posix_cpu_timer_del() to release the pid reference,
causing a leak.
Fix it by calling posix_cpu_timer_del() before the unlock-and-return
on the error path, consistent with the other exit paths in the same
function.
Fixes: 1da177e4c3 ("Linux-2.6.12-rc2")
Signed-off-by: WenTao Liang <vulab@iscas.ac.cn>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Reviewed-by: Frederic Weisbecker <frederic@kernel.org>
Cc: stable@vger.kernel.org
Link: https://patch.msgid.link/20260611161738.97043-1-vulab@iscas.ac.cn
Teddy reported that a XEN HVM has a long boot delay, which was bisected to
the recent enhancements to the negative motion detection. It turned out
that the jiffies clocksource is used in early boot before it is registered,
which leaves the max_delta_raw field at zero. That causes the read out to
be clamped to the max delta of 0, which means time is not making progress.
Cure it by ensuring that it is initialized before its first usage in
timekeeping_init().
Fixes: 76031d9536 ("clocksource: Make negative motion detection more robust")
Reported-by: Teddy Astie <teddy.astie@vates.tech>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Tested-by: Teddy Astie <teddy.astie@vates.tech>
Cc: stable@vger.kernel.org
Link: https://patch.msgid.link/87y0gn3fve.ffs@fw13
Closes: https://lore.kernel.org/all/1780914594.8631fc262581453bbf619ec5b2062170.19ea6c8227b000701b@vates.tech
When a cgroup BPF program exits with 0, bpf_prog_run_array_cg() sets
the hook return value to -EPERM if it is not a valid errno. This is
correct for errno-based hooks, which return 0 on success and negative
errno on failure, but wrong for boolean and void LSM hooks. Boolean
LSM hooks should only return true or false, and void LSM hooks have
no return value at all.
Fix it by skipping setting -EPERM for hooks not returning errno.
Fixes: 69fd337a97 ("bpf: per-cgroup lsm flavor")
Signed-off-by: Xu Kuohai <xukuohai@huawei.com>
Link: https://lore.kernel.org/r/20260610201724.733943-2-xukuohai@huaweicloud.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Generic XDP devmap multi redirect uses skb_clone() for intermediate
destinations and sends the last destination with the original skb. This
can leave multiple destinations sharing the same packet data.
This becomes visible after generic devmap egress-program support was
added: a devmap egress program may mutate packet data, and another
destination sharing the same data can observe that mutation.
Native XDP broadcast redirect does not have this issue because
xdpf_clone() copies the frame data for each destination. Generic XDP
should provide the same per-destination isolation before running a
devmap egress program.
Fix this by making cloned skbs private before running the generic devmap
egress program. Use skb_copy() instead of skb_unshare() so allocation
failure does not consume the skb and the existing caller error paths keep
their ownership semantics.
Fixes: 2ea5eabaf0 ("bpf: devmap: Implement devmap prog execution for generic XDP")
Suggested-by: Jiayuan Chen <jiayuan.chen@linux.dev>
Suggested-by: Jakub Kicinski <kuba@kernel.org>
Reviewed-by: Toke Høiland-Jørgensen <toke@redhat.com>
Signed-off-by: Sun Jian <sun.jian.kdev@gmail.com>
Link: https://lore.kernel.org/r/20260612114032.244616-2-sun.jian.kdev@gmail.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Three more fixes for the DMA-mapping code, related to PCI P2PDMA, DMA
debug and DMA link ranges API (Li RongQing and Jason Gunthorpe).
-----BEGIN PGP SIGNATURE-----
iHUEABYIAB0WIQSrngzkoBtlA8uaaJ+Jp1EFxbsSRAUCaisf2gAKCRCJp1EFxbsS
RASLAP953EuMof1qqGoMSNpDMIYNO0392Tdqyiq+nm1BwnLWfgEA/Q0m9yVtxbR7
zP/dxo+H95OforpTNJYdEWnXOBaAbwM=
=o0ql
-----END PGP SIGNATURE-----
Merge tag 'dma-mapping-7.1-2026-06-11' of git://git.kernel.org/pub/scm/linux/kernel/git/mszyprowski/linux
Pull dma-mapping fix from Marek Szyprowski:
"Three more fixes for the DMA-mapping code, related to PCI P2PDMA, DMA
debug and DMA link ranges API (Li RongQing and Jason Gunthorpe)"
* tag 'dma-mapping-7.1-2026-06-11' of git://git.kernel.org/pub/scm/linux/kernel/git/mszyprowski/linux:
iommu/dma: Do not try to iommu_map a 0 length region in swiotlb
dma-debug: fix physical address retrieval in debug_dma_sync_sg_for_device
dma-mapping: direct: fix missing mapping for THRU_HOST_BRIDGE segments
Merge updates related to system sleep support, two updates of the
intel_rapl power capping driver, and a pm-graph utility fix for
7.2-rc1:
- Add sysctl interface for DPM watchdog timeouts (Tzung-Bi Shih)
- Use complete() instead of complete_all() in device_pm_sleep_init() to
avoid a false-positive warning from lockdep_assert_RT_in_threaded_ctx()
when CONFIG_PROVE_RAW_LOCK_NESTING is enabled (Jiakai Xu)
- Use a flexible array for CRC uncompressed buffers during hibernation
image saving (Rosen Penev)
- Make the LZ4 algorithm available for hibernation compression (l1rox3)
- Move the preallocate_image() call during hibernation after the
"prepare" phase of the "freeze" transition (Matthew Leach)
- Fix a memory leak in rapl_add_package_cpuslocked() in the intel_rapl
power capping driver and use sysfs_emit() in cpumask_show() in that
driver (Sumeet Pawnikar, Yury Norov)
- Fix ValueError when parsing incomplete device properties in the
pm-graph utility (Gongwei Li)
* pm-sleep:
PM: dpm_watchdog: Add sysctl interface for DPM watchdog timeouts
PM: hibernate: Use flexible array for CRC uncompressed buffers
PM: hibernate: make LZ4 available for hibernation compression
PM: sleep: Use complete() in device_pm_sleep_init()
PM: hibernate: call preallocate_image() after freeze prepare
* pm-powercap:
powercap: intel_rapl: Use sysfs_emit() in cpumask_show()
powercap: intel_rapl: Fix memory leak in rapl_add_package_cpuslocked()
* pm-tools:
PM: tools: pm-graph: fix ValueError when parsing incomplete device properties
Merge cpuidle updates, OPP (operating performance points) updates and a
PM QoS update for 7.2-rc1:
- Allow the intel_idle driver to avoid exposing C-states that are
redundant when PC6 is disabled (Artem Bityutskiy)
- Fix memory leak and a potential race in the OPP core (Abdun Nihaal,
Di Shen)
- Mark Rust OPP methods as inline (Nicolás Antinori)
- Fix misc device registration failure path in the PM QoS core (Yuho
Choi)
* pm-cpuidle:
intel_idle: Drop C-states redundant when PC6 is disabled
intel_idle: Introduce a helper for checking PC6
intel_idle: Add constants for MSR_PKG_CST_CONFIG_CONTROL
* pm-opp:
opp: rust: mark OPP methods as inline
OPP: of: Fix potential memory leak in opp_parse_supplies()
OPP: Fix race between OPP addition and lookup
* pm-qos:
PM: QoS: Fix misc device registration unwind
Add the contended_release trace event. This tracepoint fires on the
holder side when a contended lock is released, complementing the
existing contention_begin/contention_end tracepoints which fire on the
waiter side.
This enables correlating lock hold time under contention with waiter
events by lock address.
Add trace_contended_release()/trace_call__contended_release() calls to
the slowpath unlock paths of sleepable locks: mutex, rtmutex, semaphore,
rwsem, percpu-rwsem, and RT-specific rwbase locks.
Where possible, trace_contended_release() fires before the lock is
released and before the waiter is woken. For some lock types, the
tracepoint fires after the release but before the wake. Making the
placement consistent across all lock types is not worth the added
complexity.
For reader/writer locks, the tracepoint fires for every reader releasing
while a writer is waiting, not only for the last reader.
Signed-off-by: Dmitry Ilvokhin <d@ilvokhin.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Acked-by: Paul E. McKenney <paulmck@kernel.org>
Acked-by: Usama Arif <usama.arif@linux.dev>
Link: https://patch.msgid.link/02f4f6c5ce6761e7f6587cf0ff2289d962ecddd4.1780506267.git.d@ilvokhin.com
Move the percpu_up_read() slowpath out of the inline function into a new
__percpu_up_read() to avoid binary size increase from adding a
tracepoint to an inlined function.
Signed-off-by: Dmitry Ilvokhin <d@ilvokhin.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Acked-by: Usama Arif <usama.arif@linux.dev>
Link: https://patch.msgid.link/3dd2a1b9ab4f469e1892766cb63f41d6b0f53d29.1780506267.git.d@ilvokhin.com
Breno reported significant c2c HITM in a futex hash heavy workload.
It turns out that the hash bucket to private hash table reverse pointer
(futex_hash_bucket::priv) was to blame. Notably when the hash buckets are
heavily contended, the: 'fph = bh->priv;' load in futex_hash() will typically
miss and consequently become quite expensive.
Since this load in particular is quite superfluous, removing it is fairly
straight forward. However, removing it does not in fact achieve anything much.
The pain moves to the next user, notably: futex_hash_put().
Therefore rework the whole private hash refcounting to avoid needing this back
pointer (and removing it). Instead of passing around 'struct futex_hash_bucket
*hb', pass around a new structure that contains it and the related 'struct
futex_private_hash *fph' pointer in tandem.
Funnily this turns out to remove more code than it adds and significantly
improves futex hash performance (as measured by 'perf bench futex hash'):
SKL dual socket 112 threads:
Baseline Patched
shared (16k) 1571857 1641435 + 4.4%
autosize (512) 646390 903371 +39.7%
-b 256 464395 587014 +26.4%
-b 512 715687 995943 +39.2%
-b 1024 995085 1396328 +40.3%
-b 2048 1293114 1668395 +29.0%
-b 4096 2124438 2240228 + 5.5%
Zen3 dual socket 256 threads:
Baseline Patched
shared (16k) 1275840 1381279 + 8.2%
autosize (512) 1252745 1482179 +18.3%
-b 256 856274 955455 +11.5%
-b 512 1267490 1544010 +21.8%
-b 1024 1424013 1625424 +14.1%
-b 2048 1505181 1669342 +10.9%
-b 4096 1465993 1688932 +15.2%
AMD EPYC 9D64 (Zen4, single socket) 176 threads:
Baseline Patched Delta
shared (16k) 1,230,599 1,368,655 +11.2%
autosize (1024) 1,285,440 1,556,946 +21.1%
-b 256 1,341,471 1,520,303 +13.3%
-b 512 1,438,330 1,599,319 +11.2%
-b 1024 1,443,772 1,622,493 +12.4%
-b 2048 1,472,108 1,643,975 +11.7%
-b 4096 1,333,098 1,570,897 +17.8%
Reported-by: Breno Leitao <leitao@debian.org>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Tested-by: Breno Leitao <leitao@debian.org>
Tested-by: Thomas Gleixner <tglx@kernel.org>
Link: https://patch.msgid.link/20260610135510.GB1430057@noisy.programming.kicks-ass.net
While testing Prateek's throttle series, I noticed a panic issue when
coresched is enabled and bisected to this patch.
I fed the panic log and this patch to an agent and its analysis looks
correct to me(cpu56 and cpu57 are siblings in a VM):
cpu57 (holds core-wide lock)
pick_next_task() [core scheduling]
for_each_cpu_wrap(i, smt_mask, 57):
i=57: pick_task(rq_57)
pick_task_fair(rq_57)
-> picks task A
rq_57->core_pick = task A
// task_rq(A) == rq_57
i=56: pick_task(rq_56)
pick_task_fair(rq_56)
cfs_rq->nr_queued == 0
goto idle
sched_balance_newidle(rq_56)
raw_spin_rq_unlock(rq_56)
// core-wide lock released
newidle_balance() pulls
task A: rq_57 -> rq_56
// task_rq(A) == rq_56 now
raw_spin_rq_lock(rq_56)
// core-wide lock re-acquired
return > 0
goto again
pick_task_fair(rq_56)
-> picks task A
rq_56->core_pick = task A
// first loop done
// rq_57->core_pick is still task A (set before lock release)
// but task_rq(A) == rq_56 now
next = rq_57->core_pick // = task A
put_prev_set_next_task(rq_57, prev, task A)
__set_next_task_fair(rq_57, task A)
hrtick_start_fair(rq_57, task A)
WARN_ON_ONCE(task_rq(task A) != rq_57)
// task_rq(A) == rq_56
IOW: by allowing pick_task_fair() to do newidle_balance and not returning
RETRY_TASK, it can end up selecting the same task on two CPUs. Restore the
previous state by never doing newidle when core scheduling is enabled.
Tested-by: Sven Schnelle <svens@linux.ibm.com>
Signed-off-by: "Aaron Lu" <ziqianlu@bytedance.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/20260603095108.GA1684319@bytedance.com
The fix in commit abad3d0bad ("bpf: Fix oob access in cgroup local
storage") is still incomplete. The prog-array compatibility check
treats a program with no cgroup storage as compatible with any stored
storage cookie. This allows a storage-less program to bridge a tail
call chain between an entry program and a storage-using callee even
though cgroup local storage at runtime still follows the caller's
context, that is, A -> B(no storage) -> C(storage) path.
Requiring exact cookie equality would break the legitimate case of a
storage-less leaf program being tail called from a storage-using one.
Instead, only accept a zero storage cookie if the program cannot
perform tail calls itself. This keeps A -> B(no storage) working
while rejecting the A -> B(no storage) -> C(storage) bridge.
Fixes: abad3d0bad ("bpf: Fix oob access in cgroup local storage")
Reported-by: Lin Ma <malin89@huawei.com>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Acked-by: Yonghong Song <yonghong.song@linux.dev>
Link: https://lore.kernel.org/r/20260610105539.705887-1-daniel@iogearbox.net
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
The only reason for this janitorial change is that this initialization
adds unnecessary noise to "git grep exit_signal".
args.exit_signal has no effect with CLONE_THREAD, not to mention it is
zero-initialized by the compiler anyway.
Signed-off-by: Oleg Nesterov <oleg@redhat.com>
Acked-by: Jason Wang <jasowang@redhat.com>
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
Message-ID: <acAIm732QPFZs15C@redhat.com>
Map update and delete paths currently call bpf_obj_free_fields() when a
value is being replaced or recycled. That makes field destruction depend
on the context of the update/delete operation. For tracing programs this
can include NMI context, where referenced kptr destructors, uptr
unpinning, and graph root destruction are not generally safe.
Introduce bpf_obj_cancel_fields() for the reusable-value path. It only
performs NMI-safe cleanup for timer, workqueue, and task_work fields.
Fields that need full destruction are left attached to the recycled value
and are destroyed by the final cleanup path instead.
Switch array and hashtab update/delete/recycle paths to this cancel
helper. Keep bpf_obj_free_fields() for final map destruction and for
bpf_mem_alloc destructors. Preallocated hashtabs do not have allocator
destructors, so teardown continues to walk the normal and extra elements
and fully destroy their fields.
This deliberately relaxes the eager-free semantics of map update/delete
for special fields. Programs that relied on a recycled map slot becoming
empty immediately after update/delete were relying on behavior that
cannot be implemented safely from every BPF execution context without
offloading arbitrary destructors.
There is a chance this change breaks programs making assumptions
regarding the eager freeing of fields. If so, we can relax semantics to
cancellation only when irqs_disabled() is true in the future. However,
theoretically, map values that get reused eagerly already have weaker
guarantees as parallel users can recreate freed fields before the new
element becomes visible again.
Fixes: 14a324f6a6 ("bpf: Wire up freeing of referenced kptr")
Signed-off-by: Justin Suess <utilityemal77@gmail.com>
Co-developed-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Link: https://lore.kernel.org/r/20260609202548.3571690-3-memxor@gmail.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
bpf_obj_drop() runs bpf_obj_free_fields() synchronously for
program-allocated objects. When such an object contains NMI unsafe
fields, tracing programs that can run from arbitrary instrumented
context can reach that destruction from unsafe contexts, including NMI.
NMI is likely one instance of this problem, and other instances would
include possible unsafe reentrancy. Deferring bpf_obj_drop() is not
appealing either: it would add delayed-free machinery to a release
operation that otherwise has straightforward synchronous ownership
semantics.
Reject bpf_obj_drop() and bpf_percpu_obj_drop() from tracing programs
that may run from unsafe contexts unless every field in the object's BTF
record is explicitly NMI safe. Do not reject sleepable
BPF_PROG_TYPE_TRACING programs, since they are not the arbitrary/NMI
contexts that motivate the restriction.
Note that while bpf_rb_root and bpf_list_head would be NMI safe on their
own to free, the objects recursively held by them may not be; be
conservative and just mark them as not NMI safe for now.
Use a whitelist for the NMI-safe field set instead of listing only known
NMI unsafe fields. Locks, async fields, unreferenced kptrs, and
refcounts are known to be NMI safe because their destruction is either a
no-op, simple state reset, or async cancellation. Referenced kptrs,
percpu referenced kptrs, uptrs, graph roots, graph nodes, and any future
field type are rejected until audited for arbitrary tracing and NMI
contexts. This is less susceptible to future changes in fields that were
previously safe by exclusion, and to new fields being added without
updating this check.
Convert the existing recursive local-object drop success case to a
syscall program in the same commit, since this verifier change makes the
old tracing program form invalid. The test still exercises
bpf_obj_drop() releasing a referenced task kptr from a safe program
type.
Fixes: ac9f06050a ("bpf: Introduce bpf_obj_drop")
Signed-off-by: Justin Suess <utilityemal77@gmail.com>
Co-developed-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Link: https://lore.kernel.org/r/20260609202548.3571690-2-memxor@gmail.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
- Fix reset ordering on per-task destruction
Reset the task before dropping the slot instead of after, which was
causing out-of-bound memory accesses.
- Fix HA monitor synchronization and cleanup
Ensure synchronous cleanup for HA monitors by running timer callbacks
in RCU read-side critical sections and using synchronize_rcu() during
destruction.
- Avoid armed timers after tasks exit
Add automatic cleanup for per-task HA monitors to prevent timers from
firing after task exit.
- Fix memory ordering for DA/HA monitors
Fix race conditions during monitor start by using release-acquire
semantics for the monitoring flag.
- Fix initialization for DA/HA monitors
Ensure monitors are not initialized relying on potentially corrupted
state like the monitoring flag, that is not reset by all monitors type
and may have an unknown state in monitors reusing the storage
(per-task).
- Fix memory safety in per-task and per-object monitors
Prevent use-after-free and out-of-bounds access by synchronizing with
in-flight tracepoint probes using tracepoint_synchronize_unregister()
before freeing monitor storage or releasing task slots.
- Adjust monitors for preemptible tracepoints
Fix monitors that relied on tracepoints disabling preemption.
Explicitly disable task migration when per-CPU monitors handle events
to avoid accessing the wrong state and update the opid monitor logic.
- Fix incorrect __user specifier usage
Remove __user from a non-pointer variable in the extract_params()
helper.
- Fix bugs in the rv tool
Ensure strings are NUL-terminated, fix substring matching in monitor
searches, and improve cleanup and exit status handling.
- Fix several bugs in rvgen
Fix LTL literal stringification, subparsers' options handling, and
suffix stripping in dot2k.
-----BEGIN PGP SIGNATURE-----
iIoEABYKADIWIQRRSw7ePDh/lE+zeZMp5XQQmuv6qgUCaiiJqhQccm9zdGVkdEBn
b29kbWlzLm9yZwAKCRAp5XQQmuv6qlWSAQCZ6/v0Gyb+m7md9V65NWtAA3QKuJWU
WvoQIjPLx+zCoAEAmfyhRxWGMKgBYn3yZfBU5nb1CBMcMStJ5Pp8b+uwNAs=
=VjFc
-----END PGP SIGNATURE-----
Merge tag 'trace-rv-v7.1-rc6-2' of git://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace
Pull runtime verifier fixes from Steven Rostedt:
- Fix reset ordering on per-task destruction
Reset the task before dropping the slot instead of after, which was
causing out-of-bound memory accesses.
- Fix HA monitor synchronization and cleanup
Ensure synchronous cleanup for HA monitors by running timer callbacks
in RCU read-side critical sections and using synchronize_rcu() during
destruction.
- Avoid armed timers after tasks exit
Add automatic cleanup for per-task HA monitors to prevent timers from
firing after task exit.
- Fix memory ordering for DA/HA monitors
Fix race conditions during monitor start by using release-acquire
semantics for the monitoring flag.
- Fix initialization for DA/HA monitors
Ensure monitors are not initialized relying on potentially corrupted
state like the monitoring flag, that is not reset by all monitors
type and may have an unknown state in monitors reusing the storage
(per-task).
- Fix memory safety in per-task and per-object monitors
Prevent use-after-free and out-of-bounds access by synchronizing with
in-flight tracepoint probes using tracepoint_synchronize_unregister()
before freeing monitor storage or releasing task slots.
- Adjust monitors for preemptible tracepoints
Fix monitors that relied on tracepoints disabling preemption.
Explicitly disable task migration when per-CPU monitors handle events
to avoid accessing the wrong state and update the opid monitor logic.
- Fix incorrect __user specifier usage
Remove __user from a non-pointer variable in the extract_params()
helper.
- Fix bugs in the rv tool
Ensure strings are NUL-terminated, fix substring matching in monitor
searches, and improve cleanup and exit status handling.
- Fix several bugs in rvgen
Fix LTL literal stringification, subparsers' options handling, and
suffix stripping in dot2k.
* tag 'trace-rv-v7.1-rc6-2' of git://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace:
verification/rvgen: Fix ltl2k writing True as a literal
verification/rvgen: Fix options shared among commands
verification/rvgen: Fix suffix strip in dot2k
tools/rv: Fix cleanup after failed trace setup
tools/rv: Fix substring match when listing container monitors
tools/rv: Fix substring match bug in monitor name search
tools/rv: Ensure monitor name and desc are NUL-terminated
rv: Use 0 to check preemption enabled in opid
rv: Prevent task migration while handling per-CPU events
rv: Ensure synchronous cleanup for HA monitors
rv: Add automatic cleanup handlers for per-task HA monitors
rv: Do not rely on clean monitor when initialising HA
rv: Fix monitor start ordering and memory ordering for monitoring flag
rv: Ensure all pending probes terminate on per-obj monitor destroy
rv: Prevent in-flight per-task handlers from using invalid slots
rv: Reset per-task DA monitors before releasing the slot
rv: Fix __user specifier usage in extract_params()
The previous change relaxed the rcu_dereference annotations in
lpm_trie.c so the trie walks no longer trip lockdep when reached from a
sleepable BPF program holding only rcu_read_lock_trace(). By itself
that only helps tries reached as the inner map of a map-of-maps, or
from the classic-RCU syscall path: a sleepable program that references
an LPM trie directly is still rejected at load time by
check_map_prog_compatibility(), whose sleepable whitelist omits
BPF_MAP_TYPE_LPM_TRIE:
Sleepable programs can only use array, hash, ringbuf and local storage maps
LPM trie nodes are allocated from a bpf_mem_alloc (trie->ma) and freed
with bpf_mem_cache_free_rcu(), which chains a regular RCU grace period
into a Tasks Trace grace period before the node -- and the value
embedded in it that trie_lookup_elem() returns to the program -- is
released. That is the same reclaim discipline BPF_MAP_TYPE_HASH relies
on for sleepable access, so a value handed to a sleepable reader cannot
be freed while the program is still running under rcu_read_lock_trace().
The writer paths take trie->lock across the walk and never relied on the
RCU read-side lock to keep nodes alive.
Add BPF_MAP_TYPE_LPM_TRIE to the sleepable map whitelist so these
programs can use LPM tries directly.
Signed-off-by: Vlad Poenaru <vlad.wing@gmail.com>
Reviewed-by: Emil Tsalapatis <emil@etsalapatis.com>
Link: https://lore.kernel.org/r/20260609135558.193287-3-vlad.wing@gmail.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
trie_lookup_elem() annotates its rcu_dereference_check() walks with
only rcu_read_lock_bh_held(). Because rcu_dereference_check(p, c)
resolves to "c || rcu_read_lock_held()", this passes for XDP/NAPI and
classic RCU readers but fails for sleepable BPF programs, which enter
via __bpf_prog_enter_sleepable() and hold only rcu_read_lock_trace().
trie_update_elem() and trie_delete_elem() have the same problem in a
different form: they walk the trie with plain rcu_dereference(), which
asserts rcu_read_lock_held() unconditionally. Both are reachable from
sleepable BPF programs via the bpf_map_update_elem / bpf_map_delete_elem
helpers, and from the syscall path under classic rcu_read_lock(). In
the writer paths the trie is actually protected by trie->lock (an
rqspinlock taken across the walk); we never relied on the RCU read-side
lock to keep nodes alive there.
A sleepable LSM hook that ends up touching an LPM trie therefore
triggers lockdep on debug kernels:
=============================
WARNING: suspicious RCU usage
7.1.0-... Tainted: G E
-----------------------------
kernel/bpf/lpm_trie.c:249 suspicious rcu_dereference_check() usage!
1 lock held by net_tests/540:
#0: (rcu_tasks_trace_srcu_struct){....}-{0:0},
at: __bpf_prog_enter_sleepable+0x26/0x280
Call Trace:
dump_stack_lvl
lockdep_rcu_suspicious
trie_lookup_elem
bpf_prog_..._enforce_security_socket_connect
bpf_trampoline_...
security_socket_connect
__sys_connect
do_syscall_64
This is lockdep-only -- no UAF, since Tasks Trace RCU does serialize
against the trie's reclaim path -- but it spams the console once per
distinct callsite on every debug kernel running a sleepable BPF LSM
that touches an LPM trie, which is increasingly common.
For the lookup path, switch the rcu_dereference_check() annotation
from rcu_read_lock_bh_held() to bpf_rcu_lock_held(), which accepts all
three contexts (classic, BH, Tasks Trace). Other map types already
follow this convention.
For trie_update_elem() and trie_delete_elem(), annotate the walks as
rcu_dereference_protected(*p, 1) -- matching trie_free() in the same
file -- since trie->lock is held across the walk. rqspinlock has no
lockdep_map, so the predicate degenerates to '1' rather than
lockdep_is_held(&trie->lock); the protection is real but not
machine-verifiable. trie_get_next_key() also uses bare
rcu_dereference() but is reachable only from the BPF syscall, which
holds classic rcu_read_lock() before dispatching, so it is left
untouched.
Fixes: 694cea395f ("bpf: Allow RCU-protected lookups to happen from bh context")
Cc: stable@vger.kernel.org
Signed-off-by: Vlad Poenaru <vlad.wing@gmail.com>
Reviewed-by: Emil Tsalapatis <emil@etsalapatis.com>
Link: https://lore.kernel.org/r/20260609135558.193287-2-vlad.wing@gmail.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
cpu_latency_qos_init() registers cpu_dma_latency first and, when
CONFIG_PM_QOS_CPU_SYSTEM_WAKEUP is enabled, registers cpu_wakeup_latency
afterwards. The second registration overwrites the first return value.
As a result, a failure to register cpu_dma_latency can be masked if the
second registration succeeds. Conversely, if cpu_dma_latency succeeds and
cpu_wakeup_latency fails, the function returns an error while leaving the
first misc device registered.
Return immediately on the first registration failure and deregister
cpu_dma_latency if the second registration fails.
Fixes: a4e6512a79 ("PM: QoS: Introduce a CPU system wakeup QoS limit")
Signed-off-by: Yuho Choi <dbgh9129@gmail.com>
Link: https://patch.msgid.link/20260608170748.82273-1-dbgh9129@gmail.com
Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
check_mem_reg() verifies both read and write access for global subprogram
memory arguments. When the caller register is PTR_TO_BTF_ID,
check_helper_mem_access() currently forwards the access to
check_ptr_to_btf_access() as BPF_READ regardless of the requested access
type.
This lets a BTF-backed kernel object field pointer pass the caller-side
writable memory check for a global subprogram argument. The callee is then
validated with a generic writable PTR_TO_MEM argument and can store through
it, even though an equivalent direct BTF field store is rejected with "only
read is supported".
Forward the requested access type to check_ptr_to_btf_access().
This enforces existing BTF write restrictions for global subprogram memory
arguments as well.
Fixes: 3e30be4288 ("bpf: Allow helpers access trusted PTR_TO_BTF_ID.")
Signed-off-by: Nuoqi Gui <gnq25@mails.tsinghua.edu.cn>
Link: https://lore.kernel.org/bpf/20260609-f01-04-btf-writable-arg-v1-1-f449cd970669@mails.tsinghua.edu.cn
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Some workloads with different CPU capacities consume more power with
timer migration than before. The recently introduced per capacity
hierarchies were supposed to alleviate this problem. However it appears
to also regress other types of workloads, especially when plenty of
capacities live together in the same machine.
Disable the feature until a reasonable solution is found.
Fixes: 098cbaad8e ("timers/migration: Split per-capacity hierarchies")
Reported-by: Christian Loehle <christian.loehle@arm.com>
Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Link: https://patch.msgid.link/20260609123356.28449-1-frederic@kernel.org
Closes: https://lore.kernel.org/all/3b79338f-6cfc-4722-8062-9103db2c8ad1@arm.com
btf_parse_struct_metas() walks user-supplied BTF during BPF_BTF_LOAD,
and btf_repeat_fields() expands repeatable fields from array elements
into the fixed BTF_FIELDS_MAX scratch array used by btf_parse_fields().
The remaining-capacity check performs the expanded field count calculation
in u32. A malformed BTF can wrap that calculation, causing the check to
pass even when the expanded field count exceeds the scratch array
capacity. The following memcpy() can then write past the end of the
array.
Use checked addition and multiplication before copying repeated fields
and reject impossible counts.
Fixes: 797d73ee23 ("bpf: Check the remaining info_cnt before repeating btf fields")
Cc: stable@vger.kernel.org
Signed-off-by: Paul Moses <p@1g4.org>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://lore.kernel.org/bpf/20260605234301.1109063-1-p@1g4.org
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Replace the open-coded "p->on_rq == TASK_ON_RQ_MIGRATING" comparisons
in enqueue_task_dl() and dequeue_task_dl() with the existing
task_on_rq_migrating() helper, consistent with the rest of the
scheduler code.
No functional change.
Signed-off-by: Liang Luo <luoliang@kylinos.cn>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: K Prateek Nayak <kprateek.nayak@amd.com>
Link: https://patch.msgid.link/20260608075500.387271-1-luoliang@kylinos.cn
The kernel coding style recommends using 'else if' instead of
placing 'if' on a separate line after 'else'. This change makes
the code consistent with the rest of the kernel codebase.
Signed-off-by: Liang Luo <luoliang@kylinos.cn>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/20260608071842.325159-1-luoliang@kylinos.cn
If we take runnable_avg in max(runnable_avg, util_avg) in cpu_util(), we
should then add or subtract task runnable_avg, but the arithmetic below
is still with task util_avg. This mixes runnable_avg with util_avg which
is incorrect.
Fix by always doing arithmetic with runnable_avg and only take
max(runnable_avg, util_avg) at the last step.
Fixes: 7d0583cf9e ("sched/fair, cpufreq: Introduce 'runnable boosting'")
Signed-off-by: Hongyan Xia <hongyan.xia@transsion.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Vincent Guittot <vincent.guittot@linaro.org>
Link: https://patch.msgid.link/20260605094318.37931-1-hongyan.xia@transsion.com
Patch series "mm/swap, PM: hibernate: fix swapoff race in uswsusp by
pinning swap device", v8.
Currently, in the uswsusp path, only the swap type value is retrieved at
lookup time without holding a reference. If swapoff races after the type
is acquired, subsequent slot allocations operate on a stale swap device.
Additionally, grabbing and releasing the swap device reference on every
slot allocation is inefficient across the entire hibernation swap path.
This patch series addresses these issues:
- Patch 1: Fixes the swapoff race in uswsusp by pinning the swap device
from the point it is looked up until the session completes.
- Patch 2: Removes the overhead of per-slot reference counting in alloc/free
paths and cleans up the redundant SWP_WRITEOK check.
This patch (of 2):
Hibernation via uswsusp (/dev/snapshot ioctls) has a race window: after
selecting the resume swap area but before user space is frozen, swapoff
may run and invalidate the selected swap device.
Fix this by pinning the swap device with SWP_HIBERNATION while it is in
use. The pin is exclusive, which is sufficient since hibernate_acquire()
already prevents concurrent hibernation sessions.
The kernel swsusp path (sysfs-based hibernate/resume) uses
find_hibernation_swap_type() which is not affected by the pin. It freezes
user space before touching swap, so swapoff cannot race.
Introduce dedicated helpers:
- pin_hibernation_swap_type(): Look up and pin the swap device.
Used by the uswsusp path.
- find_hibernation_swap_type(): Lookup without pinning.
Used by the kernel swsusp path.
- unpin_hibernation_swap_type(): Clear the hibernation pin.
While a swap device is pinned, swapoff is prevented from proceeding.
Link: https://lore.kernel.org/20260323160822.1409904-1-youngjun.park@lge.com
Link: https://lore.kernel.org/20260323160822.1409904-2-youngjun.park@lge.com
Signed-off-by: Youngjun Park <youngjun.park@lge.com>
Reviewed-by: Kairui Song <kasong@tencent.com>
Cc: Baoquan He <bhe@redhat.com>
Cc: Barry Song <baohua@kernel.org>
Cc: Chris Li <chrisl@kernel.org>
Cc: Kemeng Shi <shikemeng@huaweicloud.com>
Cc: Nhat Pham <nphamcs@gmail.com>
Cc: "Rafael J . Wysocki" <rafael@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Commit "driver core: platform: set mod_name in driver registration" will
set struct device_driver's mod_name member for platform driver
registration. For a driver to be registered with its mod_name set,
module_kset needs to be initialized, which currently happens in a
subsys_initcall in param_sysfs_init(). The tegra cbb drivers register
themselves before module_kset init, in a core_initcall. This works
currently because lookup_or_create_module_kobject(), which dereferences
module_kset via kset_find_obj(), is not called if mod_name is not set,
which is the case now.
So in preparation for the commit "driver core: platform: set mod_name in
driver registration", move module_kset init to pure_initcall level,
ensuring it happens before tegra cbb driver registration.
Suggested-by: Gary Guo <gary@garyguo.net>
Reviewed-by: Gary Guo <gary@garyguo.net>
Co-developed-by: Rahul Bukte <rahul.bukte@sony.com>
Signed-off-by: Rahul Bukte <rahul.bukte@sony.com>
Signed-off-by: Shashank Balaji <shashank.mahadasyam@sony.com>
Link: https://patch.msgid.link/20260601101942.4002661-1-shashank.mahadasyam@sony.com
Signed-off-by: Danilo Krummrich <dakr@kernel.org>
bpf_task_from_vpid() looks up a task in the pid namespace of the
current task, via find_task_by_vpid():
find_task_by_vpid(vpid)
find_task_by_pid_ns(vpid, task_active_pid_ns(current))
find_pid_ns(nr, ns) -> idr_find(&ns->idr, nr)
cgroup_skb programs run in softirq, which may interrupt a task that is
itself in do_exit(). Once that task has passed
exit_notify() -> release_task() -> __unhash_process(), its thread_pid is
cleared, so task_active_pid_ns(current) returns NULL and find_pid_ns()
dereferences &NULL->idr:
BUG: kernel NULL pointer dereference, address: 0000000000000050
RIP: 0010:idr_find+0x11/0x30 lib/idr.c:176
Call Trace:
<IRQ>
find_pid_ns kernel/pid.c:370 [inline]
find_task_by_pid_ns+0x3b/0xe0 kernel/pid.c:485
bpf_task_from_vpid+0x5b/0x200 kernel/bpf/helpers.c:2916
bpf_prog_run_array_cg+0x17e/0x530 kernel/bpf/cgroup.c:81
__cgroup_bpf_run_filter_skb+0x12b/0x250 kernel/bpf/cgroup.c:1612
sk_filter_trim_cap+0x1dc/0x4c0 net/core/filter.c:148
tcp_v4_rcv+0x18d1/0x2200 net/ipv4/tcp_ipv4.c:2223
</IRQ>
<TASK>
do_exit+0xa63/0x1270 kernel/exit.c:1010
get_signal+0x141c/0x1530 kernel/signal.c:3037
Bail out when current has no pid namespace.
Fixes: 675c3596ff ("bpf: Add bpf_task_from_vpid() kfunc")
Signed-off-by: Sechang Lim <rhkrqnwk98@gmail.com>
Acked-by: Leon Hwang <leon.hwang@linux.dev>
Link: https://lore.kernel.org/bpf/20260608050001.2545245-1-rhkrqnwk98@gmail.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
An ARRAY_OF_MAPS can use an array created with BPF_F_INNER_MAP as its
inner map template. A concrete inner array with a different max_entries
value can then replace the template.
After a successful outer map lookup, the verifier represents the
resulting map pointer using the inner map template. Const-key lookup
nullness elision consequently uses the template max_entries even though
the runtime helper uses the concrete inner map max_entries.
Do not elide lookup result nullness for maps marked with BPF_F_INNER_MAP,
because the template max_entries does not prove that the key is in bounds
for the concrete runtime map.
Fixes: d2102f2f5d ("bpf: verifier: Support eliding map lookup nullness")
Signed-off-by: Nuoqi Gui <gnq25@mails.tsinghua.edu.cn>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Acked-by: Jiri Olsa <jolsa@kernel.org>
Cc: stable@vger.kernel.org
Link: https://lore.kernel.org/bpf/20260607-f01-v2-v2-1-da48453146e8@mails.tsinghua.edu.cn
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
