mirror of
https://github.com/torvalds/linux.git
synced 2026-10-05 19:34:02 +02:00
e3b6cb020e
53453 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Kumar Kartikeya Dwivedi
|
bfc888f045
|
bpf: Bound ownership depth through local kptrs and graph roots
Program-allocated objects can own other local objects through referenced
kptrs. bpf_obj_free_fields() follows those pointers through
__bpf_obj_drop_impl() synchronously, before the object storage is freed
through RCU. A self-referential local kptr type therefore permits arbitrarily
deep object chains, and dropping the head can exhaust the kernel stack.
Long acyclic type chains have the same problem.
btf_check_and_fixup_fields() still assumes referenced kptrs only point to
kernel types and checks ownership through list and rbtree roots only. Its
existing rule is sufficient for graph-only cycles: the target of each graph
edge must contain a node, so every type in a cycle has both a root and a
node. The rule rejects such a type owning another root, breaking every
cycle. It also limits graph-only chains to three types, or two if the first
type contains a node, and conservatively rejects longer acyclic chains.
The missing local-kptr edges, rather than a missed graph-only cycle, are the
bug introduced by support for bpf_kptr_xchg() into local kptrs.
Replace that restriction with one bounded ownership walk covering graph
roots and local referenced kptrs. Run it after all BTF records have been
fixed up, reject cycles and paths deeper than eight record-bearing types,
and cache each type's suffix depth while checking it against the remaining
budget. This also permits the longer acyclic graph-only layouts rejected
by the old rule; update their existing BTF tests accordingly.
Keep the bound independent of MAX_CALL_FRAMES because recursive destruction
can run below a BPF call chain. A plain local pointee without special-field
metadata adds only a final non-recursing drop. Non-owning kptrs and
kernel-BTF kptrs do not recurse through local records and remain outside the
walk. Include local percpu-kptr edges too, although allocation of percpu
objects with special fields is currently forbidden, so that relaxing that
restriction cannot bypass the ownership bound.
btf_check_and_fixup_fields() continues to initialize graph_root.value_rec,
including for separately allocated map records. The ownership relationships
belong to immutable program BTF and only need validation at BTF load time.
Fixes:
|
||
|
Kumar Kartikeya Dwivedi
|
8901cee931
|
bpf: Compare stack frames in regs_exact()
regs_exact() compares the register state up to id, followed by the ID
mappings, but does not compare frameno. The PTR_TO_STACK case in regsafe()
checks frameno separately, which is bypassed when exact comparison is
requested. Consequently, infinite-loop detection can treat pointers to
different stack frames as the same pointer and reject a finite loop.
For example, initialize fp-8 to zero in the caller and to one in the
callee, then pass the caller's fp-8 to the callee as r1:
loop:
r0 = *(u64 *)(r1 + 0);
if r0 != 0 goto done;
r1 = r10;
r1 += -8;
goto loop;
done:
exit;
The loop terminates after reading the callee's slot on its second
iteration. At the loop header, however, the only relevant difference is
r1's frameno, so exact comparison incorrectly reports an infinite loop.
The same problem occurs when the pointer is spilled to the stack.
Move frameno into the type-specific metadata union, ahead of id, so the
existing prefix comparison in regs_exact() covers it. Ordinary stack
pointers do not use another union member. Iterator and IRQ stack-slot
states use their dedicated union views and do not need a frame lookup.
This also keeps bpf_reg_state at 80 bytes.
Since frameno now shares storage with other pointer metadata, it is only
meaningful for PTR_TO_STACK registers. Return NULL from bpf_func() for
other register types. process_iter_arg(), get_constant_map_key() and
is_dynptr_reg_valid_init() look up the frame before checking the register
type and would otherwise index frame[] with a byte of the register's map
or BTF pointer. They dereference the frame only after their type check.
Move the states_maybe_looping() boundary from frameno to precise after the
field relocation. Its prefix comparison continues to cover the complete
value state and now includes frameno.
Continue to ignore precise. Precision marks control whether pruning may
ignore scalar ranges; they do not change the represented values, and exact
comparison already compares those ranges unconditionally. Marks can also
change through backtracking while an ancestor state is still being
explored.
Fixes:
|
||
|
Kumar Kartikeya Dwivedi
|
71919742c8 |
bpf: Assign lock identity to callback map values
A nested bpf_for_each_map_elem() callback can unlock a different element
of the same map:
static long inner(void *map, int *key, struct value *v,
struct value **outer_value)
{
bpf_spin_lock(&v->lock);
bpf_spin_unlock(&(*outer_value)->lock);
return 0;
}
static long outer(void *map, int *key, struct value *v, void *ctx)
{
bpf_for_each_map_elem(map, inner, &v, 0);
return 0;
}
Both callback values currently have ID zero and the same map_ptr.
process_spin_lock() compares those two fields, so it accepts the unlock
even though the two callbacks can receive different map elements.
Assign a fresh ID to every callback map value in the for-each,
timer/workqueue, and task-work constructors. Copies of one callback
argument retain its ID, so locking and unlocking through that argument
continues to work. Distinct callbacks also get distinct IDs for
single-element arrays, including inner arrays sharing inner_map_meta.
Preserve map_uid for every inner-map lookup and compare it through
check_ids() during state pruning. This preserves relationships between
maps, keys, and values while allowing equivalent states with different
lookup IDs to match. It avoids field-specific rules for when an inner map
needs an identity.
Move map_uid out of the metadata union and next to the other IDs, so
register comparisons can use the existing memcmp() ranges and remap the
IDs separately. Clear it when resetting a register or converting a map
lookup result to a socket pointer. Shrink frameno to u8, which is enough
for MAX_CALL_FRAMES, to make room without growing bpf_reg_state.
Fixes:
|
||
|
Kumar Kartikeya Dwivedi
|
c26e97721b |
bpf: Apply CO-RE relocations before subprogram validation
check_subprogs() verifies that each subprogram ends in an exit or an
unconditional jump before in-kernel CO-RE relocations are applied. An
unresolved relocation can then replace that terminal instruction with an
invalid helper call. The resulting fall-through into another subprogram
breaks the CFG invariant used by postorder and stack liveness analysis,
which can write past their per-subprogram arrays.
Apply CO-RE relocations immediately after preparing the program BTF, before
subprogram discovery and validation. Keep func_info and line_info validation
after subprogram discovery because those records depend on the complete
subprogram layout.
Reject an ldimm64 first slot at the end of the instruction stream before
CO-RE can inspect its missing second slot. check_subprogs() previously
rejected this form before relocation processing because it is not a valid
subprogram terminator. Moving CO-RE ahead of check_subprogs() removes that
implicit protection, so perform an explicit check before applying
relocations.
Include core_relo_cnt when deciding whether to prepare program BTF. A load
that supplied only CO-RE relocation metadata previously skipped both BTF
setup and relocation processing.
Fixes:
|
||
|
Kumar Kartikeya Dwivedi
|
fd16449a9b |
bpf: Preserve packet pointer class displacement in regsafe()
regsafe() maps packet pointer IDs between states and checks that each current register range is a subset of the corresponding explored register range. It does not, however, preserve the displacement between registers that share a packet pointer ID. This is unsound because packet range is shared by ID. A bounds check on one class member updates every member, and a later access can consume the range through another member. Commit |
||
|
Kumar Kartikeya Dwivedi
|
261b61d373 |
bpf: Make post-verification instruction rewrites killable
After do_check() returns, the verifier runs several instruction rewrite
passes. Some of them patch or remove one instruction at a time. Each
operation moves the remaining instruction and auxiliary-data arrays and
adjusts all branch offsets, making the overall work quadratic in the
program length.
A privileged loader can submit 131072 unconditional jumps by zero followed
by a valid return. Verification finishes quickly, but bpf_opt_remove_nops()
then spends a long time removing each jump separately. Since this
post-verification work neither checks for signals nor reschedules, a pending
SIGKILL cannot terminate the task until the rewrite finishes.
Make bpf_patch_insn_data() and verifier_remove_insns() common cancellation
and rescheduling points. These helpers run from BPF_PROG_LOAD process
context, and bpf_patch_insn_data() can already sleep while reallocating
auxiliary data.
Report interrupted constant blinding as -EINTR and propagate it through
both JIT paths, including kernels that permit interpreter fallback.
Other blinding failures retain the existing fallback behavior.
This does not reduce the quadratic cost of the rewrite passes, but it makes
the work preemptible and allows a killed loader to be torn down promptly.
Fixes:
|
||
|
Weiming Shi
|
50e80e2bb5 |
bpf: Skip unsettled links in link iterator
bpf_link_prime() inserts a link into link_idr before anon_inode_getfile()
succeeds and before bpf_link_settle() publishes the ID in link->id.
bpf_link_by_id() treats such an ID-zero link as unsettled, but the link
iterator takes a reference without this check.
If anon_inode_getfile() then fails, the creator removes the ID and frees
its still-private link directly. The iterator is left with a dangling
reference and its next bpf_link_put() accesses freed memory.
Treat ID-zero entries as transient in bpf_link_get_curr_or_next(), just as
bpf_link_by_id() does.
BUG: KASAN: slab-use-after-free in bpf_link_put
Write of size 8 by task exp/384
Call Trace:
bpf_link_put kernel/bpf/syscall.c:3372
bpf_link_seq_next kernel/bpf/link_iter.c:33
bpf_seq_read kernel/bpf/bpf_iter.c:158
vfs_read fs/read_write.c:572
ksys_read fs/read_write.c:716
do_syscall_64 arch/x86/entry/syscall_64.c:84
entry_SYSCALL_64_after_hwframe arch/x86/entry/entry_64.S:121
Kernel panic - not syncing: KASAN: panic_on_warn set ...
Fixes:
|
||
|
Kumar Kartikeya Dwivedi
|
40c2096961 |
bpf: Verify global subprogs in each sleepability context
Global subprograms are verified independently with a fresh verifier root. do_check_common() currently seeds that root's in_sleepable state from the program, even though a global subprogram can also run from callbacks whose execution context differs from the program's main entry point. In particular, workqueue and task-work callbacks are sleepable even when the containing program is not. A global subprogram of that program is therefore verified as non-sleepable, making in_rcu_cs() true and allowing loads of RCU-protected kptrs to produce trusted MEM_RCU pointers. The same subprogram can then be called from a sleepable callback without a classic RCU reader. It can retain such a pointer while the object is freed and use it after free. The verifier's execution-context predicates are complementary. A state is sleepable only when in_sleepable is set and no RCU, preemption, IRQ, or lock region is active. Each condition which prevents sleeping also provides RCU protection, while in_rcu_cs() treats a non-sleepable state as implicitly protected. Use this relationship to represent a global subprogram caller with only the result of in_sleepable_context(). A protected sleepable caller is normalized to in_sleepable=false at the independent verification root. This both prevents sleepable operations and makes in_rcu_cs() true without copying caller-owned lock state. Track only the contexts in which each global subprogram is actually reached. Verify it once if all reachable calls use the same context, and twice only if both sleepable and non-sleepable calls reach it. Calls found while verifying globals or asynchronous callbacks mark further contexts for checking. Repeat the existing subprogram walk until all called contexts have been verified; unreachable global calls remain unchecked. Accumulate instruction counts over those verification passes. Preserve the total recorded before each pass, since path accounting has already added this pass's synchronous instructions and its root total must also include asynchronous subprograms. This makes an unprotected callback verify the global subprogram as sleepable, turning its RCU-protected kptr load into an untrusted pointer. Protected callers and global subprograms which do not depend on implicit RCU protection remain valid. Fixes: |
||
|
Masoud Aghasi
|
953824e508 |
bpf: Fix u32 overflow issue in map batch operations
Several map batch operation implementations such as generic_map_lookup_batch() use calculations in the form of "values + cp * map->value_size" to compute the desired userspace memory address for reading or writing. This can overflow the u32 type (the result of "cp * map->value_size") when the map size exceeds 4GB. generic_map_lookup_batch() may corrupt values for some keys in userspace memory, and in some cases it mismatches values for some keys while still reporting success. Other batch operations may fail to delete or update some keys, or the syscall may return unexpected errors. Add size_t casts to prevent the affected offset and size calculations from overflowing. Fixes: |
||
|
Pu Lehui
|
1c21452d02 |
bpf: Fix UAF due to concurrent consumption of ttrace lists in alloc_bulk
Syzkaller repeatedly triggered UAF splats related to nodes in
waiting_for_gp_ttrace within the bpf memalloc:
BUG: KASAN: slab-use-after-free in llist_del_first+0x85/0x110 lib/llist.c:61
Read of size 8 at addr ffff8881572cd080 by task syz.4.470/5112
...
llist_del_first+0x85/0x110 lib/llist.c:61
alloc_bulk+0x193/0x460 kernel/bpf/memalloc.c:229
bpf_mem_refill+0x386/0x560 kernel/bpf/memalloc.c:436
Freed by task 14:
...
__free_rcu kernel/bpf/memalloc.c:281 [inline]
__free_rcu_tasks_trace+0x48/0xd0 kernel/bpf/memalloc.c:291
rcu_tasks_invoke_cbs+0x1ec/0x3e0 kernel/rcu/tasks.h:571
rcu_tasks_one_gp+0x13d/0x220 kernel/rcu/tasks.h:621
rcu_tasks_kthread+0xf3/0x120 kernel/rcu/tasks.h:651
The reason is that the UAF occurs after the RCU Tasks Trace GP expires:
when the __free_rcu() callback runs, there is no synchronization
protecting llist_del_all() against concurrent alloc_bulk() operating on
waiting_for_gp_ttrace, leading to the race condition below:
CPU0 CPU1
__free_rcu (RCU Tasks Trace callback)
alloc_bulk
llist_del_first(&c->waiting_for_gp_ttrace)
entry = smp_load_acquire(&head->first);
do {
if (entry == NULL)
return NULL;
free_all(llist_del_all(&c->waiting_for_gp_ttrace))
llist_for_each_safe(pos, t, llnode)
free_one(pos);
next = READ_ONCE(entry->next); <-- trigger UAF
} while (!try_cmpxchg(&head->first, &entry, next));
In addition, there is also a theoretical race condition on the
free_by_rcu_ttrace list. This race requires two preconditions: an
in-flight Tasks Trace GP keeping c->call_rcu_ttrace_in_progress == 1,
and concurrent cross-CPU frees repopulating c->free_by_rcu_ttrace with
new nodes. Under these conditions, the following scenario triggers UAF:
// CPU0
// irq work is still busy (on PREEMPT_RT)
alloc_bulk()
llist_del_first(&c->free_by_rcu_ttrace)
entry = smp_load_acquire(&head->first);
do {
if (entry == NULL)
return NULL;
// CPU1
bpf_mem_alloc_destroy()
WRITE_ONCE(c->draining, true)
// wait for CPU0
irq_work_sync()
// CPU2
do_call_rcu_ttrace(tgt(CPU0))
if (c->draining) {
llist_del_all(&c->free_by_rcu_ttrace)
free_all()
}
// CPU0 continue
next = READ_ONCE(entry->next); <-- trigger UAF
while (!try_cmpxchg(&head->first, &entry, next));
Fix this by introducing a raw spinlock to synchronize the concurrent
consumption on waiting_for_gp_ttrace and free_by_rcu_ttrace.
Fixes:
|
||
|
Jose Fernandez (Anthropic)
|
85136bf224 |
bpf: Avoid soft lockup in __htab_map_lookup_and_delete_batch()
__htab_map_lookup_and_delete_batch() has no rescheduling point. The batch count bounds how many entries are copied out, not how many buckets are visited, so one BPF_MAP_LOOKUP_BATCH call can walk the map end to end. The empty-bucket fast path is worse: it stays inside a single rcu_read_lock() / bpf_disable_instrumentation() section for any run of consecutive empty buckets. That holds up on small maps, but it falls apart at scale. On a 144-CPU arm64 host running a CONFIG_PREEMPT_NONE kernel, periodic BPF_MAP_LOOKUP_BATCH calls against an LRU hash map with 16,777,216 buckets held a CPU inside the batch op for 77+ seconds and triggered the soft lockup watchdog. Commit |
||
|
Siddharth Chintamaneni
|
0d7823cd4c |
bpf: Allow terminal gotox instructions
check_subprogs() treats gotox as a direct jump and validates its reserved
zero offset. When gotox is the final instruction, this produces a
synthetic successor one instruction past the end of the subprogram and
rejects an otherwise valid program.
Skip direct-offset validation for gotox and accept it as a
non-fallthrough terminal instruction. Its actual targets remain validated
from the instruction-array jump table during CFG construction.
Fixes:
|
||
|
Jiayuan Chen
|
b0b3dc6652 |
bpf: Fix divide-by-zero in btf_struct_walk()
When an access goes past the struct and the last member is a flexible
array, btf_struct_walk() folds the offset back into a single element with
(off - moff) % t->size, but never checks that the element type has a size.
BTF takes an empty struct, so this in program BTF
/* event could be empty */
struct event {
#ifdef HAVE_TIMESTAMP
__u64 ts;
#endif
};
struct batch {
int nr;
struct event events[];
};
divides by zero at prog load time. Getting there needs a PTR_TO_BTF_ID that
is not MEM_ALLOC, e.g. a plain read of a local kptr stashed in a map from a
sleepable program.
Oops: divide error: 0000 [#1] SMP KASAN PTI
RIP: 0010:btf_struct_walk+0x53f/0x1570
Call Trace:
<TASK>
btf_struct_access+0x42a/0xcd0
check_ptr_to_btf_access+0x4dc/0x1160
check_mem_access+0x3a45/0x8740
check_load_mem+0x36a/0xd10
do_check_common+0x3ef0/0xb210
bpf_check+0x6d3b/0x8580
bpf_prog_load+0xf7c/0x2720
__sys_bpf+0xa83/0x3690
__x64_sys_bpf+0xc7/0x150
x64_sys_call+0x1f3f/0x27e0
do_syscall_64+0xe5/0x610
entry_SYSCALL_64_after_hwframe+0x76/0x7e
</TASK>
Reject a zero-sized element type. The fixed array path in the same function
already bails out on the same thing:
btf_struct_walk()
...
/* skip empty array */
if (moff == mtrue_end)
continue;
msize /= total_nelems;
Fixes:
|
||
|
Daniel Borkmann
|
2936aed9b0 |
bpf: Clear scalar delta on narrowing stack spill
check_stack_write_fixed_off() breaks the scalar link on a narrowing spill
by zeroing the id of the destination stack slot, but leaves the delta in
place. save_register_state() has just copied the source register verbatim,
so the slot keeps the BPF_ADD_CONST delta of a register it is no longer
linked to. This is the same shape as the case addressed in
|
||
|
Daniel Borkmann
|
f9191460cd |
bpf: Require CAP_PERFMON for untrusted read-only memory reads
Marking bpf_rdonly_cast() KF_PERFMON CAP-limits one producer of PTR_TO_MEM | MEM_RDONLY | PTR_UNTRUSTED, but not the type itself. A global subprogram argument tagged __arg_untrusted results in the same register with no kfunc call. Reported-by: STAR Labs SG <info@starlabs.sg> Signed-off-by: Daniel Borkmann <daniel@iogearbox.net> Link: https://lore.kernel.org/r/20260910213510.49358-3-daniel@iogearbox.net Signed-off-by: Alexei Starovoitov <ast@kernel.org> |
||
|
Daniel Borkmann
|
81c975aae3 |
bpf: Require CAP_PERFMON for kfuncs reading memory
Mark fault-safe probe reading kfuncs as KF_PERFMON, similarly as we do for the old-style BPF helper equivalents. bpf_rdonly_cast() is included in this list as well as it returns PTR_TO_MEM | MEM_RDONLY | PTR_UNTRUSTED for an unchecked object and is using fault-safe BPF_PROBE_MEM. Note that only the void form of bpf_rdonly_cast() produced a register that was readable without CAP_PERFMON. For a struct type id the kfunc returns PTR_TO_BTF_ID | PTR_UNTRUSTED, whose dereference has always been gated in check_ptr_to_btf_access(). The flag is not conditional on the type id, so for the latter it only moves the rejection from the dereference to the call itself, which is the better place to report it anyway. The bpf_stream_vprintk() and bpf_stream_print_stack() kfuncs are marked as well. The former ends up in the same bpf_bprintf_prepare() as the bpf_snprintf() helper, where %pks, %pus and %pI4 read through a program- supplied address and %pB resolves one into a symbol. The latter walks the stack and prints each instruction pointer via %pS. Lastly, bpf_get_kmem_cache() takes a raw scalar address that the verifier does not constrain and dereferences the page/slab metadata for it. Field reads on the returned pointer are still blocked (PTR_TO_BTF_ID | PTR_UNTRUSTED -> check_ptr_to_btf_access() results in -EPERM without the CAP), but the NULL/non-NULL result is observable. Reported-by: STAR Labs SG <info@starlabs.sg> Signed-off-by: Daniel Borkmann <daniel@iogearbox.net> Link: https://lore.kernel.org/r/20260910213510.49358-2-daniel@iogearbox.net Signed-off-by: Alexei Starovoitov <ast@kernel.org> |
||
|
Daniel Borkmann
|
88ce88e933 |
bpf: Add KF_PERFMON kfunc flag
Tracing related BPF helpers e.g. under bpf_base_func_proto() are gated behind CAP_PERFMON. However, the same is currently not true for kfuncs and they are accessible via plain CAP_BPF. Add a new KF_PERFMON flag which can be used such that check_kfunc_call() ensures env->allow_ptr_leaks is permitted. This follows similar pattern to existing KF_DESTRUCTIVE flag. The rejection returns -EPERM to match the other CAP_PERFMON gates in the verifier, that is, check_ptr_to_btf_access() and check_ptr_to_map_access(), which report the very same policy to user space. Signed-off-by: Daniel Borkmann <daniel@iogearbox.net> Link: https://lore.kernel.org/r/20260910213510.49358-1-daniel@iogearbox.net Signed-off-by: Alexei Starovoitov <ast@kernel.org> |
||
|
Linus Torvalds
|
b1e00ffaf9 |
tracing fixes for v7.3:
- Fix several tracefs files that did not take the trace_array reference A trace instance can be created and destroyed in the tracefs "instances" directory via mkdir and rmdir respectively. The instance is represented by a trace_array descriptor. Most tracefs files pass the trace_array as the private data of the inode to the open/read/write functions. Since there is no locking between the time a task opens a file and the deletion of the instance (and the freeing of the trace_array), each open needs to get a reference to the trace_array and each close must remove it. A instance can't be removed if there's any reference taken on its trace_array. The open function uses trace_array_get() that takes a lock (preventing removal of instances) and iterates the list of all existing trace_arrays and if it finds a match, it takes the reference and releases the lock. If it doesn't find a match, it causes the open to return -ENODEV. There were some added files that did not take the trace_array reference on open that needed to be fixed. Sashiko also correctly pointed out that there were some files that took an address of an field or element of the trace_array which had a pointer back to the trace_array to take its reference on open. But this leaves a slight race between referencing this element to get the trace_array as the element itself could be freed. To solve this, some helper functions were created to look for trace_arrays with this field or element in the search so that the element did not have to be dereferenced before the trace_array's reference was taken. - Add a lock around ftrace_ops initialization When a ftrace_ops is first used by ftrace, some internal initialization is performed on the ops. But if multiple tasks were calling functions that did this initialization, it could race and perform doing the initialization more than once, corrupting the internal data. Add a lock in the initialization code to prevent this from happening. - Fix splice reads on mmapped buffers The logic in the ring buffer splice code for mmapped buffers is supposed to do a copy of the memory as the mapped buffers can't be given to splice. But there was an if statement within the copy code that would return a -1 if a request for a full page was done and it wasn't a partial read. This is because this logic was written before mmapped buffers existed and this case didn't make sense at the time. For mmapped buffers it makes perfect sense and by returning early can drop a lot of pages unnecessarily. - Have the persistent ring buffer validation check nr_subbufs Sashiko reported that the validation code was relying on the saved nr_subbufs to match the calculated nr_pages + 1 and if they were off, that the code could cause corruption. Sashiko is correct, and the saved nr_subbufs should be validated before assuming it is correct. - Do not allow more than one instance with the same name on cmdline If an admin were to add more than one trace instances with the same name they all would be created, but only the first one would be accessible via tracefs. This used to not be allowed but some restructuring of code has since made it possible. - Fix the race between subbuf resize and trace_pipe_raw readers If a task was reading trace_pipe_raw while another task was changing the ring buffer subbuf size, it could crash the reader. The trace_pipe_raw readers do get their own copy of the page from the buffer, but the code needs some restructuring to not have the resize of the subbuffers cause issues. - Cap the size of the mapped (static) ring buffer nr_pages The meta data used for ring buffer mapped buffers is 32 bit in size. A normal ring buffer could (in theory) have more than 4 billion pages. But this is not allowed by mapped buffers, so enforce it. -----BEGIN PGP SIGNATURE----- iIoEABYKADIWIQRRSw7ePDh/lE+zeZMp5XQQmuv6qgUCap3GJRQccm9zdGVkdEBn b29kbWlzLm9yZwAKCRAp5XQQmuv6qtAfAP9hDPtzctAoz+IHZq+GYejhKTu99ICG vAxTOZivuG6KYAD/UPA4P7z5ayxGM5UlT+2U81+mpCd55F2oFiIxRuFYcwk= =/d+T -----END PGP SIGNATURE----- Merge tag 'trace-v7.3-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace Pull tracing fixes from Steven Rostedt: - Fix several tracefs files that did not take the trace_array reference A trace instance can be created and destroyed in the tracefs "instances" directory via mkdir and rmdir respectively. The instance is represented by a trace_array descriptor. Most tracefs files pass the trace_array as the private data of the inode to the open/read/write functions. Since there is no locking between the time a task opens a file and the deletion of the instance (and the freeing of the trace_array), each open needs to get a reference to the trace_array and each close must remove it. An instance can't be removed if there's any reference taken on its trace_array. The open function uses trace_array_get() that takes a lock (preventing removal of instances) and iterates the list of all existing trace_arrays and if it finds a match, it takes the reference and releases the lock. If it doesn't find a match, it causes the open to return -ENODEV. There were some added files that did not take the trace_array reference on open that needed to be fixed. Sashiko also correctly pointed out that there were some files that took an address of an field or element of the trace_array which had a pointer back to the trace_array to take its reference on open. But this leaves a slight race between referencing this element to get the trace_array as the element itself could be freed. To solve this, some helper functions were created to look for trace_arrays with this field or element in the search so that the element did not have to be dereferenced before the trace_array's reference was taken. - Add a lock around ftrace_ops initialization When a ftrace_ops is first used by ftrace, some internal initialization is performed on the ops. But if multiple tasks were calling functions that did this initialization, it could race and perform doing the initialization more than once, corrupting the internal data. Add a lock in the initialization code to prevent this from happening. - Fix splice reads on mmapped buffers The logic in the ring buffer splice code for mmapped buffers is supposed to do a copy of the memory as the mapped buffers can't be given to splice. But there was an if statement within the copy code that would return a -1 if a request for a full page was done and it wasn't a partial read. This is because this logic was written before mmapped buffers existed and this case didn't make sense at the time. For mmapped buffers it makes perfect sense and by returning early can drop a lot of pages unnecessarily. - Have the persistent ring buffer validation check nr_subbufs Sashiko reported that the validation code was relying on the saved nr_subbufs to match the calculated nr_pages + 1 and if they were off, that the code could cause corruption. Sashiko is correct, and the saved nr_subbufs should be validated before assuming it is correct. - Do not allow more than one instance with the same name on cmdline If an admin were to add more than one trace instances with the same name they all would be created, but only the first one would be accessible via tracefs. This used to not be allowed but some restructuring of code has since made it possible. - Fix the race between subbuf resize and trace_pipe_raw readers If a task was reading trace_pipe_raw while another task was changing the ring buffer subbuf size, it could crash the reader. The trace_pipe_raw readers do get their own copy of the page from the buffer, but the code needs some restructuring to not have the resize of the subbuffers cause issues. - Cap the size of the mapped (static) ring buffer nr_pages The meta data used for ring buffer mapped buffers is 32 bit in size. A normal ring buffer could (in theory) have more than 4 billion pages. But this is not allowed by mapped buffers, so enforce it. * tag 'trace-v7.3-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace: ring-buffer: Use a macro for static buffer bits tracing: Fix comment in tracing_buffers_splice_read() ring-buffer: Prevent truncation of nr_pages / nr_subbufs ring-buffer: Cap static ring buffer nr_pages tracing: Fix subbuf resize races with trace_pipe_raw readers tracing: Fix to avoid creating trace instances with duplicate names ring-buffer: Add checking nr_subbufs to persistent ring buffer validation ring-buffer: Allow splice reads on static buffers tracing: Take trace_array reference when opening options file ftrace: Synchronize the initialization of ftrace_ops ftrace: Take trace_array reference before accessing its ftrace_ops tracing: Have show_event_filters/triggers files take trace array ref |
||
|
Linus Torvalds
|
2beb1b31a1 |
bpf-fixes
-----BEGIN PGP SIGNATURE-----
iQIzBAABCAAdFiEE+soXsSLHKoYyzcli6rmadz2vbToFAmqdhRMACgkQ6rmadz2v
bTo8vRAAnP+x1z6FBOgHawGmmBpXtazoipRFeok/+/YGQ4SbS+evvFNvmkeCyihZ
7EeNHGuWEbA6xQBAcMf0StMjmr3jkJvUA6lsHSxBiFLpHGqNt0Fz9IXvAL/8QWPg
F7+CBs5A+J61i66LEmINbWzW1ujDf7baU/1VWui9zfPxnmPu40V/74eXzm6DmH/V
oHTsCscCYNe+N+Wix6B6nLrObY537fcJU12N56uqpkaSc0+6H1fBQJmyFGpYroTX
+4g+DNzEHyIYmt6B3+oDpbT0Zh/pP6ROq5TPDNJSSa+5B/uPo5C5pH520HwXF9hJ
04uaD7y2fpE809Nwl3OEK4ozpJ+mEGU1NKZPvFAxckY6GCiBtfbGd1etFno7r+4F
mZhNBKbQVEQX4XJUlcoqn9n7T2OCDnjiuneZAjIi4vUdevnSnakr0rCTFriiHP54
jg4fEkXxvYjPiqs5SQbxoGxmYSfg9fYaSfK4HgFdlfbwp+3Vmv4BW/tcsIJ9dcPs
L3YWkUU3znjbu2BLy0CP9fGt/1ik8p4/tA8vvZwz0yqpDMJkLy1TziGMef51UQ7k
1Pe/Ln5J3C6xnWs+3TRF11y8PuuCF0y47DGPACfwPoQxca1ufJ5fgmLXRRQToZe4
RjDmoeQvvI9w9kxh6gVhTHoiiEeD0wcQqx2CMHKTrU/PbM7d4Co=
=oz4Z
-----END PGP SIGNATURE-----
Merge tag 'bpf-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf
Pull bpf fixes from Alexei Starovoitov:
"This mainly contains verifier fixes that address bugs reported by
Nicholas Carlini.
- Fix incorrect non-NULL inference in pointer comparisons: pointer
types that may be NULL at runtime, pointers with unbounded offsets,
JMP32 comparisons with zero, and imprecise zero registers (Eduard
Zingerman)
- Fix precision tracking for half-dead zero spills, ld_abs/ld_ind
implicit subprog exit, bpf_loop() callbacks, linked scalar ids and
NULL call arguments (Eduard Zingerman)
- Reject BPF_PSEUDO_FUNC reference to the main program, fix zero
extension of arena 32-bit cmpxchg, don't rewrite bpf_fastcall
patterns entered by a jump (Eduard Zingerman)
- Fix percpu map update and BPF_F_CPU validation with sparse CPU IDs
(Hui Su)
- Fix NULL-ptr-derefs in bpf_snprintf_btf() for void and VAR types,
and reject key-less BTF for hash maps (Jiayuan Chen)
- Various fixes (Kumar Kartikeya Dwivedi):
- Fix out-of-bounds access in disassembler on invalid LDSX
instruction
- mark siginfo of signal tracepoints as scalar and
sched_process_wait argument as nullable
- mark faultable stack helpers as sleepable
- reject tail calls and legacy packet loads from callbacks
- enforce rbtree callback lock restrictions for resilient locks
- require MEM_PERCPU for percpu kptr stores
- clear NON_OWN_REF after RCU protection ends
- mark NULL kptr stores precise
- preserve inner map identity in callback frames
- reject non-scalar bpf_loop() iteration counts
- Fix trampoline allocation slowdown on x86 by using
EXECMEM_MODULE_DATA (Mike Rapoport)
- Keep bpf_refcount_acquire() nullable for borrowed RCU kptrs and
reject untrusted allocated-object pointers (Ning Ding)
- Fix special fields handling in recycled rhtab elements (Nuoqi Gui,
Yuan Chen)"
* tag 'bpf-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf: (86 commits)
bpf, riscv: Make arena support depend on ZACAS
selftests/bpf: Test pointer bpf_loop iteration count rejection
bpf: Reject non-scalar bpf_loop iteration counts
bpf: use mark_arg_precision() in check_mem_size_reg()
bpf: propagate mark_chain_precision() errors out of loop_flag_is_zero()
selftests/bpf: precision of a NULL global subprogram BTF_ID argument
bpf: mark a NULL BTF_ID argument of a global subprogram precise
selftests/bpf: precision of a NULL kfunc argument
bpf: mark a NULL kfunc argument precise
selftests/bpf: precision of a NULL global subprogram memory argument
bpf: mark a NULL memory argument of a call precise
selftests/bpf: precision of a NULL helper argument
bpf: mark a NULL call argument precise
selftests/bpf: Test inner map identities in callbacks
bpf: Preserve inner map identity in callback frames
selftests/bpf: Test imprecise scalar kptr stores
bpf: Mark NULL kptr stores precise
selftests/bpf: Test rhtab kptr cancellation semantics
bpf: Cancel special fields when recycling rhtab elements
selftests/bpf: Test timer field on recycled rhtab element
...
|
||
|
Linus Torvalds
|
88405f0ad1 |
Miscellaneous scheduler fixes:
- Fix a timestamping bug in pick_task_fair() and
yield_task_fair() (Zhan Xusheng)
- Skip migrate-disabled tasks when picking a push
candidate in the RT and DL schedulers (Seiji Nishikawa)
- Skip rq->avg_idle update without a valid idle_stamp
(Shubhang Kaushik)
- Fix throttling bug in throttle_cfs_rq(), caused
by the recent single-runqueue conversion (Wanwu Li)
- Fix bandwidth calculation bug in distribute_cfs_runtime(),
caused by the single-runqueue conversion (Wanwu Li)
- Don't make x86 ITMT enablement depend on debugfs (Mario Limonciello)
- Avoid creating misfits during cache-aware load-balancing
on hybrid systems (Tim Chen)
Signed-off-by: Ingo Molnar <mingo@kernel.org>
-----BEGIN PGP SIGNATURE-----
iQJFBAABCgAvFiEEBpT5eoXrXCwVQwEKEnMQ0APhK1gFAmqdTLIRHG1pbmdvQGtl
cm5lbC5vcmcACgkQEnMQ0APhK1gdURAAnjK4q2xg115VyKOvg4u4epPre1XNK4a/
RmimS/2+1fm7/5zqOXOQG3qU8z2Aj+nBIZa1d1yadV7ARHSymDINobCX+2HQ1DjU
Iysk2BqbbLnQQNKh81OBWhPMyJy4yOnig/670KR5JdFAdTKFYW/Ys+pQmO9M6rEx
hc8B4SUhOGeRnonfPMTbFW7H9eopWzst8kIpF0USl58az0mla5tkooWYJsvSuUHn
g4s9IALlRe3DxgmMT3kLKoAx2+ySnrzZNRs+M8Z/sMSDKXCMFpvX36wOqBJMHwXe
ZoAow/fJh4ysoYTR7luhDIBDQAkszf4GE9a644I/Bd0sIIYKs4+1RW1T3DAoWmKv
Sx0C29MM4Ng4R3lPx2avb+c0T0VO080/3fqRFrGrck8BIpMa9HcinOxf6etI8+vt
STnz5uQZZj3V+A5vzlKCAFGUUqK/s3EJnrT5z4VPxXmY2ZSyLLub4C5SQ8RNnqRL
u+Q9ra9DYBJh/HJTgjjcAP/HxmKK5LyQXd2DzuA1wy/wwC+s7TtPrxN5ep5EdZmv
K65zGGo28BgcCKD8jc1DfVXf8cf80fMiAr4fruGdo5TVwPyBahwZBi4sbqhahtqL
D4QwCsbOfhS+evVpx29n+V7eQzZzvSdN1tFCtZ3Tg6VvT9aJ2jLSx79aCsL/cYbr
hViB9N0yhGk=
=8l0J
-----END PGP SIGNATURE-----
Merge tag 'sched-urgent-2026-09-06' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
Pull scheduler fixes from Ingo Molnar:
- Fix a timestamping bug in pick_task_fair() and yield_task_fair()
(Zhan Xusheng)
- Skip migrate-disabled tasks when picking a push candidate in the
RT and DL schedulers (Seiji Nishikawa)
- Skip rq->avg_idle update without a valid idle_stamp (Shubhang
Kaushik)
- Fix throttling bug in throttle_cfs_rq(), caused by the recent
single-runqueue conversion (Wanwu Li)
- Fix bandwidth calculation bug in distribute_cfs_runtime(),
caused by the single-runqueue conversion (Wanwu Li)
- Don't make x86 ITMT enablement depend on debugfs (Mario Limonciello)
- Avoid creating misfits during cache-aware load-balancing on hybrid
systems (Tim Chen)
* tag 'sched-urgent-2026-09-06' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip:
sched/fair: Avoid creating misfits during cache-aware balancing
x86/itmt: Don't make ITMT enablement depend on debugfs
sched/fair: Use cfs_rq->h_curr in distribute_cfs_runtime()
sched/fair: Use cfs_rq->h_curr in throttle_cfs_rq()
sched/core: Skip rq->avg_idle update without a valid idle_stamp
sched/rt,dl: Skip migrate-disabled tasks when picking a push candidate
sched/fair: Use update_curr_eevdf() for the remaining root cfs_rq callers
|
||
|
Linus Torvalds
|
c4a3928e7d |
Misc perf events fixes:
- Skip empty AUX records with only format flags
(Leo Yan)
- Fix use-after-free when perf mmap() revival races with the
last munmap() (Yilin Zhang, Weiming Shi)
Signed-off-by: Ingo Molnar <mingo@kernel.org>
-----BEGIN PGP SIGNATURE-----
iQJFBAABCgAvFiEEBpT5eoXrXCwVQwEKEnMQ0APhK1gFAmqdSokRHG1pbmdvQGtl
cm5lbC5vcmcACgkQEnMQ0APhK1gRKRAAkDxFT3BNAtAypk5CvTPeO9K530u38mqc
fIvYQYbYkqIMZaFSmIthSgsPOJD/mqkccl1sd6djlS/73mI/X8eXCwZOgjLnaCsu
jRx8atD8Q7hHeyJ78qY/hFsyHPTQel5+cniEk4BjUrYIYhHklFfpQw1AZhoJieUs
vnqkDGMY/qHQ5pGs+dnz3oI8FE6ruNU0Ps6qiBqsbsvMJI+wviCidpC6BqluyxvQ
Pp6ACwqQIiH2mPwI9WYtHKmVL5ajQJPt8fbIwIvRbX5wCY566zb79oUys2pNUh/U
zzpTNOGgqtVCTgiuMiEI2SlIIMzC36P071vBTyD8YZWUNpdYXLM20RdvAUZkqefT
Vx3cMs8rgcPnjGO4XhPJJOzfyyCi25VL/p3af9S3wSJhfQNdMF9BuqRABFMBmtDi
5adP+4p5Dw9sfc2+F2/OCT7PrXsFiNsOp+iduWJYb6i8hdSjV1mfjGLyptdBCdj2
dUYr4VrFoiYUVLPWJD5F4/dDAIljN3CEoGwHiOITNCYu0fgO7GCALf8FG5U9Ec7D
guuRGnEXDFQDCOZHm5MuVYWAR7qPhe+iVoX6Dw9ZmgecGWb4eTl4A5e8LGgbcbJ1
JMfVr3OENSyGi1dPnCX4DEsIpeXN1UTMHhrwNosWcu1OrEEUkzdQ8GQ17qU6PURG
yYSXCvhdEtY=
=Ffvz
-----END PGP SIGNATURE-----
Merge tag 'perf-urgent-2026-09-06' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
Pull perf events fixes from Ingo Molnar:
- Skip empty AUX records with only format flags (Leo Yan)
- Fix use-after-free when perf mmap() revival races with the
last munmap() (Yilin Zhang, Weiming Shi)
* tag 'perf-urgent-2026-09-06' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip:
perf: Fix use-after-free when perf mmap() revival races with the last munmap()
perf/core: Skip empty AUX records with only format flags
|
||
|
Linus Torvalds
|
c8990f3179 |
Misc locking fixes:
- Fix a softirq processing delay bug in local_interrupt_disable(),
which should mostly only affect the Rust runtime (Boqun Feng)
- Remove the hardirq_disable_count() function which caused the
previous bug and is now unused & unnecessary (Boqun Feng)
- lockdep: Invalidate stale class_cache entries for zapped classes
(Eric Dumazet)
- Fix rt_mutex specific futex scheduling helpers
(Sebastian Andrzej Siewior)
- Fix rcuwait use-after-free race during futex requeue PI (Yao Kai)
Signed-off-by: Ingo Molnar <mingo@kernel.org>
-----BEGIN PGP SIGNATURE-----
iQJFBAABCgAvFiEEBpT5eoXrXCwVQwEKEnMQ0APhK1gFAmqdSKMRHG1pbmdvQGtl
cm5lbC5vcmcACgkQEnMQ0APhK1gh6hAAtIhAm10pkx7JG4Kl+SQArmoAxNMfW9X1
grx50dn1/LFuIF0upooJwJmLMBpPMmxcpYPWbg5nTUsx+wDqcJTT0T7Aw2iZN2f+
OdODMOGNuQU4IUxDC+qX+fkgHvRFzng0zhSXYL2kGpQJHktrFMAHn90+5aO9SEgA
XcSjmDacTxfJADNk43snrlHy6t+yOXBEmIrhicnbh11GFNIXLbvdGW/0SObU0Q5o
WdwGHTZRFAM3mpX7xMSFOKypM+Gt4pXdQ/uC1PQg0B/AT35p4zsYIdcht/cBMHp0
93eTeJd9ede410t3hL1Hnsuo1zf5qHtCgBukS+UHJWoWpnSOMgT2iaBx2X3jIsev
ptPIqV5Alq8O+tnNOSDOwD2HH2drAdUBVH+gZ/yKMDbk9jfw9eZKqRw628U2P7rG
mD5CLzpmfj0mMAuqcOhhdqSQmPhjp2Y8Ju/8UAMSmcocvqjQBxvouNS3Tby9U5v+
R4CsuUwtnYpMyy5xyRrnwVsaE3le5fvneXCmCokk8FtQ8as/4ATd+cG+RROSa3vW
/AAjrB1HBQqyseE8lki4LFFmlr01Otc8u6pASy4/pxeuKMA9hSuLC3RuEfVetW6W
BcK4BCIJBy6chcsCBxv0F7FgI/y7/bpVyBmoTkW3Sb0I23xswXzTzbucn/dPJNdl
oqFj3KQygxw=
=vEuZ
-----END PGP SIGNATURE-----
Merge tag 'locking-urgent-2026-09-06' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
Pull locking fixes from Ingo Molnar:
- Fix a softirq processing delay bug in local_interrupt_disable(),
which should mostly only affect the Rust runtime (Boqun Feng)
- Remove the hardirq_disable_count() function which caused the
previous bug and is now unused & unnecessary (Boqun Feng)
- lockdep: Invalidate stale class_cache entries for zapped classes
(Eric Dumazet)
- Fix rt_mutex specific futex scheduling helpers
(Sebastian Andrzej Siewior)
- Fix rcuwait use-after-free race during futex requeue PI (Yao Kai)
* tag 'locking-urgent-2026-09-06' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip:
futex: Prevent rcuwait use-after-free during requeue PI
futex: Provide rt_mutex_.*_schedule() equivalents for futex scheduling
locking/lockdep: Invalidate stale class_cache entries for zapped classes
preempt: Remove hardirq_disable_count()
interrupt: Disable interrupt before modifying hardirq_disable counter
|
||
|
Kumar Kartikeya Dwivedi
|
c3fd8e5fd1 |
bpf: Reject non-scalar bpf_loop iteration counts
bpf_loop() declares its nr_loops argument as ARG_ANYTHING. Privileged programs may pass pointer values to such arguments, so check_func_arg() lets a pointer-valued R1 reach the helper-specific checks. Since commit |
||
|
Kees Cook
|
3a2c4d55e3 |
treewide: refresh kmalloc_obj() conversions
This is another run of the Coccinelle script for converting kmalloc() family of allocations to kmalloc_obj() via the existing rules in scripts/coccinelle/api/kmalloc_objs.cocci This catches both the set of kmalloc() uses added since the first kmalloc_obj() conversions in v7.0 and adds a large group missed in the first pass due to Coccinelle not interacting well with the cleanup.h scoped_...() family of macros[1]. I worked around this with spatch's "--macro-file" argument to a file with all the scoped_...() macros mapped to Coccinelle's YACFE_ITERATOR[2] as that was the closest viable control flow indicator I could find. Build tested allmodconfig on x86, arm64, arm, loongarch, mips, powerpc, riscv, and s390 with no new warnings. Link: https://lore.kernel.org/lkml/202609021314.8A9C0B8@keescook/ [1] Link: https://github.com/coccinelle/coccinelle/blob/master/standard.h [2] Signed-off-by: Kees Cook <kees+treewide@kernel.org> |
||
|
Eduard Zingerman
|
cf2475616b |
bpf: use mark_arg_precision() in check_mem_size_reg()
Use newly added mark_arg_precision() helper in check_mem_size_reg(). Signed-off-by: Eduard Zingerman <eddyz87@gmail.com> Link: https://lore.kernel.org/r/20260904-register-is-null-precise-fixes-v1-10-0f5a360ff15d@gmail.com Signed-off-by: Alexei Starovoitov <ast@kernel.org> |
||
|
Eduard Zingerman
|
1d7f8f191c |
bpf: propagate mark_chain_precision() errors out of loop_flag_is_zero()
Stop verification if mark_chain_precision() fails when called from
loop_flag_is_zero(). No functional change intended for the paths where
backtracking succeeds.
Fixes:
|
||
|
Eduard Zingerman
|
e726fc6b9a |
bpf: mark a NULL BTF_ID argument of a global subprogram precise
btf_check_func_arg_match() accepts a NULL register for an
ARG_PTR_TO_BTF_ID argument tagged __arg_nullable and skips
check_reg_type() and check_func_arg_reg_off() without marking the
register precise. Hence a checkpoint created on such a path would
prune against arbitrary scalar value.
Fixes:
|
||
|
Eduard Zingerman
|
506ada8962 |
bpf: mark a NULL kfunc argument precise
check_kfunc_arg() allows bpf_register_is_null() for nullable arguments
w/o marking the underlying scalar register precise. Hence a checkpoint
created on such a path would prune against arbitrary scalar value.
Fixes:
|
||
|
Eduard Zingerman
|
f1e418129f |
bpf: mark a NULL memory argument of a call precise
check_mem_reg() allows bpf_register_is_null() for nullable arguments
w/o marking the underlying scalar register precise. Hence a checkpoint
created on such a path would prune against arbitrary scalar value.
The argument may live on the stack rather than in a register when a
call has more than MAX_BPF_FUNC_REG_ARGS arguments, hence the new
mark_arg_precision() helper.
Fixes:
|
||
|
Eduard Zingerman
|
1a3a10b030 |
bpf: mark a NULL call argument precise
check_func_arg() allows bpf_register_is_null() for nullable arguments
w/o marking the underlying scalar register precise. Hence a checkpoint
created on such a path would prune against arbitrary scalar value.
check_helper_call() enforces second parameter of the
bpf_get_local_storage() to be zero, w/o marking the underlying scalar
register precise. Hence a checkpoint created on such a path would
prune against arbitrary scalar value.
Grouping these two into one patch, as they share the same fixes tag.
Fixes:
|
||
|
Steven Rostedt
|
d80e12156f |
ring-buffer: Use a macro for static buffer bits
Instead of hard coding 30 for the number of bits used for the static buffer ids in two places, create a macro. This way if it changes in the future, it will change in all the locations that use it. Link: https://patch.msgid.link/20260904151641.17eae0aa@gandalf.local.home Signed-off-by: Steven Rostedt <rostedt@goodmis.org> |
||
|
Steven Rostedt
|
5cbea50077 |
tracing: Fix comment in tracing_buffers_splice_read()
The comment about returning an error if the read fails on the first
iteration is slightly incorrect. It makes it sound like the only reason it
could fail on a later iteration is if the subbuf order changed. That is
incorrect, it could also fail if the length passed in was not a multiple
of the subbuf size. Fix the comment.
Link: https://lore.kernel.org/all/20260904143527.40e73d36@gandalf.local.home/
Link: https://patch.msgid.link/20260904144902.506862a1@gandalf.local.home
Fixes:
|
||
|
Vincent Donnefort
|
c843fd3c73 |
ring-buffer: Prevent truncation of nr_pages / nr_subbufs
Although ring_buffer_per_cpu::nr_pages is defined as unsigned long, it is capped to 32-bits in a few places, limiting the operations possible on a very large buffer. Use `unsigned long` where appropriate and prevent truncation of values using nr_pages (or nr_subbufs). While at it, subbuf_size must be at least `unsigned int`. Note that persistent, remote and user-mapped ring buffers are capping the number of pages to 30 bits already, making "int" safe in many places. Link: https://patch.msgid.link/20260904164450.1345852-5-vdonnefort@google.com Signed-off-by: Vincent Donnefort <vdonnefort@google.com> Signed-off-by: Steven Rostedt <rostedt@goodmis.org> |
||
|
Vincent Donnefort
|
f2b2b64559 |
ring-buffer: Cap static ring buffer nr_pages
Static ring buffers (i.e. persistent, user-mapped and remote) rely on
the bpage::id field. The number of pages for those ring buffers must fit
into that variable. Enforce this limit on ring buffer creation or
user-mapping.
While at it, prevent nr_pages underflow when allocating a persistent
buffer.
Link: https://patch.msgid.link/20260904164450.1345852-4-vdonnefort@google.com
Fixes:
|
||
|
Vincent Donnefort
|
dae8dda341 |
tracing: Fix subbuf resize races with trace_pipe_raw readers
Concurrent subbuffer resizes may crash trace_pipe_raw readers or leak
uninitialized memory to userspace due to stale size values.
Modify ring_buffer_alloc_read_page() to handle the resizing of an
existing buffer_data_read_page if necessary and add a new
ring_buffer_read_page_size(). This new function enables ring-buffer
buffer_data_read_page users to not call the racy
ring_buffer_subbuf_size_get(). This makes the spare_size member of
ftrace_buffer_info redundant.
Finally, handle buffer_data_read_page/reader_page order discrepancy in
ring_buffer_read_page(). On a mismatch simply copy manually the data to
the buffer_data_read_page.
Link: https://lore.kernel.org/all/20260817140812.2C7D41F00A3A@smtp.kernel.org/
Link: https://patch.msgid.link/20260904164450.1345852-3-vdonnefort@google.com
Fixes:
|
||
|
Kumar Kartikeya Dwivedi
|
b90c5d770d |
bpf: Preserve inner map identity in callback frames
Callback frame constructors initialize map-typed argument registers with __mark_reg_known_zero() and then restore map_ptr. This clears map_uid, which is the only field distinguishing inner maps that share an inner_map_meta template. When a timer callback invokes bpf_for_each_map_elem() on a second inner map, both the saved first map and the second map value can reach the nested callback as the same template with map_uid zero. bpf_timer_init() then accepts pairing the timer from the second map with the first map. The runtime records the first map in the timer without taking a reference. Freeing that map does not find the timer stored in the second map, so a later timer callback dereferences the freed map. Copy map_uid from the same caller register as map_ptr when constructing for-each, timer/workqueue, and task-work callback arguments. The existing identity check can then reject mismatched inner maps while allowing a callback value to be paired with its actual map. Fixes: |
||
|
Kumar Kartikeya Dwivedi
|
ecdc504379 |
bpf: Mark NULL kptr stores precise
check_map_kptr_access() permits a scalar store into an untrusted kptr
field only when the register is known to contain zero. Unlike other
verifier checks whose outcome depends on a scalar value, it does not mark
that register precise.
A state checkpoint reached with an imprecise zero can therefore prune a
second path that reaches the store with an arbitrary nonzero scalar. The
program can write attacker-controlled bits into the kptr field and load
them back as a PTR_TO_BTF_ID.
Call mark_chain_precision() before accepting a known-zero register. This
forces state equivalence to compare its scalar range and makes the verifier
visit and reject a path carrying a nonzero value.
Fixes:
|
||
|
Nuoqi Gui
|
65cc95eba9 |
bpf: Cancel special fields when recycling rhtab elements
rhtab_map_update_existing() and rhtab_delete_elem() call
bpf_obj_free_fields() when replacing or deleting a value. These map
operations can run from BPF programs in NMI context, where releasing a
referenced kptr or another complex field is not generally safe.
Array and hash maps avoid that problem by cancelling only the asynchronous
fields which can be stopped safely in the caller context. Other ownership
state remains attached to the allocation until its memory allocator
destructor performs the final cleanup.
Use bpf_obj_cancel_fields() for the corresponding rhtab paths as well. This
cancels timers, workqueues, and task work while allowing rhtab_mem_dtor() to
release referenced kptrs when the allocation is eventually destroyed.
Fixes:
|
||
|
Yuan Chen
|
5df46ddcb7 |
bpf: Preserve special fields in recycled rhtab elements
rhtab_map_update_elem() initializes special fields after obtaining an element from bpf_mem_cache_alloc(). The allocator can return a fresh, zeroed unit, or recycle one from its RCU-pending lists before the registered destructor has run. A BPF program can retain a map-value pointer after deleting its element and initialize and arm a timer through that pointer. If the deleted unit is recycled, check_and_init_map_value() clears the only pointer to the timer. Neither a later deletion nor rhtab_mem_dtor() can then cancel it, and the callback can run with its key and value pointing into freed memory. Do not reinitialize special fields on insertion. Fresh allocator units are already zeroed. For recycled units, the special fields are ownership state that must remain visible to the eventual destructor. copy_map_value() already skips those fields, matching the non-preallocated hash-map path and the lifecycle established by commit |
||
|
Masami Hiramatsu (Google)
|
d7dbdd2ee0 |
tracing: Fix to avoid creating trace instances with duplicate names
Since commit |
||
|
Linus Torvalds
|
421066905c |
Probes fixes for v7.3-rc1:
- kprobes: Protect kprobe_blacklist with RCU
. RCU-protect kprobe_blacklist and use kfree_rcu() to prevent UAF
races during module unloading and enable safe atomic lookups.
- tracing/probes: Fix multi-probe field use-after-free and BTF parsing
. Multi-probe UAF fix: Duplicate field and type strings on
trace_probe_event to prevent UAF when freeing primary probe.
. BTF member lookup fixes:
- Check the containing inner struct/union kflag when resolving
anonymous members to ensure correct bitfield offset calculation.
- Prevent unnamed bitfields from being pushed to anon_stack in
btf_find_struct_member(), avoiding false lookup errors.
- Fix code block indentation in get_bitoffset_of_field().
- uprobes: Error pointer safety
. Guard free_trace_uprobe() with IS_ERR_OR_NULL() to avoid crashing
during automatic cleanup when an error pointer is returned.
-----BEGIN PGP SIGNATURE-----
iQFPBAABCgA5FiEEh7BulGwFlgAOi5DV2/sHvwUrPxsFAmqahXQbHG1hc2FtaS5o
aXJhbWF0c3VAZ21haWwuY29tAAoJENv7B78FKz8bOJIH/1RuAq2y8fvfqWKwDBNG
9CrSIMmZ0915s4LVSGQrrjNYfpj2rFYkEMsJcFo2pavKwWNyaxXFjXu8Vy9JGckx
VFAHA52x2QaEYdwBeoo/Jd3+7Ks/3zH1XwfSILFa0PMn86/JCKHx/+5ah6Sk4vcu
/he61Auyp6lJtvv88n95j1evCJNouU6lJ3fnvm8mNYTeLOIvPZ3qku6SsiOqNdeQ
Ln8bcNP2Iis33PqfeydiRWv9nPog/ifH4a9WJ+fdqKA+06AHKVsfHB+fP0hxyUDp
TS4oxKrIk6HIbEQRjgo8YcOPWurHAm0GQ2ZslgLFuWDE9rDQCtPdOYv/YzJe0ByG
nHc=
=hX9F
-----END PGP SIGNATURE-----
Merge tag 'probes-fixes-v7.3-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace
Pull probes fixes from Masami Hiramatsu:
- Protect kprobe_blacklist with RCU
RCU-protect kprobe_blacklist and use kfree_rcu() to prevent UAF races
during module unloading and enable safe atomic lookups.
- Fix multi-probe field use-after-free
Duplicate field and type strings on trace_probe_event to prevent UAF
when freeing primary probe
- Fix probe BTF member lookup:
Check the containing inner struct/union kflag when resolving
anonymous members to ensure correct bitfield offset calculation
Prevent unnamed bitfields from being pushed to anon_stack in
btf_find_struct_member(), avoiding false lookup errors
Fix code block indentation in get_bitoffset_of_field()
- uprobes error pointer safety
Guard free_trace_uprobe() with IS_ERR_OR_NULL() to avoid crashing
during automatic cleanup when an error pointer is returned
* tag 'probes-fixes-v7.3-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace:
kprobes: Protect kprobe_blacklist with RCU
tracing/probes: Fix use-after-free on field name/type of events with multiple probes
tracing/probes: Fix code indent in get_bitoffset_of_field()
tracing/probes: Fix BTF kflag check for anonymous struct member access
tracing/probes: Fix anon_stack check for unnamed bitfields in btf_find_struct_member
uprobes: guard trace cleanup against error pointers
|
||
|
Kumar Kartikeya Dwivedi
|
cd6f72d7f3 |
bpf: Clear NON_OWN_REF after RCU protection ends
A local kptr load of an object containing a graph node is marked MEM_RCU
and NON_OWN_REF while protected by RCU. When the last RCU read-side critical
section ends, invalidate_rcu_protected_refs() removes MEM_RCU and marks the
pointer PTR_UNTRUSTED, but leaves NON_OWN_REF set.
The stale flag lets graph kfunc argument checks continue treating the
pointer as a live borrowed reference. In particular, bpf_rbtree_remove()
can accept a pointer after its protection ended and return it as a new
owning reference, even though the object may already have been freed.
Clear NON_OWN_REF when an RCU-protected pointer is demoted. A spin lock also
provides implicit RCU protection, so invalidate non-owning references before
demoting RCU-protected pointers when releasing the lock. Otherwise the
demotion would clear the flag before invalidate_non_owning_refs() can find
and invalidate those aliases.
The demoted pointer remains available for fault-protected reads. Exempt such
reads from the allocated-object reference-state assertion; writes through a
fault-prone pointer are already rejected, and bpf_may_fault_on_deref() makes
the surviving loads use BPF_PROBE_MEM.
Fixes:
|
||
|
Ning Ding
|
dc36739e5c |
bpf: Keep refcount_acquire nullable for borrowed RCU kptrs
bpf_refcount_acquire() is fallible for a borrowed reference because the
object may have reached a zero refcount. The verifier therefore keeps
KF_RET_NULL on the return value unless the argument is an owning reference.
An RCU-protected load of a local kptr is marked MEM_ALLOC, but it only
receives NON_OWN_REF when the pointee contains a graph node. A refcounted
object without a graph node consequently looks like an owning reference
even though the loaded register has no acquired reference state. If the
program drops the last real reference while remaining in the RCU critical
section, refcount_inc_not_zero() returns NULL while the verifier treats the
result as non-NULL.
Only classify the argument as owning when it is backed by a verifier-tracked
reference. This retains the non-NULL return for pointers from bpf_obj_new(),
bpf_kptr_xchg(), or an earlier successful acquisition, while requiring a
NULL check for borrowed RCU kptrs.
Fixes:
|
||
|
Kumar Kartikeya Dwivedi
|
048029ba1c |
bpf: Require MEM_PERCPU for percpu kptr stores
map_kptr_match_type() treats perm_flags as the set of register type flags
that a kptr field permits. Adding MEM_PERCPU to that set for
BPF_KPTR_PERCPU does not require the source register to carry it, however.
The subset test consequently accepts both a plain bpf_obj_new() allocation
and a referenced kernel pointer into a __percpu_kptr map field.
Loads from the field are always marked MEM_PERCPU. Consumers then treat the
stored value as the cookie returned by bpf_percpu_obj_new(): per-CPU pointer
helpers relocate it, and map teardown selects the per-CPU free path. A plain
allocation can therefore provide an arbitrary kernel read/write, while a
kernel pointer can be relocated into an invalid address or sent through a
missing destructor.
Require the source MEM_PERCPU flag to match the destination field kind.
This preserves valid bpf_percpu_obj_new() stores and rejects both the
program-BTF and kernel-BTF variants.
Fixes:
|
||
|
Eduard Zingerman
|
6aed0134d3
|
bpf: Mark the zero register precise for a register-form NULL check
check_cond_jmp_op() accepts "if rA <op> rB" as a NULL check for a
nullable pointer rA when rB is a scalar known to be zero,
lifts PTR_MAYBE_NULL from rA in the corresponding branch and does not
mark rB precise. Consider the following program:
r0 = bpf_get_prandom_u32();
r6 = 1; /* the r6 == 0 path is explored first */
if (r0 == 0) goto 1f;
r6 = 0;
1:
r0 = bpf_map_lookup_elem(map, &0); /* absent, NULL at runtime */
if (r0 == r6) goto 2f; /* taken as a NULL check for r0 */
*(u8 *)(r0 + 0); /* verifier: map value; runtime: zero */
2:
return 0;
The r6 == 0 path is explored first and the dereference is accepted.
The r6 == 1 path is pruned at the checkpoint recorded for (1),
so the comparison is never verified with a non-zero r6. At runtime a
failed lookup returns NULL, NULL != 1 takes the non-NULL edge and the
program dereferences a pointer that is zero.
Fixes:
|
||
|
Eduard Zingerman
|
e51179a4e0
|
bpf: Don't predict JMP32 pointer vs zero comparisons
Consider the following program:
r1 = map_value; /* low 32 bits are zero at runtime */
r6 = 0xdead000000000000;
if w1 != 0 goto l1;
l0: r1 += r6;
r2 = *(u64 *)(r1 + 0);
exit;
l1: r6 = 0;
goto l0;
At the moment is_branch_taken() reports the jump as always taken,
because it does not distinguish between BPF_JMP and BPF_JMP32
comparisons when processing 'if w1 != 0 ...'.
Fixes:
|
||
|
Eduard Zingerman
|
73a98f9681
|
bpf: Don't resurrect a scalar id dropped by collect_linked_regs()
check_cond_jmp_op() copies the compared registers into
env->{false,true}_reg{1,2} before collect_linked_regs() runs and copies
those snapshots back into both branch states afterwards.
collect_linked_regs() records at most LINKED_REGS_MAX members of a
linked registers group in the jump history and calls clear_scalar_id()
for every member that does not fit. The compared register is not exempt
from that.
As a consequence, sync_linked_regs() might adjust ranges for more
registers than bpf_bt_sync_linked_regs() can propagate precision to.
Collect the linked registers before the snapshots are taken instead.
This might lead to some unnecessary clear_scalar_id's, but from
previous testing situations with many linked registers are
extremely rare.
Fixes:
|
||
|
Eduard Zingerman
|
67b529f521
|
bpf: Don't infer non-NULL from a pointer with an unbounded offset
reg_not_null() decides that a register holds a non-NULL value by
looking at its type alone. For pointer types that allow arithmetic the
type only guarantees a non-NULL base, in case of an unbound offset
the runtime offset value might still add up to NULL.
Consider the followng program:
r6 = bpf_map_lookup_elem(map, &0); /* present */
if (r6 == 0) return 0;
r7 = bpf_map_lookup_elem(map, &1); /* absent, NULL at runtime */
r8 = r7;
r8 -= r6; /* pointer - pointer: unknown scalar, -r6 */
r8 <<= 1;
r8 >>= 1; /* any non-negative offset is accepted by */
/* check_reg_sane_offset_ptr() */
r6 += r8; /* verifier: map value; runtime: zero */
if (r7 != r6) return 0;
*(u8 *)(r7 + 0); /* r7 is inferred non-NULL, both are zero */
At runtime both registers are zero, the comparison is true and the
load faults with NULL pointer dereference.
Require the offset to be within +-BPF_MAX_VAR_OFF in reg_not_null().
Fixes:
|
||
|
Yao Kai
|
a3b8d46fe4 |
futex: Prevent rcuwait use-after-free during requeue PI
On PREEMPT_RT, FUTEX_CMP_REQUEUE_PI can trigger a KASAN report
(slab-out-of-bounds) in futex_requeue_pi_complete() invocation of
rcuwait_wake_up().
The futex_q used by futex_wait_requeue_pi() is allocated on the waiter's
stack. An early wakeup can race with a PI requeue as follows:
waiter requeue task
------ ------------
futex_wait_requeue_pi()
futex_do_wait()
schedule()
futex_requeue
futex_proxy_trylock_atomic()
futex_requeue_pi_prepare()
Q_REQUEUE_PI_NONE -> Q_REQUEUE_PI_IN_PROGRESS
* timeout/ signal wakes waiter *
futex_requeue_pi_wakeup_sync()
Q_REQUEUE_PI_IN_PROGRESS -> Q_REQUEUE_PI_WAIT
requeue_pi_wake_futex
futex_requeue_pi_complete()
cmpxchg Q_REQUEUE_PI_WAIT -> Q_REQUEUE_PI_LOCKED
rcuwait_wait_event()
if (atomic_read(&q->requeue_state) != Q_REQUEUE_PI_WAIT)
break /* no schedule() */
/* q.pi_state->owner == current */
futex_private_hash_put()
/* return from syscall */
rcuwait_wake_up(&q->requeue_wait)
/* q is gone */
futex_requeue_pi_complete() publishes Q_REQUEUE_PI_LOCKED before
calling rcuwait_wake_up(). The waiter observes this state in
rcuwait_wait_event() before invoking schedule() in rcuwait_wait_event().
Here, the waiter is free leave the syscall before requeue task can
complete the wake.
To address this race skip rcuwait_wake_up() in the Q_REQUEUE_PI_LOCKED
case.
This state is only published by requeue_pi_wake_futex(), which saves
q->task before futex_requeue_pi_complete() and wakes the waiter via
wake_up_state().
This wake is intended to wake the waiter from its futex_do_wait() sleep.
If the waiter is still sleeping there, it can not get into the
Q_REQUEUE_PI_WAIT state (and require this removed wake).
Should the waiter be woken up from futex_do_wait() by other means (as in
this example) and sleep in futex_requeue_pi_wakeup_sync() then the
wake_up_state() from requeue_pi_wake_futex() will wake it, too.
Should the waiter task terminate before wake_up_state() had a chance to
wake the task then the task pointer does not become invalid because the
futex_hash_bucket::lock is held and the task pointer is RCU protected.
[bigeasy: Updated comment and commit message]
Fixes:
|
||
|
Sebastian Andrzej Siewior
|
912edebe85 |
futex: Provide rt_mutex_.*_schedule() equivalents for futex scheduling
There is rt_mutex_{pre|post}_schedule() around
rt_mutex_wait_proxy_lock() to ensure that sched_submit_work()/
sched_update_worker() is invoked before we schedule out and block on
rt_mutex while waiting for it become available.
The reason is that blocking on rt_mutex assigns a pi_waiter for the PI
chain and sched_submit_work() will also assign a pi_waiter if it blocks
on lock but a this point we already have a waiter assigned.
We can't skip sched_submit_work() entirely because I/O relies on the
fact that I/O queue is flushed while it blocks on a sleeping lock.
Therefore sched_submit_work() is moved before we block on the lock.
Sleeping lock in this context means mutex or rw_semaphore not spinlock_t
on PREEMPT_RT. Because the mutex abstraction on PREEMPT_RT uses the same
abstraction as the futex proxy lock, the futex code ended up using
rt_mutex_{pre|post}_schedule(), too.
Using it is/ was just to keep the task_struct::sched_rt_mutex assertion
happy. Futex proxy lock is used only in the syscall context of a task.
At this point it never got any I/O that needs to be flushed and it can't
be a workqueue that needs to notify that it will be scheduled out.
Therefore sched_submit_work() does nothing here.
By mistake futex_wait_requeue_pi() -> rt_mutex_wait_proxy_lock() did not
get the rt_mutex_{pre|post}_schedule() annotation. This was not noticed
because in this callchain the lock is (usually) not contended and so
rt_mutex_slowlock_block() does not schedule, triggering the assert.
Adding rt_mutex_pre_schedule() here looks wrong (as noted by PeterZ)
because at this point there is a pi_waiter recorded and invoking
sched_submit_work() with a possible lock contention would be wrong.
Add rt_mutex_futex_{pre|post}_schedule() which toggles the
sched_rt_mutex assert and does not involve sched_submit_work(). Add
asserts here to ensure that sched_submit_work() would do nothing. Use it
only in futex proxy lock case which is rt_mutex_wait_proxy_lock().
Remove it from futex_lock_pi().
Fixes:
|