mirror of
https://github.com/torvalds/linux.git
synced 2026-11-07 06:17:02 +01:00
v7.4-rc1
5218 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Linus Torvalds
|
46806af187 |
bpf-fixes
-----BEGIN PGP SIGNATURE-----
iQJRBAABCgA7FiEE+soXsSLHKoYyzcli6rmadz2vbToFAmrjbZEdHGFsZXhlaS5z
dGFyb3ZvaXRvdkBnbWFpbC5jb20ACgkQ6rmadz2vbTqGEQ//bqWuSeYGypshirYQ
4rxejupvJBJmrD+pI3MSDKVDaLgslm2vW8+vYB+rxpwzje4EsBtVqoHVxzk4yMoQ
WyuJ7988W3RRZw5x5EEruS65C2pM5H2Ilzj2f8Z5gYUfvB72/TGJD2sDXxlTAtUb
ESujBo/M9JRyYZBE5SMJT00r17clwf8NHoT3nr1FKAaDLQopE2zAnE/RwB1lEm3V
MtUcnyDWEwu9ITOxMAwymzYI4FHadqxLJ5MCyCFUeqWAkJcPoVGHWN9Z/0cLTBYt
46PZIotJUEio1PsF6/3y0ZAS2whCAzBuuQCcRG1qBoOdczaDN8fl7RAuTvFmSpPJ
H5Z1GUAXvoFAPpfb9upwYwrEc5hH7+4w9LsGjEumKZ2kHzPccH8K3jtUJRRvYiL8
W7LH4iiu/yPvYHKsgt89csGy6uoACUTn3Xbgfho4DzMoOO8Q8Ahw3SKMpPLIXPk2
1q8Olv9d2pdaVRVJ4qXGPV4510LXW5KwDBHiztZKij8HvasPJS4goaTRKOSIqYOx
GCWCuyM4F22N1suHyUZea507PPVFBKcNBRXWt3nNbKWZNFXpayyS2UI/3kusFFuD
+EFcu0sNDENJ04/SGgujswvGGYrahT8AsHSkwM8jMpiajBiI0ozjWyLwVYk17mAY
WrPIg+X/Uh5nxk5o5RIbniOJ8l8=
=kHds
-----END PGP SIGNATURE-----
Merge tag 'bpf-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf
Pull bpf fixes from Alexei Starovoitov:
- Remove BPF_F_ARENA_SCALAR flag and .arena.data section in libbpf that
were added for rust-bpf in this merge window. We found a cleaner
approach that does it on compiler side instead of kernel/libbpf. No
released kernel has them (Alexei Starovoitov)
- Fix bpf_send_signal_task() returning -EPERM when it is called from a
kernel thread (Aditya Sharma)
- Various fixes (Alexei Starovoitov):
- Use emit_rsb_call() to call bpf_unwind_target() in x86 JIT to fix
objtool warning during the build
- Fix scalar evolution: clear delta of a widened register, don't
widen a pointer to a buffer, don't clamp a register whose type
changed in the loop, and reset the range of a clamped packet
pointer
- Finish the insn in the state that maybe_fork_scalars() splits
off, so that pruning doesn't drop it at the head of a counted
loop
- Preserve the registers that carry stack arguments across
arch_bpf_timed_may_goto() on x86-64, arm64 and riscv64
- Don't re-sort subprog_info[] for every new subprogram
- Fix flaky map_kptr and rcu_reentry selftests
- Use memmove() in bpf_skb_load_bytes(), bpf_skb_load_bytes_relative()
and bpf_xdp_load_bytes(), since the destination can be a slice of the
same packet (Ayushman Rout)
- Fixes and follow-ups for scalar evolution: reject pointer operands in
32-bit recurrences and latches, don't intersect with the current
register range on clamp to avoid a false 'verifier bug', avoid false
infinite loop reports, and recognize a 32-bit copy of the induction
variable (Eduard Zingerman)
- Fix out-of-bounds write to btf->types[] for split BTF and propagate
btf_add_type() errors in btf_check_all_metas() (Feng Yang)
- Fix use-after-free in bpf_timer callback that fires after its inner
map was freed (Mani Tofigh)
- Fix x86 JIT stopping before jumps converge: size backward jumps by
the insns they jump over (Mykyta Yatsenko)
- Fix use-after-free of struct_ops trampoline: dissociate struct_ops
progs before the grace period of the map (Qiliang Yuan)
- Validate stack argument reads against the caller state to fix
out-of-bounds read in the verifier, and reject callbacks with stack
arguments (Weiming Shi)
- Tie borrowed iterator results and pointers loaded from trusted fields
to the lifetime of their source, so that iterator next, destroy and
reference release invalidate them (Xu Yunxiang)
* tag 'bpf-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf: (48 commits)
bpf: Don't re-sort subprog_info[] for every new subprogram
bpf: Fix out-of-bounds btf->types write for split BTF
selftests/bpf: Test stack arguments across may_goto
bpf: Keep stack arguments in arch_bpf_timed_may_goto()
selftests/bpf: Test maybe_fork_scalars() at the head of a counted loop
bpf: Finish the insn in the state that maybe_fork_scalars() splits off
selftests/bpf: Test the clamp of pointers in scalar evolution
bpf: Reset the range of a packet pointer that scalar evolution clamps
bpf: Don't clamp a register whose type changed in the loop
selftests/bpf: Test a loop that moves a pointer to a buffer
bpf: Don't widen a pointer to a buffer in scalar evolution
selftests/bpf: Avoid a page fault in rcu_reentry while the handler is armed
selftests/bpf: Wait for the last maps of map_kptr to be freed
selftests/bpf: Test that a widened register is not linked
bpf: Clear delta of a register that scalar evolution widens
bpf, x86: Remove r0_on_stack from fetching atomics in arena
bpf: Remove BPF_F_ARENA_SCALAR
selftests/bpf: Remove tests for arena access through numbers
libbpf: Keep format strings of bpf_printk() in .rodata
libbpf: Remove .arena.data
...
|
||
|
Alexei Starovoitov
|
c8a8a68d3a
|
bpf: Don't re-sort subprog_info[] for every new subprogram
add_subprog() appends new subprog to env->subprog_info[] and calls sort()
on the whole array, though only the last entry can be out of order.
Move subprogs that start after the new one up by one entry instead.
'start' is unique, so the order of the array doesn't change. The new entry
can land in a used slot, so initialize all of it.
add_subprog() returns the index of the new subprog instead of the last
one. The callers check for error only.
It's a trivial prep for rust-bpf [1] that bumps BPF_MAX_SUBPROGS to 1024.
Rust crates have more than 256 functions that are not inlined.
Time in add_subprogs() when veristat loads sched_ext schedulers from scx
repo, 8 cpu VM, usec:
subprogs before after
arena_selftest 101 386 27
lavd_enqueue 52 96 16
lavd_dispatch 48 78 11
[1] https://github.com/4ast/rust-bpf
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Reviewed-by: Mahe Tardy <mahe.tardy@gmail.com>
Link: https://patch.msgid.link/20261023150302.4097865-1-alexei.starovoitov@gmail.com
|
||
|
Feng Yang
|
6c476d2b73
|
bpf: Fix out-of-bounds btf->types write for split BTF
btf_add_type() stores the types of a BTF in btf->types[], which can hold at
most BTF_MAX_TYPE entries. The types of a split BTF get IDs start_id ..
start_id + nr_types - 1, so only BTF_MAX_TYPE - start_id of them fit, but
neither the limit check nor the size used to grow the array accounts for
start_id: the check is an equality test against BTF_MAX_TYPE
('start_id + btf->types_size == BTF_MAX_TYPE') and the new size is clamped
to BTF_MAX_TYPE.
The check therefore only fires if the array happens to grow to exactly
BTF_MAX_TYPE - start_id entries. Otherwise, if the module has more than
BTF_MAX_TYPE types, the array is filled up to BTF_MAX_TYPE entries. The
next type that needs to be added reallocates the array to the same size,
after which btf->types_size no longer equals btf->nr_types and the growth
path is not taken anymore, so every following type is written past the end
of the array:
BUG: unable to handle page fault for address: ffffc90006200000
RIP: 0010:btf_add_type+0x74/0x270
Call Trace:
btf_check_all_metas
btf_parse_module
Bound both the array size and the limit check by BTF_MAX_TYPE - start_id.
Base BTF (start_id == 0) is unaffected.
Fixes:
|
||
|
Alexei Starovoitov
|
19caa38daf
|
bpf: Finish the insn in the state that maybe_fork_scalars() splits off
When a register is -1 or 0, maybe_fork_scalars() splits the state in two
before an 'and' or an 'or' with a constant. The result is more accurate for
each value alone. The verifier continues with -1. It pushes a copy of the
state with 0 and verifies the copy later, from the same insn.
today this patch
N: r1 &= 8 r1 is -1 or 0 N: r1 &= 8 r1 is -1 or 0
/ \ / \
r1 = -1 r1 = 0 r1 = -1 r1 = 0
r1 &= 8 push to N r1 &= 8 r1 &= 8
| | push to N+1
N+1: ... N+1: ...
At insn N the copy is a part of the state that it came from: 0 is in
[-1, 0]. If pruning compares the two states, it drops the copy, and no one
verifies the case r1 = 0.
The first state has branches that are not explored. Usually pruning does
not use such states. In a loop that scalar evolution counted it does, and
the head of such a loop always keeps its state. Thus a fork at the head of
a counted loop loses the case r1 = 0:
r9 = a map value of 8 bytes
r6 = -1
1: r6 &= 8 r6 is -1 or 0, only -1 is verified: 8
r6 ^= 8 0. It is 8 when r6 was 0.
r1 = r9
r1 += r6
r0 = *(u8 *)(r1 + 0)
r6 = -1 or 0
r7 += 1
if r7 < 100 goto 1b
from 11 to 11: safe
The verifier accepts this prog. The prog reads the byte that follows the
map value. With a constant of 64 Tbyte in a register it reads an address
that is not mapped, and the kernel crashes.
Do the insn in the copy too, and push the copy to the next insn. All other
code that pushes a state does the same.
No verdict changes in selftests, Cilium and scx. Each fork processes one
insn less, and the sequence in which the verifier explores states changes
slightly. 62 of 161 progs of Cilium process a different number of insns,
from -0.7% to +3.3%. The total is 986970 -> 987013. 2 of 563 progs of scx
process one insn less.
Fixes:
|
||
|
Alexei Starovoitov
|
41b06cfd1c
|
bpf: Reset the range of a packet pointer that scalar evolution clamps
A compare with pkt_end sets 'range' in all packet pointers that have the
same id. Then 'range - offset' bytes from the pointer are in the packet.
When a state returns to the head of a counted loop, bpf_clamp_scev_regs()
gives a new id to a packet pointer, and the offsets of all iterations. It
does not change the range. But the range is valid only together with the
old id and the old offsets.
1: ; R6=pkt(id=1,r=0,off=0..100)
r5 = r6
r5 += 100
if r5 > pkt_end goto 2f ; R6=pkt(id=1,r=200,off=0..100)
r6 += 100 ; R6=pkt(id=1,r=200,off=100..200)
r9 += 1
if r9 < 2 goto 1b ; clamp: R6=pkt(id=2,r=200,off=0..100)
after bytes from r6 that the verifier accepts
---------------------- ---------------------------------------
the compare 100, correct
r6 += 100 0, correct
the clamp 100, incorrect. Nothing compared this r6
Usually the verifier prunes the state at the head, and the range has no
effect. The verifier continues with the state when a different part of it
is not within the state at the head. A flag in a 32-bit spill is
sufficient. Then the prog reads and writes after pkt_end. A write to
skb_shared_info crashes the kernel.
Take the range that the pointer has at the entry of the loop. No compare in
the loop found this range, so it is valid in all iterations. The state at
the head has the same range, so the verifier prunes as before.
Fixes:
|
||
|
Alexei Starovoitov
|
1536e9b437
|
bpf: Don't clamp a register whose type changed in the loop
When a state returns to the head of a counted loop, bpf_clamp_scev_regs()
replaces the range of each register that changes by a constant step. The
new range is the range over all iterations. The function calculates it from
the register in the state that entered the loop. For a pointer this range
is the range of its offsets.
A call in the loop can invalidate the pointer. bpf_skb_pull_data() does it
to packet pointers, bpf_ringbuf_discard() does it to ringbuf memory. Then
the register is an unknown scalar. Scalar evolution does not see that, and
the clamp gives the offsets of the pointer to the scalar.
r6 = bpf_ringbuf_reserve() ; R6=ringbuf_mem
r7 = r6
1: if first iteration goto 2f
r1 = map_value
r1 += r6 ; accepted: R6 is 0 or 1
r0 = *(u8 *)(r1 + 0)
goto 3f
2: bpf_ringbuf_discard(r7) ; R6=scalar()
3: r6 += 1
r9 += 1
if r9 < 2 goto 1b ; clamp: R6=scalar(umax=1)
At run time r6 is the address of the ringbuf memory plus 1. The prog adds
it to a pointer to a map value, and the kernel crashes in the access.
Do not clamp a register that has a different type than at the entry of the
loop. Such a state is not within the state at the head. Thus the verifier
continues with it, and r6 stays an unknown scalar.
Fixes:
|
||
|
Alexei Starovoitov
|
41750e5d68
|
bpf: Don't widen a pointer to a buffer in scalar evolution
Scalar evolution widens counted loops. Since that change the verifier
rejects a loop over the per-cpu values of a map element:
SEC("iter/bpf_map_elem")
...
for (i = 0; i < n; i++) {
out[i] = *(__u32 *)pptr;
pptr += 8;
}
loop header at 12, widening r2 to 0..56 step 8
R2 invalid variable buffer offset: off=0, var_off=(0x0; 0x38)
pptr is a pointer to a buffer (PTR_TO_BUF). Scalar evolution replaces its
offset with the range over all iterations. But __check_buffer_access()
accepts only a constant offset. Thus a prog cannot use a widened
PTR_TO_BUF.
Do not widen it. Then the verifier verifies the loop one iteration at a
time, as it did before.
loop before SCEV with SCEV this patch
-------------------------------- ----------- ---------- ----------
counts in i, reads through pptr loads rejected loads
counts in p ('p < end'), no read 1M insns loads 1M insns
counts in p, reads through p 1M insns rejected 1M insns
The verifier cannot decide a comparison of two pointers to a buffer. Thus
it follows 'p < end' until the limit of 1M insns.
bpf_iter_bpf_percpu_array_map.c and bpf_iter_bpf_percpu_hash_map.c have the
first loop. They load today because num_cpus is volatile. The prog reads
the bound again in each iteration. Scalar evolution cannot count such a
loop, and it widens nothing.
Fixes:
|
||
|
Alexei Starovoitov
|
3f9924747c
|
bpf: Clear delta of a register that scalar evolution widens
After 'r3 = r8; r3 += 4' the verifier knows that r3 == r8 + 4. The two
registers have the same id, and r3 keeps the difference in reg->delta. When
the prog compares one of them with a constant, sync_linked_regs()
calculates the range of the other register from the difference of the two
deltas.
At a loop head scalar evolution widens a register: it replaces the register
with its range over all iterations. Then the register is not linked to
other registers. scratch_widened_reg_id() clears reg->id, but it does not
clear reg->delta. A subsequent copy of the register gets the stale delta.
Then the range of the copy is incorrect by this delta.
r3 = r8
r3 += 4 ; R3=scalar(id=1+4)
1: ; r3 is widened: no id, delta is still 4
r4 = r3 ; id=2 and delta 4 in both
r4 += 1 ; R4=scalar(id=2+1)
if r3 > 10 goto 3f ; R3 <= 10, R4 = R3 + 1 - 4 <= 7
if r4 <= 7 goto 2f
... ; not verified, r4 is up to 11 here
The verifier does not verify code that can run. Use clear_scalar_id(). It
clears id and delta together. Move it to bpf_verifier.h, so that scev.c can
call it.
Fixes:
|
||
|
Alexei Starovoitov
|
b62dee3a8b
|
bpf: Remove BPF_F_ARENA_SCALAR
Rust has no address spaces. Thus a prog in Rust accesses arena through
scalars. With BPF_F_ARENA_SCALAR the verifier accepts each load and each
store through a scalar as an access to arena.
The verifier rewrites one insn at a time. Thus it must add one insn to each
access. The compiler knows the source of each pointer. It can emit one
addr_space_cast for each pointer, as it does for the __arena pointers of C.
flag: one move for each access cast: one cast for each pointer
ax = (u32)r1 r1 = addr_space_cast(r1, 0, 1)
r2 = *(u64 *)(ax + 0) r2 = *(u64 *)(r1 + 0)
ax = (u32)r1 r3 = *(u64 *)(r1 + 8)
r3 = *(u64 *)(ax + 8)
20 progs of rust-bpf, with the flag and with the casts that a prototype of
an LLVM pass emits:
flag casts
moves or casts 1568 571
insns 14864 13938
jited bytes 72351 69111
No released kernel has the flag. LLVM does not emit the casts for Rust at
this time. Until it does, Rust progs that access arena do not load.
scx_mavd and scx_nitosis set the flag too. They are experimental schedulers
in C in sched-ext/scx, and they use the flag to omit __arena. They must use
__arena again.
This reverts commit
|
||
|
Linus Torvalds
|
8a36f53fd5 |
mm.git review status for linus..mm-stable
Everything:
Total patches: 213
Fixes: tags: 69
cc:stable patches: 43
Reviews/patch: 2.04
Reviewed rate: 86%
Excluding DAMON:
Total patches: 192
Fixes: tags: 63
cc:stable patches: 39
Reviews/patch: 2.18
Reviewed rate: 89%
Excluding DAMON and selftests:
Total patches: 182
Fixes: tags: 57
cc:stable patches: 39
Reviews/patch: 2.23
Reviewed rate: 89%
Summary of series in this merge:
- "mm: make VMA flag semantics explicit, eliminate VM_SPECIAL"
(Lorenzo Stoakes (ARM)):
Replace VM_SPECIAL with helpers describing mapping ownership and
behavior. Enforce the mmap flag rules and convert affected drivers,
removing ambiguous tests shared by driver mappings and ordinary memory.
- "support kselftest on nommu platform" (Hajime Tazaki):
Add opt-in mmap and mremap tests for NOMMU systems. Teach the runner to
avoid its fork-dependent timeout command and document the build and
runtime requirements.
- "mm/collapse: separate a collapse from its callers"
(Kiryl Shutsemau (Meta)):
Separate huge-page collapse from caller policy. Replace khugepaged
identity checks with explicit permissions, and separate scanning from
collapse with the mmap_lock drop in one place.
- "mm: implement and use vma_has_anon_rmap(), silence KCSAN"
(Lorenzo Stoakes (ARM)):
Provide a common helper for testing whether a VMA has anonymous reverse
mappings. Document the locking rules, annotate intentional concurrent
reads for KCSAN, and convert the core MM callers.
- "mm/sparse: remove SECTION_MARKED_PRESENT and further cleanups"
(David Hildenbrand (Arm)):
Remove SECTION_MARKED_PRESENT in favor of the early-section flag.
Simplify sparse memory initialization, section metadata handling and
related PFN validity checks.
- "sh: remove NUMA and SPARSEMEM support" (Mike Rapoport (Microsoft)):
Remove SuperH NUMA and SPARSEMEM support. The extra memory banks have
been unusable since the NO_BOOTMEM conversion, leaving only the code and
data overhead.
- "mm: Unify device DAX and HugeTLB vmemmap population paths"
(Muchun Song):
Unify the generic device DAX and HugeTLB vmemmap population paths and
shared tail-page handling. Remove the duplicate DAX machinery while
retaining the powerpc radix implementation.
- "kselftest: mm: fix some failure of split_huge_page_test" (Yeoreum Yun):
Fix huge-page split tests racing with khugepaged recollapse. Use pagemap
and page flags instead of VMA-boundary assumptions which fail when glibc
has already enabled THP.
- "memcg: move memcgid refcount to objcg to unpin dying memcgs"
(Bingfang Guo):
Let dying memcgs be freed while swapped-out shmem still holds their IDs.
Move ID references to reparentable objcgs, with the necessary swap
accounting and RCU lifetime changes.
- "mm/truncate: fix data loss when truncating straddling large folios"
(Zhang Yi):
Preserve out-of-range data when truncating straddling large folios. Fix
a lookup race and split-failure handling, and handle minimum folio orders
for the forthcoming ext4 iomap buffered I/O path.
- "mm/damon: introduce damos quota goal target metric complement flag"
(SJ Park):
Let DAMOS quota goals use a metric's complement, so utilization can also
express free-memory goals without adding separate metrics. Add the sysfs
interface, documentation and tests.
- "arm64/mm: Standardize printing for pgtable entries" (Anshuman Khandual):
Use the generic formatter for arm64 page-table entries, ready for future
wider entries. Move the safe-PTE-update check out of the header to avoid
an include cycle.
- "mm: kmsan: fix iounmap metadata teardown" (Dima Koziuk):
Fix KMSAN metadata leaks on iounmap and failed ioremap allocations. Free
backing pages before removing their mappings, and share the teardown with
partial-allocation cleanup.
- "mm/damon/core: preserve quota state when constructing schemes"
(Karl Mehltretter):
Preserve running DAMOS quota state while validating parameter updates.
Even rejected updates could reset accounting and exceed the quota window;
add tests for both accepted and rejected updates.
- "mm/vmalloc: fix vmalloc_dump_obj VA lookup" (Ye Liu):
Fix vmalloc_dump_obj() lookups for last-page addresses and allocations
spanning vmap zones. These allocations were being missed in diagnostic
output.
- "mm: various (urgent?) fixes" (David Hildenbrand (Arm)):
Fix mmap charge leaks and hugetlb use-after-free, wrong-address
unmapping, stuck locks and endless faults. Also preserve reservations on
failed mremap and handle stale userfaultfd bits.
- "maple_tree: fix maple_range_64 crashes in RCU mode" (Dimitris Charisis):
Fix pointer and metadata confusion during RCU destruction of
maple_range_64 nodes, avoiding double frees and invalid accesses. Add a
regression test; no in-tree user currently uses this combination.
- "mm: fix hugetlb NR_HUGETLB accounting on folio migration" (Hongfu Li):
Fix hugetlb migration and preallocation accounting so /proc/vmstat
retains the correct count and memory.numa_stat follows the folio to its
new node.
- "mm/damon: fix a minor bug, clean code, and add a test" (SJ Park):
Fix 32-bit DAMON age overflow which made old regions look young. Share
hugetlb folio acquisition code and add a regression test for the
previously fixed last-region skipping bug.
- "mm/damon/core: cleanup code, reduce stack usage, and add kunit"
(SJ Park):
Reduce kdamond stack usage and remove unused enums. Add deterministic
PSI quota-goal tests which do not depend on the machine's current memory
pressure.
- "mm/damon: fix the temporal goal tuner's size quota conversion"
(Donggeun Yoo):
Bound the temporal tuner's quota conversion to prevent overflow and zero
effective quotas, which could stop schemes making progress. Add boundary
tests.
- "Don't use GFP_DMA when calling dma_alloc_coherent" (Baoquan He):
Remove redundant GFP_DMA flags from coherent DMA allocation calls. The
DMA API already selects the zone from device limits and ignores the
caller's flag.
- "kselftest: mm: fix intermittent failure khugepaged test" (Yeoreum Yun):
Propagate child-process failures in the khugepaged tests and prevent
background collapse from racing their checks. This removes false passes
and intermittent failures.
- "MAINTAINERS: cover all hugetlb headers under HUGETLB SUBSYSTEM"
(Enze Li):
Cover all hugetlb headers in MAINTAINERS so get_maintainer.pl finds the
right maintainers. Also label the closing hugetlb_cgroup.h include
guard.
Summary of the 65 singletons in this merge:
The singletons include a few worthwhile performance fixes. Avoiding
unnecessary reverse-map tree updates gave a reported 15.9% improvement
in UnixBench's execl test on a 192-core x86 system. A reclaim retry fix
stops defrag_mode from spinning under memory.low protection; a reported
file-creation stress test ran 5.7 times faster, with far less memory
pressure. Narrower hugetlb invalidations also spare KVM guests needless
refaults: the reported Redis-in-VM P99 QPS penalty while punching holes
fell from 7.09% to 1.45%.
Several fixes prevent lost data or corrupted memory. Lazyfree huge-page
races with mremap()/munmap() could lose data or access freed memory, and
MADV_COLD could lose a concurrent write's dirty bit. Another fix
prevents stale migration entries from outliving large folios. On
filesystems with blocks smaller than a page, extending a file by 4 GiB
or more could expose bytes written past the old EOF through mmap instead
of zeroes. That is fixed, along with use-after-free races in
PAGEMAP_SCAN during process exit and in memory-cgroup accounting on
PREEMPT_RT kernels.
Other fixes stop UFFDIO_MOVE spinning forever on pinned large folios,
which could wedge other memory operations, and stop mremap() leaking
commit charges until allocations fail under strict overcommit. A further
fix keeps userfaultfd tracking intact on neighboring mappings. NUMA
binding no longer prevents long-term pins when the chosen node has only
movable memory, fixing QEMU/VFIO startup and io_uring buffer
registration failures. MGLRU also gets a fix for early-boot crashes with
memory cgroups disabled at boot, notably in small crash kernels.
-----BEGIN PGP SIGNATURE-----
iHUEABYKAB0WIQTTMBEPP41GrTpTJgfdBJ7gKXxAjgUCat66sAAKCRDdBJ7gKXxA
jrQrAP99kgSXRdlmRZbHeRBHQ6KvI4XnLvQ78MDIzZ3zmBaOugEA5jiIsK2/snFD
3VaV0oipVZpuwJkVJE6zFkh0tcHgTAo=
=DHrN
-----END PGP SIGNATURE-----
Merge tag 'mm-stable-2026-10-25-19-27' of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm
Pull more MM updates from Andrew Morton:
- "mm: make VMA flag semantics explicit, eliminate VM_SPECIAL" (Lorenzo
Stoakes):
Replace VM_SPECIAL with helpers describing mapping ownership and
behavior. Enforce the mmap flag rules and convert affected drivers,
removing ambiguous tests shared by driver mappings and ordinary
memory.
- "support kselftest on nommu platform" (Hajime Tazaki):
Add opt-in mmap and mremap tests for NOMMU systems. Teach the runner
to avoid its fork-dependent timeout command and document the build
and runtime requirements.
- "mm/collapse: separate a collapse from its callers" (Kiryl
Shutsemau):
Separate huge-page collapse from caller policy. Replace khugepaged
identity checks with explicit permissions, and separate scanning from
collapse with the mmap_lock drop in one place.
- "mm: implement and use vma_has_anon_rmap(), silence KCSAN" (Lorenzo
Stoakes):
Provide a common helper for testing whether a VMA has anonymous
reverse mappings. Document the locking rules, annotate intentional
concurrent reads for KCSAN, and convert the core MM callers.
- "mm/sparse: remove SECTION_MARKED_PRESENT and further cleanups"
(David Hildenbrand):
Remove SECTION_MARKED_PRESENT in favor of the early-section flag.
Simplify sparse memory initialization, section metadata handling and
related PFN validity checks.
- "sh: remove NUMA and SPARSEMEM support" (Mike Rapoport):
Remove SuperH NUMA and SPARSEMEM support. The extra memory banks have
been unusable since the NO_BOOTMEM conversion, leaving only the code
and data overhead.
- "mm: Unify device DAX and HugeTLB vmemmap population paths" (Muchun
Song):
Unify the generic device DAX and HugeTLB vmemmap population paths and
shared tail-page handling. Remove the duplicate DAX machinery while
retaining the powerpc radix implementation.
- "kselftest: mm: fix some failure of split_huge_page_test" (Yeoreum
Yun):
Fix huge-page split tests racing with khugepaged recollapse. Use
pagemap and page flags instead of VMA-boundary assumptions which fail
when glibc has already enabled THP.
- "memcg: move memcgid refcount to objcg to unpin dying memcgs"
(Bingfang Guo):
Let dying memcgs be freed while swapped-out shmem still holds their
IDs. Move ID references to reparentable objcgs, with the necessary
swap accounting and RCU lifetime changes.
- "mm/truncate: fix data loss when truncating straddling large folios"
(Zhang Yi):
Preserve out-of-range data when truncating straddling large folios.
Fix a lookup race and split-failure handling, and handle minimum
folio orders for the forthcoming ext4 iomap buffered I/O path.
- "mm/damon: introduce damos quota goal target metric complement flag"
(SJ Park):
Let DAMOS quota goals use a metric's complement, so utilization can
also express free-memory goals without adding separate metrics. Add
the sysfs interface, documentation and tests.
- "arm64/mm: Standardize printing for pgtable entries" (Anshuman
Khandual):
Use the generic formatter for arm64 page-table entries, ready for
future wider entries. Move the safe-PTE-update check out of the
header to avoid an include cycle.
- "mm: kmsan: fix iounmap metadata teardown" (Dima Koziuk):
Fix KMSAN metadata leaks on iounmap and failed ioremap allocations.
Free backing pages before removing their mappings, and share the
teardown with partial-allocation cleanup.
- "mm/damon/core: preserve quota state when constructing schemes" (Karl
Mehltretter):
Preserve running DAMOS quota state while validating parameter
updates. Even rejected updates could reset accounting and exceed the
quota window; add tests for both accepted and rejected updates.
- "mm/vmalloc: fix vmalloc_dump_obj VA lookup" (Ye Liu):
Fix vmalloc_dump_obj() lookups for last-page addresses and
allocations spanning vmap zones. These allocations were being missed
in diagnostic output.
- "mm: various (urgent?) fixes" (David Hildenbrand):
Fix mmap charge leaks and hugetlb use-after-free, wrong-address
unmapping, stuck locks and endless faults. Also preserve reservations
on failed mremap and handle stale userfaultfd bits.
- "maple_tree: fix maple_range_64 crashes in RCU mode" (Dimitris
Charisis):
Fix pointer and metadata confusion during RCU destruction of
maple_range_64 nodes, avoiding double frees and invalid accesses. Add
a regression test; no in-tree user currently uses this combination.
- "mm: fix hugetlb NR_HUGETLB accounting on folio migration" (Hongfu
Li):
Fix hugetlb migration and preallocation accounting so /proc/vmstat
retains the correct count and memory.numa_stat follows the folio to
its new node.
- "mm/damon: fix a minor bug, clean code, and add a test" (SJ Park):
Fix 32-bit DAMON age overflow which made old regions look young.
Share hugetlb folio acquisition code and add a regression test for
the previously fixed last-region skipping bug.
- "mm/damon/core: cleanup code, reduce stack usage, and add kunit" (SJ
Park):
Reduce kdamond stack usage and remove unused enums. Add deterministic
PSI quota-goal tests which do not depend on the machine's current
memory pressure.
- "mm/damon: fix the temporal goal tuner's size quota conversion"
(Donggeun Yoo):
Bound the temporal tuner's quota conversion to prevent overflow and
zero effective quotas, which could stop schemes making progress. Add
boundary tests.
- "Don't use GFP_DMA when calling dma_alloc_coherent" (Baoquan He):
Remove redundant GFP_DMA flags from coherent DMA allocation calls.
The DMA API already selects the zone from device limits and ignores
the caller's flag.
- "kselftest: mm: fix intermittent failure khugepaged test" (Yeoreum
Yun):
Propagate child-process failures in the khugepaged tests and prevent
background collapse from racing their checks. This removes false
passes and intermittent failures.
- "MAINTAINERS: cover all hugetlb headers under HUGETLB SUBSYSTEM"
(Enze Li):
Cover all hugetlb headers in MAINTAINERS so get_maintainer.pl finds
the right maintainers. Also label the closing hugetlb_cgroup.h
include guard.
Summary of the 65 singletons in this merge:
The singletons include a few worthwhile performance fixes. Avoiding
unnecessary reverse-map tree updates gave a reported 15.9% improvement
in UnixBench's execl test on a 192-core x86 system. A reclaim retry
fix stops defrag_mode from spinning under memory.low protection; a
reported file-creation stress test ran 5.7 times faster, with far less
memory pressure. Narrower hugetlb invalidations also spare KVM guests
needless refaults: the reported Redis-in-VM P99 QPS penalty while
punching holes fell from 7.09% to 1.45%.
Several fixes prevent lost data or corrupted memory. Lazyfree
huge-page races with mremap()/munmap() could lose data or access freed
memory, and MADV_COLD could lose a concurrent write's dirty bit.
Another fix prevents stale migration entries from outliving large
folios. On filesystems with blocks smaller than a page, extending a
file by 4 GiB or more could expose bytes written past the old EOF
through mmap instead of zeroes. That is fixed, along with
use-after-free races in PAGEMAP_SCAN during process exit and in
memory-cgroup accounting on PREEMPT_RT kernels.
Other fixes stop UFFDIO_MOVE spinning forever on pinned large folios,
which could wedge other memory operations, and stop mremap() leaking
commit charges until allocations fail under strict overcommit. A
further fix keeps userfaultfd tracking intact on neighboring mappings.
NUMA binding no longer prevents long-term pins when the chosen node
has only movable memory, fixing QEMU/VFIO startup and io_uring buffer
registration failures. MGLRU also gets a fix for early-boot crashes
with memory cgroups disabled at boot, notably in small crash kernels.
* tag 'mm-stable-2026-10-25-19-27' of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm: (213 commits)
mm: multi-gen LRU: fix NULL lruvec->pgdat deref with memcg disabled on cmdline
mm/page_counter: use WRITE_ONCE() for emin/elow when parent is the root
mm/memcontrol: publish null pointer before dropping css reference
test_maple_tree: test maple_range_64 node in RCU mode
maple_tree: fix maple_range_64 crashes in RCU mode
mm/vma: avoid broken uffd merge on mremap
tools/testing/shared: catch a failed posix_memalign()
mm/damon/core: remove unreachable damos charge_{target,addr}_from
mm/shrinker: allocate shrinker_info_unit from a dedicated kmem_cache
riscv: mm: exclude invalid THP PMDs from page table check
mm/damon/core: filter out damos address filter-split failed region
fs/proc/task_mmu: pin mm_users in do_pagemap_scan()
mm/mempolicy: apply bind policies against the scoped allocation zone
mm/vmscan: reclaim every zone in per-node proactive reclaim
mm/mremap: correctly uncharge accountable VMAs in error path
mm/mremap: populate calculated fields in vrm state correctly
MAINTAINERS: add an entry for MEMORY MANAGEMENT - MEMORY MAP
mm: bdi: reset min_ratio after the sysfs attributes are gone
mm/page_vma_mapped: continue past nonmatching huge PMDs
selftests/mm: skip hugetlb cgroup v2 tests without the needed controllers
...
|
||
|
Mani Tofigh
|
815a0a2a40
|
bpf: Fix bpf_timer callback running on a freed map
A bpf_timer in an inner map can fire after the map has been freed, and bpf_timer_cb() then reads the freed map: [ 50.709520] BUG: KASAN: slab-use-after-free in bpf_timer_cb+0x249/0x290 [ 50.709540] Read of size 4 at addr ff11000003bc1030 by task bpftimer/72 ... [ 50.710783] Freed by task 12: ... [ 50.710826] bpf_map_free+0x1a2/0x380 When user space closes the map, its timers are cancelled and their callbacks are cleared. Since commit |
||
|
Eduard Zingerman
|
899cbaae6e
|
bpf: Print 32-bit SCEV bounds as 32-bit numbers
- Print large 32-bit bounds without sign extension. - Omit 32-bit bounds when they coincide with 64-bit bounds. Before: loop header at ..., widening r0 to 0x80000000..0xffffffff (s32 0xffffffff80000000..-1) step 1 loop header at ..., widening r0 to 0..3 (s32 0..3) step 1 After: loop header at ..., widening r0 to 0x80000000..0xffffffff (s32 0x80000000..-1) step 1 loop header at ..., widening r0 to 0..3 step 1 Signed-off-by: Eduard Zingerman <eddyz87@gmail.com> Signed-off-by: Alexei Starovoitov <ast@kernel.org> Link: https://patch.msgid.link/20261026-scev-simpler-clamp-v2-6-f373aec9825f@gmail.com |
||
|
Eduard Zingerman
|
abc9a76a88
|
bpf: (zext32 (zext32 X)) -> (zext32 X) SCEV simplification rule
The compiler can preserve an induction variable across a call by
copying it to a callee-saved register. For example:
w1 = 0;
loop:
w7 = w1;
call foo;
w1 = w7;
w1 += 2;
if w7 < 10 goto loop;
When building expressions for the loop body, each 32-bit move adds a
zero extension:
w7 = w1; r7 = (zext32 r1)
w1 = w7; r1 = (zext32 (zext32 r1))
w1 += 2; r1 = (zext32 (+ (zext32 (zext32 r1)) 2))
The existing ALU32 simplification rules allow recognizing the header
SCEV for r1 as:
(zext32 (linear r1 2))
Substituting this SCEV into r7's expression at the latch produces:
(zext32 (zext32 (linear r1 2)))
Add the simplification rule:
(zext32 (zext32 X)) -> (zext32 X)
This allows recognizing the saved induction variable in r7 as a
32-bit linear SCEV and computing the loop iteration bound.
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261026-scev-simpler-clamp-v2-5-f373aec9825f@gmail.com
|
||
|
Eduard Zingerman
|
86d654957d
|
bpf: Check the iteration count before the predicted exit value
The comment before `total_change` computation:
/*
* compute_backedge_taken_count() guarantees that 'total_change'
* does not overflow in 64-bit domain.
*/
total_change = initial + step * backedge_taken_count;
Is not true before the check:
if (backedge_taken_count >= U32_MAX - 1)
return false;
Move the check before `total_change` computation in order to reconcile
the code with comments. Old position is not a bug the check was
performed either way.
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261026-scev-simpler-clamp-v2-4-f373aec9825f@gmail.com
|
||
|
Eduard Zingerman
|
5dd32c8c10
|
bpf: Avoid false infinite loop reports after changes in SCEV clamping
Clamping now reproduces the widened range at every back edge,
so induction variable ranges no longer vary between iterations.
This can produce identical checkpoint states in terminating programs,
for example:
int main(...) {
for (int i = 0; i < 10; i++) {
foo(...);
for (int j = 0; j < 3; j++)
...;
}
}
void foo(...) {
...;
}
Handle two cases in states.c:bpf_is_state_visited():
- For checkpoints inside foo(), look for active loops in upper frames
as well. Otherwise current_loop() returns NULL and an identical
callee state can trigger an infinite loop report.
- For checkpoints inside the inner loop, skip the infinite loop check
when the current state is inside a terminating loop in any frame.
That loop bounds how many times its body is re-executed, so an
identical state on a later iteration does not imply nontermination.
This covers the inner loop above, re-entered with a fresh entry
state on each outer iteration, and inner loops SCEV cannot analyze.
The nested_terminating_prune test still rejects such an outer loop,
but now reaches the jump complexity limit instead of reporting an
infinite loop. Update its expected message accordingly.
Fixes:
|
||
|
Eduard Zingerman
|
0c33d612bd
|
bpf: Do not intersect with current register range on clamp
scev.c:bpf_clamp_scev_regs() is called each time verifier returns to a
loop header via a backedge. It's purpose is to re-establish computed
variable ranges for registers with linear SCEVs.
It can be defined as:
- For each register R with linear SCEV compute:
- loop_range(R) := entry_range(R) * iterations_count
- range(R) := range(R) ∩ loop_range(R)
Such formulation is necessary to avoid ranges over-approximation in
examples like below:
r1 = 0; SCEVs: r1 = (linear r1 1)
r3 = 0; r3 = (linear r3 1)
loop: r3 += 1;
r1 += 1;
if r1 < 5 goto loop;
r0 = 0;
exit;
- At loop entry r3 is widened to be in a range r3 ∈ {0,1,2,3,4}.
- After increment it becomes r3 ∈ {1,2,3,4,5}.
- On a second iteration the bpf_clamp_scev_regs() forces r3 to become
r3 ∈ {1,2,3,4}. W/o clamping logic r3 range wouldn't shrink
preventing loop from converging in states.c:states_equal().
However, intersection logic might lead to assigning an empty range to
a register, as in example below:
r1 = 0; verification
r3 = 0; path
loop: r3 += 1; | <---------.
if r3 != 5 goto +0; | r3 ∈ {5} |
r1 += 1; | |
if r1 != 5 goto loop; '-----------'
r0 = 0;
exit;
- At header r3 ∈ {0,1,2,3,4}.
- While exploring the branch `if r3 != 5 goto ...` with r3 ∈ {5}.
- On a second iteration:
r3 := {0,1,2,3,4} ∩ {5} = ∅
Which stops verification with the following error:
verifier bug: scev clamp produced empty range for r3
Technically this means that the current verification path is
impossible at runtime and should not be traced further.
This commit mitigates the above by removing the intersection logic,
the function is now defined as:
- For each register R with linear SCEV compute:
- range(R) := entry_range(R) * iterations_count
This is still sound, provided that SCEV expressions loop execution
counts are correct.
Fixes:
|
||
|
Eduard Zingerman
|
c463b378ea
|
bpf: Reject pointer operands for 32-bit recurrences/latches
Avoid inferring loop bounds for the following program shapes:
32-bit latch (A) | 32-bit recurrence (B)
-------------------------+--------------------------
r1 = ... pointer ... | r1 = ... pointer ...
r2 = ... pointer ... | r2 = ... pointer ...
l: r1 += 8 | l: w1 += 8
... | ...
if w1 < w2 goto l | if r1 < r2 goto l
(A) Scalar bounds for pointers represent only offsets from their
common origin values. The 32-bit operations also depend on the
runtime base address: the addresses can wrap at 32 bits without
the tracked offsets wrapping. Thus, offsets alone do not determine
the iteration count.
(B) 32-bit operations on pointers either prohibited (wX += 1)
or convert them to scalars (wX = wX). No need to deal with
this at SCEV level.
Fixes:
|
||
|
Weiming Shi
|
90aedf8854
|
bpf: Validate stack arg reads against caller state
check_stack_arg_read() bounds a stack argument index with the callee's BTF-derived argument count, then uses the index in the caller's stack_arg_regs[] array. The array's actual logical length is recorded in caller->out_stack_arg_cnt, so the bound and indexed object can diverge. A crafted BPF_PROG_LOAD can use an argument mismatch to mark a static subprogram's BTF information unreliable. Later calls return from btf_check_subprog_call() before check_outgoing_stack_args(), while check_func_call() continues the static call on -EINVAL. The callee can then read a slot that the caller never allocated or initialized. The earlier commit |
||
|
Xu Yunxiang
|
d09c2aa310
|
bpf: Track borrowed iterator results by default
An open-coded iterator can return a struct pointer borrowed from its current element. The verifier has not tied such results to iterator state since the task_vma iterator was introduced, so a saved result can still be used after destroy. Commit |
||
|
Lorenzo Stoakes (ARM)
|
0c2c9885c5 |
bpf: arena: mark arena_map_mmap() mappings VM_MIXEDMAP
The bpf_map->ops->map_mmap callback invoked by bpf_map_mmap() can be set to one of ringbuf_map_mmap_kern(), ringbuf_map_mmap_user(), array_map_mmap() or arena_map_mmap(). It is convention in mm to mark mappings whose pages the kernel manages itself with VM_MIXEDMAP, so core mm knows not to treat them as ordinary page cache or anonymous memory. The map_mmap callbacks ringbuf_map_mmap_kern() and ringbuf_map_mmap_user() use remap_vmalloc_range(), which ultimately invokes vm_insert_page() and so marks the ranges VM_MIXEDMAP, and array_map_mmap() sets VM_MIXEDMAP explicitly. However, the exception to this is arena_map_mmap(), which doesn't set the flag. This patch corrects this and updates the comment to reflect it. The pages are refcounted and vm_normal_page() finds them regardless of the flag, and VM_DONTEXPAND remains set (marking the memory as VM_SPECIAL and thus unmergeable). The one effect is that NUMA balancing now skips these VMAs, as it already does for the other bpf map mappings, which is the reason array_map_mmap() gives for setting the flag. The intent of this patch is to be able to establish the invariant that only PFN-mapped or mixed map ranges may clear the VM_MAYWRITE flag, as is done in bpf_map_mmap(). Link: https://lore.kernel.org/20261013-b4-mmap-prepare-vma-flag-sanify-v6-14-1aa3c9b139f9@kernel.org Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Reviewed-by: Emil Tsalapatis <emil@etsalapatis.com> Reviewed-by: Suren Baghdasaryan <surenb@google.com> Acked-by: David Hildenbrand (Arm) <david@kernel.org> Cc: Liam R. Howlett <liam@infradead.org> Cc: Vlastimil Babka <vbabka@kernel.org> Cc: Jann Horn <jannh@google.com> Cc: Pedro Falcato <pfalcato@suse.de> Cc: Mike Rapoport <rppt@kernel.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Jonathan Corbet <corbet@lwn.net> Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org> Cc: Dennis Dalessandro <dennis.dalessandro@cornelisnetworks.com> Cc: Jason Gunthorpe <jgg@ziepe.ca> Cc: Leon Romanovsky <leon@kernel.org> Cc: Paul Moore <paul@paul-moore.com> Cc: Stephen Smalley <stephen.smalley.work@gmail.com> Cc: Jaroslav Kysela <perex@perex.cz> Cc: Takashi Iwai <tiwai@suse.com> Cc: Alexei Starovoitov <ast@kernel.org> Cc: Daniel Borkmann <daniel@iogearbox.net> Cc: Andrii Nakryiko <andrii@kernel.org> Cc: Eduard Zingerman <eddyz87@gmail.com> Cc: Kumar Kartikeya Dwivedi <memxor@gmail.com> Cc: Zi Yan <ziy@nvidia.com> Cc: Baolin Wang <baolin.wang@linux.alibaba.com> Cc: Nico Pache <nico.pache@linux.dev> Cc: Ryan Roberts <ryan.roberts@arm.com> Cc: Dev Jain <dev.jain@arm.com> Cc: Barry Song <baohua@kernel.org> Cc: Lance Yang <lance.yang@linux.dev> Cc: Usama Arif <usama.arif@linux.dev> Cc: Kiryl Shutsemau <kas@kernel.org> Cc: Doug Gilbert <dgilbert@interlog.com> Cc: James Bottomley <james.bottomley@hansenpartnership.com> Cc: Martin K. Petersen <mkp@kernel.org> Cc: Simona Vetter <simona@ffwll.ch> Cc: Helge Deller <deller@gmx.de> Cc: Sebastian Reichel <sre@kernel.org> Cc: John Hubbard <jhubbard@nvidia.com> Cc: Peter Xu <peterx@redhat.com> Cc: Masami Hiramatsu <mhiramat@kernel.org> Cc: Oleg Nesterov <oleg@redhat.com> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@kernel.org> Cc: Ingo Molnar <mingo@redhat.com> Cc: Borislav Petkov <bp@alien8.de> Cc: Dave Hansen <dave.hansen@linux.intel.com> Cc: Arnaldo Carvalho de Melo <acme@kernel.org> Cc: Namhyung Kim <namhyung@kernel.org> Cc: Mark Rutland <mark.rutland@arm.com> Cc: Rik van Riel <riel@surriel.com> Cc: Harry Yoo <harry@kernel.org> Cc: Juri Lelli <juri.lelli@redhat.com> Cc: Vincent Guittot <vincent.guittot@linaro.org> Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com> Cc: Maxime Ripard <mripard@kernel.org> Cc: Thomas Zimmermann <tzimmermann@suse.de> Cc: David Airlie <airlied@gmail.com> Cc: Will Deacon <will@kernel.org> Cc: Aneesh Kumar K.V <aneesh.kumar@kernel.org> Cc: Nicholas Piggin <npiggin@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Muchun Song <muchun.song@linux.dev> Cc: Oscar Salvador <osalvador@suse.de> Cc: Matthew Wilcox (Oracle) <willy@infradead.org> Cc: Jan Kara <jack@suse.cz> Cc: Marc Zyngier <maz@kernel.org> Cc: Oliver Upton <oupton@kernel.org> Cc: Catalin Marinas <catalin.marinas@arm.com> Cc: Madhavan Srinivasan <maddy@linux.ibm.com> Cc: Anup Patel <anup@brainfault.org> Cc: Paul Walmsley <pjw@kernel.org> Cc: Palmer Dabbelt <palmer@dabbelt.com> Cc: Albert Ou <aou@eecs.berkeley.edu> Cc: Christian Borntraeger <borntraeger@linux.ibm.com> Cc: Janosch Frank <frankja@linux.ibm.com> Cc: Claudio Imbrenda <imbrenda@linux.ibm.com> Cc: Alexander Gordeev <agordeev@linux.ibm.com> Cc: Gerald Schaefer <gerald.schaefer@linux.ibm.com> Cc: Heiko Carstens <hca@linux.ibm.com> Cc: Vasily Gorbik <gor@linux.ibm.com> Cc: David S. Miller <davem@davemloft.net> Cc: Andreas Larsson <andreas@gaisler.com> Cc: Al Viro <viro@zeniv.linux.org.uk> Cc: Christian Brauner <brauner@kernel.org> Cc: Matthew Brost <matthew.brost@intel.com> Cc: Joshua Hahn <joshua.hahnjy@gmail.com> Cc: Rakie Kim <rakie.kim@sk.com> Cc: Byungchul Park <byungchul@sk.com> Cc: Gregory Price <gourry@gourry.net> Cc: Ying Huang <ying.huang@linux.alibaba.com> Cc: Alistair Popple <apopple@nvidia.com> Cc: Chris Li <chrisl@kernel.org> Cc: Kairui Song <kasong@tencent.com> Cc: Kemeng Shi <shikemeng@huaweicloud.com> Cc: Nhat Pham <nphamcs@gmail.com> Cc: Baoquan He <baoquan.he@linux.dev> Cc: Youngjun Park <youngjun.park@lge.com> Cc: Johannes Weiner <hannes@cmpxchg.org> Cc: Qi Zheng <qi.zheng@linux.dev> Cc: Shakeel Butt <shakeel.butt@linux.dev> Cc: Axel Rasmussen <axelrasmussen@google.com> Cc: Yuanchu Xie <yuanchu@google.com> Cc: Wei Xu <weixugc@google.com> Cc: Chengming Zhou <chengming.zhou@linux.dev> Cc: Michal Hocko <mhocko@kernel.org> Cc: Miklos Szeredi <miklos@szeredi.hu> Cc: Xu Xin <xu.xin@linux.dev> Cc: Ihor Solodrai <ihor.solodrai@linux.dev> Cc: Bert Karwatzki <spasswolf@web.de> |
||
|
Feng Yang
|
fb2b27d214
|
bpf: Propagate btf_add_type() errors in btf_check_all_metas()
btf_check_all_metas() ignores the return value of btf_add_type(), so a
type that could not be registered is silently dropped while the loop
keeps walking over the type section.
Both failure modes are reachable:
- -E2BIG. Once the types array reaches its capacity (BTF_MAX_TYPE),
every following btf_add_type() fails the same way, so
btf_check_all_metas() still returns 0 and the BTF object is created
with btf->nr_types capped while hdr->type_len claims more records.
- -ENOMEM. btf->nr_types is not incremented and the array is not
grown, so the next type is recorded at the slot of the dropped one
and btf->types[] stays off by one with respect to the type ids for
all remaining types. btf_type_by_id() then returns the wrong type
for those ids for the whole lifetime of the object, and the last
type is silently left unresolved.
Propagate the error so that the object is rejected instead.
Fixes:
|
||
|
Qiliang Yuan
|
377ab5f2ef
|
bpf: Dissociate struct_ops progs before the grace period of the map
A struct_ops prog is associated with its map without a reference on the
map, and finds the struct_ops through bpf_prog_get_assoc_struct_ops()
under RCU. For a struct_ops map, bpf_map_put() waits for an RCU grace
period before it calls bpf_struct_ops_map_free(), which clears the
association and then frees the trampoline and the kdata right away.
Nothing waits for a prog that reads the association after that grace
period started and before the association is cleared. A timer callback
of a struct_ops prog that calls test_1 of bpf_testmod's multi_st_ops
through the association while the map is freed returns into the freed
trampoline, whose image is filled with int3. RAX holds MAP_MAGIC (1234),
which test_1 has just returned:
Oops: int3: 0000 [#1] SMP KASAN NOPTI
CPU: 2 UID: 0 PID: 31 Comm: ksoftirqd/2 ...
RIP: 0010:0xffffffffc043114d
Code: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc <cc> cc cc cc
RAX: 00000000000004d2 ...
Call Trace:
bpf_prog_ec591f4f9d13f364_timer_cb+0x96/0x1e2
bpf_timer_cb+0x15d/0x240
__hrtimer_run_queues+0x264/0x5a0
hrtimer_run_softirq+0x1a7/0x3b0
handle_softirqs+0x18c/0x5b0
Dissociate the progs in map_free_pre_rcu(), so that the grace period of
bpf_map_put() follows it, and make that an RCU tasks trace grace period
for every struct_ops map. It then also waits for the progs that read
the association under RCU tasks trace, e.g. from a bpf_wq callback or a
sleepable prog, as a tasks trace grace period implies an RCU one.
Stop setting free_after_rcu_gp, which bpf_map_put() no longer reads for
a struct_ops map, and update the comment of free_after_mult_rcu_gp in
struct bpf_struct_ops, which now only picks the grace period of a cgroup
prog array.
The st_ops_assoc_in_timer_free subtest of test_progs reproduces it. Its
timer callback keeps calling test_1 through the association while the
map is freed, and test_1 spins in bpf_loop() when called from there:
$ cd tools/testing/selftests/bpf
$ make test_progs
$ ./test_progs -t struct_ops_assoc/st_ops_assoc_in_timer_free
On kernels of the same bpf-next commit with KASAN enabled, x86_64 VM
with 32 vCPUs, before and after this patch:
before after
outcome Oops (int3) on run 1 100 of 100 runs passed, no Oops
Fixes:
|
||
|
Qiliang Yuan
|
232475c329
|
bpf: Protect the struct_ops association with a spinlock
bpf_prog_assoc_struct_ops() and bpf_prog_disassoc_struct_ops() update prog->aux->st_ops_assoc under st_ops_assoc_mutex, and the latter puts the map of a non-struct_ops prog while holding it. A struct_ops map is about to dissociate its progs from map_free_pre_rcu(), which bpf_map_put() calls in whatever context drops the last reference, including softirq, e.g. bpf_struct_ops_put() from tcp_cleanup_congestion_control(). A mutex can't be taken there, and the put of the last reference under the lock of one prog would take the lock of other progs. Replace the mutex with a spinlock taken with interrupts disabled, and put the map after dropping it. Signed-off-by: Qiliang Yuan <odys.yuan@gmail.com> Signed-off-by: Alexei Starovoitov <ast@kernel.org> Link: https://patch.msgid.link/20261022-selftests-bpf-struct-ops-assoc-timer-v6-1-2c0f18e82582@gmail.com |
||
|
Linus Torvalds
|
55fbedcec6 |
bpf-next-7.4
-----BEGIN PGP SIGNATURE-----
iQJRBAABCgA7FiEE+soXsSLHKoYyzcli6rmadz2vbToFAmrY/GYdHGFsZXhlaS5z
dGFyb3ZvaXRvdkBnbWFpbC5jb20ACgkQ6rmadz2vbTrF3w//W1IXtb7BHM5DqyRY
+IUpLlVdJETLJpdxHGq4SMGU7T+JpBJ7e58HIa9bNfyFaeOFuIvH4dy284RHhr8T
3wPR61KmX4hZlmqz17lSKM4E4cnaYW/NMYfp4acA/VRNcvcM+IiGnuhzcFyIMs3w
mPlw/r3rW7Cn7E2Ri2hRV/ZQoT7FyI7LtUEtHU8TUE9kh+T2LTut+FWMJrHe5w0J
LDFwhVqZE0dWZ4695PK6cEIh8uxRJBnmIyMtN+zLBOtqAjiOSNNhFssdSCIkjPEe
X9Xz1Zvqa8Z5QsyRsrwK4S62pnh0pVshQ8fDjrXKY7fkMPpzTvOfqOGS3qhOlw2y
sz853Repug0ZerePFnPnfTIhRIcXagTT11BJUDhUzgy9/Cra3s0vY6c9/MrxTamh
3Xpu/UNl8AIqC/OnbAo8kJ9aKAG6olIBHYIsVOUYk3tH8m2HaVqpKXKRKfqshSDT
YyKoGQE8g9gemS44/RSHKi6QI2jUxNwcOI8FBzViDZaeE476Zvv/fHNC5Tv7lwmm
5qGbcem9sakcd+n17aLy0xKYE2kUSQQLi/J80AY7gMR9VlGsbYPZ8KX8HeSaSLRy
7SMn2dBvNmFaFNgWdLQn4dmBFhvuVgZP6A/ZTreCLuXjBcPo1TnYkVwi3eye4eY/
CwgWBoUJTIqMMt+bRyNgBK0+5rQ=
=kneb
-----END PGP SIGNATURE-----
Merge tag 'bpf-next-7.4' of git://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next
Pull bpf updates from Alexei Starovoitov:
"Major changes:
- Introduce scalar evolution (SCEV) analysis in the verifier.
It infers how registers and stack slots change in a loop and lets
the verifier check bounded loops with induction variables as ranges
instead of walking every iteration. The number of processed insns
drops ten fold in programs with loops (Eduard Zingerman)
- Add 'callx' instruction: indirect call of a static BPF subprogram.
The address is taken with ld_imm64 or loaded from read-only data.
Supported by x86-64, arm64 and riscv64 JITs (Alexei Starovoitov, Pu
Lehui)
- Add unwind_goto and unwind_exit instructions, so that Rust panic or
C++ exception can run destructors of every frame it passes through.
The normal path is free of unwind code. Supported by x86-64 and
arm64 JITs (Alexei Starovoitov)
- Support programs compiled by rust-bpf: treat load and store through
a number as arena access with BPF_F_ARENA_SCALAR, accept rust names
in BTF (Alexei Starovoitov)
- Raise BPF program stack size from 512 bytes to 2 KiB on x86-64 and
arm64 (Kumar Kartikeya Dwivedi)
- Support struct, union and __int128 up to 16 bytes passed and
returned by value in BPF subprograms and kfuncs, and fix x86-64 and
arm64 trampolines for such arguments (Yonghong Song)
- Introduce a common way to attach struct_ops to a cgroup via
bpf_link. bpf_tcp_ops is the first user and mirrors TCP sock_ops
callbacks (Martin KaFai Lau, Amery Hung)
- Describe inlined functions in BTF. New kinds tell where the inline
sites are and how to fetch the arguments there.
CONFIG_DEBUG_INFO_BTF_INLINE keeps it apart from the main BTF in
.BTF.inline section or in btf_vmlinux_inline.ko (Alan Maguire)
- Fix many memory safety, lifetime and locking bugs in BPF core,
verifier, iterators, local storage, dynptr, devmap, cpumap and
seg6local (Josef Bacik, Alexei Starovoitov)
- Diff stat is dominated by selftests and by removal of
lib/test_bpf.c. Its 16k lines of instruction tests were converted
to BPF selftests that BPF CI runs on every architecture it covers
(Daniel Borkmann)
Other features and fixes:
- Add KASAN checks to memory accesses of JITed programs on x86-64
(Alexis Lothoré)
- Complete unification of helper, kfunc and subprog argument checks
in the verifier (Amery Hung)
- Add bpf_copy_from_user_mm[_str]() kfuncs (Anastasios Papagiannis)
- Various features and fixes (Daniel Borkmann):
- Add .bpf keyring for program signature verification, modelled
after dm-verity keyring, and support ML-DSA in signing tools
- Emit shorter and faster code in x86-64 JIT for patterns that
LLVM generates
- Fix various issues in gotox handling
- Let BPF LSM set xattrs at inode creation with bpf_inode_init_xattr()
(David Windsor, Daniel Borkmann)
- Use sleepable page allocation in sleepable arena paths. Fix races
between arena free, allocation and user page fault (Emil Tsalapatis)
- Switch BPF crypto from lskcipher to AES-CBC and AES-ECB libraries
(Eric Biggers)
- Add bpf_proactive_reclaim() kfunc (Hui Zhu)
- Render decl_tags in libbpf btf_dump and add tests for bpftool C
dump (Ihor Solodrai)
- Speed up tracing_multi attach on x86 (Jiri Olsa)
- Various features and fixes (Kumar Kartikeya Dwivedi):
- Add BPF_PROG_STREAM_OPEN command to read BPF streams through a
file descriptor with poll support
- Let kernel structs publish arena pointers with __arena type tag
- Fix __uninit kfunc output buffers and fastcall rewrite with
indirect stack accesses
- Refuse CO-RE relocations for signed programs
- Add enqueue_rcvq and dequeue_rcvq hooks to bpf_tcp_ops to adjust
sk_rcvlowat and avoid unnecessary wakeups (Kuniyuki Iwashima)
- Add bpf_ksock_sendto() kfunc and allow ksock connect and send from
bpf_wq (Mahe Tardy)
- Fix racy plt target update in arm64 bpf_arch_text_poke()
(Matthew Wood)
- Restructure BPF selftests Makefile as a layered build
(Mykola Lysenko)
- Speed up hash map lookups for u32 and u64 keys (Mykyta Yatsenko)
- Add signed div/mod, movsx, bswap and gotol to MIPS JITs, signed
div/mod to parisc JITs (Nicholas Dudar, Johan Almbladh)
- Fix stack overflow in build ID parser on 32-bit (Nickolai
Zeldovich)
- Add stack arguments, signed arena loads and gotox to riscv64 JIT.
Pu Lehui is now co-maintainer of riscv BPF JITs (Pu Lehui, Chen
Pei, Feng Jiang)
- Add bpf_call_rcu() and bpf_call_rcu_tasks_trace() kfuncs. Support
fetching and/or/xor atomics in arena on x86-64 (Puranjay Mohan)
- Fix flaky selftests, selftests with clang 23 and bpf-gcc
(Qiliang Yuan, KaFai Wan, Vineet Gupta)
- Fix builds broken by fatal resolve_btfids warnings, BTF generation
on ARC, and define arch_cmpxchg_local() on arc, hexagon and
openrisc (Sasha Levin)
- Reduce memory use of hash maps by up to 50% per element
(T.J. Mercier)
- Mark pointer fields of sched_ext ops arguments as trusted (Tejun
Heo)
- Remove legacy bpf_asm, bpf_dbg and bpf_jit_disasm tools (Xia Yan)
- Fix index overflow in queue and stack maps, arena range tree after
allocation failure, tail call loop after freplace detach, lost
signal in bpf_send_signal() (Yuan Chen)
- Many more fixes and cleanups"
* tag 'bpf-next-7.4' of git://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next: (660 commits)
selftests/bpf: Time tc_edt from its first packet with data
selftests/bpf: Test unwind_exit of an exception callback
selftests/bpf: Tests for 32-bit scalar evolution bounds computation
bpf: Reject unwind_goto and unwind_exit in offloaded progs
bpf: Support 32-bit linear scalar evolution
bpf: Make unwind_exit of an exception callback exit in JITs
selftests/bpf: Test forbidden bpf_ksock_sendto() LSM attach
selftests/bpf: Test ksock sendto()
bpf: Add bpf_ksock_sendto() kfunc
selftests/bpf: Test unwinding from R0: R2 subprog
selftests/bpf: Add a C++ prog with exceptions
selftests/bpf: Add tests for unwind_goto and unwind_exit
bpf, arm64: Support unwind_goto and unwind_exit
bpf, x86: Support unwind_goto and unwind_exit
bpf: Add unwind_goto and unwind_exit insns
tools/bpf: Remove legacy BPF tools
bpf: Annotate data races on bpf_lru_node type
selftests/bpf: Add arena allocation race tests
bpf: Fix arena race between page free and alloc leading to incoherency
bpf: Revalidate faulted arena pages before installing user mappings
...
|
||
|
Linus Torvalds
|
c5fd5e67c7 |
mm.git review status for linus..mm-stable
Everything:
Total patches: 539
Fixes: tags: 101
cc:stable patches: 67
Reviews/patch: 1.62
Reviewed rate: 76%
Excluding DAMON:
Total patches: 407
Fixes: tags: 84
cc:stable patches: 44
Reviews/patch: 2.00
Reviewed rate: 87%
Excluding DAMON and selftests:
Total patches: 360
Fixes: tags: 78
cc:stable patches: 44
Reviews/patch: 2.02
Reviewed rate: 89%
Summary of patch series in this merge, ordered by descending size:
- "selftests/mm: improve khugepaged coverage" (Kiryl Shutsemau (Meta)):
Give khugepaged a proper workout. Extend the tests to all supported mTHP
orders, check exactly which windows collapsed, and make scan completion
deterministic. Race collapse against faults, pinning, fork and remap;
verify that the contents survive.
Add coverage for whole-page-table zaps and memory pressure too.
- "selftests/mm: separate GUP microbenchmarking from functional testing"
(Sarthak Sharma):
Untangle GUP benchmarking from correctness testing. Move the benchmark
into tools/mm, add a standalone harness-based GUP selftest, and move shared
helpers into tools/lib/mm. Give each tool one job instead of mixing timing
measurements with correctness checks.
Cover the mapping, pinning and batch-size combinations explicitly.
- "mm: Switch device DAX to section-based vmemmap optimization" (Muchun Song):
Apply section-based vmemmap optimization to device DAX. Rework the backing
metadata layout to save memory while preserving the handling of device
memory hotplug and its different alignment and lifetime requirements.
- "mm: Introduce section-based vmemmap optimization for HugeTLB" (Muchun
Song):
Cut the memory consumed by HugeTLB's struct page array with section-based
vmemmap optimization. Arrange the metadata so large huge pages need fewer
backing pages, while keeping initialization, teardown and the various
hugepage configurations straight.
- "mm/huge_memory: clean up and decouple the anon and file split helpers"
(Kairui Song):
Separate anonymous and file-backed folio splitting, which have rather
different locking and bookkeeping requirements. Move the work into
separate helpers, drop dead checks and tidy swap-cache handling. Fix error
paths, including one that could leave a folio frozen.
Keep the swap cache cases separate from file-cache splitting.
- "mm: Unconditional per-VMA locks and cleanups" (Dave Hansen):
Make per-VMA locking unconditional and clean up the old conditional code
around it. This removes configuration-dependent plumbing and lets callers
depend on one locking model rather than carrying both cases along.
- "memcg: remove the v1 soft limit" (Shakeel Butt):
Remove the old cgroup v1 memory soft-limit machinery. This has long since
been superseded by the newer memory-control interfaces; hauling around a
separate reclaim scheme for it no longer buys us much.
- "read proc/pid/smaps_rollup under per-vma lock" (Suren Baghdasaryan):
Read /proc/PID/smaps_rollup using per-VMA locks where possible, rather than
holding mmap_lock across the entire walk. Retain the contention fallback
and add tearing tests. This considerably reduces latency when another
thread is busy changing its mappings.
- "mm/damon: introduce data access-as-a-data attribute" (SJ Park):
Teach DAMON to expose memory access observations as data attributes, making
them available to the probe and filtering machinery. Extend the
interfaces, documentation and tests so the new information can be used
rather than merely collected.
- "Remove PG_private by using page/folio->private checks instead" (Zi Yan):
Get rid of PG_private and use the folio's private pointer to say whether
private data is attached. Convert the filesystem users, including erofs,
f2fs and md, and clean up the old accessors and flags. There are a few
special cases, notably swapcache and hugetlb, which need explicit care.
- "mm: make userland page table freeing RCU-safe" (Lorenzo Stoakes (ARM)):
Make freeing userspace page tables safe against RCU readers. Sort out the
page-table lifetime and locking rules so walkers cannot stumble over freed
tables while mappings are being torn down.
- "make unused huge shrinker memcg aware" (Qi Zheng):
Make shmem's unused-huge-folio shrinker aware of memory cgroups, so reclaim
can find those folios under the right memcg. Wire up the per-memcg
tracking and its lifetime rules, and give the shrinklist its own lockdep
class to avoid spurious dependency reports.
- "mm: Drop pxd_ERROR()" (Anshuman Khandual):
Remove the old pxd_ERROR() page-table checks and their
architecture-specific baggage. Simplify the common page-table interfaces
and callers now that these checks no longer serve a useful purpose.
- "memcg: group struct fields by access pattern" (Shakeel Butt):
Rearrange memcg's hot and cold fields according to how they're actually
accessed. Keep frequently updated counters and their neighboring state
from fighting over cachelines and reduce needless cacheline traffic.
- "mm/damon/core: fix the size charged for a filter-trimmed region" (Donggeun
Yoo):
Fix DAMOS quota accounting when a filter trims a monitoring region. Charge
only the part left after filtering, not the original region, and add KUnit
coverage. Previously the excluded memory could eat into the quota and make
schemes run more slowly than requested.
- "mm/damon: add kunit and selftests for probes and probe weights" (SJ Park):
Add more tests for DAMON probes and their weights, including validation and
the results reported to userspace. This is mostly about catching
regressions in the newer probe infrastructure before they become harder to
diagnose.
- "mm: memcontrol: constify the read side of the read side of the memcg API"
(Tal Zussman):
Mark the read-only parts of the memcg interface const, from folio and
lruvec accessors to counters and swap helpers. Most of this is mechanical,
but it makes the contracts clearer and lets the compiler catch accidental
writes.
- "mm/damon/vaddr: support {prep,apply}_probes" (SJ Park):
Support the new prepare/apply probe operations in DAMON's virtual-address
backend. This brings vaddr monitoring into line with the probe
infrastructure and makes the added probe types useful there too.
- "mm/mglru: clean up folio counters and flag usage" (Kairui Song):
Simplify MGLRU's folio counters and flag handling. Remove redundant
accounting paths and make the state transitions easier to follow without
changing the intended reclaim policy.
- "arch, mm/execmem: resolve confusion about set_direct_map_valid_noflush()"
(Mike Rapoport (Microsoft)):
Clarify and straighten out the set_direct_map_valid_noflush() interface
across architectures and execmem. Make the requested mapping changes and
the callers' expectations agree, rather than relying on subtly different
architecture behavior.
- "mm: zswap: free cold writeback folios promptly" (Alexandre Ghiti):
Free zswap writeback folios as soon as their disk write completes. They
are cold temporary buffers, so putting them on the LRU only makes reclaim
rediscover them later. Extend swap-cache dropbehind and let zswap keep
these folios off the LRU altogether.
- "mm: gup: cleanup gup_fast call chain" (Aristeu Rozanski):
Clean up the GUP-fast call chain and simplify how the fast-path helpers
pass their arguments and results around. Keep the page-pin behavior the
same while making the call sequence easier to maintain.
- "mm: optimize zone-device memmap initialization" (Li Zhe):
Speed up initialization of zone-device struct pages. Avoid redundant work
when setting up large device-memory ranges, where touching every descriptor
can get expensive.
- "mm/zswap: shrink zswap_entry via a pool id" (Jianyue Wu):
Reduce the size of zswap_entry by storing a compact pool identifier instead
of a full pointer. Keep the pool lookup and lifetime handling correct
while saving per-entry metadata.
- "mm/damon: support access monitoring of hugetlb-backed memory" (Krishna
Iyer):
Let DAMON monitor hugetlb-backed memory too. Extend the relevant access
checks and add coverage so those mappings no longer fall through a hole in
the monitoring machinery.
- "mm/execmem: fixes and cleanups for the ROX cache" (Mike Rapoport
(Microsoft)):
Tidy execmem's read-only-executable cache and fix its handling of the
awkward allocation and release cases. The aim is to make the cache's
ownership and permission transitions more predictable.
- "zsmallc: remove old object read API" (Sergey Senozhatsky):
Drop zsmalloc's old object-read API and move its remaining users to the
supported access mechanism. This removes a duplicate interface which we no
longer need.
- "mm/mempolicy: stop copying state in the interleave paths" (Gregory Price):
Stop copying mempolicy state in the interleave paths when a reference to
the existing policy will do. This avoids extra work and makes the
ownership rules less obscure.
- "mm/mglru: speed up inc_min_seq() and fix cold/hot inversions" (Barry Song
(Xiaomi)):
Make MGLRU's generation advancement cheaper and fix cases where cold and
hot folios end up in the wrong order. Adjust the counter and list handling
so reclaim doesn't keep passing over the better eviction candidates.
- "mm: Fix MF_DELAYED handling on memory failure" (Lisa Wang):
Treat MF_DELAYED as a partially completed memory-failure operation, not an
outright failure. Propagate that result correctly through shmem and the
generic handler, fix the extra-reference accounting, and add tests for the
resulting SIGBUS behavior.
- "tools/mm/page_owner_sort: fix --sort, add module filter, improve usage" (Ye
Liu):
Fix page_owner_sort's --sort handling, add module-based filtering and make
the usage output more helpful. This makes the tool less frustrating when
tracking down who is holding on to pages.
- "mm/damon: move damos filter range arguments validation to core" (SJ Park):
Move DAMOS filter range validation into the core, where all callers can
share it. Also close the sysfs read-twice races which could pass
inconsistent ranges, and add tests for invalid input.
- "mm/damon: misc cleanups" (Zenghui Yu (Huawei)):
Clean up a collection of small DAMON rough edges in the core, interfaces
and tests. No large behavior change here, just simpler code and fewer
opportunities for confusion.
- "mm/damon: cleanup code, add test cases, and update guidances in docs" (SJ
Park):
Clean up DAMON's core and interfaces, add missing test cases and bring the
documentation into line with the code. These changes should make the newer
facilities easier to understand and maintain.
- "mm/damon: introduce hugepage_size probe filter" (SJ Park):
Add a DAMON probe filter for hugepage size. Let users select a range of
folio sizes through sysfs, validate the range and document the new
interface, so monitoring can concentrate on memory backed by the desired
hugepage sizes.
- "selftests/mm: pagemap_ioctl test fixes and cleanups" (Zenghui Yu (Huawei)):
Fix some pagemap ioctl test assumptions and clean up their setup and
reporting. Make failures point to the operation that went wrong rather
than to test harness bookkeeping.
- "mm: avoid large folio splits when swap is unavailable" (Johannes Weiner):
Avoid splitting large folios merely to attempt swap when no swap space is
available. There is little point paying for an expensive split when
reclaim cannot actually swap the resulting pages out.
- "mm: workingset: fix the shadow node budget under MGLRU" (Hui Zhu):
Fix the workingset shadow-node budget with MGLRU enabled. Keep the shadow
entries within the intended limits so refault tracking does not quietly
consume more metadata than it should.
- "selftests/mm: TAP output and global-state fixes" (Song Hu):
Fix selftests which produce misleading TAP output or fail to restore global
settings after a run. A test ought to leave the system as it found it,
particularly when a whole suite follows behind.
- "mm: remove page_swap_entry()" (Tal Zussman):
Remove page_swap_entry() and convert its remaining users to the folio and
swap-entry helpers. Lose an old page-level interface and make the swap
paths a little more consistent.
- "mm: refactor zonelist constructors and iterators" (Gregory Price):
Refactor the zonelist construction and walk helpers to reduce duplicated
logic. Keep the allocation ordering rules intact while making it clearer
how zones are selected.
- "mm: kmemleak: batch the struct page scan" (Breno Leitao):
Scan adjacent struct pages in batches instead of taking kmemleak_lock once
for every single page. This preserves scan coverage and the lock-hold
bound, while cutting overhead substantially on debug kernels.
- "mm/damon: misc improvements in tests and documents" (Zenghui Yu (Huawei)):
Fix some DAMON selftests which leave kdamond running or fail under Python
safe-path mode, and improve the KUnit coverage and documentation. Nothing
especially glamorous, but these are worth doing.
- "mm/damon: introduce probe_hits_wsum DAMOS core filter" (SJ Park):
Add a DAMOS core filter using weighted probe-hit counts. This gives
schemes another way to select regions based on what the monitoring probes
observed.
- "mm/mglru: fix ineffective memory protection for non-kswapd reclaim" (Ridong
Chen):
Fix MGLRU's memory protection handling for reclaim outside kswapd. The
protection settings weren't having the intended effect in these paths,
leaving protected memory more exposed than requested.
- "mm/damon: improve readability, clarity and test coverage" (SJ Park):
Simplify a handful of DAMON core helpers, clarify their behavior and expand
test coverage. There are no intended userspace-visible changes in this
batch.
- "mm: replace PF_KCOMPACTD/PF_KSWAPD with kthread_func()" (Kefeng Wang):
Stop identifying reclaim and compaction workers through the PF_KSWAPD and
PF_KCOMPACTD task flags. Use their kthread functions instead and remove
the redundant flag plumbing.
- "mm/hugetlb: fix surplus accounting and availability checks during demotion"
(Longlong Xia):
Fix HugeTLB surplus accounting and availability checks when demoting huge
pages. Keep the counters and reservation decisions consistent as one huge
page turns into several smaller ones.
- "mm/damon: add kunit tests for probe_hits handling and probe params
validation" (Jason Angelov):
Add KUnit tests for DAMON probe-hit handling and invalid probe parameters.
These cover edge cases that were too easy to miss in ordinary monitoring
tests.
- "mm/damon: fix misc bugs in kunit, quota goals and sysfs refresh_ms" (SJ
Park):
Fix a few DAMON bugs in KUnit tests, quota-goal handling and the sysfs
refresh interval. Small fixes, but the incorrect results and control
behavior were real.
- "mm: zswap: optimize zswap invalidate and store" (Kefeng Wang):
Simplify zswap's invalidate and store paths to avoid avoidable lookups and
work. Keep the entry lifetime rules intact while making those frequent
operations cheaper.
- "mm: replace custom ratelimiting logic" (Pedro Falcato):
Replace hand-rolled MM rate limiting with the common kernel helpers. There
is no need to maintain another local version of the same mechanism.
- "mm/vmalloc: minor cleanups" (Ye Liu):
Tidy some vmalloc initialization and reporting code, and pull duplicated
vmap-area insertion work into a helper. No functional changes are
intended.
- "mm/mglru: clean up isolate_folios for readability and clarity" (Ridong
Chen):
Clean up MGLRU's isolate_folios() so the decisions are easier to read and
verify. Keep reclaim behavior unchanged while removing unnecessary twists
in the control flow.
- "Catch automatic storage in IDA and Maple Tree definitions" (Yury Norov):
Catch IDA and maple-tree definitions placed in automatic storage where the
lifetime rules are wrong or easy to get wrong. Add checks so such mistakes
are harder to slip through unnoticed.
- "mm, swap: some random fixes and cleanups" (Kemeng Shi):
Fix and simplify a few unrelated swap paths, removing redundant work and
tidying up error handling. A small collection of things which were better
fixed than left lying around.
- "mm/page_io: folio conversion cleanups" (Tal Zussman):
Finish some page-to-folio conversions in swap I/O, removing obsolete
wrappers and using folio operations consistently. The I/O behavior is
intended to remain the same.
- "mm/swap: skip empty clusters in the swapoff scan" (Youngjun Park):
Skip empty swap clusters during swapoff instead of walking slots which have
nothing to release. This avoids pointless work on sparsely used swap
devices.
- "mm: trace: decode architecture-specific VMA flags" (Meijing Zhao):
Teach the VMA trace output to decode architecture-specific flags, rather
than leaving them as unexplained bits. That should make tracing a good
deal more useful when debugging unusual mappings.
- "mm, swap: don't spin on a bad swap entry" (Breno Leitao):
Stop repeatedly retrying a swap entry which is already known to be bad.
Make the error path terminate cleanly instead of wasting CPU chasing an
entry that cannot be used.
- "mm/page_isolation: fix UBSAN shift-out-of-bounds in
isolate_single_pageblock" (Qi Xi):
Fix an out-of-range shift in isolate_single_pageblock() and tighten its
associated boundary handling. UBSAN found the issue; the result should be
more robust pageblock isolation.
- "mm: khugepaged: fix tracepoint UAF" (Vernon Yang):
Fix a use-after-free in khugepaged tracepoint handling. Make sure the
information recorded by tracing stays valid for as long as the trace event
needs it.
- "mm/damon: fix DAMOS bugs in core, paddr and vaddr" (SJ Park):
Fix several DAMOS problems across the core and physical- and
virtual-address backends. The changes close error cases and bring the
implementations back into agreement.
- "mm/hugetlb_cgroup: move the per-node usage along with the folio" (Hongfu
Li):
Move HugeTLB per-node usage accounting when folios migrate between nodes or
are reparented to another cgroup. Otherwise numa_stat can retain stale
charges and even underflow when the folio is freed.
- "docs/core-api: memory-allocation: add k[mz]alloc_obj() and clarify kmalloc"
(Mike Rapoport (Microsoft)):
Document the typed k[mz]alloc_obj() allocation helpers and clear up some
kmalloc guidance. Make it easier for callers to choose the right
interface.
- "selftests/mm: Validate selections and scope memfd_secret setup" (Tianyi
Chen):
Check selftest selections before running and confine memfd_secret setup to
the cases that actually need it. Avoid confusing failures caused by the
test harness itself.
- "mm/damon: move zero damos quota target_value handling to the core layer"
(SJ Park):
Handle a zero DAMOS quota goal target in the core rather than making each
user of the interface invent its own rule. Keep quota decisions consistent
across callers.
- "mm/damon: Introduce a huge page collapsing mechanism using auto tuning"
(Asier Gutierrez):
Add an auto-tuned DAMON mechanism for collapsing huge pages. Use
monitoring feedback to decide when collapse is worthwhile rather than
forcing one fixed policy onto every workload.
- "kmemleak: fix stale documentation and raise the verbose default" (Breno
Leitao):
Bring the kmemleak documentation up to date and require a third confirming
scan before verbose kernels print a suspected leak. The extra scan helps
suppress transient false positives in noisy debug builds.
- "mm/memory: improve insert_pages() error handling" (Avi Weiss):
Tighten insert_pages() error handling so partial failures are reported and
cleaned up consistently. The caller should not have to guess how far the
operation got.
- "mm/damon: allow NULL or unstarted damon_ctx parameter for damon_call()" (SJ
Park):
Make damon_call() handle NULL and not-yet-started contexts safely.
Simplify the rules for callers and avoid an indefinite wait on an invalid
state.
- "mm/damon: introduce pgidle_set probe filter type" (SJ Park):
Add a pgidle_set DAMON probe filter, giving monitoring users another way to
select pages by their idle-tracking state. Wire it through the filter
interfaces and tests.
- "mm/damon/core: remove unused helper functions" (Cheng-Han Wu):
Remove DAMON core helpers which no longer have callers. Keep the API
surface smaller rather than retaining dead code just in case.
- "mm: stop calling pmd_folio() on special PMDs" (Gregory Price):
Avoid calling pmd_folio() on special PMD mappings that have no struct page.
The old assumption could fault in procfs walkers; use the normal-folio
checks and handle those mappings properly.
- "mm: remove three unused helpers from mm.h" (Tal Zussman):
Remove three unused helpers from mm.h. Fewer obsolete interfaces for the
next person to stumble over.
- "mm/damon: improvements in efficiency, error handling, documents" (Liew Rui
Yan):
Skip needless DAMOS quota work once the quota is full, propagate a missing
sysfs error and tidy a couple of comments. The ignored error could make a
monitoring-start request appear successful when it wasn't.
- "docs/mm/damon/design: add explanation of nr_snapshots" (Liew Rui Yan):
Explain DAMON's nr_snapshots accounting in the design documentation and
straighten out the associated terminology. Make it clearer what users
should expect from that counter.
- "selftests/cgroup: fixes for test_zswap on single core VM" (Wilson Felipe
Pereira):
Fix two test_zswap failures on single-CPU VMs. Wait for the exiting
child's cgroup to become unpopulated before reconfiguring it, and avoid
another test assumption which breaks under constrained scheduling.
- "docs/ksm: fix advisor documentation and comment" (Jinjiang Tu):
Correct the KSM advisor documentation and a misleading code comment. Keep
the description aligned with how the advisor actually works.
- "Fix bugs in HugeTLB allocation when mem_cgroup_charge_hugetlb() fails"
(Ackerley Tng):
Fix HugeTLB allocation cleanup when the memcg charge fails. Make sure
failure exits undo what was allocated rather than leaving stale accounting
or resources behind.
- "mm, memcg: fix memory.peak reset clobbering other fds' watermark" (Ridong
Chen):
Fix memory.peak reset handling so resetting one file descriptor doesn't
overwrite the watermark kept for another. Keep those per-open measurements
independent as intended.
- "mm: reject zone device folios in more folio walkers" (Gregory Price):
Reject zone-device folios in page walkers that cannot safely handle them.
Extend the existing checks rather than assuming every folio has ordinary
system-memory backing.
- "percpu: remove code with no effect" (Sang-Heon Jeon):
Remove a few pieces of percpu code which no longer do anything. There is
no reason to keep maintaining dead paths.
-----BEGIN PGP SIGNATURE-----
iHUEABYKAB0WIQTTMBEPP41GrTpTJgfdBJ7gKXxAjgUCatbvywAKCRDdBJ7gKXxA
jrEfAP91yviLKCh8AEazeeBo4D65ntp0FmZB9qqCj46+YuEkowD9HPAyTpD+PjWS
yp/lLvvS4RIg6chID4S2U/pLGzlypg4=
=RIh/
-----END PGP SIGNATURE-----
Merge tag 'mm-stable-2026-10-19-21-33' of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm
Pull MM updates from Andrew Morton:
- "selftests/mm: improve khugepaged coverage" (Kiryl Shutsemau):
Give khugepaged a proper workout. Extend the tests to all supported
mTHP orders, check exactly which windows collapsed, and make scan
completion deterministic. Race collapse against faults, pinning, fork
and remap; verify that the contents survive.
Add coverage for whole-page-table zaps and memory pressure too.
- "selftests/mm: separate GUP microbenchmarking from functional testing"
(Sarthak Sharma):
Untangle GUP benchmarking from correctness testing. Move the
benchmark into tools/mm, add a standalone harness-based GUP selftest,
and move shared helpers into tools/lib/mm. Give each tool one job
instead of mixing timing measurements with correctness checks.
Cover the mapping, pinning and batch-size combinations explicitly.
- "mm: Switch device DAX to section-based vmemmap optimization" (Muchun
Song):
Apply section-based vmemmap optimization to device DAX. Rework the
backing metadata layout to save memory while preserving the handling
of device memory hotplug and its different alignment and lifetime
requirements.
- "mm: Introduce section-based vmemmap optimization for HugeTLB"
(Muchun Song):
Cut the memory consumed by HugeTLB's struct page array with
section-based vmemmap optimization. Arrange the metadata so large
huge pages need fewer backing pages, while keeping initialization,
teardown and the various hugepage configurations straight.
- "mm/huge_memory: clean up and decouple the anon and file split
helpers" (Kairui Song):
Separate anonymous and file-backed folio splitting, which have rather
different locking and bookkeeping requirements. Move the work into
separate helpers, drop dead checks and tidy swap-cache handling. Fix
error paths, including one that could leave a folio frozen.
Keep the swap cache cases separate from file-cache splitting.
- "mm: Unconditional per-VMA locks and cleanups" (Dave Hansen):
Make per-VMA locking unconditional and clean up the old conditional
code around it. This removes configuration-dependent plumbing and
lets callers depend on one locking model rather than carrying both
cases along.
- "memcg: remove the v1 soft limit" (Shakeel Butt):
Remove the old cgroup v1 memory soft-limit machinery. This has long
since been superseded by the newer memory-control interfaces; hauling
around a separate reclaim scheme for it no longer buys us much.
- "read proc/pid/smaps_rollup under per-vma lock" (Suren Baghdasaryan):
Read /proc/PID/smaps_rollup using per-VMA locks where possible,
rather than holding mmap_lock across the entire walk. Retain the
contention fallback and add tearing tests. This considerably reduces
latency when another thread is busy changing its mappings.
- "mm/damon: introduce data access-as-a-data attribute" (SJ Park):
Teach DAMON to expose memory access observations as data attributes,
making them available to the probe and filtering machinery. Extend
the interfaces, documentation and tests so the new information can be
used rather than merely collected.
- "Remove PG_private by using page/folio->private checks instead" (Zi
Yan):
Get rid of PG_private and use the folio's private pointer to say
whether private data is attached. Convert the filesystem users,
including erofs, f2fs and md, and clean up the old accessors and
flags. There are a few special cases, notably swapcache and hugetlb,
which need explicit care.
- "mm: make userland page table freeing RCU-safe" (Lorenzo Stoakes):
Make freeing userspace page tables safe against RCU readers. Sort out
the page-table lifetime and locking rules so walkers cannot stumble
over freed tables while mappings are being torn down.
- "make unused huge shrinker memcg aware" (Qi Zheng):
Make shmem's unused-huge-folio shrinker aware of memory cgroups, so
reclaim can find those folios under the right memcg. Wire up the
per-memcg tracking and its lifetime rules, and give the shrinklist
its own lockdep class to avoid spurious dependency reports.
- "mm: Drop pxd_ERROR()" (Anshuman Khandual):
Remove the old pxd_ERROR() page-table checks and their
architecture-specific baggage. Simplify the common page-table
interfaces and callers now that these checks no longer serve a
useful purpose.
- "memcg: group struct fields by access pattern" (Shakeel Butt):
Rearrange memcg's hot and cold fields according to how they're
actually accessed. Keep frequently updated counters and their
neighboring state from fighting over cachelines and reduce needless
cacheline traffic.
- "mm/damon/core: fix the size charged for a filter-trimmed region"
(Donggeun Yoo):
Fix DAMOS quota accounting when a filter trims a monitoring region.
Charge only the part left after filtering, not the original region,
and add KUnit coverage. Previously the excluded memory could eat into
the quota and make schemes run more slowly than requested.
- "mm/damon: add kunit and selftests for probes and probe weights" (SJ
Park):
Add more tests for DAMON probes and their weights, including
validation and the results reported to userspace. This is mostly
about catching regressions in the newer probe infrastructure before
they become harder to diagnose.
- "mm: memcontrol: constify the read side of the read side of the memcg
API" (Tal Zussman):
Mark the read-only parts of the memcg interface const, from folio and
lruvec accessors to counters and swap helpers. Most of this is
mechanical, but it makes the contracts clearer and lets the compiler
catch accidental writes.
- "mm/damon/vaddr: support {prep,apply}_probes" (SJ Park):
Support the new prepare/apply probe operations in DAMON's
virtual-address backend. This brings vaddr monitoring into line with
the probe infrastructure and makes the added probe types useful there
too.
- "mm/mglru: clean up folio counters and flag usage" (Kairui Song):
Simplify MGLRU's folio counters and flag handling. Remove redundant
accounting paths and make the state transitions easier to follow
without changing the intended reclaim policy.
- "arch, mm/execmem: resolve confusion about set_direct_map_valid_noflush()"
(Mike Rapoport (Microsoft)):
Clarify and straighten out the set_direct_map_valid_noflush()
interface across architectures and execmem. Make the requested
mapping changes and the callers' expectations agree, rather than
relying on subtly different architecture behavior.
- "mm: zswap: free cold writeback folios promptly" (Alexandre Ghiti):
Free zswap writeback folios as soon as their disk write completes.
They are cold temporary buffers, so putting them on the LRU only
makes reclaim rediscover them later. Extend swap-cache dropbehind and
let zswap keep these folios off the LRU altogether.
- "mm: gup: cleanup gup_fast call chain" (Aristeu Rozanski):
Clean up the GUP-fast call chain and simplify how the fast-path
helpers pass their arguments and results around. Keep the page-pin
behavior the same while making the call sequence easier to maintain.
- "mm: optimize zone-device memmap initialization" (Li Zhe):
Speed up initialization of zone-device struct pages. Avoid redundant
work when setting up large device-memory ranges, where touching every
descriptor can get expensive.
- "mm/zswap: shrink zswap_entry via a pool id" (Jianyue Wu):
Reduce the size of zswap_entry by storing a compact pool identifier
instead of a full pointer. Keep the pool lookup and lifetime handling
correct while saving per-entry metadata.
- "mm/damon: support access monitoring of hugetlb-backed memory"
(Krishna Iyer):
Let DAMON monitor hugetlb-backed memory too. Extend the relevant
access checks and add coverage so those mappings no longer fall
through a hole in the monitoring machinery.
- "mm/execmem: fixes and cleanups for the ROX cache" (Mike Rapoport):
Tidy execmem's read-only-executable cache and fix its handling of the
awkward allocation and release cases. The aim is to make the cache's
ownership and permission transitions more predictable.
- "zsmallc: remove old object read API" (Sergey Senozhatsky):
Drop zsmalloc's old object-read API and move its remaining users to
the supported access mechanism. This removes a duplicate interface
which we no longer need.
- "mm/mempolicy: stop copying state in the interleave paths" (Gregory
Price):
Stop copying mempolicy state in the interleave paths when a reference
to the existing policy will do. This avoids extra work and makes the
ownership rules less obscure.
- "mm/mglru: speed up inc_min_seq() and fix cold/hot inversions" (Barry
Song):
Make MGLRU's generation advancement cheaper and fix cases where cold
and hot folios end up in the wrong order. Adjust the counter and list
handling so reclaim doesn't keep passing over the better eviction
candidates.
- "mm: Fix MF_DELAYED handling on memory failure" (Lisa Wang):
Treat MF_DELAYED as a partially completed memory-failure operation,
not an outright failure. Propagate that result correctly through
shmem and the generic handler, fix the extra-reference accounting,
and add tests for the resulting SIGBUS behavior.
- "tools/mm/page_owner_sort: fix --sort, add module filter, improve
usage" (Ye Liu):
Fix page_owner_sort's --sort handling, add module-based filtering and
make the usage output more helpful. This makes the tool less
frustrating when tracking down who is holding on to pages.
- "mm/damon: move damos filter range arguments validation to core" (SJ
Park):
Move DAMOS filter range validation into the core, where all callers
can share it. Also close the sysfs read-twice races which could pass
inconsistent ranges, and add tests for invalid input.
- "mm/damon: misc cleanups" (Zenghui Yu):
Clean up a collection of small DAMON rough edges in the core,
interfaces and tests. No large behavior change here, just simpler
code and fewer opportunities for confusion.
- "mm/damon: cleanup code, add test cases, and update guidances in
docs" (SJ Park):
Clean up DAMON's core and interfaces, add missing test cases and
bring the documentation into line with the code. These changes should
make the newer facilities easier to understand and maintain.
- "mm/damon: introduce hugepage_size probe filter" (SJ Park):
Add a DAMON probe filter for hugepage size. Let users select a range
of folio sizes through sysfs, validate the range and document the new
interface, so monitoring can concentrate on memory backed by the
desired hugepage sizes.
- "selftests/mm: pagemap_ioctl test fixes and cleanups" (Zenghui Yu):
Fix some pagemap ioctl test assumptions and clean up their setup and
reporting. Make failures point to the operation that went wrong
rather than to test harness bookkeeping.
- "mm: avoid large folio splits when swap is unavailable" (Johannes
Weiner):
Avoid splitting large folios merely to attempt swap when no swap
space is available. There is little point paying for an expensive
split when reclaim cannot actually swap the resulting pages out.
- "mm: workingset: fix the shadow node budget under MGLRU" (Hui Zhu):
Fix the workingset shadow-node budget with MGLRU enabled. Keep the
shadow entries within the intended limits so refault tracking does
not quietly consume more metadata than it should.
- "selftests/mm: TAP output and global-state fixes" (Song Hu):
Fix selftests which produce misleading TAP output or fail to restore
global settings after a run. A test ought to leave the system as it
found it, particularly when a whole suite follows behind.
- "mm: remove page_swap_entry()" (Tal Zussman):
Remove page_swap_entry() and convert its remaining users to the folio
and swap-entry helpers. Lose an old page-level interface and make the
swap paths a little more consistent.
- "mm: refactor zonelist constructors and iterators" (Gregory Price):
Refactor the zonelist construction and walk helpers to reduce
duplicated logic. Keep the allocation ordering rules intact while
making it clearer how zones are selected.
- "mm: kmemleak: batch the struct page scan" (Breno Leitao):
Scan adjacent struct pages in batches instead of taking kmemleak_lock
once for every single page. This preserves scan coverage and the
lock-hold bound, while cutting overhead substantially on debug
kernels.
- "mm/damon: misc improvements in tests and documents" (Zenghui Yu):
Fix some DAMON selftests which leave kdamond running or fail under
Python safe-path mode, and improve the KUnit coverage and
documentation. Nothing especially glamorous, but these are worth
doing.
- "mm/damon: introduce probe_hits_wsum DAMOS core filter" (SJ Park):
Add a DAMOS core filter using weighted probe-hit counts. This gives
schemes another way to select regions based on what the monitoring
probes observed.
- "mm/mglru: fix ineffective memory protection for non-kswapd reclaim"
(Ridong Chen):
Fix MGLRU's memory protection handling for reclaim outside kswapd.
The protection settings weren't having the intended effect in these
paths, leaving protected memory more exposed than requested.
- "mm/damon: improve readability, clarity and test coverage" (SJ Park):
Simplify a handful of DAMON core helpers, clarify their behavior and
expand test coverage. There are no intended userspace-visible changes
in this batch.
- "mm: replace PF_KCOMPACTD/PF_KSWAPD with kthread_func()" (Kefeng
Wang):
Stop identifying reclaim and compaction workers through the PF_KSWAPD
and PF_KCOMPACTD task flags. Use their kthread functions instead and
remove the redundant flag plumbing.
- "mm/hugetlb: fix surplus accounting and availability checks during
demotion" (Longlong Xia):
Fix HugeTLB surplus accounting and availability checks when demoting
huge pages. Keep the counters and reservation decisions consistent as
one huge page turns into several smaller ones.
- "mm/damon: add kunit tests for probe_hits handling and probe params
validation" (Jason Angelov):
Add KUnit tests for DAMON probe-hit handling and invalid probe
parameters. These cover edge cases that were too easy to miss in
ordinary monitoring tests.
- "mm/damon: fix misc bugs in kunit, quota goals and sysfs refresh_ms"
(SJ Park):
Fix a few DAMON bugs in KUnit tests, quota-goal handling and the
sysfs refresh interval. Small fixes, but the incorrect results and
control behavior were real.
- "mm: zswap: optimize zswap invalidate and store" (Kefeng Wang):
Simplify zswap's invalidate and store paths to avoid avoidable
lookups and work. Keep the entry lifetime rules intact while making
those frequent operations cheaper.
- "mm: replace custom ratelimiting logic" (Pedro Falcato):
Replace hand-rolled MM rate limiting with the common kernel helpers.
There is no need to maintain another local version of the same
mechanism.
- "mm/vmalloc: minor cleanups" (Ye Liu):
Tidy some vmalloc initialization and reporting code, and pull
duplicated vmap-area insertion work into a helper. No functional
changes are intended.
- "mm/mglru: clean up isolate_folios for readability and clarity"
(Ridong Chen):
Clean up MGLRU's isolate_folios() so the decisions are easier to read
and verify. Keep reclaim behavior unchanged while removing
unnecessary twists in the control flow.
- "Catch automatic storage in IDA and Maple Tree definitions" (Yury
Norov):
Catch IDA and maple-tree definitions placed in automatic storage
where the lifetime rules are wrong or easy to get wrong. Add checks
so such mistakes are harder to slip through unnoticed.
- "mm, swap: some random fixes and cleanups" (Kemeng Shi):
Fix and simplify a few unrelated swap paths, removing redundant work
and tidying up error handling. A small collection of things which
were better fixed than left lying around.
- "mm/page_io: folio conversion cleanups" (Tal Zussman):
Finish some page-to-folio conversions in swap I/O, removing obsolete
wrappers and using folio operations consistently. The I/O behavior is
intended to remain the same.
- "mm/swap: skip empty clusters in the swapoff scan" (Youngjun Park):
Skip empty swap clusters during swapoff instead of walking slots
which have nothing to release. This avoids pointless work on sparsely
used swap devices.
- "mm: trace: decode architecture-specific VMA flags" (Meijing Zhao):
Teach the VMA trace output to decode architecture-specific flags,
rather than leaving them as unexplained bits. That should make
tracing a good deal more useful when debugging unusual mappings.
- "mm, swap: don't spin on a bad swap entry" (Breno Leitao):
Stop repeatedly retrying a swap entry which is already known to be
bad. Make the error path terminate cleanly instead of wasting CPU
chasing an entry that cannot be used.
- "mm/page_isolation: fix UBSAN shift-out-of-bounds in
isolate_single_pageblock" (Qi Xi):
Fix an out-of-range shift in isolate_single_pageblock() and tighten
its associated boundary handling. UBSAN found the issue; the result
should be more robust pageblock isolation.
- "mm: khugepaged: fix tracepoint UAF" (Vernon Yang):
Fix a use-after-free in khugepaged tracepoint handling. Make sure the
information recorded by tracing stays valid for as long as the trace
event needs it.
- "mm/damon: fix DAMOS bugs in core, paddr and vaddr" (SJ Park):
Fix several DAMOS problems across the core and physical- and
virtual-address backends. The changes close error cases and bring the
implementations back into agreement.
- "mm/hugetlb_cgroup: move the per-node usage along with the folio"
(Hongfu Li):
Move HugeTLB per-node usage accounting when folios migrate between
nodes or are reparented to another cgroup. Otherwise numa_stat can
retain stale charges and even underflow when the folio is freed.
- "docs/core-api: memory-allocation: add k[mz]alloc_obj() and clarify
kmalloc" (Mike Rapoport):
Document the typed k[mz]alloc_obj() allocation helpers and clear up
some kmalloc guidance. Make it easier for callers to choose the right
interface.
- "selftests/mm: Validate selections and scope memfd_secret setup"
(Tianyi Chen):
Check selftest selections before running and confine memfd_secret
setup to the cases that actually need it. Avoid confusing failures
caused by the test harness itself.
- "mm/damon: move zero damos quota target_value handling to the core
layer" (SJ Park):
Handle a zero DAMOS quota goal target in the core rather than making
each user of the interface invent its own rule. Keep quota decisions
consistent across callers.
- "mm/damon: Introduce a huge page collapsing mechanism using auto
tuning" (Asier Gutierrez):
Add an auto-tuned DAMON mechanism for collapsing huge pages. Use
monitoring feedback to decide when collapse is worthwhile rather than
forcing one fixed policy onto every workload.
- "kmemleak: fix stale documentation and raise the verbose default"
(Breno Leitao):
Bring the kmemleak documentation up to date and require a third
confirming scan before verbose kernels print a suspected leak. The
extra scan helps suppress transient false positives in noisy debug
builds.
- "mm/memory: improve insert_pages() error handling" (Avi Weiss):
Tighten insert_pages() error handling so partial failures are
reported and cleaned up consistently. The caller should not have to
guess how far the operation got.
- "mm/damon: allow NULL or unstarted damon_ctx parameter for
damon_call()" (SJ Park):
Make damon_call() handle NULL and not-yet-started contexts safely.
Simplify the rules for callers and avoid an indefinite wait on an
invalid state.
- "mm/damon: introduce pgidle_set probe filter type" (SJ Park):
Add a pgidle_set DAMON probe filter, giving monitoring users another
way to select pages by their idle-tracking state. Wire it through the
filter interfaces and tests.
- "mm/damon/core: remove unused helper functions" (Cheng-Han Wu):
Remove DAMON core helpers which no longer have callers. Keep the API
surface smaller rather than retaining dead code just in case.
- "mm: stop calling pmd_folio() on special PMDs" (Gregory Price):
Avoid calling pmd_folio() on special PMD mappings that have no struct
page. The old assumption could fault in procfs walkers; use the
normal-folio checks and handle those mappings properly.
- "mm: remove three unused helpers from mm.h" (Tal Zussman):
Remove three unused helpers from mm.h. Fewer obsolete interfaces for
the next person to stumble over.
- "mm/damon: improvements in efficiency, error handling, documents"
(Liew Rui Yan):
Skip needless DAMOS quota work once the quota is full, propagate a
missing sysfs error and tidy a couple of comments. The ignored error
could make a monitoring-start request appear successful when it
wasn't.
- "docs/mm/damon/design: add explanation of nr_snapshots" (Liew Rui
Yan):
Explain DAMON's nr_snapshots accounting in the design documentation
and straighten out the associated terminology. Make it clearer what
users should expect from that counter.
- "selftests/cgroup: fixes for test_zswap on single core VM" (Wilson
Felipe Pereira):
Fix two test_zswap failures on single-CPU VMs. Wait for the exiting
child's cgroup to become unpopulated before reconfiguring it, and
avoid another test assumption which breaks under constrained
scheduling.
- "docs/ksm: fix advisor documentation and comment" (Jinjiang Tu):
Correct the KSM advisor documentation and a misleading code comment.
Keep the description aligned with how the advisor actually works.
- "Fix bugs in HugeTLB allocation when mem_cgroup_charge_hugetlb()
fails" (Ackerley Tng):
Fix HugeTLB allocation cleanup when the memcg charge fails. Make sure
failure exits undo what was allocated rather than leaving stale
accounting or resources behind.
- "mm, memcg: fix memory.peak reset clobbering other fds' watermark"
(Ridong Chen):
Fix memory.peak reset handling so resetting one file descriptor
doesn't overwrite the watermark kept for another. Keep those per-open
measurements independent as intended.
- "mm: reject zone device folios in more folio walkers" (Gregory
Price):
Reject zone-device folios in page walkers that cannot safely handle
them. Extend the existing checks rather than assuming every folio has
ordinary system-memory backing.
- "percpu: remove code with no effect" (Sang-Heon Jeon):
Remove a few pieces of percpu code which no longer do anything. There
is no reason to keep maintaining dead paths.
* tag 'mm-stable-2026-10-19-21-33' of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm: (539 commits)
mm: shmem: assign a lock class to the unused huge shrinklist
mm: vmscan: put rotation-missed folios at the LRU tail
mm: kmemleak: scan the struct page array in MAX_SCAN_SIZE batches
mm: kmemleak: move the struct page scan into a helper
mm/damon: document that a zero sample_interval is accepted
mm/damon: fix typos in comments
mm/damon/sysfs: propagate damon_call() error in turn_damon_on
mm/damon/core: skip quota score setup when the quota is full
mm/damon/tests/core-kunit: test the size charged for a filter-trimmed region
mm/damon/core: charge only the part of a region the filter left
mm: disallow raw PFN mappings of huge/shared zeropage
selftests/mm: zap whole PTE tables in the khugepaged race harness
selftests/mm: add memory-pressure threads to the khugepaged race harness
selftests/mm: race the collapse of windows with holes
selftests/mm: add khugepaged race harness
selftests/mm: check that one khugepaged pass collapses one window
selftests/mm: run every supported collapse order by default
selftests/mm: cover a shared-source collapse write race
selftests/mm: parameterize the mixed-source collapse case by source order
selftests/mm: add order-parameterized khugepaged collapse cases
...
|
||
|
Linus Torvalds
|
864c07de0c |
hardening updates for v7.4-rc1
- Provide kmalloc_obj*-style API for devm_kmalloc() - Improve seq_buf API for saner C String handling with several bug fixes along the way (Kees Cook, Bill Wendling) - Overflow helper readability improvements (Bill Wendling) - Unexport native_write_cr4() (Dennis Chen) - um: fix CONFIG_GCOV for built-in code (Johannes Berg) - Improve stackinit test coverage (Justin Stitt) - KMSAN: fix memset() when using fortify-source, again (Arnd Bergmann) - lkdtm: use kmalloc() instead of __get_free_page (Mahad Ibrahim) - signalfd: don't dequeue the forced fatal signals (Oleg Nesterov) - Fix randstruct to recognize new container_of() behavior - coccinelle: kmalloc_obj: Convert multi-byte integral allocations too - Fix a couple remaining allocation return type mismatches -----BEGIN PGP SIGNATURE----- iHUEABYKAB0WIQRSPkdeREjth1dHnSE2KwveOeQkuwUCatpUqAAKCRA2KwveOeQk u6f7AQCmaAF+jGVxeljdguKRrge+bjxpsK+RIyDzf7JMUauxPwD/RvGklH+QCACS xj4tExbgBFOAo48uvqfUiGsJLaCJAQg= =uEMy -----END PGP SIGNATURE----- Merge tag 'hardening-v7.4-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/kees/linux Pull hardening updates from Kees Cook: - Provide kmalloc_obj*-style API for devm_kmalloc() - Improve seq_buf API for saner C String handling with several bug fixes along the way (Kees Cook, Bill Wendling) - Overflow helper readability improvements (Bill Wendling) - Unexport native_write_cr4() (Dennis Chen) - um: fix CONFIG_GCOV for built-in code (Johannes Berg) - Improve stackinit test coverage (Justin Stitt) - KMSAN: fix memset() when using fortify-source, again (Arnd Bergmann) - lkdtm: use kmalloc() instead of __get_free_page (Mahad Ibrahim) - signalfd: don't dequeue the forced fatal signals (Oleg Nesterov) - Fix randstruct to recognize new container_of() behavior - coccinelle: kmalloc_obj: Convert multi-byte integral allocations too - Fix a couple remaining allocation return type mismatches * tag 'hardening-v7.4-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/kees/linux: (29 commits) overflow: Match arithmetic in both parts of __builtin_choose_expr lkdtm: Make UNSET_SMEP built-in-only and unexport native_write_cr4() extcon: realtek: Remove const from type_c_cfg allocation type coresight: cti: Add const to con_groups allocation type docs: core-api: Document the seq_buf API nvdimm: ndtest: Return the string length from flags_show() powerpc/papr_scm: Return the string length from the sysfs show functions seq_buf: Add seq_buf_init_append() bpf: Remove dead newline stripping from format_disasm_line() seq_buf: Add seq_buf_terminate() seq_buf: Add seq_buf_strlen() seq_buf: Clear what a writer did not claim when a seq_buf overflows seq_buf: Copy what fits when seq_buf_puts() and seq_buf_putmem() overflow scsi: lpfc: Print rx monitor records straight into the seq_buf seq_buf: Do not pop from an overflowed seq_buf seq_buf: Do not print an empty line from an overflowed seq_buf_do_printk() .get_maintainer.ignore: Add Kees Cook devres: Provide kmalloc_obj*-style API for devm_kmalloc mm: Move default_gfp() into gfp_types.h kunit/stackinit: Clean up XFAIL under instrumentation ... |
||
|
Alexei Starovoitov
|
9b27cd0554
|
bpf: Reject unwind_goto and unwind_exit in offloaded progs
A prog with unwind_goto or unwind_exit has to be JITed by a JIT that
supports them. The driver JITs an offloaded prog, and nobody looks at
jit_required there.
nfp rejects unwind_goto in nfp_verify_insn(). netdevsim takes everything.
Nothing runs such a prog, but it's not jited for bpf_insn_prepare_dump(),
which takes unwind_goto for BPF_CALL_ARGS and shows it as a call.
Fixes:
|
||
|
Eduard Zingerman
|
32e962577d
|
bpf: Support 32-bit linear scalar evolution
Representation
--------------
Represent ALU32 recurrences as:
w0 += step -> (zext32 (linear r0 step))
Extend simplify() rules to maintain the above form for latch
expressions. E.g. for a post-condition loop such as:
w0 = 0;
l: w0 += 1;
if w0 != 3 goto l;
exit;
SCEV at header: (zext32 (linear r0 1))
Initial SCEV at latch: (zext32 (+ (zext32 (linear r0 1)) 1))
Simplified SCEV at latch: (zext32 (+ (zext32 (linear r0 1)) 1))
-> (zext32 (+ (linear r0 1) 1))
-> (zext32 (linear (+ r0 1) 1))
Iterations count computation
----------------------------
- Extract the main iteration-count logic from compute_max_iters() into
compute_backedge_taken_count(). Reuse its 64-bit arithmetic after
sign/zero extending the latch operands.
- Reject the computed count if the predicted exit value falls outside
signed/unsigned 32-bit range, e.g.:
w0 = 0x7ffffffe;
l: w0 += 1;
if w0 s<= 0x7fffffff goto l;
The 64-bit computation predicts an exit at S32_MAX + 1, but the
32-bit value wraps to S32_MIN and the comparison remains true.
- Handle mixed 64-bit/32-bit comparisons/recurrences only when initial
bounds and step fit in 32-bit ranges, e.g.:
w0 = 0;
r1 = 0x100000003 ll;
l: w0 += 1;
if r0 != r1 goto l;
Truncating the bound to 32 bits would predict three header visits.
However, `w0 += 1` keeps r0 within [0, U32_MAX], so it can never
be equal to the 64-bit bound.
Widened bounds computation
--------------------------
- Update linear_bounds() to truncate the initially computed 64-bit
range to 32 bits, then zero-extend it to obtain the 64-bit range
of iteration values. Step computation needs to handle:
- unsigned 32-bit wraparound;
- entry and zero-extended iteration values disagreeing on base/step.
- Union both ranges with the corresponding register ranges at entry,
since the entry value need not be zero-extended.
- Extend `struct bounds` to keep both {32,64}-bit ranges and pass both
to bpf_set_reg_range(). A 64-bit range alone loses useful 32-bit
information, e.g.:
entry = 0, step = -1, two header visits
r64: [0, U32_MAX]
r32: [-1, 0]
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261020-scev-32bit-v1-1-838f34f9f0cd@gmail.com
|
||
|
Alexei Starovoitov
|
a2d2747713
|
bpf: Add unwind_goto and unwind_exit insns
Rust Drop, C++ destructors and __attribute__((cleanup)) in C have to run
when a panic or an exception passes through a frame.
v1 [1] did it with a flag in r3 and 'if r3 != 0 goto pad' after the call.
Three bugs that the bots found there had the same cause: r3 is an ordinary
register, and JITs guessed the meaning from the shape of the code.
Kumar suggested new insns instead [2]:
unwind_goto pad BPF_JMP | BPF_JCOND, src_reg = BPF_UNWIND_GOTO
unwind_exit BPF_JMP | BPF_EXIT, src_reg = BPF_UNWIND_EXIT
'call foo; unwind_goto pad' goes to the pad when foo left with unwind_exit
and falls through when it left with exit. No register says which one it
was. In everything else unwind_exit is exit.
llvm lowers unwinding to them [3]:
ret exit
invoke foo unwind pad call foo; unwind_goto pad
resume r0 = 0; unwind_exit
call _Unwind_RaiseException r0 = 0; unwind_exit
invoke _Unwind_RaiseException goto pad
In a function that returns 16 bytes it's 'r0 = 0; r2 = 0; unwind_exit'.
Landing pads are ordinary blocks, so the verifier checks them like any
other code. A pad that forgets to drop a lock or a reference is a leak on
that path. There are no tables, no stack walk, no kfuncs, no BTF and
nothing to do for libbpf and other loaders.
In the verifier:
- unwind_goto has to follow a call or callx of a subprog and cannot be
a jump target. For cfg, liveness and the rest it's one more conditional
jump, like may_goto.
- How the callee left is one bit of the verifier state. It lives from
the exit of the callee to the next insn. states_equal() looks at it,
since the registers can be the same either way.
- unwind_exit is rejected when unwind_goto doesn't follow the call.
- A global subprog that has unwind_exit can leave either way. Both are
explored after the call, like it's done for might_throw.
- A sync callback cannot unwind. The helper that calls it doesn't expect
that: bpf_loop() would go on.
- unwind_exit of the main prog is exit, unless it's an extension. So it is
in async and exception callbacks. The kernel calls them, and there is no
frame to unwind to.
- An extension leaves either way, like the subprog that it replaced.
freplace is a jump, so the extension returns to the caller of
the subprog. An extension that has unwind_exit can replace only a subprog
that has it too. Otherwise there is no unwind_goto after the calls.
- Of tracing progs only fentry can attach to a subprog or an extension
that has unwind_exit. It would return to the trampoline that called it.
- unwind_exit changes its return address, so the cpu mispredicts
the return and runs what follows unwind_goto speculatively. Without
CAP_PERFMON the verifier looks at that path too.
- unwind_goto goes away together with a pad that turned out to be dead
code. It stays when what follows it is dead code. The cpu predicts that
the callee returns there, so 'goto pc-1' takes the place of the dead
code. Otherwise its removal would move other code there.
JITs don't emit a branch:
unwind_goto:
extable[n++] = { ret_addr - 1 -> pad }
unwind_exit:
my_ret_addr = bpf_unwind_target(my_ret_addr)
epilogue
So exit and the normal path after the call cost nothing. Every frame fixes
only its own return address. There is no stack walk. The lookup is what
the page fault handler does for bpf progs.
The interpreter cannot do that, so the prog has to be JITed, and JITs opt
in with bpf_jit_supports_unwind(). x86-64 and arm64 are in the next
patches.
BPF_JCOND is the same opcode as BPF_CALL_ARGS of the interpreter, and
BPF_UNWIND_GOTO is the same number as BPF_PSEUDO_CALL. may_goto is gone by
the time a prog is dumped, unwind_goto is not. bpf_insn_prepare_dump()
tells them apart: a prog with unwind_goto is always JITed.
Compared to v1 there is no BTF decl_tag and no 'r3 = 0' before every exit.
Tail call and ld_abs are fine in a subprog that can unwind, since they
leave it like exit does.
unwind_exit doesn't set r0. An implicit definition of r0 would need
special cases in backtracking, in liveness and in the check of the return
value of the main prog. A plain 'r0 = 0' in front of it needs none.
Rust
rust-bpf [4] builds with that llvm as it is. The programs are built with
-C panic=unwind. The panic handler prints the message and calls
_Unwind_RaiseException().
exc_cleanup.rs panics three frames deep, with preemption disabled in one
frame and rcu_read_lock held in another. All Drop impls run and the prog
returns 0.
scx_simple.rs is 2195 insns with 48 unwind_goto, 20 of them after callx,
and 16 unwind_exit. With r3 it was 2216 insns. scx_cosmos.rs is 3694 insns
with 68 and 18. It was 3716.
C++
Exceptions work too. This compiles, loads and runs:
struct rcu_guard {
rcu_guard() { bpf_rcu_read_lock(); }
~rcu_guard() { bpf_rcu_read_unlock(); }
};
static __noinline int inner(int len)
{
rcu_guard guard;
if (len > 100)
throw 1;
return len;
}
SEC("tc") int prog(struct __sk_buff *skb)
{
try {
return inner(skb->len);
} catch (...) {
return 42;
}
}
It returns 64 for a packet of 64 bytes and 42 for a packet of 200 bytes.
The destructor runs in both cases, otherwise the verifier would reject
the prog. 'throw' is a jump to the pad with the destructor. 'catch' is
unwind_goto:
inner:
w6 = w1
call bpf_rcu_read_lock
if w6 s< 101 goto ret
r1 = exc_obj ll
*(u32 *)(r1 + 0) = 1
goto pad
ret:
call bpf_rcu_read_unlock
w0 = w6
exit
pad:
call bpf_rcu_read_unlock
r0 = 0
unwind_exit
prog:
w6 = *(u32 *)(r1 + 0)
w1 = w6
call inner
unwind_goto catch
out:
w0 = w6
exit
catch:
w6 = 42
goto out
It needs -fexceptions -funwind-tables and four one line __cxa_*()
functions, where __cxa_throw() calls _Unwind_RaiseException().
Every pad is entered whatever was thrown, so llvm takes catch (...) only.
It's a selftest in patch 5.
[1] https://lore.kernel.org/bpf/20261018010246.1643877-1-alexei.starovoitov@gmail.com/
[2] https://lore.kernel.org/bpf/DM7KR7J8VY2T.1CZIRUF6QRCWG@gmail.com/
[3] https://github.com/llvm/llvm-project/pull/233649
[4] https://github.com/4ast/rust-bpf
Suggested-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261020181320.1284798-2-alexei.starovoitov@gmail.com
|
||
|
Linus Torvalds
|
00b04f4495 |
vfs-7.4-rc1.idmap
Please consider pulling these changes from the signed vfs-7.4-rc1.idmap tag.
Thanks!
Christian
-----BEGIN PGP SIGNATURE-----
iHUEABYKAB0WIQRAhzRXHqcMeLMyaSiRxhvAZXjcogUCatHrlgAKCRCRxhvAZXjc
opFuAPwLL5l3D3GSRyD+SyOjKYThylgowvH9c2BcLWbSAIO6yQD9F05GX+rrcbUq
9ijolxAaQZYpEWsemZIIHx6BBwdDBA4=
=IH8Z
-----END PGP SIGNATURE-----
Merge tag 'vfs-7.4-rc1.idmap' of git://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs
Pull vfs idmap constification from Christian Brauner:
"This makes struct mnt_idmap const throughout the tree.
A mount's idmapping is immutable. The only thing modified after
alloc_mnt_idmap() is the reference count and that is hidden behind
mnt_idmap_get() and mnt_idmap_put(). Everything else only reads from
it. That is the same model struct cred uses and the idmapping is
sensitive enough to deserve it.
The conversion goes bottom up through the vfs{g,u}id and fs{g,u}id
helpers, the privilege checks, acl, xattr and quota and then every
inode operation that takes an idmap, so ->permission(), ->getattr(),
->setattr(), ->create(), ->mkdir(), ->mknod(), ->symlink(),
->rename(), ->tmpfile(), ->get_acl(), ->set_acl() and ->fileattr_set()
with their implementations in every filesystem, plus the LSM hooks.
struct vfsmount, struct mount_kattr and struct kstatmount store a
const pointer and nop_mnt_idmap and invalid_mnt_idmap live in rodata,
so anything that tries to modify them faults instead of silently
corrupting the initial idmapping"
* tag 'vfs-7.4-rc1.idmap' of git://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs: (27 commits)
fs: make nop_mnt_idmap and invalid_mnt_idmap const
fs: port mnt_idmap() and file_mnt_idmap() to const mnt_idmap
fs: port vfs_*() helpers to const mnt_idmap
fs: port ->setattr() to pass const mnt_idmap
fs: port ->getattr() to pass const mnt_idmap
fs: port ->create() to pass const mnt_idmap
fs: port ->symlink() to pass const mnt_idmap
fs: port ->mkdir() to pass const mnt_idmap
fs: port ->rename() to pass const mnt_idmap
fs: port ->mknod() to pass const mnt_idmap
fs: port ->tmpfile() to pass const mnt_idmap
fs: port ->get_acl() to pass const mnt_idmap
fs: port ->set_acl() to pass const mnt_idmap
fs: port ->fileattr_set() to pass const mnt_idmap
fs: port xattr to const mnt_idmap
fs: port ->permission() to pass const mnt_idmap
fs: port acl to const mnt_idmap
fs: port inode_init_owner() to const mnt_idmap
fs: port inode_owner_or_capable() to const mnt_idmap
fs: port privilege checking helpers to const mnt_idmap
...
|
||
|
Linus Torvalds
|
a1d4585be1 |
vfs-7.4-rc1.file
Please consider pulling these changes from the signed vfs-7.4-rc1.file tag. Thanks! Christian -----BEGIN PGP SIGNATURE----- iHUEABYKAB0WIQRAhzRXHqcMeLMyaSiRxhvAZXjcogUCatHrlgAKCRCRxhvAZXjc osd6AP4nuqrNhsMzcp4zPk4rbMVhvJL9O3ccnoE6TLn8Mg7uLwD/WtjKIVtKAPkv gz58C2i14vDLLbKAAGK0P8c1EF0ZuAE= =jY1x -----END PGP SIGNATURE----- Merge tag 'vfs-7.4-rc1.file' of git://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs Pull file table updates from Christian Brauner: "This annotates struct fdtable's fd array with __counted_by_ptr so KASAN and the fortified functions can bounds check it against max_fds and adds a KUnit suite for fdtable. A follow-up uses __struct_size() in the test since sparse doesn't know __builtin_dynamic_object_size(). FD_PREPARE() is simplified. It was built as an ACQUIRE-style guard on infrastructure that was never needed. The ACQUIRE machinery and CLASS_INIT() go away, the variable is declared with __cleanup() directly, struct fd_prepare loses its err field since the fd either holds the descriptor or the error, and callers get a const pointer to the guard behind a UNIQUE_ID() name so they can't shadow it. Code generation is identical" * tag 'vfs-7.4-rc1.file' of git://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs: file: make struct fd_prepare const and kill its err field cleanup: remove CLASS_INIT() file: declare the FD_PREPARE() variable with __cleanup() directly file: simplify FD_PREPARE() vfs: use __struct_size() in the fdtable KUnit test vfs: Add KUnit tests for fdtable vfs: Annotate struct fdtable's fd field with __counted_by_ptr |
||
|
Nicholas Velten
|
f25f6319e2
|
bpf: Annotate data races on bpf_lru_node type
KCSAN reports a data race on bpf_lru_node::type between the LRU list
rotation and the free path of the common LRU:
BUG: KCSAN: data-race in __bpf_lru_list_rotate / bpf_lru_push_free
write to 0xffff88811b758022 of 1 bytes by task 4756 on cpu 1:
__bpf_lru_node_move
__bpf_lru_list_rotate_active
__bpf_lru_list_rotate
bpf_lru_list_pop_free_to_local
bpf_common_lru_pop_free
bpf_lru_pop_free
htab_lru_map_update_elem
read to 0xffff88811b758022 of 1 bytes by task 4752 on cpu 0:
bpf_lru_list_push_free
bpf_common_lru_push_free
bpf_lru_push_free
htab_lru_push_free
htab_lru_map_update_elem
value changed: 0x00 -> 0x01
bpf_lru_list_push_free() reads node->type for a WARN_ON_ONCE() sanity
check before taking l->lock. __bpf_lru_node_move(), called right after
under l->lock, does the same check, so drop the unlocked one.
bpf_common_lru_push_free() re-checks node->type under loc_l->lock after
the lockless READ_ONCE() at function entry. By then the node has been
removed from the htab, so only flush and rotate can change its type,
through __bpf_lru_node_move_in() and __bpf_lru_node_move() under
l->lock. Mark that re-check with READ_ONCE() and those two writers with
WRITE_ONCE(), following commit
|
||
|
Emil Tsalapatis
|
7244a2e58d
|
bpf: Fix arena race between page free and alloc leading to incoherency
Existing arena kfunc code has an underlying race condition
that can lead to writes being lost from the BPF program's
point of view:
a) A memory range gets freed by operation (1), and its range
is added back to the arena range tree.
b) A concurrent allocation (2) reallocates the range, and does
writes to it. Writes from that CPU may follow the stale TLB
entries into the pages that are about to be freed.
c) (1) invalidates the TLB. The old pages, and any writes done
to them, are now inaccessible. zap_pages() similarly removes the
mappings for userspace threads.
This can be triggered by particularly demanding BPF arena data
structures that constantly allocate and deallocate memory, like
hash table allocations.
Solve this ABA problem by keeping the range allocated until TLB
invalidation and unmapping are complete. Under the arena spinlock,
claim any free pages in the range and clear its kernel PTEs. Then drop
the spinlock, lock and unlock each cleared page to wait for faults that
are still installing a user PTE, flush the kernel TLB and zap the user
page tables. Finally, retake the spinlock to release the range. Frees
serialize on zap_mutex, so an overlapping free cannot release the range
early. Claiming the range only allocates a range tree node when the
whole range is free, and then there is nothing to free.
User faults on a range that is allocated without a page now raise
SIGBUS instead of populating it. That covers a range reserved with
bpf_arena_reserve_pages() and a range whose kernel PTEs a free has
already cleared. Touching a page while a BPF program frees it is a user
space bug, and in a BPF_F_SEGV_ON_FAULT arena it already raises SIGSEGV.
Sleepable frees run synchronously and are deferred only when the arena
spinlock cannot be taken. The deferred free worker retries such a free
in place instead of requeueing itself, so the flush_work() in
arena_map_free() waits for all of its work.
Fixes:
|
||
|
Emil Tsalapatis
|
05bee0638b
|
bpf: Revalidate faulted arena pages before installing user mappings
A user fault on an arena pins the page that the kernel arena maps,
drops the arena spinlock and returns the page to the core fault code,
which then locks the folio and installs the user PTE. A concurrent
free can clear the kernel mapping in that window. The page reference
prevents a use-after-free, but the fault can still install a user PTE
for a page the arena has already released. User space and BPF programs
then see different memory at the same arena offset.
Close the window from the fault side. Lock the page, recheck that the
kernel mapping still points to it, and return it with VM_FAULT_LOCKED so
that it stays locked until the core fault code has installed the user
PTE. If a free removed the page, drop it and return VM_FAULT_NOPAGE so
that the fault starts over. Return VM_FAULT_HWPOISON for a poisoned
page.
This change alone does not close the race, because the free path does
not yet wait for the page lock. The following change makes frees wait
for in-flight faults before zapping user mappings.
Fixes:
|
||
|
Kumar Kartikeya Dwivedi
|
19a17b5b2b
|
bpf: Refuse CO-RE relocations for a signed program
The signature of a program covers its insns as submitted, along with the
frozen metadata maps of a signed loader, and bpf_prog_verify_signature()
checks it before the verifier does anything else with the program. CO-RE
relocations are the one input besides the insns that changes them: the
loader hands them in with the program, and bpf_check_core_relo() applies
them after the signature check, rewriting the immediate or the offset of
every insn a record names with a value the loader chooses, a field
offset, a type size or id or an enum value of the kernel BTF, or a type
id of the BTF the loader supplies itself. Whoever holds a signed program
can so change its constants once the signature has passed, the offsets a
signed loader reads its metadata map with for instance, and run a
program the signer never saw. The exclusive metadata map of a signed
loader does not make up for it: the hash that binds the map to a program
is chosen by whoever creates the map, so it can name the relocated
program as well.
Refuse a signed load that carries relocations, before the signature is
examined, like the other checks on the shape of a signed load. A signed
loader has none to lose: the programs it embeds are relocated by the
loader itself through bpf_core_apply(), and nothing relocates the loader.
The BTF of the program itself can stay: without relocations it steers
only the verification, through func_info, line_info and the annotations
of global function arguments, and rewrites nothing.
Fixes:
|
||
|
Sasha Levin
|
ced6a0993e
|
bpf: Don't reference arena kfuncs when arena isn't built
kernel/bpf/Makefile builds arena.o only for CONFIG_MMU && CONFIG_64BIT, but special_kfunc_list references bpf_arena_alloc_pages() and bpf_arena_free_pages() unconditionally, so on 32-bit and !MMU kernels resolve_btfids can't resolve them: WARN: resolve_btfids: unresolved symbol bpf_arena_free_pages WARN: resolve_btfids: unresolved symbol bpf_arena_alloc_pages Since commit |
||
|
Sasha Levin
|
63be110a96
|
bpf: Drop unused entries from special_kfunc_list again
Commit |
||
|
Kumar Kartikeya Dwivedi
|
a8471eb074
|
bpf: Set up the arena base only in functions that use it
A JIT sets up the arena base in the prologue of every function of a program that has an arena, a callee-saved register loaded with a 64-bit immediate and spilled around the function on x86. That is paid for by every program with an arena, whether it touches the arena or not, and now that libbpf binds the arena of an object to every program in it, by every program of an object that happens to contain an arena. Decide per function, on its final insns, right before it is JITed. bpf_prog_has_arena_insns() looks for an insn that works with the base: an access through an arena pointer, or through a number in a program loaded with BPF_F_ARENA_SCALAR, has been rewritten to a BPF_PROBE_MEM32 form by then, a cast to the user address space keeps BPF_ADDR_SPACE_CAST, and a kfunc call with an __arena argument, which the JIT rebases, has a function model that says so. A load of an __arena struct member is rewritten with an immediate and needs none; dereferencing the result does. Looking at the final insns rather than recording the uses during verification keeps the answer in step with what the JIT sees. A constant NULL passed to a nullable __arena kfunc argument takes the early exit for NULL arguments in the verifier, and the JIT rebases the call all the same; an insn that is dead code has disappeared by then. bpf_prog_jit_compile() caches the answer in bpf_prog_aux for a single function program, jit_subprogs() for each subprogram once it has its kfunc table and arena. It implies that the program has an arena, as the scan says no without one. The JITs with arena support pick up the arena base and user address only when bpf_prog_uses_arena_insns() says so, and so generate the prologue and epilogue of a program bound to an arena it never touches as if it had none. The other users of the arena address of a program stay as they are on purpose. The struct_ops trampoline converts __arena arguments on entry and arena_member_fixup() converts loaded __arena members after the load, both with the base as an immediate. Both have to happen whether or not the program dereferences the result, it may compare, store or pass it on, so neither depends on the base register nor on the flag. The arena fault report reads the user address of the arena the faulting insn belongs to, and such an insn sets the flag anyway. check_map_prog_compatibility() checks the user address when the arena is bound, before any use is known. The rest, in arena.c and sched_ext, works with the map and not with a program. Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com> Signed-off-by: Alexei Starovoitov <ast@kernel.org> Link: https://patch.msgid.link/20261019132203.3034698-14-memxor@gmail.com |
||
|
Kumar Kartikeya Dwivedi
|
7efe1d4e11
|
bpf: Turn loads of __arena pointer members into arena pointers
A kernel struct shared with BPF programs may want to point at arena memory, e.g. a sched_ext argument struct that the kernel builds in the scheduler's arena and that refers to another arena object. A pointer member reaches the program typed as a kernel pointer, so sched_ext publishes a plain u64 holding a pre-translated arena address that the program casts before use. Translate in the verifier instead. When a load through a kernel struct pointer lands on a member tagged __arena, btf_struct_walk() reports it and the destination register becomes PTR_TO_ARENA. The member holds the kernel address of the arena memory; the fixup pass appends the conversion to the arena pointer form that the struct_ops trampoline applies to __arena arguments, after a plain load or a load-acquire of the member: r1 = *(u64 *)(r2 + off) if r1 == 0 goto +1 // __nullable members and probed loads w1 -= lo32(kern_vm_start) A probed load, through a plain or an untrusted pointer, keeps the NULL test as well: it reads as 0 when it faults, and the subtraction must not turn that into an arena pointer. An acquire load only comes through a pointer that cannot fault. Stores to kernel structs are rejected as before, and with them the atomics that write such a member. A member inside a multi-member union is rejected too: an offset into a union resolves to its first member, so a program reading another member there would receive an arena pointer it did not ask for. Loads through program-allocated objects and program BTF are unaffected, there the tag is what the compiler emits for the arena address space. The rewrite is per insn while verification is per path, so the aux data records what an insn loaded on any path: an insn that loaded an arena pointer member on one path and a different member on another is rejected by the fixup pass, while paths disagreeing only on __nullable keep the NULL test, which a never-NULL member can afford. As for __arena arguments, the program needs an associated arena, which also supplies the base to subtract. Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com> Signed-off-by: Alexei Starovoitov <ast@kernel.org> Link: https://patch.msgid.link/20261019132203.3034698-8-memxor@gmail.com |
||
|
Kumar Kartikeya Dwivedi
|
bc7c282468
|
bpf: Recognize __nullable on pointer members of kernel structs
A pointer member of a kernel struct that the verifier derives as trusted or RCU-protected is assumed to be non-NULL, so a load through it is a plain dereference. The BTF_TYPE_SAFE_*_OR_NULL lists exist for members known to be NULL at times, but they live in the verifier and have to be extended by hand. Teach btf_struct_walk() to report the "nullable" BTF type tag on a pointer member as PTR_MAYBE_NULL. The tag combines with any other tag on the member; it only says that the pointer may be NULL. The verifier keeps the flag where NULL would otherwise be assumed away: on trusted and RCU results, which a program dereferences as they are, and on per-CPU results, which the per-CPU helpers shift without a check. Such a pointer has to be tested before it is dereferenced or passed on. Plain and untrusted results are probed anyway and lose the flag, as the BTF_TYPE_SAFE_*_OR_NULL lists never applied to them either, so a tagged member of an existing struct asks no more of the programs that read it than a list entry does, and the tag can take the place of the lists. The next patch uses the tag on __arena members to decide whether the injected address translation has to preserve NULL. Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com> Signed-off-by: Alexei Starovoitov <ast@kernel.org> Link: https://patch.msgid.link/20261019132203.3034698-7-memxor@gmail.com |
||
|
Kumar Kartikeya Dwivedi
|
55f17779d6
|
bpf: Check exclusive maps once the program digest is final
An exclusive map is bound to the digest of its program, which the verifier compared against prog->digest in check_map_prog_compatibility() as the map was added to the program. For a map an insn refers to that happens in check_and_resolve_insns(), after the digest is computed at the top of that function. For a map bound through a continuous fd_array it happens in process_fd_array(), which runs earlier, so for an unsigned program the comparison saw a zeroed digest and the load failed with -EACCES, "program's hash doesn't match map's excl_prog_hash", for the very program the map is exclusive to. Computing the digest earlier would not do: it has to cover the insns as the kernel verifies and runs them, after the CO-RE relocations it applies, or a loader could submit the insns of the exclusive program together with relocations that change them and pass the check with a different program. Leave the digest where it is and move the comparison instead: check every used map against the digest at the end of check_and_resolve_insns(), whether an insn referred to it or the array bound it. The claims a program makes on its maps move behind the check as well: bpf_map_claim(), which records the program as the user of the map for resolve_func_ptrs(), and the takeover of an insn array by bpf_insn_array_init(). Both ran as the map was added, so a program that is not the owner of an exclusive map could have flipped the owner's single-user state to shared, which a failed load does not undo, or held the owner's insn array while it was being verified. Now they run once every map has passed the check, and a rejected program leaves the maps as it found them. Nothing consults a claim before resolve_func_ptrs() and the fixup passes, which run later. The early digest of a signed program goes with the early comparison. Commit |
||
|
Alexei Starovoitov
|
4f842631f1
|
bpf: Refuse the prologue for an extension if callers keep packet pointers
An extension that writes to the packet of a tc, lwt_xmit or sk_skb program
starts with bpf_unclone_prologue(). If the skb is cloned, that gives it
a new head.
It does not count as changing packet data. changes_pkt_data goes by the
helpers and kfuncs that a program calls, and whether it gets a prologue
is known only after it has been compared with its target. So the extension
can replace a global function that leaves the packet alone, and the callers
of that function keep their packet pointers across the call. These point
into the old head, which is left to the other clone. Once that is freed,
by the reader of a packet socket for example:
BUG: KASAN: slab-use-after-free in bpf_prog_53bc21b9810cadbc_t_read+0x2e8/0x36e
Read of size 1 at addr ffff888107bfe73c by task tt11/102
Freed by task 112:
kfree+0x161/0x500
skb_release_data+0x2dc/0x340
consume_skb+0xa6/0x1c0
packet_recvmsg+0x4b3/0xa20
Nothing is wrong if the target program writes to the packet too: its own
prologue has uncloned the skb, and that of the extension does nothing.
Nor if the callers hold no packet pointers, like a dispatcher, or if the
main function is replaced.
Note for every global function that leaves the packet alone if it is
called while packet pointers, or slices of the dynptr of a packet, are
held. What the callers of a global function hold is not known while it
is verified, so the functions that it calls are taken for such as well.
Refuse an extension with a prologue for such a function, unless the target
program has a prologue, both when the extension is loaded and when it
is attached to another target.
Functions can share a type id in func_info. The first of them is the one
that is replaced, so look it up in one place.
Fixes:
|
||
|
Yiyang Chen
|
c3a0bfddae
|
bpf: Enforce trusted BTF ID argument provenance
A global subprogram parameter tagged __arg_trusted is verified with
PTR_TRUSTED, but the common call-site type check accepts bare PTR_TO_BTF_ID
and MEM_RCU registers. The callee can consequently pass a pointer with
weaker provenance to a kfunc that does not accept RCU-protected pointers.
RCU protection only guarantees that the object remains allocated through a
grace period. It does not guarantee that an object's refcount is nonzero,
so an RCU-protected pointer cannot satisfy the trusted-pointer contract.
For example, an RCU-protected bpf_cpumask loaded from a map may have a
zero refcount. Passing it through __arg_trusted lets the callee pass it
to bpf_cpumask_acquire(), whose plain refcount_inc() can retain an object
that will be freed after the grace period.
Make the existing provenance check contract-driven. Mark generated BTF ID
kfunc argument types with PTR_TRUSTED, and add MEM_RCU only for KF_RCU
kfuncs. Normalize PTR_MAYBE_NULL when the argument contract permits it.
Then use the argument flags to require trusted or referenced provenance
and allow MEM_RCU only when the contract says so.
Split is_trusted_reg() so the normalized type is used for type and modifier
checks while reference lookup still uses the original register. Conversion
of scalar-struct BTF ID arguments to fixed-size memory continues to
construct a new ARG_PTR_TO_MEM type without PTR_TRUSTED.
Fixes:
|
||
|
Masoud Aghasi
|
4a9b25fe8d
|
bpf: Fix incorrect handling of user flags by percpu hash map updates
For BPF_MAP_TYPE_PERCPU_HASH and BPF_MAP_TYPE_LRU_PERCPU_HASH maps,
htab_map_check_update_flags() and check_flags() are not considering
the possibility of a combination of (BPF_NOEXIST, BPF_EXIST) flags
with (BPF_F_CPU, BPF_F_ALL_CPUS) flags. This causes the
(BPF_NOEXIST, BPF_EXIST) flags to lose their effect in some cases.
For example, when using (BPF_F_CPU | BPF_EXIST) or
(BPF_F_CPU | BPF_NOEXIST) flag combinations with bpf_map_update_elem()
on a percpu hash map, the BPF_EXIST flag does not prevent new
insertions as expected and BPF_NOEXIST flag does not prevent
modification of existing entries as expected.
This patch fixes the bug by adding proper flag validations and checks.
Before this patch, htab_map_check_update_flags() rejected
(BPF_F_ALL_CPUS | BPF_EXIST) and (BPF_F_ALL_CPUS | BPF_NOEXIST) with
-EINVAL. After the patch those combinations are accepted.
This patch also starts returning -EINVAL for (BPF_EXIST | BPF_NOEXIST),
with or without BPF_F_CPU, which was previously accepted.
Fixes:
|
||
|
Masoud Aghasi
|
a898ff0f3e
|
bpf: Fix incorrect handling of user flags in bpf_percpu_array_update
For BPF_MAP_TYPE_PERCPU_ARRAY map, bpf_percpu_array_update()
is not considering the possibility of a combination of
(BPF_NOEXIST, BPF_EXIST) flags with (BPF_F_CPU, BPF_F_ALL_CPUS) flags.
This causes the (BPF_NOEXIST, BPF_EXIST) flags to lose their effect
in some cases.
For example, using the (BPF_F_ALL_CPUS | BPF_EXIST) flag combination
with bpf_map_update_elem() results in an incorrect EINVAL error
response, even though the flag combination is valid.
This patch fixes the bug by adding proper flag validations and checks.
Before this patch, bpf_percpu_array_update() rejected
BPF_F_ALL_CPUS | BPF_EXIST and BPF_F_ALL_CPUS | BPF_NOEXIST with
-EINVAL. Also the BPF_F_CPU | BPF_NOEXIST were accepted.
After the patch BPF_F_ALL_CPUS | BPF_EXIST is accepted and
BPF_F_ALL_CPUS | BPF_NOEXIST and BPF_F_CPU | BPF_NOEXIST
result in -EEXIST.
This patch also starts returning -EINVAL for (BPF_EXIST | BPF_NOEXIST),
with or without BPF_F_CPU, which was previously accepted.
Fixes:
|
||
|
KaFai Wan
|
0ceb2f21d3
|
bpf: Determine freplace staticness from BTF linkage
bpf_check_attach_target() rejects a freplace whose target is a static
function, using the subprog "unreliable" flag as a proxy for "static".
That flag is raised whenever a subprogram's BTF prototype disagrees with
the registers at its call site.
`147400e9c27c ("bpf: Compute parameters liveness in
bpf_compute_live_registers()")` made that work for static callees: a
static function with an unused argument no longer matches its BTF
prototype at the call site and is marked unreliable.
Since LLVM commit `29b36a16f111 ("[BPF] Emit ABI-accurate BTF
prototypes for DW_CC_nocall")` in 23.1 [1] that premise is gone:
BTFDebug may now drop arguments removed by the optimizer, so for
__noinline static int subprog(struct __sk_buff *ctx)
{
return bpf_ktime_get_ns();
}
compiled with clang 23 the BTF describes subprog() as
[75] FUNC_PROTO '(anon)' ret_type_id=2 vlen=0
[76] FUNC 'subprog' type_id=75 linkage=static
There is then no argument left to compare, the subprog stays reliable,
and the check in bpf_check_attach_target() is skipped. The freplace is
then rejected by the later linkage check instead, reporting "subprog()
is not a global function", and the selftest "exceptions" fails.
Determine staticness from the BTF linkage instead, and keep the
unreliable check for global functions whose BTF does not match their
call site.
There is no functional change: freplacing a static function was rejected
before and is still rejected with the same -EINVAL. Selftest "exceptions"
is updated to match.
[1] https://github.com/llvm/llvm-project/pull/198426
Fixes:
|
||
|
Woojin Ji
|
1c1dda9ba9
|
bpf: Zero-extend the index in inlined bpf_get_func_arg()
bpf_get_func_arg() takes a u32 index, but the inlined expansion compared
only the low 32 bits of R2 with the argument count and then shifted the
full 64-bit register to compute the context address. R2 is ARG_ANYTHING,
so nothing guarantees that its upper bits are zero: an index of
1ULL << 32 passes the bounds check as index zero while the load is
performed at ctx + (1ULL << 35).
Shift the index as a 32-bit value. When the 32-bit bounds check passes,
the low 32 bits of R2 are below R0, and R0 is at most 255: the trampoline
arm masks the argument count with 0xFF and the raw_tp arm uses the BTF
argument count minus one, which the verifier caps at MAX_BPF_FUNC_ARGS.
The 3-bit shift therefore cannot overflow and it clears the upper bits of
the register. Widening the comparison instead would reject indices whose
low 32 bits are valid. The check itself is unchanged, so this cannot
change which programs are accepted, only the address that is computed.
Fixes:
|
||
|
Josef Bacik
|
a6f8a67c13
|
bpf: Refuse packet slices for packet-changing helpers
The helpers which may change packet data do not take a packet pointer as a
memory argument: they have no pkt_access. They make the skb writable, or
make room in it, before they read what the argument points to, and that
can reallocate the head which it points into.
A slice from bpf_dynptr_slice() of the dynptr of the skb points into the
same head, but it is a PTR_TO_MEM, and the verifier lets it pass.
bpf_lwt_push_encap(), bpf_lwt_seg6_store_bytes() and bpf_lwt_seg6_action()
then read the old head, which has been freed. bpf_skb_store_bytes() follows
the head by now, but a slice is no better than a packet pointer there.
Refuse a slice of a packet where a packet pointer is refused for that
reason. All other helpers take slices as before. bpf_dynptr_write() and
bpf_dynptr_copy() can be used to copy inside of a packet.
Fixes:
|
||
|
Alexei Starovoitov
|
bb72c28668
|
bpf: Unclone the skb for kfuncs that write through a dynptr
bpf_dynptr_copy(), bpf_dynptr_memset() and the kfuncs that read user or kernel memory into a dynptr ask bpf_dynptr_slice_rdwr() for a slice and write to it. For the linear part of an skb that is a pointer into its head. bpf_dynptr_slice_rdwr() counts on the verifier to have uncloned the skb, see bpf_unclone_prologue(). The verifier does so if the program itself calls bpf_dynptr_slice_rdwr() or writes to the packet. A program that only uses these kfuncs gets no prologue, and changes the data of all the clones of the skb. A packet socket, whose clone is taken before tc ingress runs, sees what bpf_dynptr_memset() has written there. Treat a call that may write to an skb like a direct write. Fixes: |
||
|
Alexei Starovoitov
|
aa486cff49
|
bpf: Refuse the prologue for an extension that cannot take it
A tc, lwt_xmit or sk_skb program that writes to packet data starts with
bpf_unclone_prologue(). It takes the skb from R1, and if the skb is cloned
it calls bpf_skb_pull_data() and restores R1 only.
That suits the main function of a program. An extension gets the prologue
too, but it starts with the arguments of the function that it replaces.
If the first one is not the context, something else is taken for an skb:
BUG: KASAN: stack-out-of-bounds in skb_ensure_writable+0x84/0x1a0
Read of size 8 at addr ffffc90000270ad0 by task uncl/105
skb_ensure_writable+0x84/0x1a0
bpf_skb_pull_data+0x32/0xf0
bpf_prog_853eb1991d35e093_ext_slice+0x4f/0x321
bpf_prog_9eb18752b5d0959a_t_glob+0x2ac/0x2de
BUG: unable to handle page fault for address: 0000000082298821
And whatever R2-R5 held is lost, while the verifier goes on trusting
pointers in them.
Refuse an extension that needs a prologue unless it gets the context as
its first argument and scalars beside it. A scalar is still clobbered if
the skb is cloned, which the verifier can cope with. Extensions of XDP,
sk_msg and sockopt programs get no prologue and load as before.
Fixes:
|