mirror of
https://github.com/torvalds/linux.git
synced 2026-11-07 06:17:02 +01:00
v7.4-rc1
5926 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Linus Torvalds
|
46fec1c302 |
Scheduler fix:
- Fix a false positive cputime related warning in kcpustat_dyntick_stop()
that can trigger sporadically after a clocksource switch (Stian Halseth)
Signed-off-by: Ingo Molnar <mingo@kernel.org>
-----BEGIN PGP SIGNATURE-----
iQJFBAABCgAvFiEEBpT5eoXrXCwVQwEKEnMQ0APhK1gFAmrm5OsRHG1pbmdvQGtl
cm5lbC5vcmcACgkQEnMQ0APhK1hYJQ//QXUMx7mhToCfl06aAQGKONCSbM/o40Kl
N775KWhpcWXoJEJ5GvwJzs/c9/jplA6MzWv8m8ff6dPpq57vqa3GhdoDjR0w5mB9
vxxTxYFkYgRcg3pkh7TbJ6olaYfzZTg6HyxUDqxhrSPHt4tgK+izCbM5JmSrsxgN
yDDogrG5KfTKF43SE2whVFhS86L4Jlyq3ctD7ka8PqAglu+He4D0sc6y9seAFS9E
QXNCi4tlgAccK+iDTO1jU23InZ856smMqkaGSLoXDqkMfuJe/dunDWD0nU0dYcCY
gLSmHNAf/e73cexnMEdvYE4XkOb7mOd31zCmpLpxfyit5J7XGfoIzZ7kKoqx3OQp
o8I98DVd+HU9Qw8JoEnaxcBurnPE4jsM0ZGZExbGPlIVIiny9uusVQYJfOCnTfAg
P5gB2Z0r5SzzECE2Tw13oZfFWVkBvvaXutk3Ps/t0Yp01Zhac7FihmAFMJWMfM0H
fgAOrd749DZG9976Tnj2dxpZtveOPN90fkpJRiTH12SC+NHumbnoY7MmPljD+QEa
gqxecbpfW+CpxHjypmH5GTYBTwuBhKTl3YVItRO5wegS21g4CkpigukHaExpgOhn
8o/UILjHOSY/8/YcvzraKctDh/JguUzbkffIuxG6TBczLfPlYK6pz4C1uyxa46rS
MkQkuZaennc=
=hgbA
-----END PGP SIGNATURE-----
Merge tag 'sched-urgent-2026-11-01' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
Pull scheduler fix from Ingo Molnar:
- Fix a false positive cputime related warning in kcpustat_dyntick_stop()
that can trigger sporadically after a clocksource switch (Stian Halseth)
* tag 'sched-urgent-2026-11-01' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip:
sched/cputime: Don't warn when the tick period starts after idle exit
|
||
|
Linus Torvalds
|
7a3d2740d1 |
Misc timer fixes:
- Fix on-stack hrtimer cleanup in play_idle_precise(), and
add the hrtimer_on_stack cleanup class to facilitate the fix
(Hui Su)
- Fix clock_getres() system call's handling of NULL output
pointer in the !CONFIG_POSIX_TIMERS case (Hungyu Lin)
- Fix get_cpu_itimer() to return TICK_NSEC when the sampled CPU time
is exactly equal to the nonzero expiry (Hungyu Lin)
- Fix do_timens_ktime_to_host() time offset subtraction underflow
check (Hungyu Lin)
- Report zero remaining time for disarmed SIGEV_NONE posix timers,
and add a self-test to detect this bug in the future (Hung-Jui Lin)
- Fix NOHZ 'full kick' handling bug that can cause missed
tick updates (Peng Wang)
- Fix field access bug in the vdso/gettimeofday code that works
only by accident (Thomas Weißschuh)
- Fix & prevent immediate clocksource watchdog rerun after a delay
(Karl Mehltretter)
Signed-off-by: Ingo Molnar <mingo@kernel.org>
-----BEGIN PGP SIGNATURE-----
iQJFBAABCgAvFiEEBpT5eoXrXCwVQwEKEnMQ0APhK1gFAmrm49IRHG1pbmdvQGtl
cm5lbC5vcmcACgkQEnMQ0APhK1iTDA/5Aebi9AA+RG07pniYhgx8D47I+mEB+5EG
Jthzq2OZ2rdH4IlI+YKN4KsQNnksNEkXzFmOe4KjVdogfr4JJqJ3UOv0HeNSaOtK
2MD68CswEOZQR9L/8X6VaVoCZp3TkoRP6OPU0DSJeD/ELU8xeKhyBlerA0isgluU
Oc9fePsQniN3zay7QkfViBM5D1D9SA54rp8TPaq9ZxstLqPFjzFcMtFLILm1syeZ
vnsCz1dX5sUkEKkcDClcv3gp8LL7l3r+M+z7MuplO7NYYqCFzD/DFodKVIVMBr1u
pd9XuGHo+6qF9+WXzC3wlLd6ZauC0gSFqOGdqeQoiQrOwt2xxNNFtr2NvHPtCmOL
2014sHRk2KxZP1Hc7VvwGRD/fSTrkTYlQMXkuP5zQMzlpSiWLzmffH5Y2RohS6BB
NyEw7bI9LaQ1zUatuYPkwKKcZQNhM6uaUIjfWvCIQYC0WmgIhN2r+4gTDc4jenU4
6YDnxmE3dyTHlPgksBViDN/p6sNszeoq/KgRSRuM8H7VD6TlKgcHEmGXF3LhgAtY
U7tfQ63ReDWuj2KmNyhT1lP0kBEJKbTouXn3CuGvtsmupLR+gmgZ13p8iPPpqgbO
P8GH+5l2Cz9SieFbJqoXZ5D8dOEB5sVjAden7iZol2ALjIlWySDPaFWFJjqsDUfy
Lmfu0V5K5ZU=
=WzWb
-----END PGP SIGNATURE-----
Merge tag 'timers-urgent-2026-11-01' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
Pull timer fixes from Ingo Molnar:
- Fix on-stack hrtimer cleanup in play_idle_precise(), and
add the hrtimer_on_stack cleanup class to facilitate the fix
(Hui Su)
- Fix clock_getres() system call's handling of NULL output
pointer in the !CONFIG_POSIX_TIMERS case (Hungyu Lin)
- Fix get_cpu_itimer() to return TICK_NSEC when the sampled CPU time
is exactly equal to the nonzero expiry (Hungyu Lin)
- Fix do_timens_ktime_to_host() time offset subtraction underflow
check (Hungyu Lin)
- Report zero remaining time for disarmed SIGEV_NONE posix timers,
and add a self-test to detect this bug in the future (Hung-Jui Lin)
- Fix NOHZ 'full kick' handling bug that can cause missed
tick updates (Peng Wang)
- Fix field access bug in the vdso/gettimeofday code that works
only by accident (Thomas Weißschuh)
- Fix & prevent immediate clocksource watchdog rerun after a delay
(Karl Mehltretter)
* tag 'timers-urgent-2026-11-01' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip:
clocksource: Prevent immediate watchdog rerun after a delay
vdso/gettimeofday: Correct the read of tz_minuteswest and tz_dsttime
tick/nohz: Reevaluate the tick for kicks flushed outside hardirq
selftests: timers: Add a test for disarmed SIGEV_NONE timers
posix-timers: Report zero remaining time for disarmed SIGEV_NONE timers
time/namespace: Fix overflow when converting absolute deadlines
timers/itimer: Keep a nonzero value at CPU timer expiry
posix-timers: Accept NULL clock_getres() output in stubs
sched/idle: Use hrtimer cleanup class in play_idle_precise()
hrtimer: Add cleanup class for on-stack timers
|
||
|
Stian Halseth
|
b327e63683 |
sched/cputime: Don't warn when the tick period starts after idle exit
kcpustat_dyntick_stop() warns if idle ends before the tick period
that the next tick will account has started. That happens on x86 in
a CPU's first idle period with the tick stopped, right after the
switch from the jiffies clocksource to tsc-early.
On the jiffies clocksource ktime only moves at the tick. When the
clocksource switches some time d after a tick, that d is lost: ktime
continues from the value of the last tick. The next tick comes
TICK_NSEC after the last one, but ktime has only moved TICK_NSEC - d,
while tick_periodic() still adds a full TICK_NSEC to tick_next_period,
which is now d ahead of ktime. When NOHZ starts,
tick_init_jiffy_update() rounds it up to a multiple of TICK_NSEC and
the sched timer is armed there, up to nearly two ticks after now. A
tick stop saves that as ts->last_tick, and an idle exit before now
reaches ts->last_tick - TICK_NSEC sees the tick period in the future.
Nothing is accounted twice then. The first tick covers a period that
starts after the idle exit, and the code already handles it. Drop the
warning.
Fixes:
|
||
|
Linus Torvalds
|
729beb5aa8 |
sched_ext: Fixes for v7.4
This is a big pull for fixes, but much of it was in the works before the
merge window. Most of these fixes are in sub-scheduler support, which nests
BPF schedulers under a parent that hands them CPUs. It was heavily reworked
over recent cycles and is now functionally complete. Porting real schedulers
on it drove the latest additions and keeps finding corner cases. For
example, once CPUs are shared and can be taken back, the kernel must keep
every task moving when its scheduler can't run it or is failing, and those
fallbacks interact subtly with each other and with other new core features.
- The fallback paths meant to keep a task moving when its scheduler can't
run it or is failing could themselves leave it stuck.
- The stall watchdog could blame a scheduler for time it wasn't responsible
for, even taking down all of sched_ext.
- Sub-schedulers were told late, or not at all, which CPUs they could use.
- Deferred callbacks could stay queued or be dropped under core scheduling
and proxy execution, with a sched/core fix to the duplicate check.
- Hardening: tracing programs can no longer call kfuncs that could deadlock
the CPU.
-----BEGIN PGP SIGNATURE-----
iIQEABYKACwWIQTfIjM1kS57o3GsC/uxYfJx3gVYGQUCauEJww4cdGpAa2VybmVs
Lm9yZwAKCRCxYfJx3gVYGah7AQD1rUZbNZrXHvV1cJ/WpB9UB6Ex05n4s5hDEbmZ
uMmCWgEAz3zfoly/Tbo4VvUPNVAyWj3gfpIwIPpwCklfODhIjAo=
=rDNB
-----END PGP SIGNATURE-----
Merge tag 'sched_ext-for-7.4-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/sched_ext
Pull sched_ext fixes from Tejun Heo:
"This is a big pull for fixes, but much of it was in the works before
the merge window.
Most of these fixes are in sub-scheduler support, which nests BPF
schedulers under a parent that hands them CPUs. It was heavily
reworked over recent cycles and is now functionally complete. Porting
real schedulers on it drove the latest additions and keeps finding
corner cases.
For example, once CPUs are shared and can be taken back, the kernel
must keep every task moving when its scheduler can't run it or is
failing, and those fallbacks interact subtly with each other and with
other new core features.
- The fallback paths meant to keep a task moving when its scheduler
can't run it or is failing could themselves leave it stuck
- The stall watchdog could blame a scheduler for time it wasn't
responsible for, even taking down all of sched_ext
- Sub-schedulers were told late, or not at all, which CPUs they could
use
- Deferred callbacks could stay queued or be dropped under core
scheduling and proxy execution, with a sched/core fix to the
duplicate check
- Hardening: tracing programs can no longer call kfuncs that could
deadlock the CPU"
* tag 'sched_ext-for-7.4-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/sched_ext: (34 commits)
sched_ext: Reject scx_bpf_sub_grant/revoke() in tracing programs
sched_ext: Fix the rescue overload pick ejecting the wrong sub under bypass
sched_ext: Fix use-after-free of a NONE rescuee leaving its sched while running
sched_ext: Fix a lone IMMED task waiting behind a kept prev
sched_ext: Fix queued bypassed tasks stalling when the host loses SCX_CAP_BASE
sched_ext: Let the bypass load balancer move tasks homed on other cpus
sched_ext: Factor out bypass_lb_move() and insert with the donor DSQ unlocked
sched_ext: Fix the bypass load balancer stranding tasks on cpus where the host lacks SCX_CAP_BASE
sched_ext: Fix bypassed tasks stalling when the host lacks SCX_CAP_BASE
sched_ext: Expire prev's slice in dispatch_one() under bypass
sched_ext: Fix a cpu idling after put_prev_task_scx() queues its task on the local DSQ
sched_ext: Refuse extending a kernel slice
sched_ext: Mark the rescue grant as a kernel slice with SCX_TASK_KERNEL_SLICE
sched_ext: Fix a cpu idling with a runnable rescuee
sched_ext: Restore a rescuee's slice after ops.stopping()
sched_ext: Fix rescue served time disagreeing with the slice under proxy execution
sched_ext: Move refill_task_slice_dfl() below apply_slice_vtime()
sched_ext: Fix bypassed tasks looping on a host without SCX_CAP_ENQ
sched_ext: Fix bypassed tasks stranded by the reenqueue limit
sched_ext: Clear holding_cpu where the claim is confirmed
...
|
||
|
Linus Torvalds
|
8a36f53fd5 |
mm.git review status for linus..mm-stable
Everything:
Total patches: 213
Fixes: tags: 69
cc:stable patches: 43
Reviews/patch: 2.04
Reviewed rate: 86%
Excluding DAMON:
Total patches: 192
Fixes: tags: 63
cc:stable patches: 39
Reviews/patch: 2.18
Reviewed rate: 89%
Excluding DAMON and selftests:
Total patches: 182
Fixes: tags: 57
cc:stable patches: 39
Reviews/patch: 2.23
Reviewed rate: 89%
Summary of series in this merge:
- "mm: make VMA flag semantics explicit, eliminate VM_SPECIAL"
(Lorenzo Stoakes (ARM)):
Replace VM_SPECIAL with helpers describing mapping ownership and
behavior. Enforce the mmap flag rules and convert affected drivers,
removing ambiguous tests shared by driver mappings and ordinary memory.
- "support kselftest on nommu platform" (Hajime Tazaki):
Add opt-in mmap and mremap tests for NOMMU systems. Teach the runner to
avoid its fork-dependent timeout command and document the build and
runtime requirements.
- "mm/collapse: separate a collapse from its callers"
(Kiryl Shutsemau (Meta)):
Separate huge-page collapse from caller policy. Replace khugepaged
identity checks with explicit permissions, and separate scanning from
collapse with the mmap_lock drop in one place.
- "mm: implement and use vma_has_anon_rmap(), silence KCSAN"
(Lorenzo Stoakes (ARM)):
Provide a common helper for testing whether a VMA has anonymous reverse
mappings. Document the locking rules, annotate intentional concurrent
reads for KCSAN, and convert the core MM callers.
- "mm/sparse: remove SECTION_MARKED_PRESENT and further cleanups"
(David Hildenbrand (Arm)):
Remove SECTION_MARKED_PRESENT in favor of the early-section flag.
Simplify sparse memory initialization, section metadata handling and
related PFN validity checks.
- "sh: remove NUMA and SPARSEMEM support" (Mike Rapoport (Microsoft)):
Remove SuperH NUMA and SPARSEMEM support. The extra memory banks have
been unusable since the NO_BOOTMEM conversion, leaving only the code and
data overhead.
- "mm: Unify device DAX and HugeTLB vmemmap population paths"
(Muchun Song):
Unify the generic device DAX and HugeTLB vmemmap population paths and
shared tail-page handling. Remove the duplicate DAX machinery while
retaining the powerpc radix implementation.
- "kselftest: mm: fix some failure of split_huge_page_test" (Yeoreum Yun):
Fix huge-page split tests racing with khugepaged recollapse. Use pagemap
and page flags instead of VMA-boundary assumptions which fail when glibc
has already enabled THP.
- "memcg: move memcgid refcount to objcg to unpin dying memcgs"
(Bingfang Guo):
Let dying memcgs be freed while swapped-out shmem still holds their IDs.
Move ID references to reparentable objcgs, with the necessary swap
accounting and RCU lifetime changes.
- "mm/truncate: fix data loss when truncating straddling large folios"
(Zhang Yi):
Preserve out-of-range data when truncating straddling large folios. Fix
a lookup race and split-failure handling, and handle minimum folio orders
for the forthcoming ext4 iomap buffered I/O path.
- "mm/damon: introduce damos quota goal target metric complement flag"
(SJ Park):
Let DAMOS quota goals use a metric's complement, so utilization can also
express free-memory goals without adding separate metrics. Add the sysfs
interface, documentation and tests.
- "arm64/mm: Standardize printing for pgtable entries" (Anshuman Khandual):
Use the generic formatter for arm64 page-table entries, ready for future
wider entries. Move the safe-PTE-update check out of the header to avoid
an include cycle.
- "mm: kmsan: fix iounmap metadata teardown" (Dima Koziuk):
Fix KMSAN metadata leaks on iounmap and failed ioremap allocations. Free
backing pages before removing their mappings, and share the teardown with
partial-allocation cleanup.
- "mm/damon/core: preserve quota state when constructing schemes"
(Karl Mehltretter):
Preserve running DAMOS quota state while validating parameter updates.
Even rejected updates could reset accounting and exceed the quota window;
add tests for both accepted and rejected updates.
- "mm/vmalloc: fix vmalloc_dump_obj VA lookup" (Ye Liu):
Fix vmalloc_dump_obj() lookups for last-page addresses and allocations
spanning vmap zones. These allocations were being missed in diagnostic
output.
- "mm: various (urgent?) fixes" (David Hildenbrand (Arm)):
Fix mmap charge leaks and hugetlb use-after-free, wrong-address
unmapping, stuck locks and endless faults. Also preserve reservations on
failed mremap and handle stale userfaultfd bits.
- "maple_tree: fix maple_range_64 crashes in RCU mode" (Dimitris Charisis):
Fix pointer and metadata confusion during RCU destruction of
maple_range_64 nodes, avoiding double frees and invalid accesses. Add a
regression test; no in-tree user currently uses this combination.
- "mm: fix hugetlb NR_HUGETLB accounting on folio migration" (Hongfu Li):
Fix hugetlb migration and preallocation accounting so /proc/vmstat
retains the correct count and memory.numa_stat follows the folio to its
new node.
- "mm/damon: fix a minor bug, clean code, and add a test" (SJ Park):
Fix 32-bit DAMON age overflow which made old regions look young. Share
hugetlb folio acquisition code and add a regression test for the
previously fixed last-region skipping bug.
- "mm/damon/core: cleanup code, reduce stack usage, and add kunit"
(SJ Park):
Reduce kdamond stack usage and remove unused enums. Add deterministic
PSI quota-goal tests which do not depend on the machine's current memory
pressure.
- "mm/damon: fix the temporal goal tuner's size quota conversion"
(Donggeun Yoo):
Bound the temporal tuner's quota conversion to prevent overflow and zero
effective quotas, which could stop schemes making progress. Add boundary
tests.
- "Don't use GFP_DMA when calling dma_alloc_coherent" (Baoquan He):
Remove redundant GFP_DMA flags from coherent DMA allocation calls. The
DMA API already selects the zone from device limits and ignores the
caller's flag.
- "kselftest: mm: fix intermittent failure khugepaged test" (Yeoreum Yun):
Propagate child-process failures in the khugepaged tests and prevent
background collapse from racing their checks. This removes false passes
and intermittent failures.
- "MAINTAINERS: cover all hugetlb headers under HUGETLB SUBSYSTEM"
(Enze Li):
Cover all hugetlb headers in MAINTAINERS so get_maintainer.pl finds the
right maintainers. Also label the closing hugetlb_cgroup.h include
guard.
Summary of the 65 singletons in this merge:
The singletons include a few worthwhile performance fixes. Avoiding
unnecessary reverse-map tree updates gave a reported 15.9% improvement
in UnixBench's execl test on a 192-core x86 system. A reclaim retry fix
stops defrag_mode from spinning under memory.low protection; a reported
file-creation stress test ran 5.7 times faster, with far less memory
pressure. Narrower hugetlb invalidations also spare KVM guests needless
refaults: the reported Redis-in-VM P99 QPS penalty while punching holes
fell from 7.09% to 1.45%.
Several fixes prevent lost data or corrupted memory. Lazyfree huge-page
races with mremap()/munmap() could lose data or access freed memory, and
MADV_COLD could lose a concurrent write's dirty bit. Another fix
prevents stale migration entries from outliving large folios. On
filesystems with blocks smaller than a page, extending a file by 4 GiB
or more could expose bytes written past the old EOF through mmap instead
of zeroes. That is fixed, along with use-after-free races in
PAGEMAP_SCAN during process exit and in memory-cgroup accounting on
PREEMPT_RT kernels.
Other fixes stop UFFDIO_MOVE spinning forever on pinned large folios,
which could wedge other memory operations, and stop mremap() leaking
commit charges until allocations fail under strict overcommit. A further
fix keeps userfaultfd tracking intact on neighboring mappings. NUMA
binding no longer prevents long-term pins when the chosen node has only
movable memory, fixing QEMU/VFIO startup and io_uring buffer
registration failures. MGLRU also gets a fix for early-boot crashes with
memory cgroups disabled at boot, notably in small crash kernels.
-----BEGIN PGP SIGNATURE-----
iHUEABYKAB0WIQTTMBEPP41GrTpTJgfdBJ7gKXxAjgUCat66sAAKCRDdBJ7gKXxA
jrQrAP99kgSXRdlmRZbHeRBHQ6KvI4XnLvQ78MDIzZ3zmBaOugEA5jiIsK2/snFD
3VaV0oipVZpuwJkVJE6zFkh0tcHgTAo=
=DHrN
-----END PGP SIGNATURE-----
Merge tag 'mm-stable-2026-10-25-19-27' of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm
Pull more MM updates from Andrew Morton:
- "mm: make VMA flag semantics explicit, eliminate VM_SPECIAL" (Lorenzo
Stoakes):
Replace VM_SPECIAL with helpers describing mapping ownership and
behavior. Enforce the mmap flag rules and convert affected drivers,
removing ambiguous tests shared by driver mappings and ordinary
memory.
- "support kselftest on nommu platform" (Hajime Tazaki):
Add opt-in mmap and mremap tests for NOMMU systems. Teach the runner
to avoid its fork-dependent timeout command and document the build
and runtime requirements.
- "mm/collapse: separate a collapse from its callers" (Kiryl
Shutsemau):
Separate huge-page collapse from caller policy. Replace khugepaged
identity checks with explicit permissions, and separate scanning from
collapse with the mmap_lock drop in one place.
- "mm: implement and use vma_has_anon_rmap(), silence KCSAN" (Lorenzo
Stoakes):
Provide a common helper for testing whether a VMA has anonymous
reverse mappings. Document the locking rules, annotate intentional
concurrent reads for KCSAN, and convert the core MM callers.
- "mm/sparse: remove SECTION_MARKED_PRESENT and further cleanups"
(David Hildenbrand):
Remove SECTION_MARKED_PRESENT in favor of the early-section flag.
Simplify sparse memory initialization, section metadata handling and
related PFN validity checks.
- "sh: remove NUMA and SPARSEMEM support" (Mike Rapoport):
Remove SuperH NUMA and SPARSEMEM support. The extra memory banks have
been unusable since the NO_BOOTMEM conversion, leaving only the code
and data overhead.
- "mm: Unify device DAX and HugeTLB vmemmap population paths" (Muchun
Song):
Unify the generic device DAX and HugeTLB vmemmap population paths and
shared tail-page handling. Remove the duplicate DAX machinery while
retaining the powerpc radix implementation.
- "kselftest: mm: fix some failure of split_huge_page_test" (Yeoreum
Yun):
Fix huge-page split tests racing with khugepaged recollapse. Use
pagemap and page flags instead of VMA-boundary assumptions which fail
when glibc has already enabled THP.
- "memcg: move memcgid refcount to objcg to unpin dying memcgs"
(Bingfang Guo):
Let dying memcgs be freed while swapped-out shmem still holds their
IDs. Move ID references to reparentable objcgs, with the necessary
swap accounting and RCU lifetime changes.
- "mm/truncate: fix data loss when truncating straddling large folios"
(Zhang Yi):
Preserve out-of-range data when truncating straddling large folios.
Fix a lookup race and split-failure handling, and handle minimum
folio orders for the forthcoming ext4 iomap buffered I/O path.
- "mm/damon: introduce damos quota goal target metric complement flag"
(SJ Park):
Let DAMOS quota goals use a metric's complement, so utilization can
also express free-memory goals without adding separate metrics. Add
the sysfs interface, documentation and tests.
- "arm64/mm: Standardize printing for pgtable entries" (Anshuman
Khandual):
Use the generic formatter for arm64 page-table entries, ready for
future wider entries. Move the safe-PTE-update check out of the
header to avoid an include cycle.
- "mm: kmsan: fix iounmap metadata teardown" (Dima Koziuk):
Fix KMSAN metadata leaks on iounmap and failed ioremap allocations.
Free backing pages before removing their mappings, and share the
teardown with partial-allocation cleanup.
- "mm/damon/core: preserve quota state when constructing schemes" (Karl
Mehltretter):
Preserve running DAMOS quota state while validating parameter
updates. Even rejected updates could reset accounting and exceed the
quota window; add tests for both accepted and rejected updates.
- "mm/vmalloc: fix vmalloc_dump_obj VA lookup" (Ye Liu):
Fix vmalloc_dump_obj() lookups for last-page addresses and
allocations spanning vmap zones. These allocations were being missed
in diagnostic output.
- "mm: various (urgent?) fixes" (David Hildenbrand):
Fix mmap charge leaks and hugetlb use-after-free, wrong-address
unmapping, stuck locks and endless faults. Also preserve reservations
on failed mremap and handle stale userfaultfd bits.
- "maple_tree: fix maple_range_64 crashes in RCU mode" (Dimitris
Charisis):
Fix pointer and metadata confusion during RCU destruction of
maple_range_64 nodes, avoiding double frees and invalid accesses. Add
a regression test; no in-tree user currently uses this combination.
- "mm: fix hugetlb NR_HUGETLB accounting on folio migration" (Hongfu
Li):
Fix hugetlb migration and preallocation accounting so /proc/vmstat
retains the correct count and memory.numa_stat follows the folio to
its new node.
- "mm/damon: fix a minor bug, clean code, and add a test" (SJ Park):
Fix 32-bit DAMON age overflow which made old regions look young.
Share hugetlb folio acquisition code and add a regression test for
the previously fixed last-region skipping bug.
- "mm/damon/core: cleanup code, reduce stack usage, and add kunit" (SJ
Park):
Reduce kdamond stack usage and remove unused enums. Add deterministic
PSI quota-goal tests which do not depend on the machine's current
memory pressure.
- "mm/damon: fix the temporal goal tuner's size quota conversion"
(Donggeun Yoo):
Bound the temporal tuner's quota conversion to prevent overflow and
zero effective quotas, which could stop schemes making progress. Add
boundary tests.
- "Don't use GFP_DMA when calling dma_alloc_coherent" (Baoquan He):
Remove redundant GFP_DMA flags from coherent DMA allocation calls.
The DMA API already selects the zone from device limits and ignores
the caller's flag.
- "kselftest: mm: fix intermittent failure khugepaged test" (Yeoreum
Yun):
Propagate child-process failures in the khugepaged tests and prevent
background collapse from racing their checks. This removes false
passes and intermittent failures.
- "MAINTAINERS: cover all hugetlb headers under HUGETLB SUBSYSTEM"
(Enze Li):
Cover all hugetlb headers in MAINTAINERS so get_maintainer.pl finds
the right maintainers. Also label the closing hugetlb_cgroup.h
include guard.
Summary of the 65 singletons in this merge:
The singletons include a few worthwhile performance fixes. Avoiding
unnecessary reverse-map tree updates gave a reported 15.9% improvement
in UnixBench's execl test on a 192-core x86 system. A reclaim retry
fix stops defrag_mode from spinning under memory.low protection; a
reported file-creation stress test ran 5.7 times faster, with far less
memory pressure. Narrower hugetlb invalidations also spare KVM guests
needless refaults: the reported Redis-in-VM P99 QPS penalty while
punching holes fell from 7.09% to 1.45%.
Several fixes prevent lost data or corrupted memory. Lazyfree
huge-page races with mremap()/munmap() could lose data or access freed
memory, and MADV_COLD could lose a concurrent write's dirty bit.
Another fix prevents stale migration entries from outliving large
folios. On filesystems with blocks smaller than a page, extending a
file by 4 GiB or more could expose bytes written past the old EOF
through mmap instead of zeroes. That is fixed, along with
use-after-free races in PAGEMAP_SCAN during process exit and in
memory-cgroup accounting on PREEMPT_RT kernels.
Other fixes stop UFFDIO_MOVE spinning forever on pinned large folios,
which could wedge other memory operations, and stop mremap() leaking
commit charges until allocations fail under strict overcommit. A
further fix keeps userfaultfd tracking intact on neighboring mappings.
NUMA binding no longer prevents long-term pins when the chosen node
has only movable memory, fixing QEMU/VFIO startup and io_uring buffer
registration failures. MGLRU also gets a fix for early-boot crashes
with memory cgroups disabled at boot, notably in small crash kernels.
* tag 'mm-stable-2026-10-25-19-27' of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm: (213 commits)
mm: multi-gen LRU: fix NULL lruvec->pgdat deref with memcg disabled on cmdline
mm/page_counter: use WRITE_ONCE() for emin/elow when parent is the root
mm/memcontrol: publish null pointer before dropping css reference
test_maple_tree: test maple_range_64 node in RCU mode
maple_tree: fix maple_range_64 crashes in RCU mode
mm/vma: avoid broken uffd merge on mremap
tools/testing/shared: catch a failed posix_memalign()
mm/damon/core: remove unreachable damos charge_{target,addr}_from
mm/shrinker: allocate shrinker_info_unit from a dedicated kmem_cache
riscv: mm: exclude invalid THP PMDs from page table check
mm/damon/core: filter out damos address filter-split failed region
fs/proc/task_mmu: pin mm_users in do_pagemap_scan()
mm/mempolicy: apply bind policies against the scoped allocation zone
mm/vmscan: reclaim every zone in per-node proactive reclaim
mm/mremap: correctly uncharge accountable VMAs in error path
mm/mremap: populate calculated fields in vrm state correctly
MAINTAINERS: add an entry for MEMORY MANAGEMENT - MEMORY MAP
mm: bdi: reset min_ratio after the sysfs attributes are gone
mm/page_vma_mapped: continue past nonmatching huge PMDs
selftests/mm: skip hugetlb cgroup v2 tests without the needed controllers
...
|
||
|
Tejun Heo
|
af4ed5682e |
sched_ext: Reject scx_bpf_sub_grant/revoke() in tracing programs
A tracing program can call scx_bpf_sub_grant() and scx_bpf_sub_revoke() through scx_kfunc_ids_any, which is registered on the tracing kfunc hook. Before returning, both invoke ops.sub_caps_updated() of the sub-schedulers whose caps changed, so those ops run in the tracing program's context. No SCX op expects to run from an arbitrary tracing context. With an rq lock held, this leads to the same deadlock that |
||
|
Tejun Heo
|
b7723dedac |
sched_ext: Fix the rescue overload pick ejecting the wrong sub under bypass
An earlier patch sends a bypassed task to rescue when its host lacks SCX_CAP_BASE on the cpu. The task's rescue time is charged to its owner, the bypassing sub, which is usually exiting. The overload pick skips exiting subs and ejects an unrelated sub or nobody. The host runs the task and lacks BASE, but carries none of the charge. Charge the sched running the rescuee, the host while the owner bypasses. SCX_EV_SUB_RESCUE already counts on that sched and the stall watchdog blames it. Reported-by: Sashiko <sashiko-bot@kernel.org> Signed-off-by: Tejun Heo <tj@kernel.org> Reviewed-by: Andrea Righi <arighi@nvidia.com> |
||
|
Tejun Heo
|
a298fd5f4e |
sched_ext: Fix use-after-free of a NONE rescuee leaving its sched while running
Since an earlier patch sends a bypassed task to rescue when its host lacks SCX_CAP_BASE, a task scx_fail_parent() punted to its bypassing parent at NONE can be a rescuee. When a cgroup migration into a root-owned cgroup re-homes it while running, nothing resets it: the DEQUEUE_SAVE dequeue and put_prev_task_scx() keep a running task's slice, and __scx_disable_and_exit_task() skips a NONE task. rq->scx.rescue.curr keeps pointing at the task, now on the root. An RT task, for example, can keep the task from serving its grant until the last sub is freed. From then on scx_rescuee() returns NULL and the rescue never ends. scx_rescue_timerfn() keeps accessing the task after it is freed. Move the reset out of scx_disable_task() into scx_reset_task_fields() and call it on every path that takes a task out of its sched's control, the exit of a NONE task included. Reported-by: Sashiko <sashiko-bot@kernel.org> Signed-off-by: Tejun Heo <tj@kernel.org> Reviewed-by: Andrea Righi <arighi@nvidia.com> |
||
|
Tejun Heo
|
5173b29658 |
sched_ext: Fix a lone IMMED task waiting behind a kept prev
While dispatch is in progress, rq_is_open() returns true and dsq_inc_nr()
schedules no reenqueue for an IMMED task inserted on an empty local DSQ. The
check at the end of dispatch_one() schedules the reenqueue only when more
than one task is queued, so a dispatch that keeps prev leaves a lone IMMED
task behind:
1. a parent's ops.dispatch() sub-dispatches into a child, which consumes a
task with SCX_ENQ_IMMED onto the empty local DSQ
2. the parent extends prev's slice and dispatch keeps prev
3. the local DSQ holds one task, so no reenqueue is scheduled
The IMMED task waits on the local DSQ behind prev.
Schedule the reenqueue when dispatch keeps prev and an IMMED task is queued,
as even a lone IMMED task can't run right away behind prev.
v2: The description and dispatch_one()'s comment are reworded.
Fixes:
|
||
|
Tejun Heo
|
5c23d17106 |
sched_ext: Fix queued bypassed tasks stalling when the host loses SCX_CAP_BASE
A task already queued on a bypass DSQ stalls when the host loses
SCX_CAP_BASE on that cpu, as the cpu no longer consumes the DSQ. Sending a
bypassed task to rescue at enqueue time covers only a task enqueued after
the loss, and the balancer no longer donates to a capless cpu.
Have the bypass load balancer drain such a DSQ: a cpu where the host lacks
BASE is a donor with a target of zero, and the donees take its tasks under
their ordinary fill.
A drained task no donee can take is reenqueued at its home: to rescue when
the host lacks BASE there and rescue is enabled, and to the home's bypass
DSQ otherwise.
A balancing donor leaves a task that can't move to its donee where it is.
Reenqueueing it at home would put a task homed on the donor cpu at the tail
of its own DSQ, and a blocked task that proxy execution migrated there,
which every remote cpu refuses, would go to the tail on every pass.
Without rescue, a task homed on the drained cpu goes back onto the drained
DSQ, once per pass, and waits for the host to regain BASE, the bypass to
end, or the stall watchdog to eject the host. Its reenqueues are not counted
against the reenqueue limit, as its scheduler is bypassing.
A BASE loss kicks the balancer so that the drain doesn't wait for the next
interval. With a zero interval the kick is the only pass, and a task that
lands on the drained DSQ after it waits the same way.
v2: The description drops an in-flight affinity change from the causes of a
refused balancer move.
v3:
- The description says that a stuck task's reenqueues are not counted
against the reenqueue limit (sashiko).
- The description names the stall watchdog as the way out for a task stuck
without rescue, and the tasks a zero interval leaves on a drained DSQ.
- bypass_lb_reenq_home()'s kerneldoc and the description give the placement
with rescue disabled and for a task that must run at home, and the other
two comments on that placement point to the kerneldoc.
v4:
- Skip a task the donee would refuse on a balancing donor instead of
reenqueueing it at home (Andrea).
- bypass_lb_reenq_home()'s kerneldoc is corrected on a migration-disabled
task homed on a donee and on the local DSQ admission.
- The description is condensed.
Fixes:
|
||
|
Tejun Heo
|
a043733588 |
sched_ext: Let the bypass load balancer move tasks homed on other cpus
The following patch uses the bypass load balancer to drain a cpu's bypass DSQ when the host loses SCX_CAP_BASE there. The host no longer consumes that DSQ, so the drain has to move every task on it. The balancer skips the ones homed on another cpu. An earlier pass moved such a task in from its home cpu's DSQ, and the donor cpu has not consumed it yet. A task's DSQ membership may only change under the task's rq lock, which the balancer does not hold. Take such a task off the donor DSQ with unlink_dsq_and_switch_rq_lock(), the handshake a remote consumer uses. It drops the DSQ lock and switches to the task's rq lock. A dequeue that gets in between takes the task, and the balancer moves on. The insert then runs under the task's home rq lock, as it already does for a donor-homed task. A task whose donee is its own cpu can always run there. task_can_run_on_remote_rq() warns when asked about a task's own cpu, so the balancer skips the test for such a task. The walk holds the donor rq lock, which keeps the donor cpu's dispatch and its enqueuers away from the donor DSQ. A move of a task homed elsewhere switches to the task's rq lock, and both can reach the DSQ meanwhile. The parked cursor keeps the walk's place through a consume, and the sequence bound keeps a task enqueued meanwhile out of the pass. v2: The description says "the following patch", not "patches", uses the balancer to drain a DSQ. Signed-off-by: Tejun Heo <tj@kernel.org> Reviewed-by: Andrea Righi <arighi@nvidia.com> |
||
|
Tejun Heo
|
f0bbe21d73 |
sched_ext: Factor out bypass_lb_move() and insert with the donor DSQ unlocked
Prepare the bypass load balancer for moving tasks homed on other cpus. Such a move takes the task's own rq lock through the remote consumer's handshake, which drops the donor DSQ lock on the way, so its insert runs with the donor DSQ unlocked and under the task's rq lock. The balancer's own move holds the donor DSQ lock throughout, with the donee DSQ lock nested inside. Switch the donor-homed move to the same scheme so that one body can serve both: factor it out of bypass_lb_cpu() into bypass_lb_move(), which takes the task off the donor DSQ and inserts with the DSQ unlocked. A task that can't run on the donee goes back to the head of the donor DSQ instead of being skipped in place. The walk parks its cursor behind the task before each move and resumes from it afterwards. It is bounded to the tasks queued when it began, by their DSQ sequence numbers, so a task it put back is not visited again in the same pass. Before, the walk held both the donor rq lock and the donor DSQ lock across every move. The donor DSQ can be very long and contended, a walk competing with the cpu's dispatch and the enqueues for the DSQ lock would crawl, and the walk is what relieves that contention by moving tasks off the DSQ. The walk now cycles the DSQ lock around each insert but still holds the rq lock throughout, and that alone keeps dispatch and enqueues away. The DSQ lock is uncontended when retaken, so cycling it is cheap. Signed-off-by: Tejun Heo <tj@kernel.org> Reviewed-by: Andrea Righi <arighi@nvidia.com> |
||
|
Tejun Heo
|
53cc8dd069 |
sched_ext: Fix the bypass load balancer stranding tasks on cpus where the host lacks SCX_CAP_BASE
The bypass load balancer spreads a host's bypassed tasks over the cpus of a
node by moving them between its per-cpu bypass DSQs. It counts every cpu of
the node as capacity and takes any cpu below the target as a donee. A cpu's
dispatch only descends into schedulers holding SCX_CAP_BASE on it, so a
bypass DSQ on a cpu where the host lacks BASE is never consumed. A task
moved there sits until the host regains the cap or the bypass window ends.
Count only cpus where the host holds BASE as capacity and pick donees from
them. The target rises to spread the node's tasks over the cpus that can run
them, and a capless cpu receives nothing. A capless cpu with tasks is still
an ordinary donor, offloaded down to the donor target when its queue grows
past the threshold; draining it is left to the following patches.
Fixes:
|
||
|
Tejun Heo
|
947dd3eb80 |
sched_ext: Fix bypassed tasks stalling when the host lacks SCX_CAP_BASE
A bypassed task stalls on a cpu where its host lacks SCX_CAP_BASE. The host,
the nearest ancestor that is not bypassing, runs a bypassing sub-scheduler's
tasks through its per-cpu bypass DSQs. A cpu's dispatch only descends into
schedulers holding BASE on it, so the host's bypass DSQ on such a cpu is
never consumed until the host regains the cap.
Resolve a bypassed task against its host at enqueue time. If the host lacks
BASE on the cpu, send the task to rescue on that cpu, which runs it at
bounded bandwidth without the host's dispatch having to reach it.
The enqueue does not look above the host for an ancestor that holds BASE.
That ancestor would run the task on cpus it never granted the host.
With rescue disabled the task goes to the host's bypass DSQ as before and
stalls there until the host regains BASE or the bypass ends, or until the
stall watchdog ejects the host, which it blames for a task stuck on its DSQ.
A task that must run on the cpu, migration-disabled or with a migration
pending, is admitted to the local DSQ instead, as any insert of such a task
is.
Bypass also used to end a rescue, sending a preempted rescuee to the bypass
DSQ with every other preempted task. Now that rescue is how a bypassed task
runs while its host lacks BASE, keep a preempted rescuee on the local DSQ
under bypass too. dispatch_one() also keeps such a rescuee running, as an
earlier patch's bypass expiry leaves a kernel slice alone. The grant stays
bounded by served time.
A task scx_fail_parent() punted to its bypassing parent at NONE can be a
rescuee too. switched_from_scx() ends its slice as it does for an enabled
task, so that a NONE rescuee leaving sched_ext while running doesn't leave
rq->scx.rescue.curr pointing at it.
While at it, the rescue test in __scx_resolve_local_dsq() goes through the
new scx_rescue_enabled() helper.
v2:
- The description notes that dispatch_one() keeps a bypassed rescuee too,
through the new patch before this one (sashiko).
- scx_rescue_keep()'s kerneldoc drops dispatch_one() declining a rescuee
with slice left, which the new patch removes.
v3:
- With rescue disabled, a task that must run on the cpu is admitted to the
local DSQ instead of waiting on the host's bypass DSQ.
- The description names the stall watchdog as the way out for a task stuck
without rescue.
v4:
- switched_from_scx() ends the slice of a NONE task, which can now be a
rescuee (sashiko).
- scx_punt_task()'s kerneldoc no longer says that switched_from_scx() skips
a NONE task.
- scx_bypass()'s kerneldoc and the ops.sub_ecaps_updated() doc say where a
bypassed task can be besides the bypass DSQs.
Fixes:
|
||
|
Tejun Heo
|
b9843b0399 |
sched_ext: Expire prev's slice in dispatch_one() under bypass
The next patch keeps a bypassed rescuee on the local DSQ and needs
dispatch_one() to keep it running too. dispatch_one()'s keep test refuses
any prev whose scheduler is bypassing, so another local task could run ahead
of the rescuee, even an escalated SCX_TASK_PROTECTED one. The test exists
because a cpu-form scheduler can write p->scx.slice directly from BPF, so
the slice can't be trusted under bypass.
The refusal doesn't hold anyway. Since
|
||
|
Tejun Heo
|
20a61e4142 |
sched_ext: Fix a cpu idling after put_prev_task_scx() queues its task on the local DSQ
put_prev_task_scx() enqueues a still-runnable task when the pick has settled on idle. The insert can land the task on the cpu's own local DSQ, and then the cpu must not go idle over it. |
||
|
Tejun Heo
|
8197fa420d |
sched_ext: Refuse extending a kernel slice
A rescuee can keep running after its grant is served, on a cpu its scheduler
has lost. The write rules deny a slice extension only where the scheduler
lacks SCX_CAP_BASE on the cpu, and a scheduler holding BASE but not ENQ can
have a task under rescue there:
1. the scheduler extends the rescuee's slice
2. BASE is revoked. The cap-revoke eviction skips the rescuee.
3. the grant is served and scx_rescue_charge() ends the rescue
4. the task runs out the extension on the cpu its scheduler lost, outside
the rescue's bandwidth
A scheduler shortens or zeroes a rescuee's slice to preempt it, and
scx_rescue_keep() resumes the grant afterwards. An extension serves no such
purpose.
Refuse extending a kernel slice in both scheduler write paths, the direct
write in scx_bpf_task_set_slice() and the out-of-band application, counted
in SCX_EV_SLICE_DENIED like an extension without BASE. Shortening stays
allowed.
scx_rescue_keep()'s restore of the remainder raises the slice from zero.
Like the kernel's other slice writes, it goes through scx_set_task_slice()
and not the scheduler write paths.
v2:
- scx_rescue_admit()'s kerneldoc no longer mentions a served-time snapshot.
- SCX_EV_SLICE_DENIED's comment lists every refusal it counts.
- scx_bpf_task_set_slice()'s kerneldoc says that shortening is refused on a
protected slice.
Fixes:
|
||
|
Tejun Heo
|
98889f2cae |
sched_ext: Mark the rescue grant as a kernel slice with SCX_TASK_KERNEL_SLICE
In the following patches, bypassed tasks run under rescue when their host lacks SCX_CAP_BASE on the cpu. Under bypass the kernel zeroes the running task's slice at every tick, as it does not trust the scheduler's slices. A rescue grant has to survive that, or it is served a tick at a time with the rescuee sent to the tail of the local DSQ each time. The generic slice handling can't tell a kernel grant from a scheduler's slice. Mark the grant as a kernel slice with SCX_TASK_KERNEL_SLICE, set at rescue admission and cleared when the slice ends. The generic slice handling leaves it alone. The default refill does nothing for it, and the three zeroings go through the new expire_task_slice(), which skips it: - the bypass tick's zeroing of the running task - the eviction of a running task that can't stay on the cpu when bypass lifts - the cap-revoke eviction of a running task Schedulers keep their control. Preemption and scx_bpf_task_set_slice() still apply, and the rescue stays bounded by served time. The sites that recognized the rescuee by identity, the reenqueue scan and put_prev_task_scx()'s reinsert, test the flag instead. scx_task_has_kernel_slice() hides the test behind the scx_has_subs() static key, as only a sub-scheduler's task can carry the flag. The flag's entries join the enum tables in tools/sched_ext. No kernel slice reaches a zeroing until bypassed tasks run under rescue. The cap-revoke eviction skipped the rescuee already, and bypass entry ends an open rescue, so the bypass tick and the eviction at bypass lift see none. The refill is the one behavior change. The pick refills a zero slice only for a task that can stay on the cpu, so it concerns a rescuee whose scheduler holds BASE there. A scheduler may zero such a queued rescuee's slice, and the next pick used to refill it to the default. Now the rescuee runs on the zero slice until the next tick. There put_prev_task_scx() restores the remainder when another task is runnable. When nothing else is runnable, dispatch_one() keeps the rescuee as the last task. The pick then ends its zero slice, which ends the rescue, and refills the default. A scheduler with SCX_OPS_ENQ_LAST skips the keep-last test. scx_rescue_keep() restores the remainder there instead and the rescue goes on. v2: - scx_rescue_admit()'s kerneldoc no longer mentions a served-time snapshot. - The description covers a scheduler with SCX_OPS_ENQ_LAST, whose rescuee dispatch_one() keeps on the remainder. Signed-off-by: Tejun Heo <tj@kernel.org> Reviewed-by: Andrea Righi <arighi@nvidia.com> |
||
|
Tejun Heo
|
7837315b88 |
sched_ext: Fix a cpu idling with a runnable rescuee
A scheduler preempts a running rescuee like any other task by zeroing its
slice. The rescue grant survives the preemption. put_prev_task_scx()
restores the unserved remainder through scx_rescue_keep() and queues the
rescuee at the tail of the local DSQ, behind whatever the scheduler wanted
to run first.
That breaks down when the scheduler zeroes the slice but dispatches nothing.
dispatch_one() keeps the previous task running only with slice left or, when
nothing else is runnable, with its scheduler holding SCX_CAP_BASE on the cpu
and not taking the last task through SCX_OPS_ENQ_LAST. A rescuee fails both
once its slice is zeroed if its scheduler lacks BASE there or takes the last
task:
1. ops.tick() or scx_bpf_task_set_slice() zeroes the rescuee's slice
2. the pick finds nothing else to run and settles on idle
3. put_prev_task_scx() restores the remainder and queues the rescuee on the
local DSQ
4. nothing reschedules the cpu, as the pick has settled and __schedule()
clears need_resched right after it
The cpu idles with the rescuee queued until an unrelated wakeup or the
rescue timer's escalation.
Restore the remainder from dispatch_one() too. When nothing else is
runnable, its last keep now keeps the rescuee running on the remainder, so
the pick never settles on idle with a rescuee runnable. put_prev_task_scx()
keeps handling a displaced rescuee, where another task won the pick.
A rescuee that passes the existing keep-last test is kept there instead and
continues as an ordinary task, its rescue ended with the zeroed slice. That
is fine as its scheduler holds SCX_CAP_BASE on the cpu.
The new keep also covers bypassed tasks run under rescue in the following
patches.
v2: The dispatch_one() comment and description now say that a rescuee whose
scheduler holds SCX_CAP_BASE on the cpu passes the keep-last test and
continues as an ordinary task (sashiko).
Fixes:
|
||
|
Tejun Heo
|
3ce6316379 |
sched_ext: Restore a rescuee's slice after ops.stopping()
put_prev_task_scx() restores a preempted rescuee's unserved slice through
scx_rescue_keep() before it calls ops.stopping(). A rescuee whose slice the
scheduler zeroes there is left to the reinsert instead:
1. ops.stopping() zeroes the rescuee's slice
2. the reinsert condition special-cases the rescuee and keeps it on the
local DSQ with a zero slice
3. the pick's default refill supplies a slice, or the task runs on a zero
slice until the tick when its scheduler lacks BASE
Move the keep after the stopping call so that scx_rescue_keep() restores the
remainder whether a preemption or ops.stopping() zeroed the slice, and drop
the special case from the reinsert condition as a rescuee reaching it always
has a slice. A rescuee whose slice ops.stopping() zeroed now goes to the
tail of the local DSQ with its remainder restored, behind what the scheduler
wanted to run first, instead of staying at the head on a zero slice.
The slice end in the same block moves with it, which changes nothing for a
protected task. Its slice is the unserved remainder, and scx_rescue_charge()
ends the protection in the same update_curr_scx() call that zeroes the
slice, so the block never sees a protected task in either position.
v2: The description no longer says that a protected task's protection
outlasts ops.stopping().
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
|
||
|
Tejun Heo
|
006e9e9336 |
sched_ext: Fix rescue served time disagreeing with the slice under proxy execution
Rescue measures served time as the rescuee's sum_exec_runtime minus a
snapshot taken at admission. Under proxy execution that measure follows the
wrong task. update_se() credits sum_exec_runtime to rq->curr, the mutex
owner that executes, while update_curr_scx() drains the slice of rq->donor
and charges its rescue. A rescuee can be either, so its served time and its
slice disagree in both directions.
As the owner running for another donor, the rescuee accrues served time that
drained no slice and charged no budget, and its rescue ends early. As the
donor, its rescue runs long:
1. the rescuee blocks on a mutex and the owner runs in its place
2. the owner's run drains the rescuee's slice and the budget, but not the
served time
3. the slice runs out with the rescue still open, and scx_rescue_keep()
restores a remainder that the owner's run didn't reduce
4. the owner keeps running on the rescue until it releases the mutex, past
the admitted slice and overdrawing the budget
Count served time in scx_rescue_charge() from the delta that drains the
slice. Charging the owner's run to the donor's rescue matches proxy
execution, which drains the donor's slice and accounts the cgroup time to
the donor for the owner's run.
scx_rescue_keep() also ignored the return of scx_set_task_slice(). The call
refuses the restore on a rescuee that escalation marked SCX_TASK_PROTECTED,
and the rescuee was kept with a zero slice and its protection set.
That refusal is now unreachable. Escalation sets the slice to the unserved
remainder before it protects the task. From then on every slice write a
scheduler can reach is refused. Only a sub-scheduler's task is rescued, and
sub-schedulers are cid-form, which rejects direct slice writes.
The slice then moves only by what update_curr_scx() drains, and that drain
charges the rescue by the same amount, so the slice runs out exactly when
the charge ends the rescue. WARN and end the rescue if the restore is
refused anyway.
Fixes:
|
||
|
Tejun Heo
|
867d0ac6d7 |
sched_ext: Move refill_task_slice_dfl() below apply_slice_vtime()
Move refill_task_slice_dfl() below apply_slice_vtime(), with the other slice writers, to prepare for the kernel-slice handling the following patches add to them. No functional change. Signed-off-by: Tejun Heo <tj@kernel.org> Reviewed-by: Andrea Righi <arighi@nvidia.com> |
||
|
Tejun Heo
|
a676ada0d6 |
sched_ext: Fix bypassed tasks looping on a host without SCX_CAP_ENQ
Bypass exists so that runnable tasks make progress without trusting the BPF
scheduler. A host with SCX_CAP_ENQ_IMMED (BASE) but not SCX_CAP_ENQ on a cpu
cannot run a single bypassed task there. The task loops instead:
1. a bypassed task lands on the host's bypass DSQ for the cpu
2. the cpu's dispatch descends into the host, which holds BASE there
3. the host consumes the task into the local DSQ as a plain insert
4. the insert needs ENQ and the host lacks it, so the task is rejected
5. the reject drain reenqueues the task and it is back at step 1
A bypass consume puts a task on the cpu being dispatched for, to run right
away. That is the IMMED contract, and BASE is the cap it needs. Consume with
SCX_ENQ_IMMED. BASE is then the one cap a bypassed task ever needs. Dispatch
descends into a host only on a cpu where the host holds BASE, so a consume
never lacks it.
A consumed task is first on the local DSQ when dispatch enters the host with
the DSQ empty. That is always the case for the root and normally for a
parent dispatching into a child.
A consumed task that cannot run right away is reenqueued like any IMMED task
and goes back to the bypass DSQ. Waiting on the local DSQ would need ENQ.
The task cannot run right away when a higher class or a HEAD insert took the
cpu meanwhile, or when the parent queued its own task before dispatching
into the host.
Each such task waits on the bypass DSQ for the next dispatch. The reenqueue
is not counted against the reenqueue limit, as the task's scheduler is
bypassing.
v2: The description says that the reenqueue of a consumed task is not
counted against the reenqueue limit (sashiko).
Fixes:
|
||
|
Tejun Heo
|
40c9ad7de8 |
sched_ext: Fix bypassed tasks stranded by the reenqueue limit
scx_do_enqueue_task() counts a task's reenqueues until the task runs. Past
SCX_REENQ_MAX_REPEAT it ejects the owning scheduler and leaves the task off
every DSQ for the scheduler's exit to requeue. The count comes before the
bypass test, so it also counts the reenqueues of a task whose scheduler is
bypassing.
Those reenqueues are not the scheduler's placements, and ejecting the
scheduler for them fixes nothing. Under bypass the kernel queues the
scheduler's tasks on the bypass DSQs. A host calling
scx_bpf_reenqueue_local() reenqueues a bypassed child's tasks on the cpu's
local DSQ along with its own, and the reject drain reenqueues a consumed
task that a cap check rejected.
In the following patches, the consume itself reenqueues a task it can't run
right away, and the bypass load balancer reenqueues a task at its home.
The count is reset only when the task runs or leaves sched_ext, so an
ejection can lead to a stranding the exit does not recover:
1. the task's reenqueues pass the limit, which ejects its scheduler and
strands the task
2. the exit's scx_bypass() cycles the task through dequeue and enqueue,
which requeues it on a bypass DSQ with the count still past the limit
3. the task's next reenqueue under bypass strands it again
4. scx_bypass() requeues stranded tasks only when the scheduler wasn't
bypassing yet, so the task stays off every DSQ until the disable walks
the tasks
5. the disable takes scx_enable_mutex, the cgroup lock and scx_fork_rwsem
before the walk, and a sub-scheduler's disable the cpu hotplug lock too.
It hangs if the stranded task holds one of them, forking, attaching to a
cgroup or doing a cpu hotplug
Test bypass before the count. A task reenqueued under bypass stays queued
until a later dispatch or balancer pass, so there is no tight loop to bound,
and the task's first run resets the count. The offline test moves up with it
to stay ahead of the bypass test. A reenqueue on an offline cpu goes to the
local DSQ and isn't counted either.
A host that keeps calling scx_bpf_reenqueue_local() could get a bypassed
child ejected, as the child's tasks on the local DSQ counted against the
child. The reenqueues are the host's, and its own tasks still count against
it.
v2: The description lists the cpu hotplug lock among the locks a
sub-scheduler's disable takes before the walk.
Fixes:
|
||
|
Tejun Heo
|
15c5a52c1f |
sched_ext: Clear holding_cpu where the claim is confirmed
The bypass load balancer will take tasks off a remote cpu's bypass DSQ with unlink_dsq_and_switch_rq_lock() in the following patches and insert them into another bypass DSQ. Clearing holding_cpu after the claim is up to each claimant, so the balancer would need a clear of its own. unlink_dsq_and_switch_rq_lock() marks a task with holding_cpu for the gap between dropping the DSQ lock and taking the task's rq lock, so that a dequeue in that gap can tell the claimant it lost. Once the claimant holds the rq lock and finds the marker intact, nothing can take the task away under it and the marker has no reader left. Each claimant clears it at its own site after that point: - dispatch_to_local_dsq(), which runs the handshake inline, in its same-rq and fallback branches - scx_proxy_reject_task() for the proxy reject - the dequeue inside deactivate_task() for both remote moves Clear it where the claim is confirmed instead and drop the explicit clears. No behavior change. Signed-off-by: Tejun Heo <tj@kernel.org> Reviewed-by: Andrea Righi <arighi@nvidia.com> |
||
|
Junbin Zhang
|
b07a7450f6 |
sched_ext: Reject rq-lock-taking kfuncs in tracing programs
scx_bpf_cpuperf_set() and scx_bpf_cidperf_set() are members of
scx_kfunc_ids_any, which is registered for BPF_PROG_TYPE_TRACING.
A tracing program can run in a context which already holds the target
rq lock, in which case scx_cpuperf_set() attempts to acquire the same
lock again:
__schedule()
rq_lock(rq); /* rq->__lock is held */
trace_sched_switch();
BPF tracing program
scx_bpf_cpuperf_set(smp_processor_id(), 1024);
scx_cpuperf_set();
locked_rq = scx_locked_rq(); /* NULL */
rq_lock_irqsave(rq, &rf); /* same lock */
scx_locked_rq() only tracks rq locks held during SCX callback
invocations, so it cannot detect an rq lock held by the surrounding
kernel code.
The deadlock can be reproduced with two BPF programs. The first is a
minimal SCX scheduler, which only needs to be registered and enabled:
SEC(".struct_ops.link")
struct sched_ext_ops minimal_ops = {
.name = "minimal",
};
The second is a tracing program attached to tp_btf/sched_switch,
which fires from __schedule() with rq->__lock already held:
SEC("tp_btf/sched_switch")
int BPF_PROG(on_sched_switch, bool preempt, struct task_struct *prev,
struct task_struct *next, unsigned int prev_state)
{
scx_bpf_cpuperf_set(bpf_get_smp_processor_id(), 1024);
return 0;
}
With both programs attached, the next context switch on any CPU
deadlocks that CPU. Lockdep reports:
WARNING: possible recursive locking detected
7.3.0-rc7 #3 Not tainted
swapper/3/0 is trying to acquire lock:
(&rq->__lock){-.-.}-{2:2}, at: raw_spin_rq_lock_nested+0x1e/0x30
but task is already holding lock:
(&rq->__lock){-.-.}-{2:2}, at: __schedule+0xeb/0x1430
Possible unsafe locking scenario:
CPU0
----
lock(&rq->__lock);
lock(&rq->__lock);
*** DEADLOCK ***
Call Trace:
<TASK>
print_deadlock_bug.cold+0xc0/0xd0
__lock_acquire+0x12fa/0x1b80
lock_acquire+0xd1/0x2f0
_raw_spin_lock_nested+0x32/0x40
raw_spin_rq_lock_nested+0x1e/0x30
scx_cpuperf_set+0x1e9/0x310
scx_bpf_cpuperf_set+0x9b/0x230
bpf_prog_..._on_sched_switch+0x3d/0x41
bpf_trace_run4+0x125/0x450
__traceiter_sched_switch+0x4b/0x70
__schedule+0x94a/0x1430
schedule_idle+0x26/0x50
cpu_startup_entry+0x2d/0x30
start_secondary+0xfc/0x100
</TASK>
The CPU remains locked up with interrupts disabled.
scx_bpf_cidperf_set() is similarly affected, as it resolves the CID
through scx_cid_to_cpu() before reaching the same rq-locking path.
There is no reliable way for these kfuncs to determine whether the
surrounding tracing context already holds an rq lock. Reject calls to
both kfuncs from tracing programs at verification time, while
preserving their availability to SCX struct_ops and syscall programs.
Fixes:
|
||
|
Tejun Heo
|
93b1512ca6 |
sched_ext: Fix the stall watchdog testing and blaming the wrong sched under bypass
While a sched is bypassing, its tasks are run by its host, the nearest ancestor that isn't bypassing. check_rq_for_timeouts() tests a runnable task against its owner's timeout. Since |
||
|
Linus Torvalds
|
6e723c4f94 |
Scheduler fixes:
- Fix MM CID affinity-triggered switch back to per-task mode
(Hui Su)
- Fix scheduling regression observed on AMD Ryzen
AI HX 370 systems, caused by cache aware scheduling
incorrectly overriding asym packing placement (Tim Chen)
- Fix stale get_pref_llc() prototype for !CONFIG_SCHED_CACHE
(Yafang Shao)
Signed-off-by: Ingo Molnar <mingo@kernel.org>
-----BEGIN PGP SIGNATURE-----
iQJFBAABCgAvFiEEBpT5eoXrXCwVQwEKEnMQ0APhK1gFAmrd5QsRHG1pbmdvQGtl
cm5lbC5vcmcACgkQEnMQ0APhK1jFgRAAgBzljKOa92JW7jVUpiIYgQEnnvQkH6yY
4QWS4wzdmioAJu3YEfYm+rfgn0Mjgm33rpxdkSB4+tlhid8+Aq1jEtLKnfv3RfBj
g6uz60RtnVve9MUbtmcC8IEaZdEILuwsJfSD6tsZwZpDQGSbpAIoEcTPfhL4YnyC
T3riIlxiqFwp7r5pMtjzOeXJa+ZqaZC0LWJ1oTQ8oDispbT5AUY6hp8y/9Wz47rH
F0ZUfe/5cytDHk0YQ8z2LgEavM+YRZNooHXdlVQtUrvbqd3rFoY59LdjJx9h8lao
HNRqPpE1SyTnoKjf6x5V30Ko4dWQ4ldr9/YSQtHHQeQqRm4Zag6JNh+QxpPOrB4o
DJcPNbzXrt5b7jKgIVrZn0i04D82Pxev6Fujm6nYQYyIHZBCKDuzRIsAKJqgDN75
uRkn+Upz2rS/llN2kqu6/OFnfV/8VGwbtE5xEdhp20BewnHP2WX5dN0ovxFvu3TS
NKXBMw9OCG/XpIRHGRG2g35Y+196zcG/rg1LSjiVg3yf8N8kiYR0BoWFvItzCM1e
f6bWjABt5lN1HeUK3MhKWEFEcTzRn60VRTAoFKZoXxhgN4AnnQ0wwJyWj6veIjZL
VWBn2xzjYM9eNusW/EbTBH0pVm2cUh1c82Y0D+xCTA6FWO+Tel38uHo3J302G1Pm
Yq97B9EPS/Y=
=HLg5
-----END PGP SIGNATURE-----
Merge tag 'sched-urgent-2026-10-25' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
Pull scheduler fixes from Ingo Molnar:
- Fix MM CID affinity-triggered switch back to per-task mode
(Hui Su)
- Fix scheduling regression observed on AMD Ryzen
AI HX 370 systems, caused by cache aware scheduling
incorrectly overriding asym packing placement (Tim Chen)
- Fix stale get_pref_llc() prototype for !CONFIG_SCHED_CACHE
(Yafang Shao)
* tag 'sched-urgent-2026-10-25' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip:
sched: fix stale get_pref_llc() stub for !CONFIG_SCHED_CACHE
sched/cache: Honor asym packing over cache aware scheduling on hybrid systems
sched/mmcid: Fix affinity-triggered switch back to per-task mode
|
||
|
Hui Su
|
bd09134697 |
sched/idle: Use hrtimer cleanup class in play_idle_precise()
Pair on-stack setup with destruction using the hrtimer_on_stack cleanup
class in play_idle_precise().
Keep the cleanup object's scope around timer start and the idle wait. Start
the timer in pinned hard mode, so its callback runs as a hardirq on the
current CPU. The interrupted task observes it.done only after the callback
returns. Since the callback returns HRTIMER_NORESTART, the timer is not
requeued before the cleanup object tears down its on-stack debug object.
Fixes:
|
||
|
Lorenzo Stoakes (ARM)
|
c911e7ebeb |
mm: rename is_vm_hugetlb_page() to vma_is_hugetlb()
The is_vm_hugetlb_page() predicate is badly named - the mapping can span more than a page and it is inconsistent with other VMA predicates that typically are prefixed by vma_. Rename to vma_is_hugetlb() for consistency, and while we're here update some VM_BUG_ON_VMA() to VM_WARN_ON_ONCE_VMA() as to avoid unnecessary oopses. No functional change intended. Link: https://lore.kernel.org/20261013-b4-mmap-prepare-vma-flag-sanify-v6-28-1aa3c9b139f9@kernel.org Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Acked-by: Marc Zyngier <maz@kernel.org> Acked-by: Claudio Imbrenda <imbrenda@linux.ibm.com> Acked-by: Anup Patel <anup@brainfault.org> Acked-by: Zi Yan <ziy@nvidia.com> Acked-by: David Hildenbrand (Arm) <david@kernel.org> Cc: Liam R. Howlett <liam@infradead.org> Cc: Vlastimil Babka <vbabka@kernel.org> Cc: Jann Horn <jannh@google.com> Cc: Pedro Falcato <pfalcato@suse.de> Cc: Mike Rapoport <rppt@kernel.org> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Michal Hocko <mhocko@suse.com> Cc: Jonathan Corbet <corbet@lwn.net> Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org> Cc: Dennis Dalessandro <dennis.dalessandro@cornelisnetworks.com> Cc: Jason Gunthorpe <jgg@ziepe.ca> Cc: Leon Romanovsky <leon@kernel.org> Cc: Paul Moore <paul@paul-moore.com> Cc: Stephen Smalley <stephen.smalley.work@gmail.com> Cc: Jaroslav Kysela <perex@perex.cz> Cc: Takashi Iwai <tiwai@suse.com> Cc: Alexei Starovoitov <ast@kernel.org> Cc: Daniel Borkmann <daniel@iogearbox.net> Cc: Andrii Nakryiko <andrii@kernel.org> Cc: Eduard Zingerman <eddyz87@gmail.com> Cc: Kumar Kartikeya Dwivedi <memxor@gmail.com> Cc: Baolin Wang <baolin.wang@linux.alibaba.com> Cc: Nico Pache <nico.pache@linux.dev> Cc: Ryan Roberts <ryan.roberts@arm.com> Cc: Dev Jain <dev.jain@arm.com> Cc: Barry Song <baohua@kernel.org> Cc: Lance Yang <lance.yang@linux.dev> Cc: Usama Arif <usama.arif@linux.dev> Cc: Kiryl Shutsemau <kas@kernel.org> Cc: Doug Gilbert <dgilbert@interlog.com> Cc: James Bottomley <james.bottomley@hansenpartnership.com> Cc: Martin K. Petersen <mkp@kernel.org> Cc: Simona Vetter <simona@ffwll.ch> Cc: Helge Deller <deller@gmx.de> Cc: Sebastian Reichel <sre@kernel.org> Cc: John Hubbard <jhubbard@nvidia.com> Cc: Peter Xu <peterx@redhat.com> Cc: Masami Hiramatsu <mhiramat@kernel.org> Cc: Oleg Nesterov <oleg@redhat.com> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@kernel.org> Cc: Ingo Molnar <mingo@redhat.com> Cc: Borislav Petkov <bp@alien8.de> Cc: Dave Hansen <dave.hansen@linux.intel.com> Cc: Arnaldo Carvalho de Melo <acme@kernel.org> Cc: Namhyung Kim <namhyung@kernel.org> Cc: Mark Rutland <mark.rutland@arm.com> Cc: Rik van Riel <riel@surriel.com> Cc: Harry Yoo <harry@kernel.org> Cc: Juri Lelli <juri.lelli@redhat.com> Cc: Vincent Guittot <vincent.guittot@linaro.org> Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com> Cc: Maxime Ripard <mripard@kernel.org> Cc: Thomas Zimmermann <tzimmermann@suse.de> Cc: David Airlie <airlied@gmail.com> Cc: Will Deacon <will@kernel.org> Cc: Aneesh Kumar K.V <aneesh.kumar@kernel.org> Cc: Nicholas Piggin <npiggin@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Muchun Song <muchun.song@linux.dev> Cc: Oscar Salvador <osalvador@suse.de> Cc: Matthew Wilcox (Oracle) <willy@infradead.org> Cc: Jan Kara <jack@suse.cz> Cc: Oliver Upton <oupton@kernel.org> Cc: Catalin Marinas <catalin.marinas@arm.com> Cc: Madhavan Srinivasan <maddy@linux.ibm.com> Cc: Paul Walmsley <pjw@kernel.org> Cc: Palmer Dabbelt <palmer@dabbelt.com> Cc: Albert Ou <aou@eecs.berkeley.edu> Cc: Christian Borntraeger <borntraeger@linux.ibm.com> Cc: Janosch Frank <frankja@linux.ibm.com> Cc: Alexander Gordeev <agordeev@linux.ibm.com> Cc: Gerald Schaefer <gerald.schaefer@linux.ibm.com> Cc: Heiko Carstens <hca@linux.ibm.com> Cc: Vasily Gorbik <gor@linux.ibm.com> Cc: David S. Miller <davem@davemloft.net> Cc: Andreas Larsson <andreas@gaisler.com> Cc: Al Viro <viro@zeniv.linux.org.uk> Cc: Christian Brauner <brauner@kernel.org> Cc: Matthew Brost <matthew.brost@intel.com> Cc: Joshua Hahn <joshua.hahnjy@gmail.com> Cc: Rakie Kim <rakie.kim@sk.com> Cc: Byungchul Park <byungchul@sk.com> Cc: Gregory Price <gourry@gourry.net> Cc: Ying Huang <ying.huang@linux.alibaba.com> Cc: Alistair Popple <apopple@nvidia.com> Cc: Chris Li <chrisl@kernel.org> Cc: Kairui Song <kasong@tencent.com> Cc: Kemeng Shi <shikemeng@huaweicloud.com> Cc: Nhat Pham <nphamcs@gmail.com> Cc: Baoquan He <baoquan.he@linux.dev> Cc: Youngjun Park <youngjun.park@lge.com> Cc: Johannes Weiner <hannes@cmpxchg.org> Cc: Qi Zheng <qi.zheng@linux.dev> Cc: Shakeel Butt <shakeel.butt@linux.dev> Cc: Axel Rasmussen <axelrasmussen@google.com> Cc: Yuanchu Xie <yuanchu@google.com> Cc: Wei Xu <weixugc@google.com> Cc: Chengming Zhou <chengming.zhou@linux.dev> Cc: Michal Hocko <mhocko@kernel.org> Cc: Miklos Szeredi <miklos@szeredi.hu> Cc: Xu Xin <xu.xin@linux.dev> Cc: Ihor Solodrai <ihor.solodrai@linux.dev> Cc: Bert Karwatzki <spasswolf@web.de> |
||
|
Lorenzo Stoakes (ARM)
|
94c8656453 |
mm: remove hugetlb_inline.h
This header really makes little sense - every place it is included mm.h is also included, and the header itself includes mm.h, so it does nothing to reduce header size. It also oddly does an #ifdef around checking VMA_HUGETLB_BIT, however VMA_HUGETLB_BIT is unconditionally available, and will never be set if hugetlb is not enabled. Simply remove the header, eliminate the odd ifdeffery and place the predicates in mm.h. The naming of these predicates is odd, but to keep changes separate, we will address this in a separate patch. The file was never put into MAINTAINERS so there's no change required there. No functional change intended. Link: https://lore.kernel.org/20261013-b4-mmap-prepare-vma-flag-sanify-v6-27-1aa3c9b139f9@kernel.org Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Reviewed-by: Zi Yan <ziy@nvidia.com> Acked-by: David Hildenbrand (Arm) <david@kernel.org> Cc: Liam R. Howlett <liam@infradead.org> Cc: Vlastimil Babka <vbabka@kernel.org> Cc: Jann Horn <jannh@google.com> Cc: Pedro Falcato <pfalcato@suse.de> Cc: Mike Rapoport <rppt@kernel.org> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Michal Hocko <mhocko@suse.com> Cc: Jonathan Corbet <corbet@lwn.net> Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org> Cc: Dennis Dalessandro <dennis.dalessandro@cornelisnetworks.com> Cc: Jason Gunthorpe <jgg@ziepe.ca> Cc: Leon Romanovsky <leon@kernel.org> Cc: Paul Moore <paul@paul-moore.com> Cc: Stephen Smalley <stephen.smalley.work@gmail.com> Cc: Jaroslav Kysela <perex@perex.cz> Cc: Takashi Iwai <tiwai@suse.com> Cc: Alexei Starovoitov <ast@kernel.org> Cc: Daniel Borkmann <daniel@iogearbox.net> Cc: Andrii Nakryiko <andrii@kernel.org> Cc: Eduard Zingerman <eddyz87@gmail.com> Cc: Kumar Kartikeya Dwivedi <memxor@gmail.com> Cc: Baolin Wang <baolin.wang@linux.alibaba.com> Cc: Nico Pache <nico.pache@linux.dev> Cc: Ryan Roberts <ryan.roberts@arm.com> Cc: Dev Jain <dev.jain@arm.com> Cc: Barry Song <baohua@kernel.org> Cc: Lance Yang <lance.yang@linux.dev> Cc: Usama Arif <usama.arif@linux.dev> Cc: Kiryl Shutsemau <kas@kernel.org> Cc: Doug Gilbert <dgilbert@interlog.com> Cc: James Bottomley <james.bottomley@hansenpartnership.com> Cc: Martin K. Petersen <mkp@kernel.org> Cc: Simona Vetter <simona@ffwll.ch> Cc: Helge Deller <deller@gmx.de> Cc: Sebastian Reichel <sre@kernel.org> Cc: John Hubbard <jhubbard@nvidia.com> Cc: Peter Xu <peterx@redhat.com> Cc: Masami Hiramatsu <mhiramat@kernel.org> Cc: Oleg Nesterov <oleg@redhat.com> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@kernel.org> Cc: Ingo Molnar <mingo@redhat.com> Cc: Borislav Petkov <bp@alien8.de> Cc: Dave Hansen <dave.hansen@linux.intel.com> Cc: Arnaldo Carvalho de Melo <acme@kernel.org> Cc: Namhyung Kim <namhyung@kernel.org> Cc: Mark Rutland <mark.rutland@arm.com> Cc: Rik van Riel <riel@surriel.com> Cc: Harry Yoo <harry@kernel.org> Cc: Juri Lelli <juri.lelli@redhat.com> Cc: Vincent Guittot <vincent.guittot@linaro.org> Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com> Cc: Maxime Ripard <mripard@kernel.org> Cc: Thomas Zimmermann <tzimmermann@suse.de> Cc: David Airlie <airlied@gmail.com> Cc: Will Deacon <will@kernel.org> Cc: Aneesh Kumar K.V <aneesh.kumar@kernel.org> Cc: Nicholas Piggin <npiggin@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Muchun Song <muchun.song@linux.dev> Cc: Oscar Salvador <osalvador@suse.de> Cc: Matthew Wilcox (Oracle) <willy@infradead.org> Cc: Jan Kara <jack@suse.cz> Cc: Marc Zyngier <maz@kernel.org> Cc: Oliver Upton <oupton@kernel.org> Cc: Catalin Marinas <catalin.marinas@arm.com> Cc: Madhavan Srinivasan <maddy@linux.ibm.com> Cc: Anup Patel <anup@brainfault.org> Cc: Paul Walmsley <pjw@kernel.org> Cc: Palmer Dabbelt <palmer@dabbelt.com> Cc: Albert Ou <aou@eecs.berkeley.edu> Cc: Christian Borntraeger <borntraeger@linux.ibm.com> Cc: Janosch Frank <frankja@linux.ibm.com> Cc: Claudio Imbrenda <imbrenda@linux.ibm.com> Cc: Alexander Gordeev <agordeev@linux.ibm.com> Cc: Gerald Schaefer <gerald.schaefer@linux.ibm.com> Cc: Heiko Carstens <hca@linux.ibm.com> Cc: Vasily Gorbik <gor@linux.ibm.com> Cc: David S. Miller <davem@davemloft.net> Cc: Andreas Larsson <andreas@gaisler.com> Cc: Al Viro <viro@zeniv.linux.org.uk> Cc: Christian Brauner <brauner@kernel.org> Cc: Matthew Brost <matthew.brost@intel.com> Cc: Joshua Hahn <joshua.hahnjy@gmail.com> Cc: Rakie Kim <rakie.kim@sk.com> Cc: Byungchul Park <byungchul@sk.com> Cc: Gregory Price <gourry@gourry.net> Cc: Ying Huang <ying.huang@linux.alibaba.com> Cc: Alistair Popple <apopple@nvidia.com> Cc: Chris Li <chrisl@kernel.org> Cc: Kairui Song <kasong@tencent.com> Cc: Kemeng Shi <shikemeng@huaweicloud.com> Cc: Nhat Pham <nphamcs@gmail.com> Cc: Baoquan He <baoquan.he@linux.dev> Cc: Youngjun Park <youngjun.park@lge.com> Cc: Johannes Weiner <hannes@cmpxchg.org> Cc: Qi Zheng <qi.zheng@linux.dev> Cc: Shakeel Butt <shakeel.butt@linux.dev> Cc: Axel Rasmussen <axelrasmussen@google.com> Cc: Yuanchu Xie <yuanchu@google.com> Cc: Wei Xu <weixugc@google.com> Cc: Chengming Zhou <chengming.zhou@linux.dev> Cc: Michal Hocko <mhocko@kernel.org> Cc: Miklos Szeredi <miklos@szeredi.hu> Cc: Xu Xin <xu.xin@linux.dev> Cc: Ihor Solodrai <ihor.solodrai@linux.dev> Cc: Bert Karwatzki <spasswolf@web.de> |
||
|
Lorenzo Stoakes (ARM)
|
88d77002e1 |
mm: remove VMA_IO_BIT check in vma[_flags]_is_mm_managed()
We have now made it such that every driver which sets VMA_IO_BIT marks it as not mm-managed. However, vma_flags_is_mm_managed() still checks for VMA_IO_BIT and, if set, treats the mapping as not mm-managed. This was previously required to handle drivers which set VMA_IO_BIT but none of the other flags which identify a mapping as not mm-managed. We have now established that a driver which sets VMA_IO_BIT must also set one of those flags, so the VMA_IO_BIT check is redundant. So remove the VMA_IO_BIT check in vma_flags_is_mm_managed() and update mmap_validate_vma_flags() to use vma_flags_is_mm_managed() rather than open-coding the VMA_PFNMAP_BIT, VMA_MIXEDMAP_BIT check. This change means that vma[_flags]_can_merge() doesn't check VMA_IO_BIT any longer (which is now redundant) as it calls vma_flags_is_mm_managed(). Now that the predicate means precisely VMA_PFNMAP_BIT or VMA_MIXEDMAP_BIT, also use it at the other sites which open-code that pair, so the intent is stated rather than the flags, with no functional change: zap_special_vma_range() only zaps mappings which are not mm-managed, as drivers use it to tear down ranges they established themselves. The mprotect() arch PFN modification check applies to mappings not managed by core mm, which may map PFNs without struct pages. NUMA balancing skips VM_MIXEDMAP mappings having already excluded VM_IO and VM_PFNMAP mappings via vma_migratable(), so it skips exactly the mappings not managed by core mm - say so. Finally, update the VMA userland merge 'special' flag tests to no longer assert that VMA_IO_BIT prevents merge as VMA_PFNMAP_BIT, VMA_MIXEDMAP_BIT now suffices. Link: https://lore.kernel.org/20261013-b4-mmap-prepare-vma-flag-sanify-v6-26-1aa3c9b139f9@kernel.org Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Acked-by: David Hildenbrand (Arm) <david@kernel.org> Cc: Liam R. Howlett <liam@infradead.org> Cc: Vlastimil Babka <vbabka@kernel.org> Cc: Jann Horn <jannh@google.com> Cc: Pedro Falcato <pfalcato@suse.de> Cc: Mike Rapoport <rppt@kernel.org> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Michal Hocko <mhocko@suse.com> Cc: Jonathan Corbet <corbet@lwn.net> Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org> Cc: Dennis Dalessandro <dennis.dalessandro@cornelisnetworks.com> Cc: Jason Gunthorpe <jgg@ziepe.ca> Cc: Leon Romanovsky <leon@kernel.org> Cc: Paul Moore <paul@paul-moore.com> Cc: Stephen Smalley <stephen.smalley.work@gmail.com> Cc: Jaroslav Kysela <perex@perex.cz> Cc: Takashi Iwai <tiwai@suse.com> Cc: Alexei Starovoitov <ast@kernel.org> Cc: Daniel Borkmann <daniel@iogearbox.net> Cc: Andrii Nakryiko <andrii@kernel.org> Cc: Eduard Zingerman <eddyz87@gmail.com> Cc: Kumar Kartikeya Dwivedi <memxor@gmail.com> Cc: Zi Yan <ziy@nvidia.com> Cc: Baolin Wang <baolin.wang@linux.alibaba.com> Cc: Nico Pache <nico.pache@linux.dev> Cc: Ryan Roberts <ryan.roberts@arm.com> Cc: Dev Jain <dev.jain@arm.com> Cc: Barry Song <baohua@kernel.org> Cc: Lance Yang <lance.yang@linux.dev> Cc: Usama Arif <usama.arif@linux.dev> Cc: Kiryl Shutsemau <kas@kernel.org> Cc: Doug Gilbert <dgilbert@interlog.com> Cc: James Bottomley <james.bottomley@hansenpartnership.com> Cc: Martin K. Petersen <mkp@kernel.org> Cc: Simona Vetter <simona@ffwll.ch> Cc: Helge Deller <deller@gmx.de> Cc: Sebastian Reichel <sre@kernel.org> Cc: John Hubbard <jhubbard@nvidia.com> Cc: Peter Xu <peterx@redhat.com> Cc: Masami Hiramatsu <mhiramat@kernel.org> Cc: Oleg Nesterov <oleg@redhat.com> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@kernel.org> Cc: Ingo Molnar <mingo@redhat.com> Cc: Borislav Petkov <bp@alien8.de> Cc: Dave Hansen <dave.hansen@linux.intel.com> Cc: Arnaldo Carvalho de Melo <acme@kernel.org> Cc: Namhyung Kim <namhyung@kernel.org> Cc: Mark Rutland <mark.rutland@arm.com> Cc: Rik van Riel <riel@surriel.com> Cc: Harry Yoo <harry@kernel.org> Cc: Juri Lelli <juri.lelli@redhat.com> Cc: Vincent Guittot <vincent.guittot@linaro.org> Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com> Cc: Maxime Ripard <mripard@kernel.org> Cc: Thomas Zimmermann <tzimmermann@suse.de> Cc: David Airlie <airlied@gmail.com> Cc: Will Deacon <will@kernel.org> Cc: Aneesh Kumar K.V <aneesh.kumar@kernel.org> Cc: Nicholas Piggin <npiggin@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Muchun Song <muchun.song@linux.dev> Cc: Oscar Salvador <osalvador@suse.de> Cc: Matthew Wilcox (Oracle) <willy@infradead.org> Cc: Jan Kara <jack@suse.cz> Cc: Marc Zyngier <maz@kernel.org> Cc: Oliver Upton <oupton@kernel.org> Cc: Catalin Marinas <catalin.marinas@arm.com> Cc: Madhavan Srinivasan <maddy@linux.ibm.com> Cc: Anup Patel <anup@brainfault.org> Cc: Paul Walmsley <pjw@kernel.org> Cc: Palmer Dabbelt <palmer@dabbelt.com> Cc: Albert Ou <aou@eecs.berkeley.edu> Cc: Christian Borntraeger <borntraeger@linux.ibm.com> Cc: Janosch Frank <frankja@linux.ibm.com> Cc: Claudio Imbrenda <imbrenda@linux.ibm.com> Cc: Alexander Gordeev <agordeev@linux.ibm.com> Cc: Gerald Schaefer <gerald.schaefer@linux.ibm.com> Cc: Heiko Carstens <hca@linux.ibm.com> Cc: Vasily Gorbik <gor@linux.ibm.com> Cc: David S. Miller <davem@davemloft.net> Cc: Andreas Larsson <andreas@gaisler.com> Cc: Al Viro <viro@zeniv.linux.org.uk> Cc: Christian Brauner <brauner@kernel.org> Cc: Matthew Brost <matthew.brost@intel.com> Cc: Joshua Hahn <joshua.hahnjy@gmail.com> Cc: Rakie Kim <rakie.kim@sk.com> Cc: Byungchul Park <byungchul@sk.com> Cc: Gregory Price <gourry@gourry.net> Cc: Ying Huang <ying.huang@linux.alibaba.com> Cc: Alistair Popple <apopple@nvidia.com> Cc: Chris Li <chrisl@kernel.org> Cc: Kairui Song <kasong@tencent.com> Cc: Kemeng Shi <shikemeng@huaweicloud.com> Cc: Nhat Pham <nphamcs@gmail.com> Cc: Baoquan He <baoquan.he@linux.dev> Cc: Youngjun Park <youngjun.park@lge.com> Cc: Johannes Weiner <hannes@cmpxchg.org> Cc: Qi Zheng <qi.zheng@linux.dev> Cc: Shakeel Butt <shakeel.butt@linux.dev> Cc: Axel Rasmussen <axelrasmussen@google.com> Cc: Yuanchu Xie <yuanchu@google.com> Cc: Wei Xu <weixugc@google.com> Cc: Chengming Zhou <chengming.zhou@linux.dev> Cc: Michal Hocko <mhocko@kernel.org> Cc: Miklos Szeredi <miklos@szeredi.hu> Cc: Xu Xin <xu.xin@linux.dev> Cc: Ihor Solodrai <ihor.solodrai@linux.dev> Cc: Bert Karwatzki <spasswolf@web.de> |
||
|
Yafang Shao
|
cac037a85c |
sched: fix stale get_pref_llc() stub for !CONFIG_SCHED_CACHE
Commit |
||
|
Tim Chen
|
ee92b7ef4c |
sched/cache: Honor asym packing over cache aware scheduling on hybrid systems
A regression was reported on an AMD Ryzen AI HX 370 running a cache
intensive Clang full-LTO link. The little cores run at a much lower
frequency (3.3 GHz vs 5.1 GHz) and have only half of the L3 cache
(8 MB vs 16 MB), so pinning such a task to the little-core LLC
hurts twice, and full-LTO builds slow down dramatically compared to
pre-cache-aware-scheduling kernels.
Asym packing and cache aware scheduling express conflicting placement
strategies. Asym packing wants a task to run on the highest priority CPU,
whereas cache aware scheduling wants to co-locate the tasks of a process
on one LLC regardless of the priority of CPUs in that LLC.
When asym packing tries to migrate task to an idle core that has higher
priority than source cpu, let asym packing win. Moving tasks to a higher
performing idle core will buy more performance than cache co-location.
Prioritize asym packing over LLC balancing for regular and active load
balancing.
Fixes:
|
||
|
Linus Torvalds
|
98aec86f8c |
sched_ext: Changes for v7.4
This depends on sched/core, which carries the sched_ext hooks for proxy execution, and should be pulled after it. The shortlog and diffstat are against v7.3 with sched/core merged. for-7.3-fixes was merged into for-7.4 six times during the cycle, once to receive a change the CID cluster work built on and otherwise to resolve conflicts with development here, mostly the proxy execution work. Five fixes went into for-7.3-fixes in the days before v7.3 was cut and missed the release. They come in through this pull, three of them marked for stable: balance callbacks left queued across a pick retry, a use-after-free on a CPU going offline, and bypass bugs that could stall a task. - Proxy execution can now be enabled together with sched_ext. A BPF scheduler can opt in to receive mutex-blocked tasks that donate their scheduling context to the lock owner and place them itself. - Lazy preemption: a BPF scheduler can preempt or kick a CPU lazily, deferring the switch to the next return to user space or tick, as the fair class can. - Sub-schedulers: a parent now learns which child runs on each CPU it delegated and when a cap change takes effect. A detaching child loses its caps before the parent restores the CPUs. - New kfuncs re-enqueue single tasks from a dispatch queue iteration and report a cgroup's CPU count and a CID's NUMA node. The CID topology now includes clusters. - Hot path optimizations: dispatch queue and task lookups compare keys inline, remote task moves no longer take a dispatch queue lock twice, and sub-scheduler hooks cost nothing without sub-schedulers. - Kfuncs callable from tracing programs are hardened against NMI and IRQ nesting, which could hard-lock the CPU or pick a wrong idle CPU. - Built-in dispatch queues no longer allocate per-CPU state that grew with the square of the CPU count. - Header syncs with the scx repo, a BTF workaround for pahole 1.32, and selftest fixes and coverage. -----BEGIN PGP SIGNATURE----- iIQEABYKACwWIQTfIjM1kS57o3GsC/uxYfJx3gVYGQUCatZJMQ4cdGpAa2VybmVs Lm9yZwAKCRCxYfJx3gVYGRgSAP4n0sb6c00/m1VhyjR/75faCpNrvW5fS0zZ5Zon fAsCGgD5AUTLMn460Lr9vTPXvS8RGS790V/Id/oH0+up5z0sHQA= =Kzaf -----END PGP SIGNATURE----- Merge tag 'sched_ext-for-7.4' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/sched_ext Pull sched_ext updates from Tejun Heo: - Proxy execution can now be enabled together with sched_ext. A BPF scheduler can opt in to receive mutex-blocked tasks that donate their scheduling context to the lock owner and place them itself. - Lazy preemption: a BPF scheduler can preempt or kick a CPU lazily, deferring the switch to the next return to user space or tick, as the fair class can. - Sub-schedulers: a parent now learns which child runs on each CPU it delegated and when a cap change takes effect. A detaching child loses its caps before the parent restores the CPUs. - New kfuncs re-enqueue single tasks from a dispatch queue iteration and report a cgroup's CPU count and a CID's NUMA node. The CID topology now includes clusters. - Hot path optimizations: dispatch queue and task lookups compare keys inline, remote task moves no longer take a dispatch queue lock twice, and sub-scheduler hooks cost nothing without sub-schedulers. - Kfuncs callable from tracing programs are hardened against NMI and IRQ nesting, which could hard-lock the CPU or pick a wrong idle CPU. - Built-in dispatch queues no longer allocate per-CPU state that grew with the square of the CPU count. - Header syncs with the scx repo, a BTF workaround for pahole 1.32, and selftest fixes and coverage. * tag 'sched_ext-for-7.4' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/sched_ext: (69 commits) sched_ext: Drop the locks periodically while walking a user DSQ for re-enqueue sched_ext: scx_qmap: Also re-enqueue stranded tasks with scx_bpf_dsq_reenq_task() sched_ext: Add scx_bpf_dsq_reenq_task() to re-enqueue single tasks from a DSQ iteration sched_ext: Factor out the rq lock switch of scx_dsq_move() sched_ext: Don't queue balance callbacks from a core-sched pick sched_ext: Don't leave balance callbacks queued across a pick retry sched_ext: Fix a bypassed task stalling on a cpu its scheduler has no caps on sched_ext: Fix the running task keeping its protection into bypass sched_ext: Fix use-after-free of a local reenq node left on an inactive cpu sched_ext: Keep proxy donors with slice left on the local DSQ selftests/sched_ext: Report reload_loop attach failures selftests/sched_ext: Bound exit waits selftests/sched_ext: Add SCX_ASSERT_ALIVE and adopt it selftests/sched_ext: Fail loudly when EXIT_KIND is missing from BTF sched_ext: Add SCX_ECODE_RSN_CGROUP_OFFLINE to user_exit_info.h selftests/sched_ext: Fix reload_loop error-path use-after-free selftests/sched_ext: Fix build error in dequeue_remote selftest sched_ext: scx_qmap: Reset a child's cpuperf targets once its PERF revoke takes effect sched_ext: Add ops.sub_child_ecaps_updated() to report a child's effective cap changes sched_ext: scx_qmap: Show actual cid use per participant ... |
||
|
Linus Torvalds
|
df37e39242 |
Power management updates for 7.4-rc1
- Fix multiple issues related to input validation, synchronization,
access safety, memory layout and register field validation, and error
handling in the ACPI CPPC library (Christian Loehle and Sumit Gupta)
- Select the frequency-invariance callback per CPU in the CPPC cpufreq
driver and create the FIE worker before enabling PCC callbacks in it
to avoid queueing work through a missing worker and dereferencing a
NULL pointer (Christian Loehle)
- Update the CPPC cpufreq driver entry in MAINTAINERS and add a new
entry for the ACPI CPPC library (Christian Loehle)
- Minor fixes in various cpufreq drivers (Alexander Koskovich, Daniel
Schwierzeck, Frank Zhang, Haotian Zhang, Kathiravan Thirumoorthy,
Sumeet Pawnikar, and Xueqin Luo)
- Fix wrong behaviors on multi-node servers in the Loongson cpufreq
driver (Huacai Chen)
- Reject NULL from cpufreq_cpu_get() in the Rust code (Mehmet Koseoglu)
- Add suspend OPP to mediatek cpufreq driver and DT (Ryan Brue)
- Fixups for a few bugs identified by Sashiko in the amd-pstate cpufreq
driver (Mario Limoncielloi, Wentao Guan)
- Initial support for maximum frequency capacity handling on Zen6
client in amd-pstate (Mario Limonciello)
- Fix for amd-pstate-ut with 'Requested CPU Min frequency' BIOS option
(K Prateek Nayak)
- New Zen6 tuning values for EPP in amd-pstate (Mario Limonciello,
Vishal Badole)
- amd-pstate documentation fix (Karl Mehltretter)
- Restrict deferred downscaling to low load for shared policies in the
conservative cpufreq governor to avoid undesirable CPU frequency
reduction in some cases (Shengming Hu)
- Serialize sugov_start() against rate limit updates in the schedutil
cpufreq governor to avoid using a stale rate limit value (Hui Su)
- Prevent cpuidle governors menu and teo from returning idle state 0
after it has been disabled (Xueqin Luo)
- Replace min_t() with the better alternatives in the intel_idle driver
and make it use 2-argument strscpy() (Andy Shevchenko)
- Fix two repeated phrase instances in cpuidle (Hemanth Selam, Syazwina
Fazrul)
- Add rugged Panther Lake support to the intel_rapl power capping
driver and make it use %pe for printing error pointers (Srinivas
Pandruvada and Sumeet Pawnikar)
- Update and improve runtime PM documentation and add a kunit test for
supplier idle/suspend (Brian Norris)
- Add a pm_runtime_dont_use_autosuspend() call to pm_runtime_reinit()
to ensure autosuspend disabling on driver teardown (Joshua Crofts)
- Ignore page-zero RAM in E820 checksum during hibernation on x86 to
avoid rejecting images created after kexec() (Matthias Goergens)
- Add resume event mapping for PMSG_POWEROFF to the PM core to allow it
to carry out device shutdown in the future (Mario Limonciello)
- Add DPM watchdog to prepare/late/early/noirq/complete phases of
device suspend and resume (Mayank Rungta)
- Update swsusp.txt references to swsusp.rst in hibernation
documentation (Li Jun)
- Fix use-after-free of perf domain in netlink doit handlers for energy
models (Fan Wu)
- Add synchronize_rcu() back to cpu_pm_unregister_notifier() after it
was removed inadvertently in previous development cycles to avoid
use-after-free (Bradley Morgan)
- Unregister CPU PM notifier outside the timer lock in the timer-ti-dm
clocksource driver (Bradley Morgan)
- Fix typo "sucessfully" in a comment in pm_clk_add_clk() kerneldoc
comment and typo "hierachy" in a comment in pm-graph (Hemanth Selam)
- Fix the return value in dev_pm_opp_get_of_node() kernel-doc (Karl
Mehltretter)
-----BEGIN PGP SIGNATURE-----
iQFGBAABCAAwFiEEcM8Aw/RY0dgsiRUR7l+9nS/U47UFAmrWNkISHHJqd0Byand5
c29ja2kubmV0AAoJEO5fvZ0v1OO1M7QH/RyTpxMUHI3eLqhPIczecdsQxoJ9gCZS
wGMQmvBRrNEw/Cp+X+j5Swo8iOJRKQ5Na+h53Ebg4EFJg5Fkh8SDO/VhQDaC0GUp
t9y9MpVfXq/Fc/gOimzJ/MGhRmBXl5HKva586pg5TdFg4adDs4kPcDj9zDFnKMhf
Kb0FHgjRjli1iDT1HSOV7KDsH1bZBWD1P04jMaVQ1zJb6Ngwi1msyBaRlE3A1mqX
W31C3pi7qtfbs1v1JPmrnaVAJEZmv38t7x/G3o5FL35P2bwlgN/KXD0aMNHZ0Rnl
A6Jz4p5ugop+gBLzyQo4W8SNcLk9cRvlp3FUwuEax4MEPbg+rhfjdDs=
=4ccu
-----END PGP SIGNATURE-----
Merge tag 'pm-7.4-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/rafael/linux-pm
Pull power management updates from Rafael Wysocki:
"This time, the most visible part are the ACPI CPPC updates from
Christian Loehle, including CPPC cpufreq driver changes, even though
updates of cpufreq drivers proper are still the largest topic group.
However, the updates of cpufreq drivers depend on the ACPI CPPC
updates, so they should really be counted together and that is well
over 1/2 of the total commit count.
Interestingly enough, almost all of the updates included here are
fixes or cleanups, with some new feature support in the amd-pstate
cpufreq driver and IPQ9650 support in the qcom-nvmem cpufreq driver.
The runtime PM documentation improvements from Brian Norris can also
be mostly regarded as fixes, but a new kunit test goes along with
them.
Specifics:
- Fix multiple issues related to input validation, synchronization,
access safety, memory layout and register field validation, and
error handling in the ACPI CPPC library (Christian Loehle and Sumit
Gupta)
- Select the frequency-invariance callback per CPU in the CPPC
cpufreq driver and create the FIE worker before enabling PCC
callbacks in it to avoid queueing work through a missing worker and
dereferencing a NULL pointer (Christian Loehle)
- Update the CPPC cpufreq driver entry in MAINTAINERS and add a new
entry for the ACPI CPPC library (Christian Loehle)
- Minor fixes in various cpufreq drivers (Alexander Koskovich, Daniel
Schwierzeck, Frank Zhang, Haotian Zhang, Kathiravan Thirumoorthy,
Sumeet Pawnikar, and Xueqin Luo)
- Fix wrong behaviors on multi-node servers in the Loongson cpufreq
driver (Huacai Chen)
- Reject NULL from cpufreq_cpu_get() in the Rust code (Mehmet
Koseoglu)
- Add suspend OPP to mediatek cpufreq driver and DT (Ryan Brue)
- Fixups for a few bugs identified by Sashiko in the amd-pstate
cpufreq driver (Mario Limoncielloi, Wentao Guan)
- Initial support for maximum frequency capacity handling on Zen6
client in amd-pstate (Mario Limonciello)
- Fix for amd-pstate-ut with 'Requested CPU Min frequency' BIOS
option (K Prateek Nayak)
- New Zen6 tuning values for EPP in amd-pstate (Mario Limonciello,
Vishal Badole)
- amd-pstate documentation fix (Karl Mehltretter)
- Restrict deferred downscaling to low load for shared policies in
the conservative cpufreq governor to avoid undesirable CPU
frequency reduction in some cases (Shengming Hu)
- Serialize sugov_start() against rate limit updates in the schedutil
cpufreq governor to avoid using a stale rate limit value (Hui Su)
- Prevent cpuidle governors menu and teo from returning idle state 0
after it has been disabled (Xueqin Luo)
- Replace min_t() with the better alternatives in the intel_idle
driver and make it use 2-argument strscpy() (Andy Shevchenko)
- Fix two repeated phrase instances in cpuidle (Hemanth Selam,
Syazwina Fazrul)
- Add rugged Panther Lake support to the intel_rapl power capping
driver and make it use %pe for printing error pointers (Srinivas
Pandruvada and Sumeet Pawnikar)
- Update and improve runtime PM documentation and add a kunit test
for supplier idle/suspend (Brian Norris)
- Add a pm_runtime_dont_use_autosuspend() call to pm_runtime_reinit()
to ensure autosuspend disabling on driver teardown (Joshua Crofts)
- Ignore page-zero RAM in E820 checksum during hibernation on x86 to
avoid rejecting images created after kexec() (Matthias Goergens)
- Add resume event mapping for PMSG_POWEROFF to the PM core to allow
it to carry out device shutdown in the future (Mario Limonciello)
- Add DPM watchdog to prepare/late/early/noirq/complete phases of
device suspend and resume (Mayank Rungta)
- Update swsusp.txt references to swsusp.rst in hibernation
documentation (Li Jun)
- Fix use-after-free of perf domain in netlink doit handlers for
energy models (Fan Wu)
- Add synchronize_rcu() back to cpu_pm_unregister_notifier() after it
was removed inadvertently in previous development cycles to avoid
use-after-free (Bradley Morgan)
- Unregister CPU PM notifier outside the timer lock in the
timer-ti-dm clocksource driver (Bradley Morgan)
- Fix typo "sucessfully" in a comment in pm_clk_add_clk() kerneldoc
comment and typo "hierachy" in a comment in pm-graph (Hemanth
Selam)
- Fix the return value in dev_pm_opp_get_of_node() kernel-doc (Karl
Mehltretter)"
* tag 'pm-7.4-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/rafael/linux-pm: (89 commits)
cpufreq: loongson3: Replace IOCSR read/write with MMIO ones
cpufreq: loongson3: Use global physical CPU ID in get/target callbacks
cpufreq: loongson3: Replace per-package mutex with per-node
cpufreq: loongson3: Adjust the width of id and val in smc_message
cpufreq: loongson3: Make this driver depend on MACH_LOONGSON64
PM: runtime: Expand introduction with core concepts and structure
cpufreq: schedutil: Serialize sugov_start() against rate limit updates
MAINTAINERS: ACPI: Add a separate CPPC entry
MAINTAINERS: cpufreq: Replace CPPC reviewer
arm64: dts: mediatek: mt8173: Mark the lowest CPU OPPs for suspend
cpufreq: mediatek: Enter system sleep at the suspend OPP
cpufreq: qcom-cpufreq-hw: fix device node refcount leak in phandle parsing
cpufreq: qcom-nvmem: Add IPQ9650 support
cpufreq: Add Milos to cpufreq-dt-platdev blocklist
cpufreq: Use %pe to print error pointers symbolically
rust: rcpufreq_dt: add module alias
cpufreq: sti: avoid NULL dereference in dev_err()
cpufreq: tegra194: fix double-pointer error in get_cpu_ndiv
cpufreq: sparc-us2e: fix frequency table index copy-paste error
rust: cpufreq: reject NULL from cpufreq_cpu_get()
...
|
||
|
Hui Su
|
651694bc47 |
sched_ext: Reject oversized automatic CID shards
Automatic topology packing starts a new shard before adding a core when
the current shard would exceed its target size. However, a first core is
always accepted so that cores remain indivisible. An indivisible core with
more than SCX_CID_SHARD_MAX_CPUS CPUs cannot be represented without
exceeding the hard shard limit.
Per-shard state uses fixed-size inline cmask storage sized for
SCX_CID_SHARD_MAX_CPUS. Passing an oversized shard to alloc_pshard() can
therefore make cmask initialization overwrite the following fields.
Reject an indivisible core larger than SCX_CID_SHARD_MAX_CPUS before adding
it to a shard. Existing non-empty shard splitting already keeps the total
within the hard cap, while an indivisible core may still exceed
cid_shard_size when it fits within the hard limit.
Report the oversized core and the hard limit with scx_error() so that the
rejection is diagnosable.
tj: The bug is only reproducible with artificial VM topologies. It can't
happen on real hardware.
Fixes:
|
||
|
Andrea Righi
|
3b1c7c74a6 |
sched/core: Avoid requeueing a singleton balance callback
With proxy execution, __schedule() can call set_next_task_scx() with
SNT_REPICK and then SNT_PICK for the same donor. Both calls queue the
kick_sync balance callback while kick_sync_pending remains set.
When the callback is the only list entry, its next pointer is NULL, so
the existing duplicate check misses it. Queueing it again links it to
itself and causes the wait to run twice.
Also check whether the callback is already the head of the list to
reject the second insertion.
This covers the kick_sync callback because nothing else is queued
between SNT_REPICK and SNT_PICK. Requeueing the tail of a longer list
remains possible and is outside the scope of this fix.
Fixes:
|
||
|
Linus Torvalds
|
cb95904b47 |
Scheduler updates:
Kconfig updates:
- Simplify PREEMPT_DYNAMIC (Mark Rutland)
Fair scheduler updates:
- Remove redundant RUN_TO_PARITY check (Elif Topuz)
- Declare augmented callbacks per field, fix rb_add_augmented_cached()
descent (Yiwei Lin, Peter Zijlstra)
- Restart fair hrtick after same-task repicks (Shubhang Kaushik)
- Improve the scheduling latency of short slice tasks (Vincent Guittot)
Deadline/RT scheduler updates:
- Check start_dl_timer expiry with ktime_before()
(Liang Hao)
- irq_work: CPU-hotplug improvements on PREEMPT_RT (Sebastian Andrzej
Siewior)
Energy-aware scheduling updates:
- Implement the steal_governor, introduce preferred CPUs and
steal-driven vCPU backoff (Shrikanth Hegde)
Multi-processor load-balancing updates:
- Enable preferred SMT siblings on NVIDIA Olympus (Andrea Righi)
- Randomize equally shallow idle CPU picks (Christian Loehle)
- Remove redundant core_sched_seq (Hui Su)
Cache-aware scheduling updates:
- Avoid publishing partial LLC utilization (Jemmy Wong)
NUMA-balancing updates:
- Reset NUMA fault locality after scan period update (Eric Kim)
Proxy execution updates:
- Ongoing work to implement sleeping owner handling for proxy
execution (John Stultz, Vasily Gorbik)
- Preparatory steps to make sched_ext compatible with proxy
execution (Andrea Righi)
Scheduler infrastructure updates & fixes:
- Remove the RCU wait from PSI trigger destruction (Shakeel Butt)
- Avoid modifying wait-entry on failed priority registration
(Hui Su)
- Use the new static key API for sched_feat (Hongyan Xia)
- Convert paravirt_steal to new static key APIs (Hongyan Xia)
- cputime: Don't account idle time twice after dyntick-idle
(Stian Halseth)
Scheduler debugging updates:
- Set TIF_NEED_RESCHED before calling __trace_set_need_resched()
(Sechang Lim)
- Fix context analysis errors in non-preferred CPU push (Shrikanth
Hegde)
Misc cleanups and fixes by Christian Loehle, Shubhang Kaushik,
Thomas Huth and Vladimir Zapolskiy.
Signed-off-by: Ingo Molnar <mingo@kernel.org>
-----BEGIN PGP SIGNATURE-----
iQJFBAABCgAvFiEEBpT5eoXrXCwVQwEKEnMQ0APhK1gFAmrUomIRHG1pbmdvQGtl
cm5lbC5vcmcACgkQEnMQ0APhK1ik+Q/+Jx0tXmwOm/wk1ba50zz/tkUL17APJzD1
0TdHSf0sqssyVnQVmGBNWoKfp30zzrnd8Yf46PH5kUmyIxX0LqazAXsPA/0lxAY6
jB2b5ZGosyQQPqyjM1q48K/BMB+GGO9xDKlFxxkBsICi+vS+6diBzXM70xla9M4O
CcAO5DHb50otN/TljfnkmbxG7vgssCVZyS7XG6bSyhd6tUrcr4DVD7gYOMbSVcaL
8Cgu+b54rOAcrcWU1454YKHI5etUOTjblCbFgISrdw7lwf4dKVtTAKSanFX9n522
q9sQ1vzuRz6JfSak9nxFkCy5SMjuHJ53DkVNR4eUXNmM0tHdAcVd8jOexmQVBrug
Yjloo/WydUqGT2TYYEVQPi4J+jMqMRyDIp3TL9UuQjwcjHRnUpd8EVmoNme0TOi+
e3SoS+xOSVBLgbMFwrjJR/tQuqDPjgBrgIyg8wL+ed5cY6Equ+J3467wp8lO7RBN
uGwChvxQAOaMlkh5tIR38Zjq9kzs79y85vz3uD2SNk17TcvijVw9UsmQKfAkpdZo
5FUDM2AdyOhOtZkzMNLHfjjBLSMVCBRo726c6WbELibhuWN7xTAJSJx8o8JNOCrC
vOBcNNAHpU7BwqA0gFIA3lL1uMXvKn9npKKjnB2C4WxhFGf23NF6uPapyA4sMZ00
YwRGvOeDvu4=
=cPCx
-----END PGP SIGNATURE-----
Merge tag 'sched-core-2026-10-18' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
Pull scheduler updates from Ingo Molnar:
"Kconfig updates:
- Simplify PREEMPT_DYNAMIC (Mark Rutland)
Fair scheduler updates:
- Remove redundant RUN_TO_PARITY check (Elif Topuz)
- Declare augmented callbacks per field, fix
rb_add_augmented_cached() descent (Yiwei Lin, Peter Zijlstra)
- Restart fair hrtick after same-task repicks (Shubhang Kaushik)
- Improve the scheduling latency of short slice tasks (Vincent
Guittot)
Deadline/RT scheduler updates:
- Check start_dl_timer expiry with ktime_before() (Liang Hao)
- irq_work: CPU-hotplug improvements on PREEMPT_RT (Sebastian Andrzej
Siewior)
Energy-aware scheduling updates:
- Implement the steal_governor, introduce preferred CPUs and
steal-driven vCPU backoff (Shrikanth Hegde)
Multi-processor load-balancing updates:
- Enable preferred SMT siblings on NVIDIA Olympus (Andrea Righi)
- Randomize equally shallow idle CPU picks (Christian Loehle)
- Remove redundant core_sched_seq (Hui Su)
Cache-aware scheduling updates:
- Avoid publishing partial LLC utilization (Jemmy Wong)
NUMA-balancing updates:
- Reset NUMA fault locality after scan period update (Eric Kim)
Proxy execution updates:
- Ongoing work to implement sleeping owner handling for proxy
execution (John Stultz, Vasily Gorbik)
- Preparatory steps to make sched_ext compatible with proxy execution
(Andrea Righi)
Scheduler infrastructure updates & fixes:
- Remove the RCU wait from PSI trigger destruction (Shakeel Butt)
- Avoid modifying wait-entry on failed priority registration
(Hui Su)
- Use the new static key API for sched_feat (Hongyan Xia)
- Convert paravirt_steal to new static key APIs (Hongyan Xia)
- cputime: Don't account idle time twice after dyntick-idle
(Stian Halseth)
Scheduler debugging updates:
- Set TIF_NEED_RESCHED before calling __trace_set_need_resched()
(Sechang Lim)
- Fix context analysis errors in non-preferred CPU push (Shrikanth
Hegde)
Misc cleanups and fixes by Christian Loehle, Shubhang Kaushik,
Thomas Huth and Vladimir Zapolskiy"
* tag 'sched-core-2026-10-18' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip: (64 commits)
sched: Replace nr_iowait_cpu() with a bool accessor
Revert "sched/fair: Remove dead code on enqueue_task_fair()"
sched/cache: Avoid publishing partial LLC utilization
sched/cputime: Don't account idle time twice after dyntick-idle
sched/psi: Remove the RCU wait from trigger destruction
sched/eevdf: Remove redundant RUN_TO_PARITY check
sched/wait: Avoid modifying entry on failed priority registration
rbtree_test: Use rb_add_augmented_cached() for the cached augmented test
rbtree: Update augmented data on the way down in rb_add_augmented_cached()
rbtree: Declare augmented callbacks per field with RB_AUG()
rbtree_test: Use rb_add() and rb_add_cached() for the basic tests
smpboot: Don't park the thread if work is pending
irq_work: Flush lazy work CPU down on PREEMPT_RT
irq_work: Update a comment regarding CPU hotplug invocation
sched/topology: Add asymmetric SMT packing override
sched/fair: Honor asymmetric SMT priority in idle selection
sched: Set TIF_NEED_RESCHED before calling __trace_set_need_resched()
sched/core: Fix context analysis errors in non-preferred CPU push
virt/steal_governor: Enable the driver
virt/steal_governor: Implement steal_governor policy loop
...
|
||
|
Andrea Righi
|
9b5ce4345b |
sched_ext: Use irq_work for deferred actions with proxy execution
maybe_queue_balance_callback() clears SCX_RQ_BAL_CB_PENDING when it
queues the deferred callback. Proxy resolution can then zap the callback
without running it, leaving the deferred actions unscheduled until a
later wakeup or deferred request.
Use the existing irq_work fallback for dispatch-time deferred requests
when proxy execution is enabled. This keeps the work scheduled even if
proxy resolution retries the pick or falls back to idle, without relying
on another SCX dispatch to restore a discarded callback.
Fixes:
|
||
|
Hui Su
|
7dbc307b6c |
sched/mmcid: Fix affinity-triggered switch back to per-task mode
MM CID switches to per-CPU ownership when the number of users of an mm
exceeds the maximum number of CIDs. When the affinity union grows, the
per-CPU threshold is recalculated and the MM can switch back to per-task
ownership through deferred work.
mm_cid_calc_pcpu_thrs() incorrectly uses min(nr_cpus_allowed, users) to
calculate that switch-back threshold. This makes pcpu_thrs no larger than
users, so the users < pcpu_thrs check in mm_update_cpus_allowed() can never
succeed.
For example, with three users restricted to two CPUs, the MM enters per-CPU
mode. On an eight-CPU system, widening the affinity union to all eight CPUs
should produce a threshold of four, but the old calculation produces three
and returns early. The MM therefore remains in per-CPU mode even though the
widened CPU set makes per-task ownership preferable.
Calculate pcpu_thrs from nr_cpus_allowed directly. The users bound in
__mm_update_max_cids() still controls when per-CPU mode is entered, so this
does not change the threshold calculated when switching from per-task to
per-CPU ownership.
Tested with an x86_64 QEMU reproducer using three users, an initial two-CPU
affinity, and an eight-CPU affinity expansion. The baseline does not invoke
mm_cid_work_fn() after the affinity expansion. The fixed kernel invokes the
deferred work once and completes the switch back to per-task mode.
Fixes:
|
||
|
Andrea Righi
|
e2859566ff |
sched_ext: Queue the kick_sync wait from set_next_task_scx()
The SCX_KICK_WAIT wait is deferred to rq->scx.kick_sync_bal_cb, which a
final pick queues on the local rq. Under core scheduling, a pick that
returns a task isn't final: pick_next_task() goes on to pick the SMT
siblings, and their dispatch can release the shared rq lock with the
callback still queued. A CPU acquiring the rq lock in that window then
finds a pending balance callback and triggers the warning in
rq_pin_lock().
Fix this by queuing the callback from set_next_task_scx() on SNT_PICK
and SNT_REPICK. put_prev_set_next_task() calls it once the pick is
final, after the core-wide selection, so the rq lock can't be dropped
with the callback queued. If the CPU ends up running idle or a higher
class task, no SCX task runs there and there's nothing to wait for:
kick_sync_pending stays set and the next pick of an SCX task waits
before running it.
Also, proxy exec zaps the balance callbacks and picks again when
find_proxy_task() fails, so clear kick_sync_pending when the callback
runs rather than when it's queued.
Fixes:
|
||
|
Michal Blaszczyk
|
993e8fbfec |
sched_ext: Reset runnable_at when a task leaves its scheduler
The watchdog measures how long a task has been waiting from
p->scx.runnable_at, which set_task_runnable() resets when the task becomes
runnable after it last ran. ops_dequeue() never asks for a reset, as the
dequeue of a waiting task is temporary, e.g. for a migration or an
attribute change.
Leaving the scheduler isn't temporary. A task which is switched out of
SCHED_EXT while it's waiting keeps its runnable_at. When it's switched back
and becomes runnable, its wait appears to have started before it left, and
the time it spent in another class counts towards the timeout. If the
watchdog checks before the task runs, it aborts the scheduler:
sched_ext: class_switch_stall: runner[418] failed to run for 3.010s
Here, the task waited for a few milliseconds before it was switched to
SCHED_OTHER, slept outside of SCHED_EXT for three seconds, and was charged
for all of it as soon as it woke up on SCHED_EXT again.
The same happens when scx_sub_enable_workfn() hands a waiting task over to
a new sub-scheduler, which can then be blamed for time the task waited
under the parent. scx_rehome_task() isn't affected, as it moves the task
through SCX_TASK_INIT, which resets runnable_at. The stale runnable_at also
skews the default core-sched ordering in scx_prio_less().
Reset runnable_at in scx_disable_task(), when the task leaves the control
of its scheduler.
Fixes:
|
||
|
Tejun Heo
|
ff343a94e4 |
sched_ext: Leave suspend and hibernation out of the ecaps reports
Suspend and hibernation take down every non-boot cpu and bring them back. A loss and a regain per cpu to every sub-scheduler would be noise. Leave them out: a cpu they take down reports no loss and keeps the caps last delivered, and the reseed after resume reports only the net change. Their downs are the ones with cpuhp_tasks_frozen set between the PM notifications. A kexec jump sets the flag too but sends no PM notification. Two things need care. A cpu that suspend or hibernation takes down doesn't wait for its syncs, and no ecaps_report() mark can be pending on it then. The attach report runs while the attaching task waits uninterruptibly, which holds off the freezer, and the detach report holds cpus_read_lock(). The detach report reaches only active cpus, and the loss on a cpu that suspend or hibernation took down was never delivered. The detach waits for those cpus to come back, so every loss is delivered. They come back before the tasks thaw, so the wait doesn't hold up the freezer. If one fails to come back, the hierarchy is aborted at the end of the PM cycle. v2: Fix why no report mark is pending at a PM down, add a data_race(). Signed-off-by: Tejun Heo <tj@kernel.org> Reviewed-by: Andrea Righi <arighi@nvidia.com> |
||
|
Tejun Heo
|
51d2730ec1 |
sched_ext: Report the ecaps loss when a cpu goes offline
When a cpu goes offline, a sub-scheduler's effective caps on its cid are cleared, but neither the sub-scheduler nor its parent is notified. The sub-scheduler keeps treating the cid as usable, and the parent can't act on the loss, e.g. reset the cpuperf target. Report the loss before the root's ops.cpu_offline() and the regain after its ops.cpu_online(). The zeroing at offline was the last change to a sched's ecaps outside a delivery, so they now change only with one. Syncs on an inactive cpu are processed instead of discarded, and the reseed reschedules the cpu itself, as they got no resched when queued. Suspend and hibernation now report their cpu downs and ups as well. This is temporary, and a later patch leaves them out. Two things need care. A cpu going down marks a sync for each sub-scheduler, and its hotplug thread waits until each marked sync has been processed. The cpu's own ext picks process them, and the ext server is started in case a fair task holds the cpu. Only the marked syncs count, so a flood of changes on the cid can't hold up the hotplug. The detach report reaches only active cpus. It holds off cpu hotplug, so a cpu it skips has reported its loss as it went down, and no unreported loss is left to drop. Drop the cpuperf reset from scx_qmap's ops.sub_detach(), as its ops.sub_child_ecaps_updated() now resets the targets on the loss reports. v2: Reschedule from the irq work and document the regain order. v3: Pack nr_ecaps_offline with kick_sync_pending to avoid a hole (Andrea). Signed-off-by: Tejun Heo <tj@kernel.org> Reviewed-by: Andrea Righi <arighi@nvidia.com> |
||
|
Tejun Heo
|
f89d11642c |
sched_ext: Deliver ecaps only while the root has the cpu online
A sub-scheduler's ecaps ops run in dispatch context, and ops.cpu_online() and ops.cpu_offline() document that the cpu doesn't call ops.dispatch() outside them. The reseed of a cpu coming back is queued before the root's ops.cpu_online(), so the ops of a sub-scheduler and its parent can run for a cpu the root still considers offline. Add SCX_RQ_OPS_ONLINE, set while the root considers the cpu online, from its ops.cpu_online() or its enable until its ops.cpu_offline(). A sync outside that span leaves the ecaps alone, and the reseed, now queued after ops.cpu_online(), reports the net change. SCX_RQ_ONLINE can't be used, as the core changes it ahead of the root's view on both edges. A cpu going offline still drops its caps without a report. While at it, fix the SCX_RQ_ONLINE comments, which claimed that the flag tracks ops.cpu_online() and ops.cpu_offline(). v2: Fix the SCX_RQ_ONLINE comments. Signed-off-by: Tejun Heo <tj@kernel.org> Reviewed-by: Andrea Righi <arighi@nvidia.com> |
||
|
Tejun Heo
|
47769db146 |
sched_ext: Report a sub-scheduler's ecaps on attach and detach
A sub-scheduler learns which cids it holds through ops.sub_ecaps_updated(), but the caps granted from ops.sub_attach() reach it only after its enable bypass lifts, when it is already scheduling its tasks. Report the caps at the end of the enable, before bypass lifts. On disable, report their loss before any task goes back to the parent, so that the parent hears of the loss of every cap it was told of on an active cpu. ops.sub_detach() remains where the parent cleans up after the sub-scheduler. Deliveries no longer depend on bypass, so ops.sub_ecaps_updated() and ops.sub_child_ecaps_updated() also run while a scheduler bypasses. The replay at unbypass goes, and the enable's bypass returns to after ops.sub_attach(). A sub-scheduler's ecaps now change only with the deliveries that report them, so that whenever its ops run under the rq lock, its caps are the ones it was last told of. A sync before the attach report leaves them alone, and the detach report zeroes them as it delivers the loss. A cpu going offline still drops its caps without a report, and those are the only changes not delivered. clear_all_caps() drops that unreported loss on inactive cpus. The following things need care. A report reschedules only the cpus the sub-scheduler was ever granted a cap on. Rescheduling every cpu would disrupt a large machine split many ways, and which cpus still owe a delivery can't be read off the caps. The report and a queued sync reschedule the cpu from hard irq work on that cpu, not with a kick or a direct resched. dispatch_one() processes the syncs only at its start, and a kick only guarantees that a scheduling event completes after it, which can be the pick in progress. That pick has already processed the syncs and clears the resched, and a cpu that then idles or runs a task with an infinite slice leaves the report waiting. A pick runs with IRQs off, so the irq work runs after any pick in progress on that cpu. Under core scheduling, a sibling's committed pick can switch the cpu without dispatch_one(), and a sync queued after that pick is processed only at the cpu's next scheduling event. A cpu running a task of a higher class isn't rescheduled, as its pick would return that task again. While the root bypasses, a queued sync doesn't reschedule the cpu, as a grant or revoke can run in the PM syscore phase, where irq work can malfunction. The report never runs there and reschedules the cpu itself. The report waits on an rcuwait woken from hard irq work. Under core scheduling, a cpu going down can process a sibling's marked sync after hotplug has parked its irq_work thread. On PREEMPT_RT, a regular irq_work queued there would be left behind, and a hard irq can't take a wait queue's lock. v2: Reschedule from hard irq work on the cpu instead of a kick (sashiko). Signed-off-by: Tejun Heo <tj@kernel.org> Reviewed-by: Andrea Righi <arighi@nvidia.com> |
||
|
Tejun Heo
|
c28c637dd8 |
sched_ext: Free a sub-scheduler a grace period after discarding its ecaps syncs
The free path removes a sub-scheduler's ecaps sync node from each cpu's list and then cycles the rq lock until the node reads off-list, as a sync batch can hold the node privately. Cycling the lock is correct only because the sched is bypassing and a bypassing sched's sync calls no op, so the batch touches the pcpu within one rq lock hold. Tying the free path to what the sync does for a bypassing sched is unusual and brittle, and it gets in the way of changing the sync. Delivering ecaps to a bypassing sched or its parent would break the wait, as a delivery drops the rq lock. Wait for the batches with RCU instead, the usual way to wait for lockless users before a free. Discard the nodes after the first grace period as before, and free the sched after a second one. The first grace period waits for the paths that found the sched before it was unlinked, such as the sub-cap kfuncs and the cpu hotplug hooks. Until they finish, they can queue its syncs, and a cpu can take a queued node into a sync batch that the first grace period doesn't wait for. After it, nothing queues the syncs, and the discards remove the nodes still queued. A batch that took a node keeps irqs off from llist_del_all() to its last pcpu write, even across the rq lock drops of a delivery. The second grace period waits for it whether or not it called an op. The exit and disable irq_works, the helper kthread and the bypass load balancing timer are stopped ahead of the per-cpu checks as before, so they move into scx_sched_quiesce_rcu_work() with the checks. Signed-off-by: Tejun Heo <tj@kernel.org> Reviewed-by: Andrea Righi <arighi@nvidia.com> |
||
|
Tejun Heo
|
c71ecdf0d8 |
sched_ext: Factor out cid_pshard()
Factor the cid to pshard lookup out of the ecaps sync paths into cid_pshard() so that new sync paths can share it. No functional change. Signed-off-by: Tejun Heo <tj@kernel.org> Reviewed-by: Andrea Righi <arighi@nvidia.com> |
||
|
Rafael J. Wysocki
|
eece15c579 |
Merge branch 'pm-cpufreq'
Merge cpufreq updates for 7.4: - Minor fixes in various cpufreq drivers (Alexander Koskovich, Daniel Schwierzeck, Frank Zhang, Haotian Zhang, Kathiravan Thirumoorthy, Sumeet Pawnikar, and Xueqin Luo) - Fix wrong behaviors on multi-node servers in the Loongson cpufreq driver (Huacai Chen) - Reject NULL from cpufreq_cpu_get() in the Rust code (Mehmet Koseoglu) - Add suspend OPP to mediatek cpufreq driver and DT (Ryan Brue) - Fixups for a few bugs identified by Sashiko in the amd-pstate cpufreq driver (Mario Limoncielloi, Wentao Guan) - Initial support for maximum frequency capacity handling on Zen6 client in amd-pstate (Mario Limonciello) - Fix for amd-pstate-ut with 'Requested CPU Min frequency' BIOS option (K Prateek Nayak) - New Zen6 tuning values for EPP in amd-pstate (Mario Limonciello, Vishal Badole) - amd-pstate documentation fix (Karl Mehltretter) - Restrict deferred downscaling to low load for shared policies in the conservative cpufreq governor to avoid undesirable CPU frequency reduction in some cases (Shengming Hu) - Serialize sugov_start() against rate limit updates in the schedutil cpufreq governor to avoid using a stale rate limit value (Hui Su) * pm-cpufreq: (34 commits) cpufreq: loongson3: Replace IOCSR read/write with MMIO ones cpufreq: loongson3: Use global physical CPU ID in get/target callbacks cpufreq: loongson3: Replace per-package mutex with per-node cpufreq: loongson3: Adjust the width of id and val in smc_message cpufreq: loongson3: Make this driver depend on MACH_LOONGSON64 cpufreq: schedutil: Serialize sugov_start() against rate limit updates arm64: dts: mediatek: mt8173: Mark the lowest CPU OPPs for suspend cpufreq: mediatek: Enter system sleep at the suspend OPP cpufreq: qcom-cpufreq-hw: fix device node refcount leak in phandle parsing cpufreq: qcom-nvmem: Add IPQ9650 support cpufreq: Add Milos to cpufreq-dt-platdev blocklist cpufreq: Use %pe to print error pointers symbolically rust: rcpufreq_dt: add module alias cpufreq: sti: avoid NULL dereference in dev_err() cpufreq: tegra194: fix double-pointer error in get_cpu_ndiv cpufreq: sparc-us2e: fix frequency table index copy-paste error rust: cpufreq: reject NULL from cpufreq_cpu_get() cpufreq: amd-pstate-ut: Fix max_freq and EPP test failures on Zen6 cpufreq: amd-pstate: Propagate cppc_set_auto_sel() errors on mode change cpufreq: amd-pstate: Restore previous mode when changing driver mode fails ... |