pick_task_fair() and yield_task_fair() call update_curr(&rq->cfs) to bring
curr up to date before they look at the eevdf state. With cgroups that
does not happen: update_curr() reads ->h_curr, which on the root cfs_rq is
the top level group entity, and returns at the !entity_is_task() check
before touching vruntime. Both then read ->curr, so the guard and the
update disagree about which entity they mean.
Counting how often ->h_curr and ->curr differ at pick_task_fair(), on one
CPU for 10s with three busy tasks and one 200us-periodic task:
all tasks in the root cgroup 43321 calls, 0 no-ops
busy tasks in G0, periodic in G1 45211 calls, 45193 no-ops
Whether that matters depends on what precedes the pick. Since
commit 68e3748781 ("sched/fair: Fix flat hierarchy") the tick and
enqueue/dequeue all update curr correctly, so on the normal reschedule
path only the microseconds between those and the pick are missing, and I
could not measure a latency difference there. Three paths have nothing
before them on that rq though:
- pick_task() on the sibling rqs of a core under core scheduling
(kernel/sched/core.c), which updates that rq's clock first for
exactly this reason
- fair_server_pick_task()
- yield_task_fair(), where the stale value feeds the entity_eligible()
test that guards forfeiting the remaining vruntime
There curr can be a full tick behind, as it was before that commit.
No new behaviour for the entity being updated: without cgroups ->h_curr
is already the task, so these two call sites already run the full
update_curr() including update_deadline(), dl_server_update() and the
resched_curr_lazy() at the end. This makes the cgroup case do the same.
Fixes: 85570f10a4 ("sched/eevdf: Move to a single runqueue")
Signed-off-by: Zhan Xusheng <zhanxusheng@xiaomi.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Vincent Guittot <vincent.guittot@linaro.org>
Link: https://patch.msgid.link/20260822105930.2352761-1-zhanxusheng1024@gmail.com
ops.cgroup_set_idle() is documented to be invoked when a cgroup
transitions between idle and non-idle states, and scx_group_set_weight()
already skips value-preserving writes. scx_group_set_idle() delivers
every write unconditionally, so rewriting an already-correct cpu.idle
value feeds the BPF scheduler a transition callback each time, which
toggle- or accounting-based schedulers miscount. Mirror the weight
guard and only deliver on an actual change.
Verified with a probe scheduler printing each callback: rewriting
cpu.idle=1 twice on an already-idle cgroup delivered two callbacks
before and none after.
Fixes: 347ed2d566 ("sched/ext: Implement cgroup_set_idle() callback")
Link: https://lore.kernel.org/r/b53c61a1-4d7d-4232-941f-d48b0563d4ed@linux.dev
Signed-off-by: Tao Cui <cuitao@kylinos.cn>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
scx_cgroup_init_args carries the initial weight and bandwidth control
parameters of a cgroup to ops.cgroup_init(), but not its cpu.idle
state. A cgroup that was already configured idle before the scheduler
was loaded (or before it was onlined under it) is presented as
non-idle, and the BPF scheduler only learns about it if cpu.idle is
written again later.
Add the sched_idle state to scx_cgroup_init_args and fill it in all
four places that build the args: scx_tg_online() for cgroups onlined
under the scheduler, scx_cgroup_init() for cgroups that already exist
when the scheduler is loaded, and the sub-scheduler handover paths
scx_cgroup_claim_subtree() and scx_cgroup_return_subtree().
Verified in a VM with a probe scheduler printing the init args: a
cgroup configured cpu.idle=1 before loading shows sched_idle=1 in
ops.cgroup_init(), the default shows 0, and later cpu.idle writes
still come through ops.cgroup_set_idle(). The sub-scheduler paths
are compile-tested only.
Fixes: 347ed2d566 ("sched/ext: Implement cgroup_set_idle() callback")
Signed-off-by: Tao Cui <cuitao@kylinos.cn>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
- The task ownership check in the dispatch queue move operation raced
against the task exiting or moving to a different sub-scheduler,
spuriously triggering scheduler aborts. Fix by moving the check under
the queue lock.
- The cgroup bandwidth change callback runs in a sleepable context but
sleepable implementations were rejected at load time. Allow them and add
a marker so userspace can detect the capability.
- Sync tooling headers with the scx repo for accumulated compatibility
improvements.
- Example scheduler fixes: ignored timer re-arm failures and vtime credit
loss on cgroup migration.
- Documentation and comment fixes.
-----BEGIN PGP SIGNATURE-----
iIQEABYKACwWIQTfIjM1kS57o3GsC/uxYfJx3gVYGQUCapXcug4cdGpAa2VybmVs
Lm9yZwAKCRCxYfJx3gVYGZFfAQCMdpcFMqLdaHkRMiWK+SopQ703AnnpoX9xC81t
vy81mQD/QlIEZimidIS2xju3/HkHfVXX6hHxAbMy8hdclhYHlgM=
=256l
-----END PGP SIGNATURE-----
Merge tag 'sched_ext-for-7.3-rc1-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/sched_ext
Pull sched_ext fixes from Tejun Heo:
- The task ownership check in the dispatch queue move operation raced
against the task exiting or moving to a different sub-scheduler,
spuriously triggering scheduler aborts. Fix by moving the check under
the queue lock
- The cgroup bandwidth change callback runs in a sleepable context but
sleepable implementations were rejected at load time. Allow them and
add a marker so userspace can detect the capability
- Sync tooling headers with the scx repo for accumulated compatibility
improvements
- Example scheduler fixes: ignored timer re-arm failures and vtime
credit loss on cgroup migration
- Documentation and comment fixes
* tag 'sched_ext-for-7.3-rc1-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/sched_ext:
sched_ext: Fix missing @slice and @vtime descriptions in finish_dispatch() kernel-doc
sched_ext: Fix several comment issues
sched_ext: Check bpf_timer_start return values in scx_qmap
sched_ext: Fix vtime delta loss in scx_flatcg cgroup migration
sched_ext: Fix timer pinning and return value in scx_central
docs/sched_ext: document that cgroup CPU knobs are scheduler-dependent
sched_ext: Fix spurious aborts in scx_bpf_dsq_move() on ownership change races
sched_ext: Sync common and compat headers from the scx repo
sched_ext: Sync tools autogen enum headers from the scx repo
Docs/admin-guide/cgroup-v2: document BPF scheduler callbacks for cpu.max and cpu.idle
sched_ext: Fix nonexistent field in sched-ext.rst example
sched_ext: Allow ops.cgroup_set_bandwidth() to be sleepable
Commit 13f1eae3b6 ("sched_ext: Synchronize slice and dsq_vtime
writes") added the slice and vtime parameters to finish_dispatch() but
did not update its kernel-doc, which produces warnings:
Warning: function parameter 'slice' not described in 'finish_dispatch'
Warning: function parameter 'vtime' not described in 'finish_dispatch'
Describe both parameters using the same wording as
dispatch_to_local_dsq(), which receives the same values.
Signed-off-by: Liang Luo <luoliang@kylinos.cn>
Signed-off-by: Tejun Heo <tj@kernel.org>
in the subtle limitations & fragility of guards (again...) and
caused a regression (Peter Zijlstra)
Signed-off-by: Ingo Molnar <mingo@kernel.org>
-----BEGIN PGP SIGNATURE-----
iQJFBAABCgAvFiEEBpT5eoXrXCwVQwEKEnMQ0APhK1gFAmqTmG0RHG1pbmdvQGtl
cm5lbC5vcmcACgkQEnMQ0APhK1iV+A/+J2IN9xHNPv0O9rLKaJvLsNnlMPQ4QJJB
kTbEOLDrAE7ozTmP2sLfUu75w1FQz0Hp00/tf1V7VFYaSeMBdMoqJs3itCtuqBrU
qkVFrx7splaNauvFgxdaHd5dYGszwFOjixMwhezQC1wn46ckW29bEHbIrmY5j1xC
zF7iSTaVp2zHBbkh0n6ozN28zSbHZcLEAD8mvGoscv3Bnb+9HRMAr4weTHC94kq5
4hDaRS48CgofxEfXT0erp6Rc7lS1YTUltSvkICORTv0tYtDqDf1GCiZKIjaW8FqW
N7Nre77fSlH6HfzzujNgNOhGAGoVO+Ki9vfqgnYhnFRc864g9OTREmdNLlJ2V5XB
yC0SCbAeAjOvbtTLQlRmDlKWJPlwKgXkatGupczkCna3EeL9eXx33fOO6kOGxNNs
RpQXv+wPv0S9EDdIkkuMwhx0dq6yUCfrA22+wlXUzFhXvHTbhqZlDneI9OtzubZm
kU+Vf8dgVU0z59C+2ZOwlp5YHVXquRfNXOz8lrRIeb0y0Iyvlo7pDuQ/t8w3Bh3c
KVlAEEQsK8maBHgHHKMv8ml4W6XJHC9KG6x0TtKDv+ntmrNBDhvq2sxWrhBA7JT4
kXdab1QG/hP2FEY3M6bChEadT64U1BeGlcI9xMkhX886PwrnOMzTd9xiAN7UtdCR
dITQG+bHeQo=
=EtPf
-----END PGP SIGNATURE-----
Merge tag 'locking-urgent-2026-08-30' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
Pull locking fix from Ingo Molnar:
- Revert a commit to spinlock cleanup guards that got caught up
in the subtle limitations & fragility of guards (again...) and
caused a regression (Peter Zijlstra)
* tag 'locking-urgent-2026-08-30' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip:
locking: Revert switching guards to _irq_{disable,enable}()
- Fix a kernel panic during PMU unbind in the intel_rapl power capping
driver and sign-extend the PMU delta on counter wraparound in it to
avoid misreporting energy (Sumeet Pawnikar and Yifan Li)
- Unblock runtime PM when device prepare fails that was not done by
mistake (Shibo Zhu)
- Fix possible rate limit overflow on 32-bit systems in the schedutil
cpufreq governor (Hui Su)
- Consolidate HWP P-states initialization in the intel_pstate cpufreq
driver and make that driver avoid using the DESIRED_PERF HWP hint
when the Dynamic Efficiency Control (DEC) is enabled in the processor
to avoid inconsistent behavior (Rafael Wysocki)
-----BEGIN PGP SIGNATURE-----
iQFGBAABCAAwFiEEcM8Aw/RY0dgsiRUR7l+9nS/U47UFAmqPLkgSHHJqd0Byand5
c29ja2kubmV0AAoJEO5fvZ0v1OO1rjkIAJHjnU5/ak8dVrKfDNdK7vcP656sNJkx
gsVjdrU0ki4JSE9n/PE2Z1SjqSz0DvWnk1RYxXBKYbwTswuBE7xqcT5M2g1RycbA
LyEQRRUbFUHJANpNko1y431BPyiSzX/YjdFadC9vRi/IhTVxJ4SpEp0aXnqy7ANV
JppXhyRDpgEcH2OjXhjKmKFnYD2VBw0zcIidok5uTZg6rftlxLRpzabMiGJ+T4x4
h5l5ZDejoMnL/A6MUyrJO5cXi5E+moYXFZj7ofTgX5aNzxxu0rdQ130VMlChQI19
nc1GdStcYO74xTguexlxU6nKWL3eiLaLqulEqghInAvVq+HEQyghsLc=
=kZl3
-----END PGP SIGNATURE-----
Merge tag 'pm-7.3-rc1-2' of git://git.kernel.org/pub/scm/linux/kernel/git/rafael/linux-pm
Pull more power management updates from Rafael Wysocki:
"These fix two issues in the intel_rapl power capping driver, fix a
potential issue in the schedutil cpufreq governor on 32-bit systems,
fix a runtime PM issue related to failing system suspend, and update
the intel_pstate cpufreq driver:
- Fix a kernel panic during PMU unbind in the intel_rapl power
capping driver and sign-extend the PMU delta on counter wraparound
in it to avoid misreporting energy (Sumeet Pawnikar and Yifan Li)
- Unblock runtime PM when device prepare fails that was not done by
mistake (Shibo Zhu)
- Fix possible rate limit overflow on 32-bit systems in the schedutil
cpufreq governor (Hui Su)
- Consolidate HWP P-states initialization in the intel_pstate cpufreq
driver and make that driver avoid using the DESIRED_PERF HWP hint
when the Dynamic Efficiency Control (DEC) is enabled in the
processor to avoid inconsistent behavior (Rafael Wysocki)"
* tag 'pm-7.3-rc1-2' of git://git.kernel.org/pub/scm/linux/kernel/git/rafael/linux-pm:
powercap: intel_rapl: Fix kernel panic during PMU unbind
PM: sleep: Unblock runtime PM when device prepare fails
powercap: intel_rapl: Sign-extend the PMU delta on counter wraparound
cpufreq: intel_pstate: Avoid using DESIRED_PERF when DEC is enabled
cpufreq: intel_pstate: Consolidate HWP P-states initialization
cpufreq: schedutil: Fix rate limit overflow
Merge additional cpufreq updates and one update related to system sleep
for 7.3-rc1:
- Unblock runtime PM when device prepare fails that was not done by
mistake (Shibo Zhu)
- Fix possible rate limit overflow on 32-bit systems in the schedutil
cpufreq governor (Hui Su)
- Consolidate HWP P-states initialization in the intel_pstate cpufreq
driver and make that driver avoid using the DESIRED_PERF HWP hint
when the Dynamic Efficiency Control (DEC) is enabled in the processor
to avoid inconsistent behavior (Rafael Wysocki)
* pm-cpufreq:
cpufreq: intel_pstate: Avoid using DESIRED_PERF when DEC is enabled
cpufreq: intel_pstate: Consolidate HWP P-states initialization
cpufreq: schedutil: Fix rate limit overflow
* pm-sleep:
PM: sleep: Unblock runtime PM when device prepare fails
Revert commit 1b08668748 ("locking: Switch to _irq_{disable,enable}()
variants in cleanup guards").
While the guards are properly nested, not all wrapped code is nice, as already
highlighted by that fair.c hunk.
Syzbot found another instance of this pattern in posix_timer_delete(), which
does spin_unlock_irq()+spin_lock_irq() inside scoped_guard(spinlock_irq).
Combined with this patch, that goes sideways most spectacular.
Undo this until we've developed stronger tools / debug for such issues.
Fixes: 1b08668748 ("locking: Switch to _irq_{disable,enable}() variants in cleanup guards")
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/20260824105523.GA4121620%40noisy.programming.kicks-ass.net
- Add missing cpus_read_lock locking to rebuild_sched_domains()
(Sebastian Andrzej Siewior)
- Fix division by zero bug in tg_cpus() that can be triggered
with empty cpusets (Jake Steinman)
Signed-off-by: Ingo Molnar <mingo@kernel.org>
-----BEGIN PGP SIGNATURE-----
iQJFBAABCgAvFiEEBpT5eoXrXCwVQwEKEnMQ0APhK1gFAmqKGdMRHG1pbmdvQGtl
cm5lbC5vcmcACgkQEnMQ0APhK1jQ3w/9Hpnbu78VJjicFBH/De/xXJ10FBJrZMI2
WzS7XXs2AeSPM9aUrhwkVgus/6x0rkAShht0tb9+ax6/DcDDxcSW2tBzCUDZ++Wu
g1oUDlbEP8OAtotJQLx3ss+/vgEjTEv7gidMNEIdqGRVlMyfUmAvBZ0C9REdFZT1
Lq8I0KtEVNKCLMi5LfcafQTHld4o67Wb2d6DqMDbSCkT45sjGvlVOmYnJPhVtUqs
JaOdvit7uheOgc/mtlWsdJXd4yYtKlMpqq5tzQSCGEfReoMdsFa8FXpaZiT/kn6N
XuCaqgtUG5J94bREzNxpsfqKclK4QWqUj3MmD2i5vzoGJkEiXDyY6B6WNqGex2KG
gJvglaW+HfyDjh8RB5n3KOGX6ikYtDBYc8HtmJewOfEGQnX/c+E7Xhl4/9u1CUnN
6z47IP80Ch8tWQNz5XZUplzbm8DL+gYMFdh8L8oRXab3rFby72h+Ftxz/epNP14W
1D1mKZE+TkeyGnKk1cidf/cz60qPBPrLp3gYzbOeaJPk5y388t/gmO+V07Mwzl2Q
O17hkIk0+4ySiRvnd8IxDowTWz53FE9qPoky5zPRGf+SvZXu9GAbY1Hwo+jl6Qle
DTg1jptYkTe5Mnttmxgi2gSTPofrGloUqXAMP/T+U+Qos+dF/ujm4pgVdWsLLD/F
rhBjdJRybrE=
=RQRK
-----END PGP SIGNATURE-----
Merge tag 'sched-urgent-2026-08-22' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
Pull scheduler fixes from Ingo Molnar:
- Add missing cpus_read_lock locking to rebuild_sched_domains()
(Sebastian Andrzej Siewior)
- Fix division by zero bug in tg_cpus() that can be triggered with
empty cpusets (Jake Steinman)
* tag 'sched-urgent-2026-08-22' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip:
sched/fair: Floor tg_cpus() at 1
sched/topology: Add a cpus_read_lock to rebuild_sched_domains()
- Series to enforce that private futex owner shares the mm when
attaching. (Kyle Zeng, Thomas Gleixner)
- Fix race on the initial mm->futex.phash.ref allocation
(Hyunwoo Kim)
- Fix might_sleep() warning in futex_pivot_pending()
(Peter Zijlstra)
Signed-off-by: Ingo Molnar <mingo@kernel.org>
-----BEGIN PGP SIGNATURE-----
iQJFBAABCgAvFiEEBpT5eoXrXCwVQwEKEnMQ0APhK1gFAmqKGJwRHG1pbmdvQGtl
cm5lbC5vcmcACgkQEnMQ0APhK1iKxhAAj+A3L8a1OHRCKCMIE1gbk+bL5CYYrDqf
TO642/rGmvvrMdNtocK3IbzIady0umhZFrPqIRJcN5SAnoZf6mL5uD4XaR4xgaTp
prbNTXJrxzPMN5u3DcqnlWfGwFYXZxJIN6WBZCT7N6D/czr0Mqh7+Isdxey6nDWp
zWZfJ4DeVbSbf+2zft6pjwzXQAljyvEYEilV8+xgzwNfJjmG1QRS+MisPjjuGMZh
4C5bTTo966WgyKCu0MdtuP9a/WFT4ZUMWXXCB89Dz3mXImntltu+0rYEsg00I/NA
lSjEOnTpozqQwLPAUGBwGQRCP8tbPxRW9k+dmWUKY7/Ox5QtoNRzMt1n2hcG2uXk
VkefzABd0HWvUOYy8LL1o32rxdv78iqmfVxKyORHZKBS59OM+CAahCvgV9bVgE+g
Rbp1RzsPZJOEBkDY6bncyUN0Qm7bQqoCXt4ZEaX/Cj7lPRrCjzj+NXzuYKUReib1
z79Xnrps4r5QlY1PVICF4lKAWnUsLRSr/A+vPYgTJRQKTo0DwDaPTcEs4DrcAZ1j
0uTlk2B/mcThEDEd+kB7vBZcc7QnNsh8Qp5objXNPsq/7+8cJdHwXY1/qxsXgLFf
aQMhKJY/kfoE24yyb18SosXqMoxc8rDZkMKv91z597PlqIWbPMhNeg/dGlx8ptcl
S4DIaIZgJ5E=
=9ORp
-----END PGP SIGNATURE-----
Merge tag 'locking-urgent-2026-08-22' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
Pull futex fixes from Ingo Molnar:
- Enforce that the private futex owner shares the mm when attaching
(Kyle Zeng, Thomas Gleixner)
- Fix race on the initial mm->futex.phash.ref allocation (Hyunwoo Kim)
- Fix might_sleep() warning in futex_pivot_pending() (Peter Zijlstra)
* tag 'locking-urgent-2026-08-22' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip:
futex: Fix might_sleep() warning in futex_pivot_pending()
futex: Fix race on the initial mm->futex.phash.ref allocation
futex: Clean up the redundant exit/exec functions
futex/pi: Plug private futex exec() race
futex: Sanitize and document task_struct::futex::state transitions
futex/pi: Reject cross-mm private futex owners
scx_dsq_move() verifies that the task belongs to the calling scheduler
before taking any locks and aborts the scheduler on mismatch. The task can
lose the sched association at any point: It can run and fully exit, which
clears the association, or get rehomed to a different sub-sched. Both are
benign races, but the early ownership check escalates them into scheduler
aborts.
Move the ownership check below the cursor-lost check. Every ownership change
dequeues the task first, so a task that is still on the iterated DSQ under
the lock while owned elsewhere indicates a genuine violation and should
abort.
Also fix two stale comments still referencing sched_ext_free(), which has
been renamed to sched_ext_dead().
Fixes: bb4d9fd551 ("sched_ext: scx_dsq_move() should validate the task belongs to the right scheduler")
Signed-off-by: Tejun Heo <tj@kernel.org>
-----BEGIN PGP SIGNATURE-----
iQJEBAABCAAuFiEEwPw5LcreJtl1+l5K99NY+ylx4KYFAmqGD2AQHGF4Ym9lQGtl
cm5lbC5kawAKCRD301j7KXHgpomLD/9rr3SIQvdQRTnSCJlw1cFypSrnRhGAx1g8
v1F7UXuf4Abt4GIerbdXmBTx9zZRctpcOC41ReQk3YKkYuukuvna+Z6tBGf5I5+e
3SR3Zkrj2k4zyNC49g0/QkUJkTTS4tlsBFhNJtY3Q5BBIRMRd/sPvZ+cLnD7g/wH
3zM9IaEMk9xu9mF1nGI7LRl1qY0IOr4ynQpqohl5sEL3TQWgIa0r2B0jhGacBAvC
GGDwIwGDrM6oGwCn6mLN78O5odhLyTufIMNy3EnVw7BcDpLApcnJzYYjGzv+ElCM
WLBJwWYj+PYeoenuJuFhbRihbOPg5gZT3Mn8CUhLxUYzwZvIdFGWOjcGffD9sVu/
JO4nB6A+2AG/wgsb2YVKefhN86FoD0F/2hhdEdgZPJE6a+7+xeyyp9IzyQEWd0cg
Seo06fxE4pwF+dczXT3jzU3LXSZb/Jbh9hzu163VMSWbGYDCpDzfipqZGeN6DWE7
nG2NssAQtjtZrgdggYbZXtwecLZhu3cMj3jbhS4wSUzhQFW42yceJstCx4wpRhW9
m9EXjHQnRFMAb2TjUC3FzEncjbxL+uACjcKhTetd1hLbTI5I8KArZXjam1OdJOHV
urdtsru79LanQLAgRwWYPeKCZGNhX7I2o3OsmVDLQgHIDaZguTcPUVQs3wVui7Dk
IDmmrtwzpg==
=G1l4
-----END PGP SIGNATURE-----
Merge tag 'for-7.3/block-20260819' of git://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux
Pull block updates from Jens Axboe:
- NVMe updates via Keith:
- Enable Clang context analysis for the nvme host driver, adding
context annotations across core, fabrics, rdma, tcp and pci
- nvmet reservation state exposed through a new namespace-level
debugfs directory, plus ABI documentation for the host sysfs and
target configfs interfaces
- nvme-tcp host memory disclosure fixes on the read path: reject a
read that transferred too few bytes, don't accept C2HData based
on blk_rq_payload_bytes() alone, and fix the R2T case for a read
command
- Parallelize nvme-rdma I/O queue allocation and startup (Surabhi)
- Apple nvme fixes and quirks: page aligned admin queue buffers,
destroy the admin queue on removal, and various DMA/NVMMU
correctness fixes
- A large pile of nvmet and host fixes for out-of-bounds reads,
refcount/resource leaks, and NULL derefs across auth, zns,
passthru, pci-epf, rdma and configfs
- Various other fixes and cleanups
- MD updates via Yu Kuai:
- llbitmap reshape support, the large series wiring exact bitmap
mapping and reshape lifecycle through raid5 and raid10, growing
the page cache in place, and remapping checkpointed bits as
reshape progresses
- raid5 fixes for lockless max_nr_stripes and recovery_offset
accesses, a reshape deadlock with more failed devices than max
degraded, and bitmap batch counter consistency
- Atomic write handling for raid1/raid10, and removal of the
REQ_NOWAIT support from raid1/10/456
- raid5-ppl use-after-free fix in ppl_do_flush()
- A batch of smaller fixes across md core and the bitmap code
- s390/dasd ESE full-track write support and the surrounding
infrastructure, plus enabling CONTEXT_ANALYSIS for s390/block
- RWF_DONTCACHE support for block devices, built on new task-context
bio completion infrastructure, and wiring it up for the iomap and
buffer dropbehind writeback paths
- Async io_uring zone reset all, plus zone management command cleanups
allowing REQ_NOWAIT and tightening conventional zone rejection
- Block integrity refactoring: lift BIP_CHECK_FLAGS to the shared
header, handle nogenerate/noverify properly in fs-integrity, and drop
the blk-integrity.h include from bdev.c
- Split out a new blk_plug.h header
- ublk improvements: add UBLK_F_IO_DESC_SIZE, split request validation
from io_desc init, reject non-power-of-2 zone sizes in SET_PARAMS,
and a series of hardening fixes around map/unmap and auto buf reg
- null_blk cleanups and configfs serialization fixes
- nbd queue freeze removal on the setup paths, and a new
pre_defined_connections module parameter for pre-created devices
- blk-cgroup fixes for the race between policy activation and blkg
destruction, and accounting per-cpu stats over possible CPUs across
blk-stat, iolatency, iocost and kyber
- Various dio fixes: leak on metadata mapping error, validate user
space vectors during extraction, and set dma_alignment from the
backing file for loop and zloop direct I/O
- bio cleanups
- Various other fixes and cleanups all over
* tag 'for-7.3/block-20260819' of git://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux: (241 commits)
nbd: add pre_defined_connections module parameter for pre-created devices
nbd: remove queue freeze for newly created nbd from netlink path
nbd: factor out a nbd_genl_foreach_sock
nbd: skip queue freeze when setting size at device startup
nbd: remove queue freeze in nbd_add_socket
nbd: clear queue limits on disconnect
nbd: disallow NBD_SET_SOCK on an active device
nbd: simplify find_fallback() by removing redundant logic
blk-mq: add missing call to srcu_barrier() in blk_mq_free_tag_set()
block: mtip32xx: synchronize ioctls with device removal
ublk: avoid teardown retry loop on xarray allocation failure
null_blk: fix UBSAN shift-out-of-bounds when zone_size is 0 or overflows
block: don't include blk-integrity.h in bdev.c
xfs: avoid double deferrals for RWF_DONTCACHE writes
loop: Fix recently introduced lock inversion
block: set QUEUE_FLAG_DYING unconditionally in blk_mark_disk_dead()
swim3: Add missing MODULE_DESCRIPTION
selftests: ublk: add SET_PARAMS validation test
selftests: ublk: add helper for SET_PARAMS
ublk: reject non-power-of-2 zone sizes in SET_PARAMS
...
This depends on the arena argument support in the BPF tree and should be
pulled after the scheduler core and BPF pulls. The patches based on bpf-next
were kept on a separate branch which was merged into for-7.3 just now. The
same merged result was in linux-next for several days.
Most of this cycle completes the enqueue-path support for hierarchical
sub-scheduling, which makes sub-scheduler support feature complete: a root
BPF scheduler can now hand a cgroup subtree over to a nested sub-scheduler
together with revocable CPU grants, and the sub-scheduler owns all
scheduling decisions for its tasks on those CPUs.
Development volume was high and a number of changes plugging holes in the
new support landed late in the cycle. Also included are core scheduling
fixes that were completed too late for the v7.2 release and are routed
through this pull request.
- Sub-scheduler CPU delegation:
- Parent schedulers now grant and revoke per-CPU capabilities (enqueueing,
preemption, CPU frequency control) on their children, enforced on every
path a scheduler can reach a CPU through. Previously only dispatching
could be delegated; this lets sub-schedulers fully schedule their CPUs.
- Rescue execution: a task whose scheduler doesn't have access to the CPUs
the task needs to run on starved until the watchdog ejected the whole
scheduler. The kernel now runs such tasks directly on a small bandwidth
budget, turning a scheduler-killing failure into bounded degradation.
- Cgroup integration: tasks migrating across a sub-scheduler boundary
weren't re-homed to the new owner, causing wrong-scheduler scheduling
and a use-after-free. Sub-schedulers now take over their cgroup subtree
and receive its cgroup callbacks.
- Arena objects now cross the kernel/BPF boundary as typed pointer
arguments, translated transparently by the BPF tree's new arena argument
support, replacing untyped arguments with manual translation.
- scx_qmap now demonstrates full hierarchical sub-scheduling.
- Robustness improvements: the abort path is now NMI-safe, fixing deadlocks
when errors are raised from NMI context and making hardlockup recovery
direct. Reenqueue loops that could monopolize a CPU ahead of the watchdog
now eject the offending scheduler, and stalls are blamed on the scheduler
actually responsible.
- Hardening: BPF-writable arena memory is validated before kernel use, and
task slice and vtime writes got explicit synchronization rules, closing
corruption vectors open to buggy or malicious schedulers.
- Core scheduling: sched_ext dispatching can drop the rq lock inside the
core-wide pick, which let interleaving selections corrupt each other's
state and hard-hang the machine. The selection now restarts when the lock
was released. The task ordering callback was also invoked with its
arguments swapped, and the default ordering is updated to work across
sub-scheduler boundaries. The fixes are marked for stable.
- Other fixes headed for stable: a task init leak on fork failure during
enable, tooling compat macros that silently failed to detect newer
kernels, and a crash on reenqueueing against a destroyed dispatch queue.
- Tooling: scx_pair moves off deprecated callbacks, and the deprecated
scx_bpf_cpu_rq() kfunc is removed.
-----BEGIN PGP SIGNATURE-----
iIQEABYKACwWIQTfIjM1kS57o3GsC/uxYfJx3gVYGQUCaoOA7w4cdGpAa2VybmVs
Lm9yZwAKCRCxYfJx3gVYGQWqAP9Sy8GwS7dRdGze/eHwYlDBt5U9ayd2ntR0Z+H1
1Hd23AEA5kYPaEN68OgCXh/XqmFljkvEgEisXtMtw8XsZA+pHQA=
=OwIm
-----END PGP SIGNATURE-----
Merge tag 'sched_ext-for-7.3' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/sched_ext
Pull sched_ext updates from Tejun Heo:
"Most of this cycle completes the enqueue-path support for hierarchical
sub-scheduling, which makes sub-scheduler support feature complete: a
root BPF scheduler can now hand a cgroup subtree over to a nested
sub-scheduler together with revocable CPU grants, and the
sub-scheduler owns all scheduling decisions for its tasks on those
CPUs.
Development volume was high and a number of changes plugging holes in
the new support landed late in the cycle. Also included are core
scheduling fixes that were completed too late for the v7.2 release and
are routed through this pull request.
Sub-scheduler CPU delegation:
- Parent schedulers now grant and revoke per-CPU capabilities
(enqueueing, preemption, CPU frequency control) on their children,
enforced on every path a scheduler can reach a CPU through.
Previously only dispatching could be delegated; this lets
sub-schedulers fully schedule their CPUs.
- Rescue execution: a task whose scheduler doesn't have access to the
CPUs the task needs to run on starved until the watchdog ejected
the whole scheduler. The kernel now runs such tasks directly on a
small bandwidth budget, turning a scheduler-killing failure into
bounded degradation.
- Cgroup integration: tasks migrating across a sub-scheduler boundary
weren't re-homed to the new owner, causing wrong-scheduler
scheduling and a use-after-free. Sub-schedulers now take over their
cgroup subtree and receive its cgroup callbacks.
- Arena objects now cross the kernel/BPF boundary as typed pointer
arguments, translated transparently by the BPF tree's new arena
argument support, replacing untyped arguments with manual
translation.
- scx_qmap now demonstrates full hierarchical sub-scheduling.
Other fixes and updates:
- Robustness improvements: the abort path is now NMI-safe, fixing
deadlocks when errors are raised from NMI context and making
hardlockup recovery direct. Reenqueue loops that could monopolize a
CPU ahead of the watchdog now eject the offending scheduler, and
stalls are blamed on the scheduler actually responsible.
- Hardening: BPF-writable arena memory is validated before kernel
use, and task slice and vtime writes got explicit synchronization
rules, closing corruption vectors open to buggy or malicious
schedulers.
- Core scheduling: sched_ext dispatching can drop the rq lock inside
the core-wide pick, which let interleaving selections corrupt each
other's state and hard-hang the machine. The selection now restarts
when the lock was released. The task ordering callback was also
invoked with its arguments swapped, and the default ordering is
updated to work across sub-scheduler boundaries. The fixes are
marked for stable.
- Other fixes headed for stable: a task init leak on fork failure
during enable, tooling compat macros that silently failed to detect
newer kernels, and a crash on reenqueueing against a destroyed
dispatch queue.
- Tooling: scx_pair moves off deprecated callbacks, and the
deprecated scx_bpf_cpu_rq() kfunc is removed"
* tag 'sched_ext-for-7.3' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/sched_ext: (144 commits)
sched_ext: Drop the dead SCX_DEQ_CORE_SCHED_EXEC test in dequeue_task_scx()
sched_ext: Make core-sched task ordering hierarchy-aware
sched_ext: Use runnable_at for the default core-sched task ordering
sched_ext: Fix inverted ops.core_sched_before() invocation
sched_ext: Move the config-off sub-cap kfunc stubs into sub.c
sched_ext: Rename balance-era identifiers to dispatch terms
sched_ext: Drop the stale keep_prev fixup in dispatch_pick()
sched_ext: Keep kick_sync waiting on the rq's own CPU
sched_ext: Make SCHED_CLASS_EXT select GENERIC_ALLOCATOR
sched_ext/scx_flatcg: Fix cvtime true-up on slice expiry
sched_ext: Don't BUG_ON a destroyed DSQ in process_deferred_reenq_users
sched_ext: Fix scx_bpf_dsq_move_to_local___v2 compat detection
sched_ext: Make scx_bpf_events() read the calling scheduler's counters
sched_ext: Drop unlocked scx_rq_clock_invalidate() from scx_root_disable()
selftests/sched_ext: Fix flaky ddsp failure tests on busy systems
selftests/sched_ext: Make numa idle validation race-free
sched_ext: Fix scx_bpf_dsq_reenq___compat kfunc extern prototype
sched_ext/scx_flatcg: expire cached hweights on weight changes
sched_ext: Fix exit_task leak on fork failure during enable
sched_ext: fix stale references in doc comments
...
* Documentation
Fix kernel-doc warnings by adjusting in file documentation.
* Consolidate do_proc_* function into do_proc_vec.
Consolidate three slightly different implementations of applying a converter
on all elements of a vector. Fixes to this function now propagate to the three
types.
* Miscellaneous
Replaced CONFIG_PROC_SYSCTL with CONFIG_SYSCTL (they were the same) and
restrict cad_pid modifications to global root (GLOBAL_ROOT_UID)
* Testing
This series was run through sysctl selftests/kunit test suite in x86_64. And
went into linux-next around rc2, giving it a bit more than a month of testing.
-----BEGIN PGP SIGNATURE-----
iQGzBAABCgAdFiEErkcJVyXmMSXOyyeQupfNUreWQU8FAmp0ga8ACgkQupfNUreW
QU9RZQv/UFBopK15/HcUPWAp8h0Yq68Reij0ryPhuOuoqRaWjLrz5iT9SvNW4IZH
WfAHal9+gVyAov2TvZUDHE0iojYa1S7l1MSBSvzlEijZnbglnt+63Ijfria3FTr2
FIqk7gOxCpDg7kRh1GkIm8yc6Ke7T2IcbMT8XW5PAs43QqMPgGfeUry14sr6H89u
pc1vXS/PcRDdsKkYKADCuljGG3ywL/7frDhSRcc+5WkCACOujzosmXTL10TVVs7H
6Lz3vujtQyCnv3UxlHicsJQXD60m2gqbCLcqa5gndfE5jO2nS3ZTqRBALT9Cimuk
43k+cKgJwOa5ry9OFXmp+in57hgOg8AZqWWENY2W0RaU6Tw7Ui5x6z36ZtE5PR9u
NlNAOprv6UnKOQ6Bc4GGtfSMh+EviqQBSNQroEz4UxLdrFx2X7SxkSgz5I6gLbzf
w+KZpQ3Y3Kjvr9ywkQUftekWqaeapv/OfrjGXRwiIBYhlurmKqvFpjjSQXmjow9u
RD1bfR9X
=aYrQ
-----END PGP SIGNATURE-----
Merge tag 'sysctl-7.03-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/sysctl/sysctl
Pull sysctl updates from Joel Granados:
- Fix kernel-doc warnings by adjusting in file documentation
- Consolidate do_proc_* function into do_proc_vec
Consolidate three slightly different implementations of applying a
converter on all elements of a vector. Fixes to this function now
propagate to the three types.
- Replace CONFIG_PROC_SYSCTL with CONFIG_SYSCTL (they were the same)
and restrict cad_pid modifications to global root (GLOBAL_ROOT_UID)
* tag 'sysctl-7.03-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/sysctl/sysctl:
sysctl: remove CONFIG_PROC_SYSCTL, it just mirrors CONFIG_SYSCTL
sysctl: move the "cad_pid" entry from pid_table[] to kern_reboot_table[]
sysctl: repair some kernel-doc comments
sysctl: add Returns: kernel-doc for all functions
sysctl: Update API function documentation
sysctl: Rename proc_doulongvec_minmax_conv to proc_doulongvec_conv
sysctl: Group proc_handler declarations and document
sysctl: Replace do_proc_do{int,ulong,uint}vec with do_proc_vec
sysctl: Add negp parameter to douintvec converter functions
sysctl: Move default converter assignment out of do_proc_dointvec
tg_cpus() returns cpuset_num_cpus() unfloored, while its sibling
tg_tasks() already floors its result at 1. calc_concur_shares() feeds
nr = min(tg_tasks(tg), tg_cpus(tg))
into __calc_smp_shares() as shares_max, so an nr of 0 makes shares_max 0.
__calc_smp_shares() ends with
return clamp_t(long, shares, MIN_SHARES, shares_max);
and clamp() yields hi when hi < lo, so a zero shares_max silently defeats
the MIN_SHARES floor and returns 0 -- the exact case the comment above
that line says must return MIN_SHARES instead of 0.
That leaves a group sched_entity with load.weight == 0, and
__calc_prop_weight() then divides by cfs_rq->load.weight:
weight *= se->load.weight;
if (parent_entity(se))
weight /= cfs_rq->load.weight;
which takes a #DE inside enqueue_task_fair():
Oops: divide error: 0000 [#1] SMP NOPTI
RIP: 0010:enqueue_task_fair+0x422/0x950
Call Trace:
<TASK>
enqueue_task+0x8e/0x250
wake_up_new_task+0x148/0x2e0
kernel_clone+0x1c6/0x390
__x64_sys_clone+0xcc/0x100
do_syscall_64+0x147/0x3c0
</TASK>
This is not survivable in practice: with panic_on_oops=0 the kernel took
the first #DE and continued for 476 ms, then faulted at the same RIP with
identical register state and an identical RSP, because the oops recovery
path (kill task -> schedule()) re-enters the same enqueue while the rq
lock is held mid-enqueue. The second fault escalates to a panic.
Flooring tg_cpus() at 1 makes it symmetric with tg_tasks() and keeps
shares_max >= tg_shares, so the MIN_SHARES floor in __calc_smp_shares()
can no longer be bypassed.
Note this only removes the division hazard. Whether cpuset_num_cpus() can
legitimately return 0 -- via the cpu hotplug/suspend path where a v2
cpuset may transiently become empty, or via an RCU race -- is a separate
question still open on the report thread.
Fixes: 90ac22ffef ("sched/fair: Add cgroup_mode: max")
Signed-off-by: Jake Steinman <j@metarealtyinc.ca>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://lore.kernel.org/all/20260818231333.1441757-1-j@metarealtyinc.ca/
Link: https://patch.msgid.link/20260819132104.2148918-1-j@metarealtyinc.ca
- Fix a subtly inconsistency in the timekeeping code, which fails to
account for the monotonicity adjustment in ntp_error. For small changes
of the clocksource multiplicator (+/-1) which are typically used by the
NTP PLL this is hardly to observe. But for larger adjustments,
e.g. caused by a direct frequency setting through adjtimex() the
one-time uncompensated offset is significant.
Cure this by adjusting ntp_error with the resulting offset so that the
discrepancy is smoothed away over time
- Make tick length calculations correct in NTP. The timekeeping core
takes the quantisation of the clocksource into account when calculating
the tick length to compensate for the deviation of the nominal
NTP_INTERVAL_LENGTH. While timekeeping gets this right, NTP is not
aware of that, which means it operates on the nominal value and not on
the actual value which is determined by the clock source frequency. The
rounding of a coarse clocksource like the ACPI PM timer results in a
+127 PPM deviation.
Cure this by exposing the deviation to the NTP code so that it can
operate on the same data as the timekeeping core. This is purely kernel
internal. User space still sees the nominal tick lenght via adjtimex().
- The accuracy of the NTP adjustments is fairly approximate as the code
assumes that the invocations are precisely in NTP interval frequency
ticks and the final adjustment can over and under-run.
Cure this by adjusting ntp_error by the intended skew on each tick to
achieve the desired rate.
- Handle the two competing skews of time offset and time adjustment
correctly by calculating the conflict portion between the skews and
adjusting both accordingly.
- A set of updates and improvements for the selftests
- The usual small fixes and improvements all over the place
-----BEGIN PGP SIGNATURE-----
iQJEBAABCgAuFiEEQp8+kY+LLUocC4bMphj1TA10mKEFAmqCwxwQHHRnbHhAa2Vy
bmVsLm9yZwAKCRCmGPVMDXSYoaOwD/45XBW2s5/L+GSy52r/APrzYOqgjYQdn2UQ
XI85iTjDqg9sfZQNFbr7toJ84gY/wQ0ZxykLMi+LlOSZVe5azSepoUHMg9XGk8S0
4kI+GLznmnDLFypTezpzk3DUgNgu8rhOYvPsgwQEgx0DuzWHRZgNoNDk4AfJ8HP6
8cRgPIJ7CpIbP7/p0Kc5IjW3ybo849wBirl7ZDxNRRao+Dkc56ArZvSfQgnQu5Kw
D3Ei+AFcxpBQ2GEJT4Z/92yNf1x2AXjkhfijgO6hIFzPAR+c049fG5sgneo3GoLQ
riduEPQHvoiJO3gVGzoICF2DAuVOwMHwIBlGtNEDd+FcMAvMU4V3lQi8lJV+9CpH
SJwa5R6KJHg2aTuoPup8lmfJN7BqmthrkZwcYVrv2Ikt5yWGYuoW0/e00So9IfUO
R7UG/VlruckK+T0GCuKBAAfm/+zm6Sbir+4liPDZwDz4dW2lkvkZnAb6xHPCev8S
ioYL/Jqr3GmIMgK7UelUkQ/gy5YCek89JAszdlD+blLAjUMcDeYsZaAN9rSdjzSu
dbEA4Hze6Q8HpIW35bHkytyyjNNhvL6xobjKBNRuOPUHTFMe+s5itMApRlhsu32J
6EL/D1TuwkiHlqaSek7ZqUAiitZoNWxa1NyreM9EuZ7qWPjUWOzlsDXkP2v298+t
p27xt+/8sA==
=prJt
-----END PGP SIGNATURE-----
Merge tag 'timers-core-2026-08-17' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
Pull timer and timekeeping core updates from Thomas Gleixner:
- Fix a subtly inconsistency in the timekeeping code, which fails to
account for the monotonicity adjustment in ntp_error.
For small changes of the clocksource multiplicator (+/-1) which are
typically used by the NTP PLL this is hard to observe. But for larger
adjustments, e.g. caused by a direct frequency setting through
adjtimex() the one-time uncompensated offset is significant.
Cure this by adjusting ntp_error with the resulting offset so that
the discrepancy is smoothed away over time
- Make tick length calculations correct in NTP.
The timekeeping core takes the quantisation of the clocksource into
account when calculating the tick length to compensate for the
deviation of the nominal NTP_INTERVAL_LENGTH.
While timekeeping gets this right, NTP is not aware of that, which
means it operates on the nominal value and not on the actual value
which is determined by the clock source frequency. The rounding of a
coarse clocksource like the ACPI PM timer results in a +127 PPM
deviation.
Cure this by exposing the deviation to the NTP code so that it can
operate on the same data as the timekeeping core. This is purely
kernel internal. User space still sees the nominal tick lenght via
adjtimex().
- The accuracy of the NTP adjustments is fairly approximate as the code
assumes that the invocations are precisely in NTP interval frequency
ticks and the final adjustment can over and under-run.
Cure this by adjusting ntp_error by the intended skew on each tick to
achieve the desired rate.
- Handle the two competing skews of time offset and time adjustment
correctly by calculating the conflict portion between the skews and
adjusting both accordingly.
- A set of updates and improvements for the selftests
- The usual small fixes and improvements all over the place
* tag 'timers-core-2026-08-17' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip: (58 commits)
selftests: timers: nsleep-lat: Check all calls to clock_nanosleep() and clock_gettime()
selftests: timers: nsleep-lat: Reuse kselftest error numbers
selftests: timers: nsleep-lat: Explicitly list the tested clocks
selftests: timers: nsleep-lat: Use NSEC_PER_MSEC define for unreasonable latency
selftests: timers: nanosleep: Report each test separately
selftests: timers: nanosleep: Explicitly handle timer_delete() failure
selftests: timers: nanosleep: Move all single clock tests out of the loop in main()
selftests: timers: nanosleep: Reuse kselftest error numbers
selftests: timers: nanosleep: Explicitly list the tested clocks
selftests: timers: nanosleep: Drop output alignment
selftests: timers: Use clock_name() and constants from clock-helpers.h
selftests: Add clock-helpers.h
timer_list: Use ktime_t over nanoseconds
timer_list: Use standard 'long long' format placeholders
hrtimer: Add a lockdep assertion to hrtimer_update_base()
timekeeping: Use u32 for clock_was_set_seq
timekeeping: Rename clockid_aux_valid() to clockid_is_aux_clock()
hrtimer: Account nr_retries on recovered interrupt retries
timers/itimer: Zero-init old itimerval before copy to userspace
nohz: Replace dead select with choice default
...
Load-balancing updates:
- Series to 'flatten the pick': improve cgroup scheduling, which has
always been problematic and painful, which has caused various
scheduling misbehavior such as the mishandling of reniced tasks et al.
Add various cgroup weight distribution methods via cgroup_mode:
'up', 'max', 'concur' and 'tasks' - with the default being 'concur'
which is the most precise yet also most expensive version.
Finally, change cgroup scheduling to a single runqueue.
(Peter Zijlstra)
- Series to improve the scheduling latency of short slice tasks
(Vincent Guittot)
- Series to fix cluster scheduling in the presence of asymmetric capacity
(Ricardo Neri)
- Prefer fully idle cores for NOHZ balancing (Andrea Righi)
- Don't trigger active load-balancing if src_rq->curr is not on_rq
(Xin Zhao)
PSI updates:
- Skip irqtime accounting when no new irq time has elapsed
(Usama Arif)
Scheduler debugging updates:
- Remove unused schedstats (Shrikanth Hegde)
- Defer freeing of cpumask memblock memory to initcall
(Waiman Long)
Misc fixes and updates by Yu C Chen, K Prateek Nayak, Peter Zijlstra,
Vincent Guittot, Xin Zhao, Yury Norov, Zhan Xusheng.
Signed-off-by: Ingo Molnar <mingo@kernel.org>
-----BEGIN PGP SIGNATURE-----
iQJFBAABCgAvFiEEBpT5eoXrXCwVQwEKEnMQ0APhK1gFAmqC3WARHG1pbmdvQGtl
cm5lbC5vcmcACgkQEnMQ0APhK1hawRAAuVAreXtsp7Ok6aRHmiOMR+Va2EsHIL4B
3MY5do2oUfiJSCDtIn3BXqjM3djOXdbQLQwm6r9/NkyJ6hsOsRgZaP1L/w07FiY0
33ilIhZIH9QZ2RzvEjZcFXVwnWH6zP8Ag/gTsch3g8TIX2JJcZBzcJjQM8ZIdcBL
oxwWNe9z3Mh0/x2N+N/s1xstaZf95bWQKeBVBGjiOAakGRWMEaQmMtOUujJNmx/p
YFtNcJL6pLGfD+ASOtLxYWts4z1JyDmwvje8SZmRsgjZdhoxRlnEIgU6jrNjmPqu
NfR5w6R193Pa+FapawYi0ttuawapHvvSDDUxjl+eIccPMw0mBdjzn1XuNIt6pclC
Q4VDUQEDBYvbeznNfLuvvsLVBq/QIjim391lg9DuAGreuRx00Ij0hXOCPk6TM4H8
tfUZ8N25vIAbYrFajSDyOYdgAa2Z2VnM1Q1ko29LPb3WFfZsEqgIg7+ukoXCZ5n8
67pUrOYLuSlLXJGlJY9WDz1nbeUh3e0f2390fif3VBcQ7roPEmoL8CYZhh8Vd2rq
ug8kOpL/91/PDgSPbxRRIEYhe9X1kzWZxqpjvO93tZ+GhXUxtLFG4PjnbQhINy1A
BMNlxcq4ktVHk9gwLaG9wcySjINnN/wK91o3dQ9YjwTuCAxFi/AqUVjY+OOmguAh
TO0mkEq99M0=
=HTPk
-----END PGP SIGNATURE-----
Merge tag 'sched-core-2026-08-17' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
Pull scheduler updates from Ingo Molnar:
"Load-balancing updates:
- 'flatten the pick': improve cgroup scheduling, which has always
been problematic and painful, which has caused various scheduling
misbehavior such as the mishandling of reniced tasks et al.
Add various cgroup weight distribution methods via cgroup_mode:
'up', 'max', 'concur' and 'tasks' - with the default being 'concur'
which is the most precise yet also most expensive version.
Finally, change cgroup scheduling to a single runqueue (Peter
Zijlstra)
- Series to improve the scheduling latency of short slice tasks
(Vincent Guittot)
- Series to fix cluster scheduling in the presence of asymmetric
capacity (Ricardo Neri)
- Prefer fully idle cores for NOHZ balancing (Andrea Righi)
- Don't trigger active load-balancing if src_rq->curr is not on_rq
(Xin Zhao)
PSI updates:
- Skip irqtime accounting when no new irq time has elapsed (Usama
Arif)
Scheduler debugging updates:
- Remove unused schedstats (Shrikanth Hegde)
- Defer freeing of cpumask memblock memory to initcall (Waiman Long)
Misc fixes and updates by Yu C Chen, K Prateek Nayak, Peter Zijlstra,
Vincent Guittot, Xin Zhao, Yury Norov, Zhan Xusheng"
* tag 'sched-core-2026-08-17' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip: (33 commits)
sched/fair: Fix flat hierarchy
sched/isolation: Defer freeing of cpumask memblock memory to initcall
sched/topology: Restore SD_PREFER_SIBLING in domains with asymmetric capacity
sched/fair: Allow load balancing between CPUs of identical capacity
sched/fair: Skip misfit load accounting when the destination CPU cannot help
sched/fair: Check CPU capacity before comparing group types during load balance
sched/fair: Also gate overloaded status update for SD_ASYM_CPUCAPACITY
sched/fair: Do not skip CPUs of similar capacity with busy SMT siblings
sched/fair: Prefer fully idle cores for NOHZ balancing
stop_machine: Make stop_one_cpu_nowait() return void
sched/eevdf: Delayed dequeue task can't preempt
sched/fair: Fix stale comments referring to removed CFS concepts
sched/debug: Remove unused schedstats
sched/psi: skip irqtime accounting when no new irq time has elapsed
sched/fair: Reflow sched_balance_rq()
sched/fair: Simplify balance_interval reset logic in sched_balance_rq()
sched/fair: Don't trigger active lb if src_rq->curr is not on_rq
sched/eevdf: Speedup short slice task scheduling
sched/eevdf: Always update slice protection
sched/eevdf: Cancel slice protection if short slice task is eligible
...
Futexes:
- Use runtime constants for futex_hash computation
(K Prateek Nayak, Peter Zijlstra)
- Optimise the size check get_futex_key() (Sebastian Andrzej Siewior)
- Avoid private hash use-after-free on final put (Felix Hoffmann)
- Tell kmemleak we're not leaking __futex_queues (Peter Zijlstra)
Rust integration updates:
- Implement refcounted interrupt disable and SpinLockIrq for Rust
(Boqun Feng, Heiko Carstens, Joel Fernandes, Lyude Paul)
- Rust sync: add helpers for mb, dma_mb and friends;
add generic memory barriers and use LKMM atomics
instead of Rust atomics in the revocable code (Gary Guo)
- Add abstraction and integrate synchronize_rcu() (Philipp Stanner)
Lock debugging:
- Add qspinlock contended_release tracepoint
(Dmitry Ilvokhin, Peter Zijlstra)
- Enable the printing of held locks of remote running tasks and print
task CPU (Ingo Molnar)
- percpu-rwsem: Annotate intentional data race in readers_active_check()
(Sun Shaojie)
Misc fixes and updates by Boqun Feng, Peter Zijlstra, Fangrui Song,
Naveen Kumar Chaudhary and Thomas Huth.
Signed-off-by: Ingo Molnar <mingo@kernel.org>
-----BEGIN PGP SIGNATURE-----
iQJFBAABCgAvFiEEBpT5eoXrXCwVQwEKEnMQ0APhK1gFAmqC2KMRHG1pbmdvQGtl
cm5lbC5vcmcACgkQEnMQ0APhK1gNwg//awvTQONfhPanAyTgl7CLDSlMSHdqmlyh
Ue0/Q8Ef1Cy4jwXY2FE2A0b1VcM6cGpDPoryVdg/wMdUXRNwinzAEXmxIkRy9kve
4LybrZwDShgLxJ7pJ6KKhgjgDiat8EdYmOwCBEE3LnP7AYhkAb8BFetA3YZJvzPa
KfA2BRYCgvBTid6yOAuXWm55Ev92AczOBamBzTxCadcaDGtNGXtQO6LfnqiQDOav
X5tVoANBeaQtSs1+LxE41WdNOiRoBuy0IFFvXtZRal6PZYuGGmZ5tbQvscD099em
haVwQyzDHQrqzglv71M0KRTXvYzdGveMRg/Au1SQnuLO3V6Vd5rMQ1g7I2M9Ln0f
Pg+tlRvQ77mLoqcgrtl0W/u0fRR4eDkiJ1pmG+98oniPwau23RdbFhC0vKFz3ikF
WHMgk3/9TcULylgF1Tj6QLmNrBY3Vx8LBdsFjhflEw7bG4cW42D91npmXIiEDE6K
tJc9CcaVdyE75o59z2Dtjj+qQVBlNPlfKQFXFL7p3jU/gFw2SzYuqon66X3kGmr0
mKJ9UNJdkLdiCjxS/QiMcDeYhwJksJqxFBkH50z3Kzmo84JsSpUFkoa6GM4aSiGn
HEwgC0Q7oOXVNIKUBYk5QaRW0HSk55hbsX2TWkvpeBYkE1zXshVZCCmgpaTSJgb5
oFmiwrGfUjo=
=slqC
-----END PGP SIGNATURE-----
Merge tag 'locking-core-2026-08-17' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
Pull locking updates from Ingo Molnar:
"Futexes:
- Use runtime constants for futex_hash computation (K Prateek Nayak,
Peter Zijlstra)
- Optimise the size check get_futex_key() (Sebastian Andrzej Siewior)
- Avoid private hash use-after-free on final put (Felix Hoffmann)
- Tell kmemleak we're not leaking __futex_queues (Peter Zijlstra)
Rust integration updates:
- Implement refcounted interrupt disable and SpinLockIrq for Rust
(Boqun Feng, Heiko Carstens, Joel Fernandes, Lyude Paul)
- Rust sync: add helpers for mb, dma_mb and friends; add generic
memory barriers and use LKMM atomics instead of Rust atomics in the
revocable code (Gary Guo)
- Add abstraction and integrate synchronize_rcu() (Philipp Stanner)
Lock debugging:
- Add qspinlock contended_release tracepoint (Dmitry Ilvokhin, Peter
Zijlstra)
- Enable the printing of held locks of remote running tasks and print
task CPU (Ingo Molnar)
- percpu-rwsem: Annotate intentional data race in readers_active_check()
(Sun Shaojie)
Misc fixes and updates by Boqun Feng, Peter Zijlstra, Fangrui Song,
Naveen Kumar Chaudhary and Thomas Huth"
* tag 'locking-core-2026-08-17' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip: (44 commits)
rust: sync: Introduce SpinLockIrq::lock_with() and friends
rust: sync: Add SpinLockIrq
rust: sync: Use super::* in spinlock.rs
rust: helper: Add spin_{un,}lock_irq_{enable,disable}() helpers
rust: Introduce interrupt module
s390/preempt: Enable HAS_SEPARATE_PREEMPT_RESCHED_BITS
arm64: sched/preempt: Enable HAS_SEPARATE_PREEMPT_RESCHED_BITS
preempt: Introduce HAS_SEPARATE_PREEMPT_RESCHED_BITS
sched: Avoid signed comparison of preempt_count() in __cant_migrate()
sched: Remove the unused preempt_offset parameter of __cant_sleep()
locking: Switch to _irq_{disable,enable}() variants in cleanup guards
irq: Add KUnit test for refcounted interrupt enable/disable
irq,spin_lock: Add counted interrupt disabling/enabling
openrisc: Include <linux/cpumask.h> in smp.h
preempt: Introduce __preempt_count_{sub,add}_return()
preempt: Introduce HARDIRQ_DISABLE_BITS
preempt: Track NMI nesting to separate per-CPU counter
futex: Tell kmemleak we're not leaking __futex_queues
x86/paravirt: Trace contended_release on unlock
tracing/lock: Use TRACE_EVENT_FN() for contended_release
...
ops.cgroup_set_bandwidth() is delivered from scx_group_set_bandwidth(),
which runs from the cpu.max cgroup interface write path (tg_set_bandwidth())
in process context. scx_group_set_bandwidth() holds
percpu_down_read(&scx_cgroup_ops_rwsem), whose read side may sleep.
The call site is therefore sleepable, like ops.cgroup_init().
bpf_scx_check_member() rejects a sleepable program on any member not on its
allow-list, so a BPF scheduler cannot allocate -- which is sleepable -- when
a cgroup gains a cpu.max limit at runtime; it must instead pre-reserve memory
for a callback that cannot allocate. Add cgroup_set_bandwidth() to the
allow-list so the callback can allocate on demand, and document that it may
block.
A scheduler must decide at load time whether to mark the callback sleepable,
but the allow-list entry is a verifier property with no symbol to probe. Add
a compatibility marker whose presence in the kernel's BTF lets userspace detect
this support: DEFINE_SCX_COMPAT_MARKER() emits an empty, callerless function,
here scx_compat_marker_cgroup_set_bandwidth_may_sleep(). It is __used
__retain so neither the compiler nor the linker (under
CONFIG_LD_DEAD_CODE_DATA_ELIMINATION) drops it. The markers share the
scx_compat_marker_ prefix and are collected near the end of ext.c so more
can be added as further capabilities appear.
Signed-off-by: Changwoo Min <changwoo@igalia.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
- Minor fixes and cleanups in assorted cpufreq drivers (Dan Carpenter,
Guru Das Srinagesh, Haoxiang Li, Karl Mehltretter, Sasha Finkelstein,
and Pan Chuang)
- Fix cpufreq table creation and bios_limits() callback in the Rust
bindings (Priya Bala Govindasamy)
- Add IPQ5210 support to qcom-nvmem driver (Varadarajan Narayanan)
- Adjust the .adjust_perf() cpufreq driver callback to allow the
maximum performance value to be passed to drivers and update the
intel_pstate driver to use it (Rafael Wysocki)
- Set policy->cur to the actual requested frequency in the intel_pstate
driver when the performance policy is used (Rafael Wysocki)
- Simplify HWP handling on Broadwell processors in intel_pstate (Rafael
Wysocki)
- Fix setting minimum P-state at init time in intel_pstate (Rafael
Wysocki)
- Consolidate frequency values computation in intel_pstate and clean up
code in that driver (Rafael Wysocki)
- Add missing kernel-doc descriptions for structure and union members
in the amd-pstate driver (David Vernet)
- Handle missing policy in dynamic EPP callbacks in the amd-pstate
driver (EDAMAMEX)
- Introduce EXPORT_SYMBOL_FOR_PSTATE_UT() to export amd-pstate driver
symbols to the amd-pstate-ut subdriver (K Prateek Nayak)
- Add dynamic EPP as an "energy_performance_preference" mode in
amd-pstate, remove the "amd_dynamic_epp" kernel command line option
and the "dynamic_epp" sysfs attribute, and update the dynamic_epp
documentation accordingly (K Prateek Nayak)
- Add unit tests for CPPC Performance Priority and the "dynamic" EPP
mode in the amd-pstate driver (K Prateek Nayak)
- Set min_limit_freq based on bios_min_perf in amd-pstate and remove
the defensive check for bios_min_perf from it (K Prateek Nayak)
- Fix EPP return type and handle errors in amd-pstate during
initialization, toggle auto_sel in active mode on shared memory
systems, and cache the firmware programmed EPP value (Marco Scardovi)
- Skip tests in amd-pstate-ut if the amd-pstate driver is not in active
use (Qianheng Peng)
- Replace sprintf() with sysfs_emit() in sysfs show in the cpufreq
schedutil governor and fix a self-contradictory comment in
sugov_iowait_apply() (Zhongqiu Han)
- Fix the usage example for the sampling_rate tunable of the ondemand
cpufreq governor in admin-guide (wangxiaodong)
- Avoid using deep idle states during initialization in the intel_idle
driver to work around device handling issues (Rafael Wysocki)
- Fix and refactor the ACPI processor driver code related to ACPI _LPI
support and add ACPI _LPI support to intel_idle based on that ACPI
processor driver update (Rafael Wysocki)
- Backup and restore governor for cpufreq sptests (Yiwei Lin)
- Remove unnecessary sudo from quick_shuffle() and remove unused local
variables from switch_show_governor() in cpufreq selftests (Jinseok
Kim)
- Rename the PM core module parameter prefix to "pm" and allow the PM
transition (DPM) watchdog to be disabled by default (Tzung-Bi Shih)
- Fix off-by-one in wakelocks number limit check in the system sleep
sysfs interface (Haowen Tu)
- Remove kernel-doc markings from helper descriptions in the core
hibernation code (Adi Nata)
- Use %pe to print error pointer values in the hibernation core (Ronan
Marchal)
- Fix memory leak in snapshot_write_next() error path (Malaya Kumar
Rout)
- Delay allocating and linking the next swap_map_page in the
hibernation image saving code until another image page actually
needs to be recorded (Haesung Kim)
- Fix cleanup ordering around scope-based pointers in OPP (Gregor
Herburger).
- Use clk_get_optional() for optional clocks in OPP (Praveen Talari).
- Stop setting runtime_error on runtime resume callback failures to
allow drivers to recover from resume issues (Praveen Talari)
- Handle PMU registration failure during probe in the intel_rapl_tpmi
driver (Sumeet Pawnikar)
- Avoid optional imports in intel_pstate_tracer unless they are really
needed (Yousef Alhouseen)
- Add generic CPPC performance display to the cpupower utility, build
and call CPPC information on non-AMD processors, make cpupower print
kernel and hardware frequency information, and add libm to cpupower
for generic CPPC view (Jeremy Linton)
- Remove conditional return with no effect from cpupower (Sang-Heon
Jeon)
-----BEGIN PGP SIGNATURE-----
iQFGBAABCAAwFiEEcM8Aw/RY0dgsiRUR7l+9nS/U47UFAmp+DXsSHHJqd0Byand5
c29ja2kubmV0AAoJEO5fvZ0v1OO10Z4IAIYwM9RKwbwCqPMKGvWExIb+1kpEy1aw
wyFnYSiQd09XCt8/YjERFR9gB7CfGduisTxEOqOdCIpGX2rT9Me5xO5aR5Gz4FE3
bML7KyulS9tVyaEVBjFx1yDZxGTNooTQVTLIxBYbCVB1RrgiCGdLyC2okn79XHlk
olTZemq59Kn0B6t7BP7UQ0WCCCAs3RlbrQIZSrVh0HydoO0DHlJvttN2fc7bjeiM
Je/dSm14o+vjglkVpY1kbxzBHJOH7/XH5nqSQRoHoAZWdllWK4+iQtll63Un4A72
2KG0O5+nFrS3d166TEjDEInnEwR012Sod/8Gk72M0IZZhG/naqs5ekE=
=mn7E
-----END PGP SIGNATURE-----
Merge tag 'pm-7.3-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/rafael/linux-pm
Pull power management updates from Rafael Wysocki:
"As has been the case for quite some time, this set of changes is
dominated by cpufreq updates including intel-pstate and amd-pstate
driver updates, minor fixes and cleanups of other assorted cpufreq
drivers, schedutil governor updates, fixes of the Rust bindings, new
hardware support (IPQ5210 in qcom-nvmem), and some updates of self
tests related to cpufreq.
The second largest group of changes are cpuidle updates consisting of
intel_idle driver updates and ACPI processor idle driver updates, both
mostly related to ACPI _LPI support.
There are also updates related to system sleep, mostly in the
hibernation core code, two operating performance points (OPP) updates,
one runtime PM framework update, one power capping update, and some
tools updates including the addition of ACPI CPPC support to cpupower.
Specifics:
- Minor fixes and cleanups in assorted cpufreq drivers (Dan
Carpenter, Guru Das Srinagesh, Haoxiang Li, Karl Mehltretter, Sasha
Finkelstein, and Pan Chuang)
- Fix cpufreq table creation and bios_limits() callback in the Rust
bindings (Priya Bala Govindasamy)
- Add IPQ5210 support to qcom-nvmem driver (Varadarajan Narayanan)
- Adjust the .adjust_perf() cpufreq driver callback to allow the
maximum performance value to be passed to drivers and update the
intel_pstate driver to use it (Rafael Wysocki)
- Set policy->cur to the actual requested frequency in the
intel_pstate driver when the performance policy is used (Rafael
Wysocki)
- Simplify HWP handling on Broadwell processors in intel_pstate
(Rafael Wysocki)
- Fix setting minimum P-state at init time in intel_pstate (Rafael
Wysocki)
- Consolidate frequency values computation in intel_pstate and clean
up code in that driver (Rafael Wysocki)
- Add missing kernel-doc descriptions for structure and union members
in the amd-pstate driver (David Vernet)
- Handle missing policy in dynamic EPP callbacks in the amd-pstate
driver (EDAMAMEX)
- Introduce EXPORT_SYMBOL_FOR_PSTATE_UT() to export amd-pstate driver
symbols to the amd-pstate-ut subdriver (K Prateek Nayak)
- Add dynamic EPP as an "energy_performance_preference" mode in
amd-pstate, remove the "amd_dynamic_epp" kernel command line option
and the "dynamic_epp" sysfs attribute, and update the dynamic_epp
documentation accordingly (K Prateek Nayak)
- Add unit tests for CPPC Performance Priority and the "dynamic" EPP
mode in the amd-pstate driver (K Prateek Nayak)
- Set min_limit_freq based on bios_min_perf in amd-pstate and remove
the defensive check for bios_min_perf from it (K Prateek Nayak)
- Fix EPP return type and handle errors in amd-pstate during
initialization, toggle auto_sel in active mode on shared memory
systems, and cache the firmware programmed EPP value (Marco
Scardovi)
- Skip tests in amd-pstate-ut if the amd-pstate driver is not in
active use (Qianheng Peng)
- Replace sprintf() with sysfs_emit() in sysfs show in the cpufreq
schedutil governor and fix a self-contradictory comment in
sugov_iowait_apply() (Zhongqiu Han)
- Fix the usage example for the sampling_rate tunable of the ondemand
cpufreq governor in admin-guide (wangxiaodong)
- Avoid using deep idle states during initialization in the
intel_idle driver to work around device handling issues (Rafael
Wysocki)
- Fix and refactor the ACPI processor driver code related to ACPI
_LPI support and add ACPI _LPI support to intel_idle based on that
ACPI processor driver update (Rafael Wysocki)
- Backup and restore governor for cpufreq sptests (Yiwei Lin)
- Remove unnecessary sudo from quick_shuffle() and remove unused
local variables from switch_show_governor() in cpufreq selftests
(Jinseok Kim)
- Rename the PM core module parameter prefix to "pm" and allow the PM
transition (DPM) watchdog to be disabled by default (Tzung-Bi Shih)
- Fix off-by-one in wakelocks number limit check in the system sleep
sysfs interface (Haowen Tu)
- Remove kernel-doc markings from helper descriptions in the core
hibernation code (Adi Nata)
- Use %pe to print error pointer values in the hibernation core
(Ronan Marchal)
- Fix memory leak in snapshot_write_next() error path (Malaya Kumar
Rout)
- Delay allocating and linking the next swap_map_page in the
hibernation image saving code until another image page actually
needs to be recorded (Haesung Kim)
- Fix cleanup ordering around scope-based pointers in OPP (Gregor
Herburger).
- Use clk_get_optional() for optional clocks in OPP (Praveen Talari).
- Stop setting runtime_error on runtime resume callback failures to
allow drivers to recover from resume issues (Praveen Talari)
- Handle PMU registration failure during probe in the intel_rapl_tpmi
driver (Sumeet Pawnikar)
- Avoid optional imports in intel_pstate_tracer unless they are
really needed (Yousef Alhouseen)
- Add generic CPPC performance display to the cpupower utility, build
and call CPPC information on non-AMD processors, make cpupower
print kernel and hardware frequency information, and add libm to
cpupower for generic CPPC view (Jeremy Linton)
- Remove conditional return with no effect from cpupower (Sang-Heon
Jeon)"
* tag 'pm-7.3-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/rafael/linux-pm: (76 commits)
cpufreq: imx6q: fix out-of-bounds write when probed more than once
cpufreq: imx6q: fix devres accumulation across driver rebind
rust: cpufreq: Fix temporary write in Registration::bios_limit_callback
rust: cpufreq: Add CPUFREQ_TABLE_END as last table entry in TableBuilder::to_table
opp: Use clk_get_optional() to avoid leaving opp_table->clk as an error pointer
intel_idle: Avoid using deep idle states during initialization
cpupower: remove conditional return with no effect
cpufreq: intel_pstate: Adjust policy->cur in active mode to policy
cpufreq/amd-pstate: Document missing kernel-doc members
cpufreq/amd-pstate-ut: Add unit test for CPPC Performance Priority
cpufreq/amd-pstate-ut: Add unit test for "dynamic" EPP mode
cpufreq/amd-pstate: Reduce the scope of exported symbols
Documentation/amd-pstate: Update dynamic_epp documentation with new behavior
cpufreq/amd-pstate: Remove "amd_dynamic_epp" cmdline and "dynamic_epp" sysfs
cpufreq/amd-pstate: Add dynamic EPP as an "energy_performance_preference" mode
cpufreq/amd-pstate: Extract platform profile to EPP conversion into a helper
cpufreq/amd-pstate: Remove the defensive check for bios_min_perf
cpufreq/amd-pstate: Set min_limit_freq based on bios_min_perf
powercap: intel_rapl_tpmi: Handle PMU registration failure during probe
PM: sleep: Allow disabling DPM watchdog by default
...
Pull to receive the __arena argument conversion:
67f1f4a48c ("sched_ext: Pass kernel arena pointers to ops_cid callbacks")
a8dc810968 ("sched_ext: Convert sub-cap kfuncs to __arena cmask arguments")
a05c5b5cb5 ("sched_ext: Convert scx_bpf_cid_override() to __arena array arguments")
along with the bpf-next branch carrying the __arena argument support they
depend on.
Conflict in kernel/sched/ext/ext.c between:
c384ab8a0b ("sched_ext: Move the config-off sub-cap kfunc stubs into sub.c")
and:
a8dc810968 ("sched_ext: Convert sub-cap kfuncs to __arena cmask arguments")
which updated the stubs in their old ext.c location. Resolved by keeping
ext.c without the stubs and applying the prototype conversion to the
relocated stubs in sub.c.
Signed-off-by: Tejun Heo <tj@kernel.org>
dequeue_task_scx() masks SCX_DEQ_CORE_SCHED_EXEC out of the
SCX_DEQ_SCHED_CHANGE decision, but the test can never fire: the incoming
flags are an int of generic DEQUEUE_* bits while the flag is bit 32, and the
core-sched execute path never goes through class dequeue anyway -
set_next_task_scx() calls ops_dequeue() with the flag directly. The test was
live when the SCX_DEQ_SCHED_CHANGE computation sat in ops_dequeue() and
became dead when 03f5304aad ("sched_ext: Pass full dequeue flags to
ops.quiescent()") moved the computation here. Drop it.
Signed-off-by: Tejun Heo <tj@kernel.org>
With sub-schedulers, tasks of different schedulers routinely share rqs and
SMT siblings, but scx_prio_less() consults ops.core_sched_before() only when
both tasks belong to the same scheduler. Every pair spanning two schedulers
falls back to the default ordering, so no scheduler can express ordering
across a scheduler boundary, including a root over its sub-schedulers'
tasks.
Order a pair spanning schedulers by the nearest common ancestor that
implements ops.core_sched_before(): both tasks are in its subtree, making
this the one op where a scheduler is called on tasks it delegated to its
sub-schedulers and may not be scheduling anymore. Same-scheduler pairs keep
using the owning scheduler's op so a parent never orders inside a subtree it
delegated. The op is skipped when the deciding scheduler is bypassing on
either task's CPU.
Update scx_qmap to fall back to the kernel's default ordering when handed a
delegated task it has no task_ctx for.
Signed-off-by: Tejun Heo <tj@kernel.org>
The default core-sched ordering runs the longest waiting task first by
comparing p->scx.core_sched_at stamps. The stamp is maintained under two
rules. touch_core_sched() stamps when a task starts waiting for a CPU and
when its slice runs out. If the scheduler implements
ops.core_sched_before(), touch_core_sched_dispatch() re-stamps on every
dispatch.
A comparison can see one stamp taken under each rule, which isn't a
meaningful ordering. The dispatch rule also buys little - it only aligns
bypass-mode comparisons with the local DSQ order. Multiple schedulers make
the mixed comparisons more common.
Wait time is what p->scx.runnable_at already tracks for the stall watchdog.
Delete core_sched_at with both touch functions and compare runnable_at in
the scx_prio_less() fallback.
runnable_at is refreshed only on enqueue and goes stale while a task keeps
occupying its CPU. Instead of re-stamping, order a running task after every
waiting task as it is the most recently serviced.
Signed-off-by: Tejun Heo <tj@kernel.org>
scx_prio_less() implements prio_less() semantics - %true means that @a is
the lower priority and should run after @b. ops.core_sched_before() is
documented to return %true when @a should run before @b. scx_prio_less()
returns the op's value as-is, inverting the documented semantics at runtime.
Call the op with the arguments swapped.
scx_qmap followed the wiring instead of the documentation and returned %true
for the younger task, so the two inversions canceled out and it behaved as
intended. Flip its comparison to match. scx_qmap is likely the only current
user in or out of the kernel tree. Any scheduler written the same way needs
the same flip, while schedulers following the documentation are fixed by
this change.
Fixes: 7b0888b7cc ("sched_ext: Implement core-sched support")
Cc: stable@vger.kernel.org # v6.12+
Signed-off-by: Tejun Heo <tj@kernel.org>
The EOPNOTSUPP stubs for the sub-cap kfuncs live in ext.c under #ifndef
CONFIG_EXT_SUB_SCHED while the real definitions live in sub.c. Move the
stubs into sub.c so all sub kfunc definitions live in one file. Pure code
move, no functional change.
Signed-off-by: Tejun Heo <tj@kernel.org>
rate_limit_us is an unsigned int, while NSEC_PER_USEC is defined as
1000L. On 32-bit systems, the multiplication is therefore performed
using 32-bit unsigned arithmetic before the result is assigned to
freq_update_delay_ns.
For example, writing 4294968 to rate_limit_us wraps the delay from
4294968000 ns to 704 ns. This makes schedutil update far more often
than configured.
Add sugov_update_rate_limit_us() to widen rate_limit_us to s64 before
converting it to nanoseconds. Use the helper when updating the tunable
through sysfs and when starting the governor, so both paths perform the
conversion without overflow.
Fixes: 9bdcb44e39 ("cpufreq: schedutil: New governor based on scheduler utilization data")
Signed-off-by: Hui Su <sh_def@163.com>
Reviewed-by: Zhongqiu Han <zhongqiu.han@oss.qualcomm.com>
Cc: All applicable <stable@vger.kernel.org>
Link: https://patch.msgid.link/20260806142304.1761454-1-sh_def@163.com
Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
sched_class->balance() is gone from sched_ext and what balance_one() does is
run dispatch to produce something pickable. Update the balance-era names to
dispatch terms:
- balance_one() -> dispatch_one()
- SCX_RQ_IN_BALANCE -> SCX_RQ_IN_DISPATCH
No BPF scheduler reads the flag. The enum autogen headers gain the new name
with the old entry retained like other removed enumerators, zero-filling at
load time. No functional changes.
Signed-off-by: Tejun Heo <tj@kernel.org>
The fixup demoting a keep verdict when @prev is not on ext_sched_class
guarded against the rq-level SCX_RQ_BAL_KEEP flag going stale back when
balancing and picking were separate operations.
The verdict now travels in the return value, created and consumed in one
invocation against the @prev it evaluated, and every keep decision tests
SCX_TASK_QUEUED under the rq lock, which implies ext_sched_class as a class
switch dequeues first. Drop the fixup along with dispatch_core_pick()'s
copy.
Signed-off-by: Tejun Heo <tj@kernel.org>
kick_sync_wait_bal_cb() assumes it runs on the rq's CPU from the
__schedule() tail: the snapshots it compares against live in that CPU's
percpu area and the busy-wait runs with the rq lock dropped and IRQs
enabled.
However, dispatch can now drop the rq lock while the callback sits queued,
and rq lock takers in that window (the sched class change paths, the scx
task iterator) flush pending balance callbacks on release, running the
callback on a foreign CPU. Such a run compares against unrelated snapshots
and can deadlock when the executing CPU is itself a wait target.
Bail on a foreign CPU and leave the wait state alone. The wait only observes
progress that the resched kicks already guarantee and the rq's next wait
picks up the stale cpus_to_sync bits.
Fixes: 4c95380701 ("sched/ext: Fold balance_scx() into pick_task_scx()")
Cc: stable@vger.kernel.org # v6.19+
Signed-off-by: Tejun Heo <tj@kernel.org>
scx_bpf_dsq_reenq() queues a deferred reenq (dru) that runs from
run_deferred(), not ops.dispatch(). If the DSQ is destroyed before the dru
runs, process_deferred_reenq_users() sees dsq->id == SCX_DSQ_INVALID and
hits the BUG_ON. destroy_dsq() doesn't flush pending drus, so just skip.
tj: Read dsq->id once with READ_ONCE(). Reading it separately in the INVALID
check and the BUG_ON would leave a window where destroy_dsq() can
invalidate the id between the two reads and still trigger the BUG_ON.
Fixes: 84b1a0ea0b ("sched_ext: Implement scx_bpf_dsq_reenq() for user DSQs")
Cc: stable@vger.kernel.org # v7.1+
Signed-off-by: Tao Cui <cuitao@kylinos.cn>
Signed-off-by: Tejun Heo <tj@kernel.org>
scx_bpf_events() always reads the root scheduler's event counters, so a
sub-scheduler program querying its own events silently gets the root's
instead and has no BPF-visible way to read its own (the per-scheduler sysfs
"events" file is the only interface). Resolve the scheduler from the calling
program with scx_prog_sched(). Unassociated programs follow the usual
scx_prog_sched() resolution: the root scheduler under a pre-sub-attach
compat root and zeroed counters otherwise.
Also fix up the malformed comment into proper kerneldoc.
Signed-off-by: Tejun Heo <tj@kernel.org>
scx_root_disable() invalidates each rq's clock before taking the rq lock.
scx_rq_clock_invalidate() is a plain read-modify-write of rq->scx.flags and
every other writer of the word runs under the rq lock, so the unlocked
update can race a concurrent flags update and lose one side's bits.
The invalidation doesn't matter in the first place. The cached clock is read
only by scx_bpf_now() from a loaded scheduler's BPF programs, nothing can
re-validate the clock while sched_ext is disabled as scx_rq_clock_update()
is gated on scx_enabled() too, and the usual rq lock cycles under the next
scheduler refresh or invalidate it before it's practically observable. Drop
the invalidation instead of fixing the locking.
v2: Description and comment updated - the invalidation is unnecessary rather
than subsumed by the rq lock cycle below.
Fixes: 3a9910b590 ("sched_ext: Implement scx_bpf_now()")
Signed-off-by: Tejun Heo <tj@kernel.org>
Cc: Changwoo Min <changwoo@igalia.com>
scx_fork() initializes tasks when scx_init_task_enabled is set, but
scx_cancel_fork() only exits them when scx_enabled() is true. A fork
that fails in the enable window (between releasing scx_fork_rwsem and
setting __scx_enabled) runs ops.init_task() but never ops.exit_task().
Gate scx_cancel_fork() on scx_init_task_enabled.
Fixes: 4269c603cc ("sched_ext: Enable scx_ops_init_task() separately")
Cc: stable@vger.kernel.org # v6.12+
Signed-off-by: fangqiurong <fangqiurong@kylinos.cn>
Signed-off-by: Tejun Heo <tj@kernel.org>
- inlines.h: scx_bpf_dispatch() doesn't exist; the comment means
scx_bpf_sub_dispatch()
- internal.h: name %SCX_DEQ_SCHED_CHANGE instead of the never-defined
%SCX_DEQ_SAVE
- internal.h: @name shows up in the ops file in the scheduler's sysfs
directory, not a "kernel.sched_ext_ops" sysctl
Signed-off-by: Tao Cui <cuitao@kylinos.cn>
Signed-off-by: Tejun Heo <tj@kernel.org>
scx_process_sync_ecaps() sets up the dispatch context for
ops.sub_ecaps_updated() in the target cpu's pcpu context recovered from the
llist node. However, the context is per executing cpu: the dispatch kfuncs
resolve it with this_cpu_ptr() and the dispatch buffer lives in it. What the
dispatches target is determined by the rq recorded in the context, not by
which cpu's context it is. Under core scheduling the pick runs balance_one()
for sibling rqs, so a sync processed for a sibling invokes the op with the
executing cpu's context not set up and its dispatch kfuncs misoperate on a
NULL or stale rq.
Set up the executing cpu's dsp_ctx instead, matching scx_dispatch_sched().
The recorded rq keeps the dispatches targeting the synced cpu.
Fixes: b81a6c018c ("sched_ext: Add sub_ecaps_updated() effective-cap change notifier")
Reported-by: David Carlier <devnexen@gmail.com>
Link: https://lore.kernel.org/all/20260813045931.8691-1-devnexen@gmail.com/
Signed-off-by: Tejun Heo <tj@kernel.org>
When a fair task is enqueued, we must update curr and more precisely
its vruntime before placing the enqueued task so avg vruntime will take
into account the last exec phase.
Example:
TA is an always running task in cgroup G0.
TB is a short running task (cyclictest) in cgroup G1.
The lag of TB always increases up the clamp limit because TB is placed
before TA(curr) is updated (since the last tick). When curr(TA) is
finally updated, its last exec phase provide positive lag to TB
Because TA and TB don't belong to the same group, enqueue_hierarchy() will not
update TA's entity when updating curr but only G0's entity at root level.
The same applies when dequeuing.
This is because update_curr() uses ->h_curr, rather than ->curr, and therefore,
while it is invoked on the root cfs_rq, which contains all the eevdf bits, it
does not do the right thing.
Fixes: 85570f10a4 ("sched/eevdf: Move to a single runqueue")
Signed-off-by: Vincent Guittot <vincent.guittot@linaro.org>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/20260812125039.1717249-1-vincent.guittot@linaro.org
In order to compute the right lag, it is required to update time to 'now'.
Without this, the delayed entity might appear younger than it really is and
receive less compensation for having waited.
Signed-off-by: Vincent Guittot <vincent.guittot@linaro.org>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
When testing a linux-next kernel with commit 59bd1d914b ("memblock:
warn when freeing reserved memory before memory map is initialized"),
the following warning was hit when there was a "nohz_full" kernel boot
parameter.
Cannot free reserved memory because of deferred initialization of the memory map
WARNING: mm/memblock.c:904 at __free_reserved_area+0xde/0xf0, CPU#0: swapper/0/0
:
Call Trace:
<TASK>
memblock_phys_free+0xcb/0x100
housekeeping_init+0x14c/0x170
start_kernel+0x207/0x450
x86_64_start_reservations+0x24/0x30
x86_64_start_kernel+0xda/0xe0
common_startup_64+0x13e/0x141
</TASK>
IOW, we shouldn't free memblock allocated memory so early
in the boot process when memory map isn't fully initialized in
deferred_init_memmap().
Fix it by saving the housekeeping cpumask memblock memory to be
freed into a llist free list in housekeeping_init() and add a new
housekeeping_late_init() helper to defer the actual freeing of memblock
memory to when initcall's are being processed. The cpumask memblock
memory is treated as a llist_node with the size of a "long" type which
is also smallest cpumask size that can be allocated.
The non-atomic version of the llist APIs are used as there is no
contention.
This commit depends on the presence of commit 7c2eee9c13 ("memblock:
don't touch memblock arrays when memblock_free() is called late")
to prevent a KASAN UAF bug report [1].
[1] https://lore.kernel.org/lkml/20260505051821.1107133-1-longman@redhat.com/
Fixes: 27c3a5967f ("sched/isolation: Convert housekeeping cpumasks to rcu pointers")
Signed-off-by: Waiman Long <longman@redhat.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Frederic Weisbecker <frederic@kernel.org>
Reviewed-by: Phil Auld <pauld@redhat.com>
Link: https://patch.msgid.link/20260701195810.477326-1-longman@redhat.com
scx_bpf_cid_override() predates the cid-form arena transition and takes its
arrays as verifier-checked mem+size buffers, forcing scx_qmap to keep the
cpu_to_cid and shard_start arrays in writable bss while the rest of its
state lives in the arena. Unify on arena arguments before cid-form
schedulers start seeing real use.
BPF now translates between BPF and kernel arena addresses for __arena
arguments. Take the arrays as __arena arguments, with the counts passed in
entries. The counts now size the snapshot copies and are bounds-checked
before them.
scx_qmap moves the arrays into struct qmap_arena. As the arena is mmapped at
load, the loader populates them between load and attach instead of before
load.
The arena argument address translation is currently implemented only on
x86-64. Schedulers calling this kfunc load only there for now.
Signed-off-by: Tejun Heo <tj@kernel.org>
The sub-cap kfuncs take their cmask arguments as __ign pointers. The values
cross the kfunc boundary as unchecked scalars and scx_cmask_ref_init()
rebases them into the arena by hand.
BPF now translates between BPF and kernel arena addresses for __arena
arguments. Tag the cmask arguments __arena so the kfuncs receive kernel
addresses and scx_cmask_ref_init() loses the hand-rolled conversion. The
optional denied_out keeps its NULL not-provided signal via
__arena__nullable. The mandatory masks use plain __arena.
scx_qmap's call sites drop the (void *)(long) casts since the BPF-side
declarations type the cmask arguments __arena and take arena pointers
directly.
The arena argument address translation is currently implemented only on
x86-64. Schedulers calling these kfuncs load only there for now.
Signed-off-by: Tejun Heo <tj@kernel.org>
The cid-form set_cmask() and sub_caps_updated() callbacks receive cmasks
that the kernel builds in the arena, and the kernel converts the kernel
addresses to the BPF arena pointer form by hand before each call.
BPF now translates between BPF and kernel arena addresses for __arena
arguments. Tag the arguments __arena in the cfi stubs and the ops_cid member
declarations and pass the kernel arena addresses directly, dropping the
manual scx_kaddr_to_arena() conversions and the now-unused helper. The
delivered value is unchanged and existing BPF-side code works as before.
The arena argument address translation is currently implemented only on
x86-64. cid-form schedulers implementing these callbacks load only there for
now.
Signed-off-by: Tejun Heo <tj@kernel.org>
scx_bpf_cid_to_cpu(), scx_bpf_cpu_to_cid() and scx_bpf_cid_topo() live in
the scx_kfunc_ids_cid set, but scx_kfunc_context_filter() doesn't check
that set. The filter's first test treats any kfunc outside its known sets
as non-SCX and allows it, so these three kfuncs can be called from any
struct_ops program - e.g. a TCP congestion control program.
Add scx_kfunc_ids_cid to the filter's known sets, matching how in_any and
in_idle are handled.
Fixes: e9b55af47e ("sched_ext: Add topological CPU IDs (cids)")
Assisted-by: Z.ai:glm-5.2
Signed-off-by: fangqiurong <fangqiurong@kylinos.cn>
Signed-off-by: Tejun Heo <tj@kernel.org>
Pull to receive:
c10b216a07 ("sched/core: Handle pick_task() releasing the rq lock")
f3629c63a4 ("sched/core: Make core-sched flips wait for in-flight selections")
ffaab58d21 ("sched_ext: Replace SCX_RQ_BAL_KEEP with a dispatch verdict return")
3dd52416e4 ("sched_ext: Fix this_rq() assumptions in dispatch kfuncs")
f2da958711 ("sched_ext: Count rq lock releases in rq->scx.lock_drop_seq")
d954004205 ("sched_ext: Fix rq->core_pick corruption under core scheduling")
for the pending core scheduling follow-ups and to resolve the conflicts
with the code reorganization and cap gate work on for-7.3.
ffaab58d21 converts scx_dispatch_sched() to a dispatch verdict return
which for-7.3 moved from ext.c into inlines.h. Resolved by applying the
conversion to the relocated copy and combining balance_one()'s verdict
returns with the scx_task_can_stay_on_cpu() gate from the cap work.
ffaab58d21 and 3dd52416e4 update scx_bpf_sub_dispatch() which
for-7.3 moved into sub.c. Resolved by applying the scx_locked_rq()
switch and the verdict test to the sub.c copy.
f2da958711 instruments the open-coded lock releases in
consume_remote_task() which for-7.3 folded into switch_rq_lock().
Resolved by keeping the accounting in switch_rq_lock() which covers all
its callers.
d954004205 widens the put_prev_task_scx() WARN suppression to all
core-sched rqs on the same condition that for-7.3 gated with
scx_task_can_stay_on_cpu(). Resolved by combining both.
Signed-off-by: Tejun Heo <tj@kernel.org>
Core scheduling's pick_next_task() picks what to run on every SMT sibling of
the core in a single pass under the shared core-wide rq lock. The selection
state is consistent only while the lock is held continuously, so
->pick_task() originally could not release it. However, since 4c95380701
("sched/ext: Fold balance_scx() into pick_task_scx()"), sched_ext runs
dispatch from inside the pick and dispatching can drop the rq lock. To
support this, pick_next_task() has been updated to restart the whole
selection when a pick returns RETRY_TASK after releasing the lock.
When selections on the same core interleave through the dropped lock, they
corrupt each other's state: one clears the other's rq->core_pick leading to
a NULL deref, or invalidates its keep-the-previous-task decision leaving a
dequeued task running, which deadlocks the next wakeup and matches the
reported hard hangs. A cookied ping-pong load on an SMT machine makes the
interleavings frequent and kills the kernel within seconds.
Fix it by making the pick return RETRY_TASK whenever dispatch released the
rq lock, so that a selection only ever commits picks made under a
continuously held lock. The previous patch's rq->scx.lock_drop_seq counts
the releases. A dispatch that touched nothing never releases the lock and
its verdict, including "nothing to run", stands: retries are bounded, each
following a dispatch that actually did something, and an idle CPU does not
loop.
If another dispatch is already in flight on the rq, skip dispatching and
pick from what is already queued locally - the in-flight dispatch has
released the lock, so its own selection will retry and re-pick this rq,
while returning RETRY_TASK here would only spin on the lock that dispatch
needs to finish.
Balance callbacks must run in the context that queued them, so they can only
be queued on the CPU's own rq. When dispatching for another rq, run the
deferred work directly instead - that rq may consume all its picks through
the core-sched fast path and never queue the callback itself.
The put_prev_task_scx() warning about a runnable task being left behind
assumed that dispatch ran as part of the very pick that is switching away.
That now only holds on the non-core path, so gate it and drop the
cookie-match test, which is always true without core scheduling, from its
condition.
Fixes: 4c95380701 ("sched/ext: Fold balance_scx() into pick_task_scx()")
Cc: stable@vger.kernel.org # v6.19+
Reported-by: ElXreno <elxreno@gmail.com>
Link: https://github.com/sched-ext/scx/issues/3715
Signed-off-by: Tejun Heo <tj@kernel.org>
Under core scheduling, pick_next_task() selects for all SMT siblings under
one continuous hold of the shared core-wide rq lock, and sched_ext's
dispatch can release that lock from inside the pick. In preparation for
making the core-sched pick detect the releases and retry, add
rq->scx.lock_drop_seq and bump it at every site that can release an rq lock
while a dispatch may be in flight. The counter is only maintained while core
scheduling is enabled. No functional changes.
Fixes: 4c95380701 ("sched/ext: Fold balance_scx() into pick_task_scx()")
Cc: stable@vger.kernel.org # v6.19+
Signed-off-by: Tejun Heo <tj@kernel.org>
Under core scheduling, dispatch runs from within the core-wide pick and can
target a sibling rq, so ops.dispatch() may execute on a CPU different from
the dispatched rq's. Several kfunc paths assumed the two always coincide:
- scx_dsq_move() decided whether an rq lock is held by testing this_rq()'s
rq flags and lock-danced accordingly. A dispatch for a sibling took the
unlocked-context branch and acquired the source rq lock on top of the
already held dispatched rq lock which could deadlock.
- scx_bpf_sub_dispatch() dispatched this_rq() with its stashed
sub_dispatch_prev, which is NULL when dispatching for a sibling.
- finish_dispatch(), scx_bpf_dsq_reenq() and scx_bpf_dsq_nr_queued()
resolved SCX_DSQ_LOCAL to this CPU's local DSQ rather than the dispatched
rq's. The latter two are callable from other rq-locked operations too,
where SCX_DSQ_LOCAL now likewise resolves to the op's rq. This changes
behavior also without core scheduling, e.g. for ops.enqueue() running a
remote wakeup on the waking CPU, and is intended: which CPU happens to
execute an operation is incidental, the op's rq is what it is operating
on, and the resolution now matches the insert side where SCX_DSQ_LOCAL
dispatches land on the task's rq.
Use the rq tracked by scx_locked_rq(), which is set to the dispatched rq
around ops invocations and NULL in unlocked contexts.
Fixes: 4c95380701 ("sched/ext: Fold balance_scx() into pick_task_scx()")
Cc: stable@vger.kernel.org # v6.19+
Signed-off-by: Tejun Heo <tj@kernel.org>
SCX_RQ_BAL_KEEP tells the pick to keep running the previous task, a leftover
from when balancing and picking were separate operations. An rq-level flag
only works while dispatches and picks pair up one to one, which core
scheduling breaks: selections interleave through dispatch's lock drops and a
pick can consume a stale flag, keeping a task that has since been dequeued.
Fixing core scheduling support requires the decision to travel with the
dispatch that made it. Make scx_dispatch_sched() and balance_one() return an
explicit verdict instead and drop the flag's plumbing from the tools autogen
enum headers.
Also factor the pick-side invocation, its follow-up queueing and the
post-dispatch checks out of do_pick_task_scx() into dispatch_pick(). No
functional changes intended.
v2: Drop the SCX_RQ_BAL_KEEP plumbing from the tools autogen enum headers
as well (Andrea).
Fixes: 4c95380701 ("sched/ext: Fold balance_scx() into pick_task_scx()")
Cc: stable@vger.kernel.org # v6.19+
Signed-off-by: Tejun Heo <tj@kernel.org>
Core scheduling's pick_next_task() operates on all sibling rqs under one
acquisition of the shared core-wide lock. A ->pick_task() that releases the
rq lock leaves every sibling __lock momentarily free, letting
__sched_core_flip(false) complete mid-selection and rebind rq_lockp() under
it. The selection resumes on the split locks, touching sibling state it no
longer protects, and __schedule() finally releases a lock that was never
taken while leaking the one that was.
Count in-flight core-wide selections in the leader's rq->core_pick_in_flight
and make __sched_core_flip() wait for the count to drain. The count only
changes under the shared lock, which the flip holds while sampling, so no
other ordering is needed. The wait can repeat while selections overlap, but
the flip backs off between samples and flips are rare cookie-lifetime
events.
sched_core_cpu_deactivate() moves the count to the new leader - a stale copy
left behind would bias it forever if that CPU later returns as its own
leader.
Fixes: 539f65125d ("sched: Add core wide task selection and scheduling")
Cc: stable@vger.kernel.org # v5.14+
Signed-off-by: Tejun Heo <tj@kernel.org>
Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Core scheduling's pick_next_task() breaks when a ->pick_task()
implementation can release the rq lock. The selection state derived on entry
is only valid while the lock is held continuously. Once a pick can drop the
lock, an interleaving selection can invalidate all of it: the single-CPU
fast path can commit an uncookied pick although the core went cookied during
the release, and forceidle committed by the interleaving selection skews the
restarted pass's accounting.
Fix it by restarting the whole selection when a pick returns RETRY_TASK
after releasing the lock: a single restart point above the state derivation
replaces the per-loop restart labels, so a retry picks up state committed by
interleaving selections and accounts and resets forceidle like a fresh
selection would.
need_sync and fi_before latch across retries. Clock validity can't be
re-derived - there is no program-ordered way to tell whether the own and
core rq clocks are still updated after the lock was released, as other
lockers' pin cycles may or may not have invalidated them. When restarting,
clear core_clock_updated so that the sibling loop re-updates the core rq,
and update the own rq clock if invalidated.
Fixes: 4c95380701 ("sched/ext: Fold balance_scx() into pick_task_scx()")
Cc: stable@vger.kernel.org # v6.19+
Signed-off-by: Tejun Heo <tj@kernel.org>
Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Pull bpf-next d114bb9893 ("Merge branch
'add-arena-argument-support-to-kfuncs-and-struct_ops'") to make the __arena
and __arena__nullable kfunc and struct_ops argument suffixes available. The
suffixed arguments will be used to convert sched_ext kfuncs and struct_ops
callbacks that currently pass arena pointers as scalars and rebase them by
hand.
With the changes that enable preempt count to track IRQ disabling
nesting, we don't have enough bits in 32-bit preempt count
implementation, as a result we move NMI nesting bits out of the 32-bit
preempt count. However on the architectures that can support 64-bit
preempt count implementation, we can keep the NMI nesting bits in the
32-bit preempt count and avoid maintaining NMI nesting bits outside of
the same cache line.
Therefore HAS_SEPARATE_PREEMPT_RESCHED_BITS is introduced to allow
architectures to select this. Note that under this Kconfig, preempt
count is maintained in a 64-bit word however preempt_count() still
remains as an int because all the effective bits still fit in
(previously we mask out NEED_RESCHED bit in preempt_count()). This
should make no functional changes for existing preempt_count() users.
Enable this for x86_64 along with the introduction of the Kconfig.
[boqun: Undo the __preempt_count_{add,sub}() optimization in 32-bit
preempt count since it may introduce {over,under}flow]
Originally-by: Peter Zijlstra <peterz@infradead.org>
Signed-off-by: Boqun Feng <boqun@kernel.org>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/20260804161447.84806-11-boqun@kernel.org
Currently preempt_count() is always a non-negative int on all archs
(PREEMPT_NEED_RESCHED archs will mask out the MSB when returning
preempt_count()), hence the checking in __cant_migrate() is in fact just
checking whether preempt_count() is 0 or not. In a future change, we are
going to use all the 32 bits of preempt_count(), which would make
negative int values possible from preempt_count(). Therefore convert the
"> 0" comparison into a zero check to prepare for the future change.
No functional changes are intended.
Signed-off-by: Boqun Feng <boqun@kernel.org>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/20260804161447.84806-10-boqun@kernel.org
The preempt_offset is always 0 in all the callsites of __cant_sleep(),
hence remove it. It also allows us to clear up the code a bit by
no longer using a "preempt_count() > .." comparison.
Signed-off-by: Boqun Feng <boqun@kernel.org>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/20260804161447.84806-9-boqun@kernel.org
The semantics of various IRQ disabling guards match what
*_irq_{disable,enable}() provide, i.e. the interrupt disabling is
properly nested, therefore it's OK to switch to use
*_irq_{disable,enable}() primitives.
[boqun: Adjust the user-side changes in do_sched_cfs_*_timer() provided
by Peter and Lyude]
Signed-off-by: Boqun Feng <boqun@kernel.org>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/20260804161447.84806-8-boqun@kernel.org
Merge cpufreq updates for 7.3-rc1:
- Minor fixes and cleanups in assorted cpufreq drivers (Dan Carpenter,
Guru Das Srinagesh, Haoxiang Li, Karl Mehltretter, Sasha Finkelstein,
and Pan Chuang)
- Fix cpufreq table creation and bios_limits() callback in the Rust
bindings (Priya Bala Govindasamy)
- Add IPQ5210 support to qcom-nvmem driver (Varadarajan Narayanan)
- Adjust the .adjust_perf() cpufreq driver callback to allow the
maximum performance value to be passed to drivers and update the
intel_pstate driver to use it (Rafael Wysocki)
- Set policy->cur to the actual requested frequency in the intel_pstate
driver when the performance policy is used (Rafael Wysocki)
- Simplify HWP handling on Broadwell processors in intel_pstate (Rafael
Wysocki)
- Fix setting minimum P-state at init time in intel_pstate (Rafael
Wysocki)
- Consolidate frequency values computation in intel_pstate and clean up
code in that driver (Rafael Wysocki)
- Add missing kernel-doc desciptions for structure and union members in
the amd-pstate driver (David Vernet)
- Handle missing policy in dynamic EPP callbacks in the amd-pstate
driver (EDAMAMEX)
- Introduce EXPORT_SYMBOL_FOR_PSTATE_UT() to export amd-pstate driver
symbols to the amd-pstate-ut subdriver (K Prateek Nayak)
- Add dynamic EPP as an "energy_performance_preference" mode in
amd-pstate, remove the "amd_dynamic_epp" kernel command line option
and the "dynamic_epp" sysfs attribute, and update the dynamic_epp
documentation accordingly (K Prateek Nayak)
- Add unit tests for CPPC Performance Priority and the "dynamic" EPP
mode in the amd-pstate driver (K Prateek Nayak)
- Set min_limit_freq based on bios_min_perf in amd-pstate and remove
the defensive check for bios_min_perf from it (K Prateek Nayak)
- Fix EPP return type and handle errors in amd-pstate during
initialization, toggle auto_sel in active mode on shared memory
systems, and cache the firmware programmed EPP value (Marco Scardovi)
- Skip tests in amd-pstate-ut if the amd-pstate driver is not in active
use (Qianheng Peng)
- Replace sprintf() with sysfs_emit() in sysfs show in the cpufreq
schedutil governor and fix a self-contradictory comment in
sugov_iowait_apply() (Zhongqiu Han)
- Fix the usage example for the sampling_rate tunable of the ondemand
cpufreq governor in admin-guide (wangxiaodong)
* pm-cpufreq: (40 commits)
cpufreq: imx6q: fix out-of-bounds write when probed more than once
cpufreq: imx6q: fix devres accumulation across driver rebind
rust: cpufreq: Fix temporary write in Registration::bios_limit_callback
rust: cpufreq: Add CPUFREQ_TABLE_END as last table entry in TableBuilder::to_table
cpufreq: intel_pstate: Adjust policy->cur in active mode to policy
cpufreq/amd-pstate: Document missing kernel-doc members
cpufreq/amd-pstate-ut: Add unit test for CPPC Performance Priority
cpufreq/amd-pstate-ut: Add unit test for "dynamic" EPP mode
cpufreq/amd-pstate: Reduce the scope of exported symbols
Documentation/amd-pstate: Update dynamic_epp documentation with new behavior
cpufreq/amd-pstate: Remove "amd_dynamic_epp" cmdline and "dynamic_epp" sysfs
cpufreq/amd-pstate: Add dynamic EPP as an "energy_performance_preference" mode
cpufreq/amd-pstate: Extract platform profile to EPP conversion into a helper
cpufreq/amd-pstate: Remove the defensive check for bios_min_perf
cpufreq/amd-pstate: Set min_limit_freq based on bios_min_perf
cpufreq: apple-soc: Calculate frequency as a 64-bit value
kselftest: cpufreq: Backup and restore governor for sptests
selftests/cpufreq: Remove unnecessary sudo from quick_shuffle()
selftests/cpufreq: Remove unused local variables from switch_show_governor()
cpufreq/amd-pstate: handle missing policy in dynamic EPP callbacks
...
Commit 9c63e84db2 ("sched/core: Disable SD_PREFER_SIBLING on asymmetric
CPU capacity domains") removed the SD_PREFER_SIBLING from the domains with
asymmetric capacity. This was done to avoid spreading tasks to sibling
scheduling groups with less capacity, but this does not happen: checks for
capacity in update_sd_pick_busiest(), sched_balance_find_src_group(), and
sched_balance_find_src_rq() prevent migrations from high- to low-capacity
CPUs if the busiest group is not overloaded.
The cluster topology is a notable example: some systems have scheduling
domains spanning CPUs of asymmetric capacity, grouped into two or more
equal-capacity clusters sharing an L2 cache. When CONFIG_SCHED_CLUSTER is
enabled, SD_PREFER_SIBLING is needed in the domain to spread load across
these clusters.
CPUs with spare capacity, big or small, have always helped overloaded
groups. Once the overloading condition disappears, misfit load will still
be used to move high-utilization tasks to bigger CPUs if they have spare
capacity.
Adding the SD_PREFER_SIBLING flag shifts load balancing in shared-LLC
domains from equalizing the number of idle CPUs to equalizing the number
of running tasks. This enables migrations among clusters from newly-idle
load balance, where the outgoing task is already dequeued but the CPU
has not yet transitioned to idle.
Signed-off-by: Ricardo Neri <ricardo.neri-calderon@linux.intel.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Vincent Guittot <vincent.guittot@linaro.org>
Tested-by: Christian Loehle <christian.loehle@arm.com>
Tested-by: Andrea Righi <arighi@nvidia.com>
Link: https://patch.msgid.link/20260720-rneri-fix-cas-clusters-v6-6-bb500bf4afd4@linux.intel.com
sched_balance_find_src_rq() avoids selecting a runqueue with a single
running task as busiest if doing so results in migrating the task to a
CPU with less than ~5% of extra capacity. It also unintentionally
prevents migrations between CPUs of identical capacity.
When CONFIG_SCHED_CLUSTER is enabled, load should be balanced across
clusters of CPUs with the same capacity. Allowing migration between CPUs
of identical capacity is necessary to meet this goal.
Use get_actual_cpu_capacity() to reflect architectural capacity as well
as diminished capacity due to hardware or cpufreq pressure. Guard this
check with the sched_cluster_active static key so that systems without
cluster topology are unaffected.
Signed-off-by: Ricardo Neri <ricardo.neri-calderon@linux.intel.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Christian Loehle <christian.loehle@arm.com>
Reviewed-by: Vincent Guittot <vincent.guittot@linaro.org>
Tested-by: Christian Loehle <christian.loehle@arm.com>
Tested-by: Andrea Righi <arighi@nvidia.com>
Link: https://patch.msgid.link/20260720-rneri-fix-cas-clusters-v6-5-bb500bf4afd4@linux.intel.com
In domains with asymmetric capacity, identifying misfit load in a
scheduling group is not useful when the destination CPU cannot help (i.e.,
its capacity exceeds the group's maximum CPU capacity by less than ~5%). In
such cases, it also prevents load balance among clusters of equal capacity
when CONFIG_SCHED_CLUSTER is enabled. This happens because
update_sd_pick_busiest() skips candidate groups of type misfit_task if the
destination CPU has similar capacity.
Skipping misfit load accounting in this situation allows the group to be
classified as has_spare or fully_busy and lets load balancing proceed. Keep
marking scheduling groups as overloaded when misfit tasks are present. The
sg_overloaded flag propagates to the root domain and allows bigger CPUs in
it to help via newly idle balance.
Signed-off-by: Ricardo Neri <ricardo.neri-calderon@linux.intel.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Christian Loehle <christian.loehle@arm.com>
Reviewed-by: Chen Yu <yu.c.chen@intel.com>
Reviewed-by: Vincent Guittot <vincent.guittot@linaro.org>
Tested-by: Christian Loehle <christian.loehle@arm.com>
Tested-by: Andrea Righi <arighi@nvidia.com>
Link: https://patch.msgid.link/20260720-rneri-fix-cas-clusters-v6-4-bb500bf4afd4@linux.intel.com
update_sd_pick_busiest() may incorrectly select a fully_busy group as the
busiest group when its per-CPU capacity exceeds that of the destination
CPU. This happens because the type of busiest group is initialized to
group_has_spare and allows the fully_busy group to win the type comparison.
update_sd_pick_busiest() should not choose a candidate scheduling group
with at most one runnable task if its per-CPU capacity is greater than that
of the destination CPU. Such a check already exists, but it is done too
late: after the type comparison, preventing a subsequent fully_busy group
of equal per-CPU capacity from being correctly selected.
Move this check to occur before comparing group types.
Fixes: 0b0695f2b3 ("sched/fair: Rework load_balance()")
Signed-off-by: Ricardo Neri <ricardo.neri-calderon@linux.intel.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Christian Loehle <christian.loehle@arm.com>
Reviewed-by: Chen Yu <yu.c.chen@intel.com>
Reviewed-by: Tim Chen <tim.c.chen@linux.intel.com>
Reviewed-by: Vincent Guittot <vincent.guittot@linaro.org>
Tested-by: Christian Loehle <christian.loehle@arm.com>
Tested-by: Andrea Righi <arighi@nvidia.com>
Link: https://patch.msgid.link/20260720-rneri-fix-cas-clusters-v6-3-bb500bf4afd4@linux.intel.com
The argument sg_overloaded of update_sg_lb_stats() is only consumed when
balancing at the root domain. It only makes sense to update it in such a
case. Commit 3229adbe78 ("sched/fair: Do not compute overloaded status
unnecessarily during lb") updated the logic accordingly but missed the case
in which the root domain has the SD_ASYM_CPUCAPACITY flag. Fix this.
Fixes: 3229adbe78 ("sched/fair: Do not compute overloaded status unnecessarily during lb")
Reported-by: Chen Yu <yu.c.chen@intel.com>
Signed-off-by: Ricardo Neri <ricardo.neri-calderon@linux.intel.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Vincent Guittot <vincent.guittot@linaro.org>
Tested-by: Christian Loehle <christian.loehle@arm.com>
Tested-by: Andrea Righi <arighi@nvidia.com>
Link: https://patch.msgid.link/20260720-rneri-fix-cas-clusters-v6-2-bb500bf4afd4@linux.intel.com
When picking a busiest CPU with only one running task, the function
sched_balance_find_src_rq() skips candidate CPUs if the destination CPU has
less than ~5% extra capacity. This condition only holds if all the SMT
siblings of a CPU are idle.
SMT siblings share the computing resources of a physical core and this
results in reduced capacity if more than one sibling is busy.
Skipping a CPU as described would prevent the load balancer from pulling
tasks from a scheduling group previously and correctly identified as
group_smt_balance (i.e., one with more than one task running).
Do not skip a candidate CPU of similar capacity if it has busy SMT
siblings.
Signed-off-by: Ricardo Neri <ricardo.neri-calderon@linux.intel.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: K Prateek Nayak <kprateek.nayak@amd.com>
Reviewed-by: Vincent Guittot <vincent.guittot@linaro.org>
Tested-by: Andrea Righi <arighi@nvidia.com>
Link: https://patch.msgid.link/20260720-rneri-fix-cas-clusters-v6-1-bb500bf4afd4@linux.intel.com
find_new_ilb() selects the first idle housekeeping CPU without
considering whether another thread is running on the same physical core.
On an SMT system, the idle load balancer can therefore activate both
siblings even when another housekeeping CPU has an entirely idle core.
On most SMT systems, this is not problematic because the idle load
balancer is a short-lived activity and the transient wakeup of a sibling
has negligible performance impact.
However, this can be particularly costly on NVIDIA Olympus cores used in
Vera. Briefly activating an otherwise idle sibling can reduce the
performance available to the other sibling and this effect does not
necessarily end once the activated sibling becomes idle: after the ILB
finishes and its CPU enters WFI, full single-thread performance is
restored only after the sibling has remained idle for a qualification
interval (10 Ki cycles on the tested Vera system). Repeated short
sibling wakeups can therefore sustain the interference even with little
actual overlap.
Prevent this by preferring an idle housekeeping CPU whose entire SMT
core is idle. Retain the first idle CPU as a fallback when no fully idle
core is available, so NOHZ balancing continues to make forward progress.
Once a partially busy core has been examined, skip its remaining SMT
siblings to avoid repeating the core-idle check on wide SMT systems.
Tests performed using an ad hoc GEMM benchmark running one CPU-intensive
task per SMT core within its CPU affinity mask improved from
approximately 6.2 TFLOP/s to 9.4 TFLOP/s.
Note that this preference may wake a fully idle physical core instead of
using an idle sibling of an active core, potentially increasing ILB
wakeup latency or energy consumption on some architectures. It may also
scan additional CPUs before selecting the one to run the ILB. The
selection falls back to the first idle CPU when no fully idle SMT core
is available. Non-SMT systems continue to select the first idle
housekeeping CPU.
Signed-off-by: Andrea Righi <arighi@nvidia.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Mete Durlu <meted@linux.ibm.com>
Reviewed-by: Vincent Guittot <vincent.guittot@linaro.org>
Link: https://patch.msgid.link/20260804151324.918020-1-arighi@nvidia.com
CONFIG_SYSCTL used to make sense as a separate hidden bool before commit
61a47c1ad3 ("sysctl: Remove the sysctl system call"); it was selected
by both CONFIG_SYSCTL_SYSCALL and CONFIG_PROC_SYSCTL.
Today CONFIG_PROC_SYSCTL is the only selector, so the two are always
equal. Kill the hidden bool, rename the PROC_SYSCTL prompt to SYSCTL,
and s/CONFIG_PROC_SYSCTL/CONFIG_SYSCTL/ tree-wide.
Signed-off-by: Oleg Nesterov <oleg@redhat.com>
Signed-off-by: Joel Granados <joel.granados@kernel.org>
blkdev.h gets included in various places outside the block layer just
for struct blk_plug and related plugging functions.
Split blk_plug into a separate helper to reduce the amount of code
that needs to get rebuilt when blkdev.h changes and to slightly
reduce compile times.
In io_uring this requires pulling in a few other headers explicitly that
previously were implicitly included through blkdev.h.
Signed-off-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Christian Brauner (Amutable) <brauner@kernel.org>
Reviewed-by: Johannes Thumshirn <johannes.thumshirn@wdc.com>
Reviewed-by: Damien Le Moal <dlemoal@kernel.org>
Link: https://patch.msgid.link/20260804125524.740996-1-hch@lst.de
Signed-off-by: Jens Axboe <axboe@kernel.dk>
When rescue demand on a cpu persistently exceeds the configured bandwidth,
tasks age on that cpu's rescue DSQ until the stall watchdog fires. The
watchdog blames the waiting task's owner, but the misbehaving party is
whoever floods the queue, not whoever happens to time out.
Track each sched's recent rescue consumption per cpu as a decaying average.
Once the oldest waiter on a cpu's rescue DSQ has been queued past a
threshold derived from the rescue knobs (4s at the defaults), the rescue
timer ejects the sub with the highest recent consumption on that cpu with
SCX_EXIT_ERROR_RESCUE. With no recent consumer there is no victim and
nothing is ejected - the generic stall watchdog eventually blames the
waiter's owner instead. Ejections on a cpu are spaced one threshold apart so
the freed bandwidth can drain the backlog before another sub is judged.
The overload check only wins the race against the stall watchdog when the
watchdog timeout clears the threshold, and a single in-budget wait must not
cross the trigger on its own. Warn on a scheduler whose timeout doesn't fit
and on knobs whose funding period exceeds half the threshold.
v2: - Track kill_at in jiffies_64 - on 32-bit, the time_before() grace check
wraps 2^31 ticks after the last ejection and suppresses ejections.
(sashiko AI)
- Track rescue_avg_at in jiffies_64 likewise - the unsigned long decay
delta truncates mod 2^32 on 32-bit and can revive a weeks-old usage
average in the victim pick.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
A local DSQ insert lacking the needed caps is diverted to the reject DSQ and
bounced back through ops.enqueue() so the scheduler can re-decide. That
recovery assumes the scheduler has somewhere legal to send the task. When it
doesn't, e.g. when the task's affinity is restricted to cids delegated away,
the task starves until the stall watchdog ejects the scheduler. An exiting
task is worse - it skips ops.enqueue() and the rejection becomes a
self-requeuing cycle that burns the CPU until the watchdog fires.
Add SCX_ENQ_RESCUE, a fallback modifier on local DSQ inserts. When the
insert would be rejected for missing caps, the kernel takes over and runs
the task on the target CPU without consulting the owning scheduler. The
kernel sets the flag itself when enqueueing an exiting task.
Rescue is a last-resort forward-progress backstop with a persistent
disadvantage, not a way around cap enforcement. A per-CPU token bucket
accrues rescue_bandwidth_ppt (default 2%) of CPU time and rescues run one at
a time in arrival order. Each is granted a slice of the rescue_quantum_us
(default 5ms) quantum divided across the waiters, waits at the tail of the
local DSQ claiming no priority, and rejoins its scheduler as a fresh arrival
once the slice is served.
The schedulers keep their normal control over an admitted rescuee and may
preempt or reslice it. Service is measured on CPU time actually received, so
neither shortens the rescue. Prolonged denial escalates - the remaining
slice turns into protected execution (SCX_TASK_PROTECTED) and the rescuee
preempts the current task. Escalation is paced by the same bucket, and
delivered service converges on the configured bandwidth no matter how
aggressively the schedulers dispatch.
Both knobs are root-only and SCX_RESCUE_DISABLE turns rescue off, making
SCX_ENQ_RESCUE inserts reject as usual.
v2: - Add SCX_OPS_OPEN() fix-ups for the new ops fields so cpu-form
schedulers setting them still load on older kernels. (Andrea)
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
A BPF scheduler can displace any of its tasks at will - cut a running one's
slice with an SCX_ENQ_PREEMPT dispatch, an SCX_KICK_PREEMPT kick or a direct
shortening, and jump a queued one with HEAD insertions. Sometimes the kernel
needs a slice and a DSQ position to stick regardless.
Add SCX_TASK_PROTECTED, guarding both:
- The slice becomes immutable. Every scheduler-reachable write is refused
and counted as SCX_EV_SLICE_DENIED. Higher scheduling classes are
unaffected. PREEMPT|IMMED can't preempt a running protected task and gets
reenqueued.
- A protected task that reached the head of its DSQ keeps it - HEAD
insertions land behind the leading run of protected tasks and reenqueue
sweeps skip them. Only rq-owned DSQs can hold protected tasks, so the walk
runs only for them.
The bit lives in p->scx.flags so that both the refusal and the head walk
read it under the rq lock that protects it.
Protection ends when the slice is consumed, when the task leaves the rq
except for a save/restore on the running task, on a yield, when the
scheduler enters bypass, and when the task leaves scx. The flag is
kernel-internal and not used yet.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
p->scx.slice and p->scx.dsq_vtime writes have no synchronization rules. The
dsq insert kfuncs write both fields synchronously from whatever context
they're called in - a direct dispatch from ops.select_cpu() writes with only
pi_lock held - and, as the kfuncs are safe to call spuriously with the
invalid dispatch discarded later, a scheduler can modify any task's slice by
spuriously calling them. The latter stands in the way of an upcoming patch
which adds kernel-granted slices that the schedulers must not be able to
modify.
Give both fields explicit rules. While the task is running, sleeping or
queued on an rq-owned DSQ, the rq lock protects them - these are the states
where the kernel consumes the slice. While queued on a user DSQ or on the
BPF side, the kernel neither consumes nor decides on the fields and every
writer acts for the BPF scheduler - synchronizing the writers is the
scheduler's responsibility and whichever write lands last wins.
To conform, an insert kfunc no longer writes the fields when called. The
values travel with the dispatch and take effect when the task is inserted. A
discarded dispatch has no side effects. The rq lock rule is asserted at the
slice store.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
SCX_ENQ_IGNORE_CAPS is kernel-internal and marks a placement the kernel
forces. scx_caps_for_enq() waives the enqueue cap for it, but a PREEMPT
insert still picks up the preemption cap requirement from
scx_caps_for_preempt(). Update scx_caps_for_preempt() to take enq_flags and
require nothing when SCX_ENQ_IGNORE_CAPS is set.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
The dsq insert kfuncs reject __SCX_ENQ_INTERNAL_MASK bits in
scx_dsq_insert_preamble() instead of scx_vet_enq_flags(). A scheduler can
smuggle internal flags such as SCX_ENQ_CLEAR_OPSS through the dsq move
kfuncs and corrupt the dispatch protocol. Move the rejection into
scx_vet_enq_flags(). The vtime move wrapper OR'd the internal
SCX_ENQ_DSQ_PRIQ bit into enq_flags before the vet; the bit now goes in
inside scx_dsq_move() after the vet.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
scx_bpf_now() couples the valid-or-fresh rq clock read to the current rq.
The read is useful for kernel-internal timing against a specific rq,
including a remotely locked one. Factor it out into __scx_bpf_now().
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
set_task_slice(), task_unlink_from_dsq(), move_local_task_to_local_dsq(),
init_dsq() and dump_line() will be used outside ext.c. Add the scx_ prefix
and declare them in internal.h. The scx_sched_all list will also be used
outside ext.c, drop its static. No functional changes.
v2: Declare scx_sched_all outside the CONFIG_EXT_SUB_SCHED block - the
definition is unconditional. (sashiko AI)
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
The following rescue execution addition gives the function a third possible
destination, making a name that enumerates the outcomes a poor fit. Rename
to the destination-neutral scx_resolve_local_dsq(). No functional changes.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
The built-in idle masks are reset with all online CPUs marked idle
before sched_ext is enabled. Busy CPUs can therefore be incorrectly
advertised as idle until their next idle transition.
Initialize the masks empty so that the initial state is conservative.
When bypass is lifted, every CPU is rescheduled and idle-to-idle
re-picks populate the masks with CPUs that are actually idle. Later
idle transitions keep the masks up to date.
Suggested-by: Tejun Heo <tj@kernel.org>
Signed-off-by: Andrea Righi <arighi@nvidia.com>
Reviewed-by: Kuba Piecuch <jpiecuch@google.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
- More lifecycle fixes for the new sub-scheduler support: a failed enable
could tear down a never-linked sub-scheduler in a way that races the root
scheduler's disable and leads to a use-after-free, tasks that were not on
the ext class could still get the enable callback, and a policy-rejection
path silently rewrote a running task's scheduling policy instead of
aborting the scheduler.
- Scheduler enable/disable could deadlock with cgroup removal and a
concurrent cgroup weight write through kernfs. Fixed by reordering lock
acquisition.
- Sync wakeups could leave the waker CPU incorrectly marked idle in the
built-in idle-CPU tracking.
- A selftest fix for sleeping tasks whose CPU affinity changes before
wakeup.
-----BEGIN PGP SIGNATURE-----
iIQEABYKACwWIQTfIjM1kS57o3GsC/uxYfJx3gVYGQUCam/mog4cdGpAa2VybmVs
Lm9yZwAKCRCxYfJx3gVYGe0NAQC2UrrXeSm97RTCv+5HBfom/8sjPdB2tiAwGiUt
fUUQ+AD9FP30urJaoiALUiL+EGWpLFejwUCXNXlT+0kN2TiFxQM=
=EuWf
-----END PGP SIGNATURE-----
Merge tag 'sched_ext-for-7.2-rc6-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/sched_ext
Pull sched_ext fixes from Tejun Heo:
- More lifecycle fixes for the new sub-scheduler support: a failed
enable could tear down a never-linked sub-scheduler in a way that
races the root scheduler's disable and leads to a use-after-free,
tasks that were not on the ext class could still get the enable
callback, and a policy-rejection path silently rewrote a running
task's scheduling policy instead of aborting the scheduler.
- Scheduler enable/disable could deadlock with cgroup removal and a
concurrent cgroup weight write through kernfs. Fixed by reordering
lock acquisition.
- Sync wakeups could leave the waker CPU incorrectly marked idle in the
built-in idle-CPU tracking.
- A selftest fix for sleeping tasks whose CPU affinity changes before
wakeup.
* tag 'sched_ext-for-7.2-rc6-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/sched_ext:
selftests/sched_ext: Handle sleeping task affinity changes in numa test
sched_ext: Mark waker CPU busy when selected in WAKE_SYNC case
sched_ext: Don't enable non-ext tasks in the sub-sched task loops
sched_ext: Skip sub-disable teardown for never-linked sub-schedulers
sched_ext: Take cgroup_lock() first in scx_cgroup_lock()
sched_ext: Reject setting disallow from init_task outside the enable path
- A pressure trigger's poll timer could be re-armed while the last trigger
was being torn down and then fire after the cgroup was freed. Tie the
timer to the cgroup's lifetime and shut it down when the cgroup is freed.
- Writing to a pressure file forked a worker kthread while holding the
cgroup mutex, creating lock dependencies from the mutex to the whole fork
path. A pressure write racing a sched_ext scheduler enable, which blocks
forks before grabbing the mutex, deadlocked. Fork the worker with the
mutex dropped.
- Documentation fix for io.latency behavior on non-rotational devices.
-----BEGIN PGP SIGNATURE-----
iIQEABYKACwWIQTfIjM1kS57o3GsC/uxYfJx3gVYGQUCam/mlw4cdGpAa2VybmVs
Lm9yZwAKCRCxYfJx3gVYGVhRAQCyNBHCHpaY9erKTezenGDK6D+LqbfLWiWuIntB
swiwLQEA6h6Rgob2GDDRHOey7+XwF6PHh6xoh4FhSYboeEZd1wc=
=Re+7
-----END PGP SIGNATURE-----
Merge tag 'cgroup-for-7.2-rc6-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/cgroup
Pull cgroup fixes from Tejun Heo:
- A pressure trigger's poll timer could be re-armed while the last
trigger was being torn down and then fire after the cgroup was freed.
Tie the timer to the cgroup's lifetime and shut it down when the
cgroup is freed.
- Writing to a pressure file forked a worker kthread while holding the
cgroup mutex, creating lock dependencies from the mutex to the whole
fork path. A pressure write racing a sched_ext scheduler enable,
which blocks forks before grabbing the mutex, deadlocked.
Fork the worker with the mutex dropped.
- Documentation fix for io.latency behavior on non-rotational devices.
* tag 'cgroup-for-7.2-rc6-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/cgroup:
Docs/admin-guide/cgroup-v2: document io.latency rotational vs non-rotational behavior
sched/psi: Shut down rtpoll_timer in psi_cgroup_free()
sched/psi: Create the psimon kthread outside of cgroup_mutex
If the SCX_ENABLING -> SCX_ENABLED cmpxchg at the tail of
scx_root_enable_workfn() fails, the function jumps to err_disable
without setting ret. At that point ret still holds the return value
of the last successful __scx_init_task() call, which is 0, so the
err_disable fallback reports the meaningless message:
scx_root_enable() failed (0)
Set ret = -EBUSY, consistent with the other enable-state guards at
the top of the same function, so the fallback always reports a real
errno.
Signed-off-by: Liang Luo <luoliang@kylinos.cn>
Signed-off-by: Tejun Heo <tj@kernel.org>
The kernel-doc comment for sched_ext_ops::sub_cgroup_id uses the old
@cgroup_id name, which no longer matches the struct member. This
produces two kernel-doc warnings:
Warning: struct member sub_cgroup_id not described in sched_ext_ops
Warning: Excess struct member cgroup_id description in sched_ext_ops
Update the @param name to match the actual member.
Signed-off-by: Liang Luo <luoliang@kylinos.cn>
Signed-off-by: Tejun Heo <tj@kernel.org>
* pm-cpufreq:
cpufreq/amd-pstate: handle missing policy in dynamic EPP callbacks
cpufreq/amd-pstate: Cache the firmware programmed EPP value
cpufreq/amd-pstate: Toggle auto_sel in active mode on shared memory systems
cpufreq/amd-pstate: Fix EPP return type and handle errors during initialization
cpufreq: amd-pstate-ut: Skip tests when amd-pstate driver is not active
cpufreq: schedutil: Replace sprintf() with sysfs_emit() in sysfs show
cpufreq: schedutil: Fix self-contradictory comment in sugov_iowait_apply()
Documentation: admin-guide: cpufreq: fix sampling_rate example command
cpufreq: intel_pstate: Move two functions closer to callers
cpufreq: intel_pstate: Consolidate frequency values computation
cpufreq: intel_pstate: Introduce intel_pstate_update_freq_limits()
cpufreq: intel_pstate: Fix setting minimum P-state at init time
cpufreq: intel_pstate: Rename INTEL_PSTATE_HWP_BROADWELL
cpufreq: intel_pstate: Simplify HWP handling on Broadwell
cpufreq: intel_pstate: Adjust the .adjust_perf() driver callback
cpufreq: intel_pstate: Rearrange checks in hybrid_get_cost()
Commit 14a8570564 ("sched/deadline: Use revised wakeup rule for
dl_server") applies the revised wakeup rule to any server, as a result
servers that are not running (dl_defer_running == 0) and start with a
deadline overflow get enqueued and can boost tasks as if they were
running, invalidating the defer rule and the documented state model.
Apply the revised wakeup rule only for deferrable servers that are
marked as running.
Fixes: 14a8570564 ("sched/deadline: Use revised wakeup rule for dl_server")
Signed-off-by: Gabriele Monaco <gmonaco@redhat.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Acked-by: Juri Lelli <juri.lelli@redhat.com>
Tested-by: Andrea Righi <arighi@nvidia.com>
Link: https://patch.msgid.link/20260522125833.264145-1-gmonaco@redhat.com
scx_link_sched() carries each failure out of the locked section through
err_msg and ret because scx_error() used to take scx_sched_lock and couldn't
be called under it. That restriction is gone, so report each failure at the
site it's detected and return directly. The scx_error() here claims the exit
on the sched being linked, which has no descendants yet, and the locked
propagation walk is deferred, so nothing reacquires scx_sched_lock inline.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
scx_hardlockup() defers the abort to an irq_work because exit claiming used
to take scx_sched_lock and couldn't run from NMI. The deferral is now
unnecessary - claiming is NMI-safe and asserting ->aborting is exactly what
breaks the live-locks that hard-lock CPUs. Call handle_lockup() directly and
drop the irq_work. This also makes the self-detected case recoverable: the
perf watchdog fires on the hard-locked CPU itself, where a queued irq_work
never runs with IRQs off.
Also fix the return value: %true used to be returned whenever sched_ext was
loaded, suppressing the kernel's hardlockup report even when the abort was
refused. Return %true only when this call initiated the abort.
Fixes: bd2d76455b ("sched_ext: Defer scx_hardlockup() out of NMI")
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
The per-cpu kick lists are protected by IRQ masking which doesn't stop NMIs,
so scx_bpf_kick_cpu() from NMI silently drops the kick after a one-time
warning. A dropped kick can leave a CPU idle when the scheduler believes it
was woken, which is a correctness problem for the scheduler even if the
kernel is fine. Now that scx_error() works from NMI, abort the scheduler
instead so that the bug is surfaced deterministically. The warned_nmi_kick
tracking is no longer needed.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
The bstr exit kfuncs format the message into a shared static buffer under a
raw spinlock before initiating the exit. The lock can't be taken from NMI
and needlessly serializes all bstr exits system-wide.
Now that exit claiming is lock-free, reverse the order: claim the exit first
and format directly into the exit_info message buffer which the claim winner
owns exclusively. The new scx_exit_bstr() implements the sequence, replacing
scx_bstr_format(), and the shared buffer and lock are deleted; the formatter
itself is what bpf_trace_printk() already runs from NMI. scx_prog_sched()
callers were relying on the lock for RCU protection, which is now provided
explicitly.
A malformed format no longer changes or fails the requested operation:
scx_bpf_exit_bstr() keeps its graceful exit kind and scx_bpf_sub_kill_bstr()
still kills the child, with a fallback message carrying the formatting
errno, while the sched that supplied the bad format is aborted for its bug.
Before this and the previous patch, an "any" category kfunc called from NMI
context could trigger scx_error() and deadlock - e.g. a tracing prog
attached to a function running in NMI calling scx_bpf_dsq_peek() on a
non-existent DSQ would try to grab scx_sched_lock, which may be held by the
interrupted CPU. This and the previous patch fix the deadlock: scx_error()
and the bstr exit kfuncs, and thus scx_bpf_error() and scx_bpf_exit(), are
now safe to call from any context including NMI.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
scx_claim_exit() claims descendants' exits by walking the subtree under
scx_sched_lock, making exit claiming, and thus scx_error(), unusable from
NMI and from under scx_sched_lock. However, kfuncs raising errors can run
from NMI-attached BPF progs, the hardlockup handler runs in NMI, and
scx_link_sched() wants to report failures under the lock.
The walk does two things with different urgencies: ->aborting must be
asserted synchronously to break IRQs-off dispatch-path live-locks, while the
descendants' exit_kind claims can happen later. Split them: sweep ->aborting
locklessly under RCU to unwedge the system and defer the locked
SCX_EXIT_PARENT walk to a new irq_work, both of which are NMI-safe.
The sweep stores each node's ->aborting and then reads its children list
while scx_link_sched() inserts and then checks the parent's ->aborting, the
two sides paired by full barriers - one side always sees the other. A link
that sees ->aborting undoes its insert and fails. As the undo's
list_del_rcu() leaves ->sibling non-empty, list_empty() can no longer
identify a never-linked sched during teardown - add sch->linked instead.
trace_sched_ext_exit can now fire from NMI and is called after the
->aborting stores so that its callbacks don't hold up live-lock recovery.
The exit backtrace is skipped for NMI exits as stack_trace_save()'s
NMI-safety is arch-dependent and undocumented.
v2: Move trace_sched_ext_exit() after the ->aborting stores (Andrea).
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
Unlike local reenqueues, cap rejections have no repeat limit. A
malfunctioning scheduler can keep re-inserting a task to a cid it lacks caps
on, cycling the task through reject and reenqueue. This was assumed safe
because a task that never runs trips the stall watchdog. However, the
reenqueue irq_work re-arms itself and outranks the timer vector, blocking
everything else on the CPU including stall detection and recovery, until the
NMI hardlockup detector fires.
Local reenqueues already have a repeat cap, SCX_REENQ_LOCAL_MAX_REPEAT,
which needs generalizing to cover all reenqueues. It also has an attribution
problem. Counted per-cpu on root, it tears down the whole hierarchy even
when a sub-scheduler caused the repeated reenqueues.
Generalize by bounding every reenqueue with one per-task counter. reenq_cnt
is bumped in scx_do_enqueue_task() on each SCX_ENQ_REENQ, the single path
every reenqueue producer passes through, and cleared in clr_task_runnable()
when the task is picked to run and in scx_disable_task() when it leaves the
scheduler's control. Past SCX_REENQ_MAX_REPEAT the task's owning scheduler
is ejected with a new SCX_EXIT_ERROR_REENQ and the task is left stranded to
be picked up during sched exit.
The SCX_EV_REENQ_LOCAL_REPEAT event becomes SCX_EV_REENQ_REPEAT, counting
repeat reenqueues from all sources.
v2: Count SCX_EV_REENQ_REPEAT only when a reenqueue leads to another
reenqueue, not on every reenqueue.
v3: - Also clear reenq_cnt in scx_disable_task() so that the count doesn't
carry over to the next owner across sched class switches, scheduler
replacement or sub-scheduler rehoming (Andrea Righi).
- Update the stale SCX_EV_REENQ_LOCAL_REPEAT references in sched-ext.rst
(Andrea Righi).
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
dsq->first_task is __rcu for the lockless scx_bpf_dsq_peek(). The task
removal path compares it against the departing task with a plain load, which
sparse flags. The comparison runs under the dsq lock and only tests
identity, so rcu_access_pointer() is the fit.
Signed-off-by: Tejun Heo <tj@kernel.org>
scx_root is __rcu and naked accesses were left as transitional markers for
the multi-scheduler transition, to be converted to accesses through the
associated scheduler instances. Most accesses have since been converted to
resolve the sched from the program or task at hand. The remaining naked
sites divide into ones that semantically always want the root sched, which
this patch resolves, and one that is left to a later patch.
The resolved sites:
- The SCX_OPS_TID_TO_TASK validation and the ecaps sync kick already hold a
sched whose ancestors[] pins the root as entry 0 with plain pointers
stable for the sched's lifetime. Reach the root through the sched at hand.
- The dispatch entry, class switch, idle notification and fork init paths
only execute while the scheduler is live and scx_root never changes inside
the live window, so no update can race them. Add scx_root_protected_live()
which documents that invariant and resolves with a plain load.
- The hotplug path, including the ecaps reseeds, runs with the hotplug lock
held, which excludes the scx_root writers. Add scx_root_protected(), which
accepts either the hotplug lock or scx_enable_mutex.
- Is-root tests use a zero level instead of comparing against the global.
touch_core_sched_dispatch() stays naked, to be resolved by a later patch.
Signed-off-by: Tejun Heo <tj@kernel.org>
cgrp->scx_sched is __rcu and published with rcu_assign_pointer() but every
reader loads it with a plain access, so sparse flags all of them. The reads
are lock-protected: enable/disable paths rewrite the field under all of
scx_enable_mutex, scx_fork_rwsem and cgroup_mutex, and cgroup creation
inherits the parent's sched under cgroup_mutex before the new cgroup is
reachable, so holding any one of the three locks makes the read stable.
Add scx_cgroup_sched() which states the protection with
rcu_dereference_check() and convert the readers. No functional changes.
Signed-off-by: Tejun Heo <tj@kernel.org>
scx_bpf_cidperf_set() reaches cpufreq with no cap check, so any cid-form
sub-sched can steer the frequency of any cid in its view, including ones it
holds nothing on.
Gate it behind a new SCX_CAP_PERF rather than SCX_CAP_BASE: hardware control
is a separate axis from queue access - a parent may well delegate scheduling
on a cid without handing over its frequency. PERF neither implies nor is
implied by the other caps. The check runs under the target rq's lock, which
ecaps updates are also folded under, so it is authoritative - a write can
never land after a revoke has taken effect. Denials are counted in
SCX_EV_SUB_CIDPERF_DENIED.
The operation is synchronous and the outcome is reported to the caller:
scx_bpf_cidperf_set() now returns 0 or -errno, -EACCES on denial. The
cid-form interface is still under initial development, so the signature is
changed in place without versioning.
scx_qmap grants PERF alongside its existing cid grants so the cpuperf demo
keeps working in sub-scheds.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
Factor the cpuperf target write out of scx_bpf_cpuperf_set() into
scx_cpuperf_set() which takes the acting sched and returns 0 or -errno, and
flatten the nested validation into early returns. No functional change.
Prep for gating the write behind a cap and reporting the outcome from the
cid-form kfunc.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
kick_one_cpu() silently skips a kick when the kicking sub-sched lacks
SCX_CAP_BASE on the target cid, as does kick_one_cpu_if_idle() for idle
kicks. The skips are sound with the same logic as the reenq gate but are
invisible today, unlike the preempt degradation counted in
SCX_EV_SUB_PREEMPT_DENIED. Count them in a new SCX_EV_SUB_KICK_DENIED event
so every cap denial is observable.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
scx_bpf_dsq_reenq() with an SCX_DSQ_LOCAL_ON target schedules deferred reenq
work on the cid's cpu, raising an IPI when the target rq isn't the locked
one. Nothing checks caps along the way, so a sub-sched holding no cap at all
on a cid can force its cpu to take IPIs and rq lock cycles at will. The
analogous scx_bpf_kick_cid() path gates delivery on SCX_CAP_BASE in
kick_one_cpu() to prevent exactly this.
Apply the same rule at the reenq scheduling point: if the calling sched
lacks SCX_CAP_BASE on the target cid, drop the reenq and count it in the new
SCX_EV_SUB_REENQ_DENIED event. The check is lockless, which is fine: a reenq
slipping through right after a revoke is harmless, and a wrong denial can't
happen - if the caller has seen its ownership of the cpu, the check sees it
too.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
