mirror of
https://github.com/torvalds/linux.git
synced 2026-10-16 21:00:02 +02:00
master
5775 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Linus Torvalds
|
db7afcbf9e |
sched_ext: Fixes for v7.3-rc7
- A task that skipped the BPF scheduler on its way back to its CPU's local queue could stall there until the watchdog fired, as the CPU had already decided to go idle. - Use-after-free of a destroyed user dispatch queue by a deferred re-enqueue request. - A task could be moved off a dispatch queue while the dispatching CPU was still running ops.dequeue() on it. - Bypass mode mishandled a sub-scheduler bypass overlapping an ancestor's, as across a system suspend. Descendants could leave bypass mid-teardown and tasks could be stranded on bypass queues. - The idle CPU selection helper could return a CPU outside the allowed CPUs of a pinned or migration-disabled task. -----BEGIN PGP SIGNATURE----- iIQEABYKACwWIQTfIjM1kS57o3GsC/uxYfJx3gVYGQUCatACow4cdGpAa2VybmVs Lm9yZwAKCRCxYfJx3gVYGdEvAP4tfwuDqRpiVDU/69SdALuWdObKorErLg/6uTxE 53qODgEA4Vvao1ZtcxzF3Fd+umA13UgiJ8WBxX66XJDMaxXtiAo= =LtsB -----END PGP SIGNATURE----- Merge tag 'sched_ext-for-7.3-rc7-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/sched_ext Pull sched_ext fixes from Tejun Heo: - A task that skipped the BPF scheduler on its way back to its CPU's local queue could stall there until the watchdog fired, as the CPU had already decided to go idle. - Use-after-free of a destroyed user dispatch queue by a deferred re-enqueue request. - A task could be moved off a dispatch queue while the dispatching CPU was still running ops.dequeue() on it. - Bypass mode mishandled a sub-scheduler bypass overlapping an ancestor's, as across a system suspend. Descendants could leave bypass mid-teardown and tasks could be stranded on bypass queues. - The idle CPU selection helper could return a CPU outside the allowed CPUs of a pinned or migration-disabled task. * tag 'sched_ext-for-7.3-rc7-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/sched_ext: sched_ext: Wait for SCX_OPSS_DISPATCHING before moving tasks off DSQs sched_ext: Rename scx_reenq_wait_dispatching() to scx_wait_dispatching() sched_ext: idle: Respect task affinity in restricted CPU selection sched_ext: Fix the bypass dispatch claim for schedulers bypassed by an ancestor sched_ext: Propagate every bypass depth change to the descendants sched_ext: Fix use-after-free of a destroyed user DSQ by a deferred reenqueue sched_ext: Fix stall when a task enqueued with SCX_ENQ_LAST lands back on its CPU's local DSQ |
||
|
Kuba Piecuch
|
2935062be7 |
sched_ext: Wait for SCX_OPSS_DISPATCHING before moving tasks off DSQs
A task can be found on a non-local DSQ while the CPU which dispatched it is still running ops.dequeue() and hasn't cleared SCX_OPSS_DISPATCHING. The DSQ consume and move paths don't wait for the dispatch to finish and can operate on the task concurrently with the dispatcher. Call scx_wait_dispatching() in scx_move_local_task_to_local_dsq(), unlink_dsq_and_switch_rq_lock() and move_task_between_dsqs() before they operate on the task. In scx_move_local_task_to_local_dsq(), wait before scx_resolve_local_dsq() as it may update p->scx.flags. Fixes: |
||
|
Kuba Piecuch
|
18f5c699d1 |
sched_ext: Rename scx_reenq_wait_dispatching() to scx_wait_dispatching()
Waiting for SCX_OPSS_DISPATCHING to clear isn't specific to the reenqueue paths. Any path which takes a task off a DSQ can find it while the dispatch is still in flight, and the next patch calls the helper from the DSQ consume and move paths. Rename it to scx_wait_dispatching() and update its comment accordingly. No functional change. Reviewed-by: Andrea Righi <arighi@nvidia.com> Assisted-by: Claude:claude-opus-5.5 Signed-off-by: Kuba Piecuch <jpiecuch@google.com> Signed-off-by: Tejun Heo <tj@kernel.org> |
||
|
Hui Su
|
6ac4544eff |
sched_ext: idle: Respect task affinity in restricted CPU selection
The restricted select_cpu kfunc path handles single-CPU and
migration-disabled tasks without going through scx_select_cpu_dfl().
It currently uses prev_cpu as the candidate, even though prev_cpu is only a
hint and may no longer be usable after an affinity change. With a
caller-provided @allowed mask, this can also return an idle CPU outside
p->cpus_ptr.
Derive the candidate from task state instead: use the sole CPU in
p->cpus_ptr for single-CPU tasks and task_cpu(p) for migration-disabled
tasks. Validate it against p->cpus_ptr and @allowed before claiming its
idle state.
Since the derived candidate may be on a different NUMA node from
prev_cpu, also preserve SCX_PICK_IDLE_IN_NODE by rejecting cross-node
candidates.
Perform all checks before scx_idle_test_and_clear_cpu(), as a successful
call consumes the CPU's idle bit.
Fixes:
|
||
|
Tejun Heo
|
5bdeca322a |
sched_ext: Fix the bypass dispatch claim for schedulers bypassed by an ancestor
scx_bypass() claims bypass dispatch when the scheduler's depth goes from
zero to one and releases it when the depth returns to zero, but propagates
only the depth to the descendants, not the claim. A descendant that starts
bypassing with its ancestor never claims, and one whose own bypass ends
under the ancestor's keeps its claim, as its depth does not return to zero.
A sub-scheduler enable whose bypass ends under a PM suspend thus leaks its
claim: the parent keeps hosting bypass dispatch and the sub-scheduler's next
bypass trips the WARN_ON_ONCE() on the claim. A sub-scheduler disable that
starts under a PM suspend has no claim, so once the resume lifts the root's
bypass its tasks sit on the parent's bypass DSQs with nobody consuming them.
Claim for every descendant whose depth goes from zero to one in the
propagation and release every scheduler in the subtree whose depth returned
to zero once the tasks have moved out of the bypass DSQs. A parent that is
bypassing itself when a descendant claims consumes only its own bypass DSQs,
empty until it leaves bypass and becomes the host.
Fixes:
|
||
|
Tejun Heo
|
dcf7f650a2 |
sched_ext: Propagate every bypass depth change to the descendants
scx_bypass() propagates a depth change to the descendants only when the
scheduler's own depth crosses zero, so an own bypass that starts under an
ancestor's never reaches them. When the ancestor's bypass ends first, the
descendants leave bypass while the scheduler is still bypassing: a
sub-scheduler disabled under a PM suspend has its children re-queue their
tasks through their own BPF schedulers at resume, mid-teardown. If the own
bypass then ends, every descendant is decremented once more than incremented
and dec_bypass_depth() warns, though no current caller gets there.
Propagate on every call so that a descendant's depth is the sum of its own
bypasses and its ancestors'. The early return follows the walk, as a
descendant's depth is at least the scheduler's and nothing changes bypass
state unless the scheduler's own does.
Fixes:
|
||
|
Tejun Heo
|
069403dff3 |
sched_ext: Fix use-after-free of a destroyed user DSQ by a deferred reenqueue
A deferred reenqueue can read a freed user DSQ. The request stays on its
CPU's deferred list until run_deferred() takes it, and the RCU callback that
frees the DSQ assumes that the grace period has flushed it, so exit_dsq()
checks the per-CPU node without the deferred list lock. A grace period waits
for readers in flight, not for a pending run_deferred(). A run_deferred()
which starts after the grace period began can detach the request right
before the callback looks and then read the freed DSQ's id.
Cancel the pending requests before the grace period instead. The irq work
that frees the DSQ unlinks every CPU's request under the per-CPU deferred
list lock, and schedule_dsq_reenq() ignores a new request under the same
lock once the id is invalid. Only a run_deferred() that started before the
grace period can hold a request, and it runs with IRQs disabled, so the
grace period waits for it and nothing touches the DSQ after the callback.
Fixes:
|
||
|
Tejun Heo
|
7a9df76679 |
sched_ext: Fix stall when a task enqueued with SCX_ENQ_LAST lands back on its CPU's local DSQ
With SCX_OPS_ENQ_LAST, the last runnable task on a CPU is passed to
ops.enqueue() instead of being kept running. Exiting tasks without
SCX_OPS_ENQ_EXITING, migration disabled tasks without
SCX_OPS_ENQ_MIGRATION_DISABLED and tasks on an offline rq skip ops.enqueue()
and go straight to the local DSQ. This happens in put_prev_task_scx() after
the pick has settled on idle, so nothing reschedules the CPU, which idles
with the task queued until an unrelated wakeup lands on it or the watchdog
fires.
scx_mitosis, which sets SCX_OPS_ENQ_LAST without SCX_OPS_ENQ_EXITING, hit
this during mass cgroup teardown: exiting tasks stuck in exit_mmap() for
over 40 seconds.
A task put on the local DSQ runs, as anywhere else. If the SCX_ENQ_LAST
enqueue left the task on this CPU's local DSQ, set need_resched on the task
being switched in, as proxy_resched_idle() does. resched_curr() would flag
the task being put instead, and __schedule() clears that right after the
pick.
Fixes:
|
||
|
Linus Torvalds
|
762122d75e |
sched_ext: Fixes for v7.3-rc6
- Taking a CPU offline could hang, or stall until the watchdog ejected the BPF scheduler, when tasks on the dying CPU were still held by the scheduler or sitting on a user dispatch queue. Re-enqueue them onto the local queue when the runqueue goes offline so that the CPU pushes them off like the other sched classes. - The sequence number guarding against stale dispatches was per runqueue, so a task re-enqueued on another CPU could get the same number and a dispatch meant for its earlier instance was applied to the new one. Use a per-task counter. - A task dispatched to another CPU's local queue got its ops.dequeue() only when picked to run and flagged as a core-sched pick. Call it at insertion like for same-CPU dispatches. -----BEGIN PGP SIGNATURE----- iIQEABYKACwWIQTfIjM1kS57o3GsC/uxYfJx3gVYGQUCasUtpQ4cdGpAa2VybmVs Lm9yZwAKCRCxYfJx3gVYGSu5AQDwrICXDyQlGjzAXzFq+bEKVhtMkHxuk8hm5tAy za4AwwD9FjhJHRy5x89TAmea5Cj++AToL6npdEHv7KDcgqNxsAU= =/aDw -----END PGP SIGNATURE----- Merge tag 'sched_ext-for-7.3-rc6-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/sched_ext Pull sched_ext fixes from Tejun Heo: - Taking a CPU offline could hang, or stall until the watchdog ejected the BPF scheduler, when tasks on the dying CPU were still held by the scheduler or sitting on a user dispatch queue. Re-enqueue them onto the local queue when the runqueue goes offline so that the CPU pushes them off like the other sched classes. - The sequence number guarding against stale dispatches was per runqueue, so a task re-enqueued on another CPU could get the same number and a dispatch meant for its earlier instance was applied to the new one. Use a per-task counter. - A task dispatched to another CPU's local queue got its ops.dequeue() only when picked to run and flagged as a core-sched pick. Call it at insertion like for same-CPU dispatches. * tag 'sched_ext-for-7.3-rc6-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/sched_ext: sched_ext: Generate qseq from a per-task counter selftests/sched_ext: Add a test for ops.dequeue() on remote local DSQ moves sched_ext: Call ops.dequeue() when a task arrives on a remote local DSQ sched_ext: Fix CPU hotplug hang when a dying CPU's tasks sit in the BPF scheduler |
||
|
Kuba Piecuch
|
e6ac89b8b1 |
sched_ext: Generate qseq from a per-task counter
finish_dispatch() uses the qseq embedded in p->scx.ops_state to tell
whether the QUEUED instance of a task it's about to claim is the one
scx_bpf_dsq_insert() saw. qseq is generated from rq->scx.ops_qseq, but
the counters of different rqs are independent, so if a task is dequeued
and re-enqueued on a different rq between scx_bpf_dsq_insert() and
finish_dispatch(), the new QUEUED instance can end up with the same
qseq as the old one:
CPU X CPU Z
----- -----
enqueue p on rq A, qseq = N
ops.dispatch()
scx_bpf_dsq_insert(p)
records qseq N
sched_setaffinity(p)
dequeue p from rq A
enqueue p on rq B, qseq = N
finish_dispatch(p, N)
qseq matches, p is claimed
The claim itself is still atomic so the core stays consistent, but an
insert issued for a previous QUEUED instance gets applied to a new one
which the BPF scheduler has just received through ops.enqueue(). This
breaks the guarantee that dispatches targeting a stale instance are
ignored.
Generate qseq from a per-task counter, p->scx.ops_qseq, instead so that
consecutive QUEUED instances of a task never share a qseq regardless of
which rq they're on. The counter is only updated in
scx_do_enqueue_task() with the task's rq locked, so no additional
synchronization is needed, and it fits in an existing hole in struct
sched_ext_entity on 64bit. Remove the now unused rq->scx.ops_qseq.
Never generate qseq 0. NONE and DISPATCHING don't carry a qseq, so
scx_bpf_dsq_insert() on a task in either state records 0. With a
per-task counter, every task's first QUEUED instance would otherwise get
qseq 0 and could be claimed by such an insert. Wrap the counter where
the QSEQ field wraps so that it can't reach a value that shifts to 0 on
32bit either.
Fixes:
|
||
|
Kuba Piecuch
|
af701b8d32 |
sched_ext: Call ops.dequeue() when a task arrives on a remote local DSQ
When a task in the BPF scheduler's custody is moved to another CPU's local DSQ, ops.dequeue() is only called once the task is picked for execution, from set_next_task_scx() with SCX_DEQ_CORE_SCHED_EXEC, even though no core-sched pick took place. It should instead be called without flags when the task is inserted into the destination DSQ, as it is for same-rq dispatches. move_remote_task_to_local_dsq() sets p->scx.sticky_cpu across the migration so that deactivate_task() on the source rq doesn't end custody. Since commit |
||
|
Tejun Heo
|
d35a535d3e |
sched_ext: Fix CPU hotplug hang when a dying CPU's tasks sit in the BPF scheduler
A CPU going down has to empty its own rq. Its hotplug thread waits in sched_cpu_wait_empty() until nothing else is left, and the only thing that wakes it is balance_push(), which runs from __schedule() on the dying CPU and pushes off the migratable tasks the CPU picks. sched_ext breaks this. A task sitting on a user DSQ or held by the BPF scheduler is still counted on its rq, but an inactive CPU no longer calls ops.dispatch() and can't pull the task back. The dying CPU goes idle with the task still counted, and one of two things happens: - Another CPU consumes the task. The rq empties without a __schedule() on the dying CPU, so the hotplug thread is never woken and cpu_down() hangs holding cpu_hotplug_lock. - The task is affine only to the dying CPU. Nothing can move it, and the offline stalls until the watchdog ejects the BPF scheduler. When the rq goes offline, re-enqueue every task on it that isn't already on the local DSQ. An enqueue on an offline rq lands on the local DSQ, so the dying CPU itself picks the tasks and balance_push() pushes them off, the same as for the other sched classes. From then on no sched_ext path on another CPU can pull a task off the rq. ops.dispatch() currently stops as soon as the CPU goes inactive, and CPU hotplug then waits for an RCU grace period before taking the rq offline. A BPF-held task affine only to the dying CPU and preempted inside an RCU read-side critical section would block that grace period, and the rq would never go offline. Test SCX_RQ_ONLINE directly so that ops.dispatch() keeps running until the rq goes offline. Only the dying CPU's own hotplug thread clears the flag during teardown, and task_can_run_on_remote_rq() still keeps other rqs' tasks off the CPU. Fixes: |
||
|
Linus Torvalds
|
7cdf91542e |
sched_ext: Fixes for v7.3-rc4
- The CPU topology helper for BPF schedulers took no buffer size, so its structure couldn't grow without breaking schedulers built against the older layout. Add a size argument. -----BEGIN PGP SIGNATURE----- iIQEABYKACwWIQTfIjM1kS57o3GsC/uxYfJx3gVYGQUCarjW+w4cdGpAa2VybmVs Lm9yZwAKCRCxYfJx3gVYGX2cAP0VlU9fUs+6zw+CPEvxDfRClCsMvwpbHEh9EtFQ 1csnhQD+KnIhvmascPtuZLhcYScKfA9ZvvYw/Nme7RVmQ/I8HA8= =UsTG -----END PGP SIGNATURE----- Merge tag 'sched_ext-for-7.3-rc4-fixes-2' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/sched_ext Pull sched_ext fix from Tejun Heo: - The CPU topology helper for BPF schedulers took no buffer size, so its structure couldn't grow without breaking schedulers built against the older layout. Add a size argument. * tag 'sched_ext-for-7.3-rc4-fixes-2' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/sched_ext: sched_ext: Add a size argument to scx_bpf_cid_topo() so struct scx_cid_topo can grow |
||
|
Linus Torvalds
|
673dab7eac |
Scheduler fixes:
- Fix LLC mis-scheduling bugs (Tim Chen, Lu Wang)
- Fix cache-grouping related scheduling statistics UAF bugs (Tim Chen)
- Skip kernel threads for cache aware scheduling to rubustify the code
(Chen Yu)
- Refresh LLC capacity across CPU hotplug, to fix capacity
underestimation bug (Davi Chaves Azevedo)
- Account PSI IRQ time to the execution context, not the scheduling
context, to fix proxy scheduling accounting bug (Zhan Xusheng)
Signed-off-by: Ingo Molnar <mingo@kernel.org>
-----BEGIN PGP SIGNATURE-----
iQJFBAABCgAvFiEEBpT5eoXrXCwVQwEKEnMQ0APhK1gFAmq4zcIRHG1pbmdvQGtl
cm5lbC5vcmcACgkQEnMQ0APhK1iN/Q/9HCvfdQ0tTjZHLVhL5f7ZnIjwxVNzmKTK
ysqfXacW3KIwuulXjEDXgAXRi85nKiAxV4sHVp8WaDfpu4NEVzekXZV7dHPf8fPZ
9LCfFJrrYhcAhKHwEIEyGKn7uSSYzFNVjUxHVfc7W7JtqNtrXOYmYKtJuaMdE8ya
fxV/wGHw3HHCmDv/v4BurqOgNyPFEzhN6vl875J0qEdsauxakwvlcIJsAWDstbxi
4kDF+47lLyDZoolrWsVCl8BoxERznK8JelmNVx0FkEpKi2DnLarWjkpAnjfGkIFg
lz2jxdQnTW5v60BTLw4yAUIam8SAzbDNnfKexzT3oG1UgdyqEOMZ4bpynOhoCu5b
3c/2E7Ah7zfMnyQrItnZ7JaXiCrHgPM+aHxjskmMV08Zr54S5hbTxws7uwfl/MA0
5r64GzrStcjswNZKKQycDFCwtkZIcsBmSyJvJqb3RPm4NVYHIBJWDd1YLlH7pM8R
BuNiW0zM9IdXl6EgD2gX41WoeESobaPPPb67b0DQ6WRSZkz7UAF0Xzfz/xu66WlO
AwMMFHF1qhAcBkUBYU9R+hGSQoaTClf7f2stuSr4zVebAZSAzGLRQob5s/ldmfel
2c1se8w3fPUC29eVB7AQMFsh1xFMzkR53r6zZEKJsG5MHu06fdKD2FhdQcfQDAob
K8Y4Yh4LhOo=
=2067
-----END PGP SIGNATURE-----
Merge tag 'sched-urgent-2026-09-27' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
Pull scheduler fixes from Ingo Molnar:
- Fix LLC mis-scheduling bugs (Tim Chen, Lu Wang)
- Fix cache-grouping related scheduling statistics UAF bugs (Tim Chen)
- Skip kernel threads for cache aware scheduling to rubustify the code
(Chen Yu)
- Refresh LLC capacity across CPU hotplug, to fix capacity
underestimation bug (Davi Chaves Azevedo)
- Account PSI IRQ time to the execution context, not the scheduling
context, to fix proxy scheduling accounting bug (Zhan Xusheng)
* tag 'sched-urgent-2026-09-27' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip:
sched/core: Account PSI IRQ time to the execution context, not the scheduling context
sched/cache: Refresh LLC capacity across CPU hotplug, to fix capacity underestimation bug
sched/cache: Skip kernel threads for cache aware scheduling to rubustify the code
sched/cache: Introduce task_struct->sched_cache_grp to fix UAF
sched/cache: Decouple sched_cache_group from mm to fix UAF
sched/cache: Honor migrate_llc_task semantics in active load balance, to fix LLC mis-scheduling bug
sched/cache: Keep nr_pref_llc_running in the runnable domain, to fix LLC mis-scheduling bug
|
||
|
Tejun Heo
|
94480606a6 |
sched_ext: Add a size argument to scx_bpf_cid_topo() so struct scx_cid_topo can grow
scx_bpf_cid_topo() copies struct scx_cid_topo into a buffer the BPF program
sized from its own vmlinux.h while the verifier sizes the write from the
running kernel's BTF. The struct may grow and each growth then breaks every
scheduler built against the older layout, rejected at load or written past
its buffer. This is the usual hole for a struct handed to BPF, closed
elsewhere with a size argument, and it was missed here.
Take the buffer size, copy the smaller of it and the kernel's struct and set
the rest to -1. Accesses to the copy are CO-RE relocated, so the struct can
grow by appending fields, which its comment now states. The kfunc changes in
place: the cid interface is still being finalized and no released scheduler
uses the current form.
Fixes:
|
||
|
Linus Torvalds
|
ee9c669f9b |
sched_ext: Fixes for v7.3-rc4
- A task reenqueued while its dispatch was still completing had its queued state clobbered by the dispatcher, dropping every later dispatch of the task. Wait for the in-flight dispatch to settle first. - A wakeup activation on another CPU marked the destination runqueue as mid-wakeup, stranding a pending local reenqueue. If the scheduler was unloaded first, the stale request pointed into freed memory that the next scheduler dereferenced. - ops.dequeue() ran with the source dispatch queue's lock held, so a scheduler iterating that queue from the callback deadlocked the CPU. - Schedulers with their own CPU ID mapping had no way to learn a task's initial CPU mask and rebuilt it themselves, which went wrong across sub-scheduler enable and re-home. Pass it to ops.enable(). - A bypass dispatch event counter missed the dispatches made by the end-of-dispatch fallback and under-reported. - Selftests for the dequeue locking and initial mask changes. -----BEGIN PGP SIGNATURE----- iIQEABYKACwWIQTfIjM1kS57o3GsC/uxYfJx3gVYGQUCarWSLQ4cdGpAa2VybmVs Lm9yZwAKCRCxYfJx3gVYGY+sAP9y6nJh6vIvFh/X9FlJtWlNo0mncOKhy93E8jii 8CKnPQEAhvX3+Gcdl+imTh4Z915kdsEByBjTTPPOXnQxI8BKUAY= =VhBF -----END PGP SIGNATURE----- Merge tag 'sched_ext-for-7.3-rc4-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/sched_ext Pull sched_ext fixes from Tejun Heo: - A task reenqueued while its dispatch was still completing had its queued state clobbered by the dispatcher, dropping every later dispatch of the task. Wait for the in-flight dispatch to settle first - A wakeup activation on another CPU marked the destination runqueue as mid-wakeup, stranding a pending local reenqueue. If the scheduler was unloaded first, the stale request pointed into freed memory that the next scheduler dereferenced - ops.dequeue() ran with the source dispatch queue's lock held, so a scheduler iterating that queue from the callback deadlocked the CPU - Schedulers with their own CPU ID mapping had no way to learn a task's initial CPU mask and rebuilt it themselves, which went wrong across sub-scheduler enable and re-home. Pass it to ops.enable() - A bypass dispatch event counter missed the dispatches made by the end-of-dispatch fallback and under-reported - Selftests for the dequeue locking and initial mask changes * tag 'sched_ext-for-7.3-rc4-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/sched_ext: sched_ext: Count SCX_EV_SUB_BYPASS_DISPATCH in the dispatch fallback selftests/sched_ext: Check the cmask cid-form ops.enable() receives sched_ext: Pass the initial cmask to cid-form ops.enable() selftests/sched_ext: Test that ops.dequeue() can iterate the consumed DSQ sched_ext: Don't run ops.dequeue() with a DSQ lock held sched_ext: Derive SCX_RQ_IN_WAKEUP from the core enqueue flags sched_ext: Wait for SCX_OPSS_DISPATCHING before reenqueueing a task |
||
|
Zhan Xusheng
|
a0bb6fac53 |
sched/core: Account PSI IRQ time to the execution context, not the scheduling context
psi_account_irqtime() has two callers which share rq->psi_irq_time, and they disagree about the context: __schedule() passes the outgoing rq->curr, sched_tick() passes rq->donor. Under proxy execution the donor is blocked on a mutex while rq->curr burns the CPU. The tick charges PSI_IRQ_FULL to the donor's cgroup and advances the timestamp, so the call from __schedule() then finds delta <= 0 and charges nothing. The delta is not counted twice, it lands on the wrong cgroup. Pass rq->curr, which is what the call read before commit |
||
|
Liang Luo
|
4409a85735 |
sched_ext: Count SCX_EV_SUB_BYPASS_DISPATCH in the dispatch fallback
When a descendant scheduler enters bypass mode, its tasks are parked in
the bypass DSQs of the nearest non-bypassing ancestor, which is then
responsible for running them. On behalf of such a non-bypassing host,
scx_dispatch_sched() consumes those bypass DSQs from two places: the
attempt made every SCX_BYPASS_HOST_NTH dispatches, and the
end-of-dispatch fallback that keeps the CPU from going idle while
bypassed descendants still have tasks queued.
The former increments SCX_EV_SUB_BYPASS_DISPATCH but the latter does
not, even though both perform the same scx_consume_dispatch_q() on the
same bypass DSQ. The descendant bypass dispatches done by the fallback
are therefore missing from the counter exposed via sysfs,
scx_dump_state() and the scx_bpf_events() kfunc, which under-reports the
actual number of such dispatches.
Add the missing __scx_add_event() so the fallback counts them too. When
@sch itself is bypassing, scx_dispatch_sched() takes the earlier
self-bypass branch and returns before reaching these host paths; that
mode is accounted for by SCX_EV_BYPASS_DISPATCH at enqueue time and is
intentionally left unchanged.
Fixes:
|
||
|
Davi Chaves Azevedo
|
3cb0243767 |
sched/cache: Refresh LLC capacity across CPU hotplug, to fix capacity underestimation bug
The scheduler scales LLC capacity by the fraction of cache-sharing CPUs
covered by a domain:
llc_bytes = cache_size * span_weight / shared_weight
During CPU teardown, sched_cpu_deactivate() rebuilds scheduler domains
before cacheinfo_cpu_pre_down() removes the CPU from shared_cpu_map. The
new domains therefore use the old sharing weight. The later call to
sched_update_llc_bytes() looks up the departing CPU's sd_llc, which has
already been detached, and returns without correcting the surviving CPUs.
On a Ryzen 5 7535U with twelve logical CPUs sharing a 16 MiB LLC,
offlining one SMT sibling left the remaining CPUs with:
llc_bytes = floor(16777216 * 11 / 12) = 15379114 bytes
The correct capacity is still 16777216 bytes. On systems with active
cache-aware scheduling, an underestimated capacity can cause
exceed_llc_capacity() to reject aggregation for a process whose footprint
would fit. Unchanged cpuset partitions sharing the physical cache can
also retain stale capacity when a CPU comes online in another partition.
Pass the cache-sharing mask already retained by cacheinfo to the
scheduler update. Refresh every surviving CPU using its own LLC domain
so that each partition receives the correct share. This also preserves
the correction needed as cache-sharing maps grow during boot.
Keep the existing CPU-hotplug and scheduler-domain synchronization. The
update remains on the hotplug path; no steady-state scheduling operation
or persistent allocation is added.
Fixes:
|
||
|
Chen Yu
|
65efcccddc |
sched/cache: Skip kernel threads for cache aware scheduling to rubustify the code
Kernel thread should not be covered by cache aware scheduling as
it borrows the statistics from the user space thread. Filter the
kernel thread in account_mm_sched().
In theory a kernel thread does not have any valid
cache group, so !grp should gate the kernel thread.
Add the PF_KTHREAD check explicitly here for safety
reasons, to guard against future modifications and
to pair with task_tick_cache().
Fixes:
|
||
|
Tim Chen
|
b636fef85b |
sched/cache: Introduce task_struct->sched_cache_grp to fix UAF
Add a sched_cache_grp pointer to task_struct so that scheduler code
can access the cache group directly via the task, without going
through mm->sched_cache_grp. This decouples the scheduler's hot-path
accesses from the mm_struct.
Each task holds its own refcount on the sched_cache_group, separate
from the reference held by its mm_struct. The reference is acquired
in copy_mm() (fork) and exec_mmap() (exec), and released in exit_mm().
This fixes the use-after-free when account_mm_sched() reaches the group
through a task whose mm is being switched, as reported by Hyunwoo:
https://lore.kernel.org/lkml/apPb-Dr4nPYuHQOK@v4bel/
Convert all scheduler code in fair.c and exit.c to use
p->sched_cache_grp instead of p->mm->sched_cache_grp.
Keep the fork/exec/exit reference management out of the generic mm
paths: add sched_cache_fork(), sched_cache_fork_cleanup(),
sched_cache_exec_mmap() and sched_cache_exit_mm() in
kernel/sched/cache_sched.c (with empty stubs for !CONFIG_SCHED_CACHE),
so fs/exec.c, kernel/fork.c and kernel/exit.c each call one helper
instead of open-coding the refcounting under #ifdef. Also add
sched_cache_group_get() and task_cache_group_get().
Fixes:
|
||
|
Tim Chen
|
28f9c0e0a0 |
sched/cache: Decouple sched_cache_group from mm to fix UAF
Currently the sched cache grouping is by mm and the scheduling statistics
sched_cache_stat lives in the mm structure. This ties the life cycle
of scheduling stats with mm.
In account_mm_sched(), the scheduling stats are accessed by
task->mm->sc_stat. However, a task may be switching mm on one CPU when
another CPU is running account_mm_sched(), and possibly accessing the
old mm that was freed. This problem was found when running tests with
KASAN by Hyunwoo:
https://lore.kernel.org/lkml/apPb-Dr4nPYuHQOK@v4bel/
Instead of serializing the mm access by introducing extra acquisition of
rq lock in the mm free path, extract sched_cache_stat from mm_struct,
rename it as sched_cache_group and manage its life cycle apart from
mm_struct with its own ref counting. This allows us in the next patch
access sched_cache_group directly from task, and add a refcount
on sched_cache_group when a task links to it. This prevents the use
after free issue when accessing stale and released old mm and its
sched cache stat a task switches to a new mm while account_mm_sched()
is done elsewhere.
The other benefit of this restructure is in the future, the grouping of
tasks to a LLC would have the flexibility to be associated with a user
defined grouping, or cgroup, cookie group, numa_group or others instead
of just with a single mm address space.
Rename sched_cache_stat to sched_cache_group and turn it into a refcounted
object allocated from mm_struct. The mm_struct now holds a pointer
(sched_cache_grp) to this object instead of embedding it.
Fixes:
|
||
|
Lu Wang
|
d6013e2465 |
sched/cache: Honor migrate_llc_task semantics in active load balance, to fix LLC mis-scheduling bug
Cache aware scheduling introduced the migrate_llc_task migration type to direct
tasks toward their preferred LLC, but its semantics can be lost when passive
load balance falls back to active load balance (ALB). This may allow ALB to
select a candidate whose preferred LLC does not match the destination, moving
it away from its preferred LLC.
Example scenario:
src_rq has two runnable tasks, p1 and p2. p1 prefers dst_rq (dst_llc), while p2
prefers src_rq (src_llc). In this case, migrate_llc_task is set because src_rq
has at least one task, p1, that wants to migrate to dst_rq. In ALB,
can_migrate_task() finds p2 and returns true for it, thus moving p2 out of its
preferred LLC.
Solution:
The CPU stopper in ALB constructs a fresh lb_env that does not inherit
migration_type from the passive load-balance pass. Two approaches are
possible:
(a) Add a new member to struct rq so ALB can inherit migrate_llc_task
from the passive LB that triggered it.
(b) Define a new flag LBF_ACTIVE_LB_LLC and select the stopper callback
at kick time to preserve the migration semantics across the
asynchronous boundary.
We choose (b) because it avoids passing migration_type through the
stopper, which would affect the meaning of migration_type for
delayed-dequeue tasks.
Fixes:
|
||
|
Tim Chen
|
0d6526f82c |
sched/cache: Keep nr_pref_llc_running in the runnable domain, to fix LLC mis-scheduling bug
alb_break_llc() decides whether to break LLC preference during active
load balance. It does so by testing that every runnable fair task on the
source rq prefers its LLC:
env->src_rq->nr_pref_llc_running == env->src_rq->cfs.h_nr_runnable
But the two counters cover different sets. nr_pref_llc_running is updated
in account_llc_enqueue()/account_llc_dequeue(), next to cfs_rq->nr_queued,
so it follows queued tasks. h_nr_runnable is updated in set_delayed()/
clear_delayed() and drops delay-dequeued tasks.
So under DELAY_DEQUEUE, a preferring task that goes to sleep stays counted
in nr_pref_llc_running while h_nr_runnable falls. The equality then breaks,
alb_break_llc() returns false, and active balance is free to pull a task
off its preferred LLC. Active balance only moves runnable tasks, and this
is the only LLC check it consults: once the stopper runs, LBF_ACTIVE_LB
skips the per-task test in can_migrate_task(). The runnable set is the one
we want.
Fix it on the counter side. A task should be counted in
nr_pref_llc_running exactly while it is both queued on its preferred LLC
(pref_llc_queued) and runnable (!sched_delayed). Define that membership
once in task_pref_llc_runnable(), and adjust the counter only through
pref_llc_running_inc()/pref_llc_running_dec() from the four sites that
change either input: account_llc_enqueue(), account_llc_dequeue(),
set_delayed() and clear_delayed(). Gating every update on the same
predicate keeps the delay, wake and dequeue paths from double-counting
or underflowing; see the comments at those sites for the ordering.
nr_llc_running and sd->llc_counts are not touched and stay on queued
semantics.
Fixes:
|
||
|
Linus Torvalds
|
fecbe78ac0 |
Scheduler fix:
- Avoid false positive migration warning for proxy donors
(Andrea Righi)
Signed-off-by: Ingo Molnar <mingo@kernel.org>
-----BEGIN PGP SIGNATURE-----
iQJFBAABCgAvFiEEBpT5eoXrXCwVQwEKEnMQ0APhK1gFAmqvqssRHG1pbmdvQGtl
cm5lbC5vcmcACgkQEnMQ0APhK1gPzBAAhE9hcpkBV6vNW9xzBDKdGPxRjch2vcfu
lQbx7da1yC2rHU7RDlcdfEgkhAqTRwEAXKz26zth3sDHLk43tusuNtqG3swELhNW
YAGbHjdn87snQlmP8AOK4EaT3uE5NjWqRSIJWMK+AhWSwqO/zzK91T6yZyQY0fM4
3Edp8CFo3IeyOzCR96vsob2x1fFQhuR//5fWul3uuB0EZ8VA5FhH3ene6VCm7P/1
dauxX0rMTb2730qNXA8cHROZq+bwhqTZOUaoOZ33WxnRPkvY9mV/hZsN8JnJuzBE
ogzyyorcl8dFH8qOapos9Cp3tQj9GkTX7mXWDbuUflt/8uOXtQMf75kFGBVI5NUw
2xNfgubTYGo6Qc1C+wyOhGYJ6T5Al+083pV/vPc4y7Z1i7RgZ94QypMuZuaR/RqS
z+RQcdNQlXiRIAIILGJqq1xdbaCJvbVx3tiFZkhPse6ioOF6UNGbQiNExAq3v5BU
ocvhBuf9p/uvRmfs+ZtQNqAAjZUL7tQPvdFAsjxKjuI2Z5YGPROy1L9NdYKfzryM
yWOEkV2mdn97CwzDS+auC0HmPkGqf8we2VI5Ub4R35UPqDcV6vc5BAnwzAbuxAmc
K7mQOMDEaRkbVQ0MoSMdidIXLL0AcN+BROBUtHv8kqJur6nADE3K3HZNdhr2ViLN
eisNI6m8pLk=
=BEte
-----END PGP SIGNATURE-----
Merge tag 'sched-urgent-2026-09-20' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
Pull scheduler fix from Ingo Molnar:
- Avoid false positive migration warning for proxy donors
(Andrea Righi)
* tag 'sched-urgent-2026-09-20' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip:
sched/core: Avoid false migration warning for proxy donors
|
||
|
Tejun Heo
|
3bd46666cf |
sched_ext: Pass the initial cmask to cid-form ops.enable()
The cid-form API has an obvious hole. A task's cid mask is only visible through ops.set_cmask(), which fires on affinity changes and class switches but not when a task enters a scheduler through fork, sub-sched enable or re-home, and there is no p->cpus_ptr equivalent to fall back on. Schedulers work around it by seeding the mask in ops.init_task() from p->cpus_ptr cid by cid, which is subtly wrong: on sub-sched enable and re-home, an affinity change between init_task() and enable() is delivered to the sched the task is still on, and nothing corrects the new sched's copy afterwards. Fix it by adding struct scx_enable_args to cid-form ops.enable() carrying the task's cmask, built in the per-cpu scratch under the rq lock as the task enters the scheduler, and calling set_cmask() with the same mask right after enable(), ahead of set_weight(). A scheduler can then track affinity in set_cmask() alone, and scx_qmap drops its init_task() seed. set_cmask() no longer fires for a cid-form task before it is enabled, and the class-switch republish in switching_to_scx() is limited to the cpu form. This changes the cid-form ops.enable() signature, which is fine as the cid-form API is still considered unreleased. An args struct rather than a bare cmask argument leaves room for more initial state without another signature change, and the cmask travels as a plain arena address because BTF can't type arena struct members yet. v2: The cmask travels as a u64 arena address, cmask_arena_addr, instead of a kernel-typed pointer, with the typing limitation and the planned typed alias documented (Sashiko review). v3: The initial set_cmask() is delivered before set_weight() so that the mask is in place when weight-dependent state is derived (Andrea Righi). Selftest added. Signed-off-by: Tejun Heo <tj@kernel.org> Reviewed-by: Andrea Righi <arighi@nvidia.com> |
||
|
Andrea Righi
|
fe3c73d7bc |
sched/core: Avoid false migration warning for proxy donors
Proxy execution can move a blocked donor's scheduling context to the
lock owner's CPU even when the donor is migration-disabled. The donor
does not execute there, and its original execution CPU remains recorded
in wake_cpu.
set_task_cpu() warns unconditionally for migration-disabled tasks, so a
subsequent proxy migration or the wakeup path returning the donor home
triggers a false positive: moving a blocked scheduling context does not
violate the migration-disabled execution context.
For example, creating a mutex owner on CPU1 and a migration-disabled
waiter on CPU0 can trigger the following warning:
proxy_migrate_repro: donor blocking on CPU0 with migration disabled
proxy_migrate_repro: donor moved from CPU0 to CPU1
WARNING: kernel/sched/core.c:3389 at set_task_cpu+0x1d3/0x280
...
Call Trace:
try_to_wake_up+0x43f/0x780
__mutex_unlock_slowpath+0x330/0x540
owner_fn+0x9f/0xc0 [proxy_migrate_repro]
...
proxy_migrate_repro: donor woke on CPU0, task_cpu=0
proxy_migrate_repro: completed
Exclude blocked proxy donors from the warning. The proxy wakeup path
restores an executable placement before clearing the blocked state.
Fixes:
|
||
|
fangqiurong
|
cb86607ada |
sched_ext: Don't run ops.dequeue() with a DSQ lock held
ops.dequeue() is invoked with the source user DSQ's lock still held on
the consume and move paths (scx_consume_dispatch_q(),
move_task_between_dsqs()). A BPF scheduler which locks the source user
DSQ from ops.dequeue() - e.g. by iterating it with bpf_iter_scx_dsq -
self-deadlocks.
ops.dequeue() can only call the "any" kfuncs and none of them can lock a
builtin DSQ, so the global and bypass paths can't deadlock; however,
all DSQ locks share one lockdep class, so iterating any user DSQ from
ops.dequeue() on those paths trips the recursion check.
Move the invocation after the DSQ unlock on all three paths.
SCX_TASK_IN_CUSTODY is cleared under the lock serializing the transfer
so that the callback is invoked exactly once.
Fixes:
|
||
|
Tejun Heo
|
df5cdc2c83 |
sched_ext: Derive SCX_RQ_IN_WAKEUP from the core enqueue flags
schedule_deferred_locked() skips scheduling a deferred action while
SCX_RQ_IN_WAKEUP is set and relies on the task_woken_scx() call that follows
a wakeup enqueue to run it. enqueue_task_scx() sets the flag from the merged
enqueue flags, which include the flags stashed for a remote activation.
move_remote_task_to_local_dsq() thus sets SCX_RQ_IN_WAKEUP on the
destination rq when the moved task was woken up, although no
task_woken_scx() follows that activation.
An IMMED insert into a busy destination requests a local reenqueue during
that enqueue. The request gets linked but not scheduled and stays pending
until an unrelated wakeup or preemption on that CPU runs the deferred
actions. The IMMED task sits behind the running task in the meantime. If
nothing runs them before the scheduler is disabled, the request outlives the
scheduler and points into its freed per-cpu area, which the next scheduler
dereferences from run_deferred().
Test the core enqueue flags for the wakeup bit. Only the core's wakeup path
is followed by task_woken_scx().
Fixes:
|
||
|
Tejun Heo
|
7de9a6fb44 |
sched_ext: Wait for SCX_OPSS_DISPATCHING before reenqueueing a task
|
||
|
Linus Torvalds
|
9b87fdc9af |
sched_ext: Fixes for v7.3-rc3
- An error raised by a BPF program before the scheduler finished enabling was consumed by the disable path's pre-enable shortcut, leaving a running scheduler that couldn't be disabled and was later freed while in use. - Two compat kfuncs dereferenced a NULL scheduler when handed an exited or idle task, oopsing the kernel. - Keep-running decisions in the dispatch path used the root scheduler's flags for tasks belonging to a sub-scheduler, causing warnings and stalls. - Schedulers with their own CPU ID mapping had no way to learn which IDs are online. Add a kernel-maintained online mask to plug the hole. - Cgroup idle state: the initial cpu.idle state wasn't passed on cgroup init and same-value rewrites delivered spurious callbacks. - Example scheduler fixes for a reenqueue loop on attach, placements on CPUs without effective grants, stalled partition work and stale idle tracking. -----BEGIN PGP SIGNATURE----- iIQEABYKACwWIQTfIjM1kS57o3GsC/uxYfJx3gVYGQUCaql+vQ4cdGpAa2VybmVs Lm9yZwAKCRCxYfJx3gVYGUTxAP9mhhePmghF/aq6SzeaVGeUXtclXRAGE1EAqea6 kzh50AEAiWXtFQu9dAF3EaXTWJkEJZkJUs0AO0wEXgghC0py9gY= =gcDW -----END PGP SIGNATURE----- Merge tag 'sched_ext-for-7.3-rc3-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/sched_ext Pull sched_ext fixes from Tejun Heo: - An error raised by a BPF program before the scheduler finished enabling was consumed by the disable path's pre-enable shortcut, leaving a running scheduler that couldn't be disabled and was later freed while in use. - Two compat kfuncs dereferenced a NULL scheduler when handed an exited or idle task, oopsing the kernel. - Keep-running decisions in the dispatch path used the root scheduler's flags for tasks belonging to a sub-scheduler, causing warnings and stalls. - Schedulers with their own CPU ID mapping had no way to learn which IDs are online. Add a kernel-maintained online mask to plug the hole. - Cgroup idle state: the initial cpu.idle state wasn't passed on cgroup init and same-value rewrites delivered spurious callbacks. - Example scheduler fixes for a reenqueue loop on attach, placements on CPUs without effective grants, stalled partition work and stale idle tracking. * tag 'sched_ext-for-7.3-rc3-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/sched_ext: sched_ext: Maintain an online cid mask in the scheduler arena sched_ext: scx_qmap: Restore unused idle claims from ops.dispatch() sched_ext: Close the pre-enable ops error claim window sched_ext: scx_qmap: Fix pending partition work handoff sched_ext: scx_qmap: Place only on cids whose caps are in effect sched_ext: scx_qmap: Do not add IMMED to rescue inserts sched_ext: Use @prev's scheduler for the keep decisions in dispatch_one() sched_ext: Rename sch to root_sch in dispatch_one() sched_ext: Fix NULL sched deref in kfunc sub-sched error paths sched_ext: Don't deliver duplicate ops.cgroup_set_idle() for same value sched_ext: Pass the initial cpu.idle state in scx_cgroup_init_args |
||
|
Tejun Heo
|
a9e3760b08 |
sched_ext: Maintain an online cid mask in the scheduler arena
Schedulers on the default cid mapping treat [0, nr_online_cids) as the online set and restart on hotplug. Schedulers that install their own mapping with scx_bpf_cid_override() have no way to learn which cids are online: the count no longer identifies members and the CPU-form cpumask is unusable from cid programs. This is an obvious hole in the cid API. Add scx_bpf_online_cmask(), a kernel-maintained cmask in the scheduler's arena, allocated alongside the per-CPU scratch masks and populated after the cid mapping is finalized and before ops.init(), for child schedulers too. The pointer stays valid through ops.exit() with no reference to take. It is the arena offset as a void pointer, the same form struct_ops arena arguments arrive in. The verifier types the void return as a scalar for the program's arena cast. The mask follows the SCX hotplug notifications: seeded from cpu_active_mask and updated before ops.cid_online/offline() runs, so it lags cpu_online_mask only inside a hotplug transition. Updates walk the scheduler list under the lock that also serializes unlinking. Reads are live, not atomic snapshots. Root initialization excludes hotplug. v2: Reworded the getter kerneldoc (Andrea Righi). Signed-off-by: Tejun Heo <tj@kernel.org> Reviewed-by: Andrea Righi <arighi@nvidia.com> |
||
|
Tejun Heo
|
9a0b159ff1 |
sched_ext: scx_qmap: Restore unused idle claims from ops.dispatch()
scx_qmap tracks idle cids itself. pick_direct_dispatch_cid() claims a cid by clearing its bit and the task is inserted into that cid's local DSQ, which kicks the CPU. When the task does not arrive, for example because the insert fell back to the global DSQ after an affinity change, the CPU wakes, finds nothing and picks idle again. That is not an idle transition, so ops.update_idle() is not called and the cid stays marked busy until an unrelated task runs on it. Restore the claim from ops.dispatch(). The kick guarantees a dispatch on the kicked CPU, and when it finds nothing to run with a NULL @prev, the CPU is going back to idle. Document the pattern in ops.update_idle(), which reports only actual transitions. Signed-off-by: Tejun Heo <tj@kernel.org> Reviewed-by: Andrea Righi <arighi@nvidia.com> Cc: Andrea Righi <arighi@nvidia.com> |
||
|
fangqiurong
|
c7a1c6e800 |
sched_ext: Close the pre-enable ops error claim window
scx_alloc_and_add_sched() publishes ops->priv before
scx_root_enable_workfn() switches the state to SCX_ENABLING. An error
claimed via scx_bpf_error_bstr() from an associated BPF program in that
window is consumed by scx_disable_workfn(), which takes the pre-enable
shortcut in scx_root_disable(). The shortcut returns without any teardown
and restores SCX_DISABLED with an unconditional scx_set_enable_state() xchg
racing the enable workfn's own transition. The enable then completes with
the claim consumed: the scheduler stays up but can never be disabled again,
and bpf_scx_unreg() frees it while still in use, resulting in a
use-after-free. Both WARN_ON_ONCE()s fire back to back:
WARNING: kernel/sched/ext/ext.c:7522 at
scx_root_enable_workfn+0xeec/0x1be0, CPU#3: scx_enable_help/276
WARNING: kernel/sched/ext/ext.c:6398 at scx_root_disable+0xb50/0xdb8,
CPU#0: sched_ext_helpe/664
scx_root_enable_workfn() switches to SCX_ENABLING before the scheduler
allocation, so ops->priv is never visible while SCX_DISABLED. The allocation
failure path restores SCX_DISABLED.
Fixes:
|
||
|
Hui Su
|
f5741d2b34 |
sched/core: Call wq_worker_tick() for the execution context
wq_worker_tick() accounts CPU time and detects CPU-intensive work for
the kworker that is actually running. With proxy execution, rq->donor
is the scheduling context while rq->curr is the execution context.
Calling the hook with rq->donor can skip workqueue accounting when a
kworker is executing on behalf of a donor task. It can also account a
blocked kworker when the donor is a worker but rq->curr is the task
actually executing. The former can delay WORKER_CPU_INTENSIVE handling
and pool concurrency management, which can delay pending kernel work
and userspace operations depending on it.
Use rq->curr for the workqueue tick hook while retaining rq->donor for
scheduler accounting.
Fixes:
|
||
|
Hui Su
|
c23810313b |
sched: Account cgroup CPU time to the execution context
Proxy execution separates the scheduling context from the execution context. Commit |
||
|
Vincent Guittot
|
51b0e68cfa |
sched/eevdf: Fix rb augmented with multi fields
The eevdf rb tree maintains 3 augmented fields but only one is currently
copied when balancing the tree.
Add a more generic define that can be used when there are several augmented
fields. In this case, we provide a function that takes care of copying all
fields.
Fixes:
|
||
|
Vincent Guittot
|
9a8bc9bb4c |
sched/eevdf: Fix augmented max_slice
Similarly to se->min_slice, init se->max_slice with se->slice before
enqueueing the entity so the augmented callback computes it correctly
at parent level.
Fixes:
|
||
|
Linus Torvalds
|
88405f0ad1 |
Miscellaneous scheduler fixes:
- Fix a timestamping bug in pick_task_fair() and
yield_task_fair() (Zhan Xusheng)
- Skip migrate-disabled tasks when picking a push
candidate in the RT and DL schedulers (Seiji Nishikawa)
- Skip rq->avg_idle update without a valid idle_stamp
(Shubhang Kaushik)
- Fix throttling bug in throttle_cfs_rq(), caused
by the recent single-runqueue conversion (Wanwu Li)
- Fix bandwidth calculation bug in distribute_cfs_runtime(),
caused by the single-runqueue conversion (Wanwu Li)
- Don't make x86 ITMT enablement depend on debugfs (Mario Limonciello)
- Avoid creating misfits during cache-aware load-balancing
on hybrid systems (Tim Chen)
Signed-off-by: Ingo Molnar <mingo@kernel.org>
-----BEGIN PGP SIGNATURE-----
iQJFBAABCgAvFiEEBpT5eoXrXCwVQwEKEnMQ0APhK1gFAmqdTLIRHG1pbmdvQGtl
cm5lbC5vcmcACgkQEnMQ0APhK1gdURAAnjK4q2xg115VyKOvg4u4epPre1XNK4a/
RmimS/2+1fm7/5zqOXOQG3qU8z2Aj+nBIZa1d1yadV7ARHSymDINobCX+2HQ1DjU
Iysk2BqbbLnQQNKh81OBWhPMyJy4yOnig/670KR5JdFAdTKFYW/Ys+pQmO9M6rEx
hc8B4SUhOGeRnonfPMTbFW7H9eopWzst8kIpF0USl58az0mla5tkooWYJsvSuUHn
g4s9IALlRe3DxgmMT3kLKoAx2+ySnrzZNRs+M8Z/sMSDKXCMFpvX36wOqBJMHwXe
ZoAow/fJh4ysoYTR7luhDIBDQAkszf4GE9a644I/Bd0sIIYKs4+1RW1T3DAoWmKv
Sx0C29MM4Ng4R3lPx2avb+c0T0VO080/3fqRFrGrck8BIpMa9HcinOxf6etI8+vt
STnz5uQZZj3V+A5vzlKCAFGUUqK/s3EJnrT5z4VPxXmY2ZSyLLub4C5SQ8RNnqRL
u+Q9ra9DYBJh/HJTgjjcAP/HxmKK5LyQXd2DzuA1wy/wwC+s7TtPrxN5ep5EdZmv
K65zGGo28BgcCKD8jc1DfVXf8cf80fMiAr4fruGdo5TVwPyBahwZBi4sbqhahtqL
D4QwCsbOfhS+evVpx29n+V7eQzZzvSdN1tFCtZ3Tg6VvT9aJ2jLSx79aCsL/cYbr
hViB9N0yhGk=
=8l0J
-----END PGP SIGNATURE-----
Merge tag 'sched-urgent-2026-09-06' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
Pull scheduler fixes from Ingo Molnar:
- Fix a timestamping bug in pick_task_fair() and yield_task_fair()
(Zhan Xusheng)
- Skip migrate-disabled tasks when picking a push candidate in the
RT and DL schedulers (Seiji Nishikawa)
- Skip rq->avg_idle update without a valid idle_stamp (Shubhang
Kaushik)
- Fix throttling bug in throttle_cfs_rq(), caused by the recent
single-runqueue conversion (Wanwu Li)
- Fix bandwidth calculation bug in distribute_cfs_runtime(),
caused by the single-runqueue conversion (Wanwu Li)
- Don't make x86 ITMT enablement depend on debugfs (Mario Limonciello)
- Avoid creating misfits during cache-aware load-balancing on hybrid
systems (Tim Chen)
* tag 'sched-urgent-2026-09-06' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip:
sched/fair: Avoid creating misfits during cache-aware balancing
x86/itmt: Don't make ITMT enablement depend on debugfs
sched/fair: Use cfs_rq->h_curr in distribute_cfs_runtime()
sched/fair: Use cfs_rq->h_curr in throttle_cfs_rq()
sched/core: Skip rq->avg_idle update without a valid idle_stamp
sched/rt,dl: Skip migrate-disabled tasks when picking a push candidate
sched/fair: Use update_curr_eevdf() for the remaining root cfs_rq callers
|
||
|
Linus Torvalds
|
c8990f3179 |
Misc locking fixes:
- Fix a softirq processing delay bug in local_interrupt_disable(),
which should mostly only affect the Rust runtime (Boqun Feng)
- Remove the hardirq_disable_count() function which caused the
previous bug and is now unused & unnecessary (Boqun Feng)
- lockdep: Invalidate stale class_cache entries for zapped classes
(Eric Dumazet)
- Fix rt_mutex specific futex scheduling helpers
(Sebastian Andrzej Siewior)
- Fix rcuwait use-after-free race during futex requeue PI (Yao Kai)
Signed-off-by: Ingo Molnar <mingo@kernel.org>
-----BEGIN PGP SIGNATURE-----
iQJFBAABCgAvFiEEBpT5eoXrXCwVQwEKEnMQ0APhK1gFAmqdSKMRHG1pbmdvQGtl
cm5lbC5vcmcACgkQEnMQ0APhK1gh6hAAtIhAm10pkx7JG4Kl+SQArmoAxNMfW9X1
grx50dn1/LFuIF0upooJwJmLMBpPMmxcpYPWbg5nTUsx+wDqcJTT0T7Aw2iZN2f+
OdODMOGNuQU4IUxDC+qX+fkgHvRFzng0zhSXYL2kGpQJHktrFMAHn90+5aO9SEgA
XcSjmDacTxfJADNk43snrlHy6t+yOXBEmIrhicnbh11GFNIXLbvdGW/0SObU0Q5o
WdwGHTZRFAM3mpX7xMSFOKypM+Gt4pXdQ/uC1PQg0B/AT35p4zsYIdcht/cBMHp0
93eTeJd9ede410t3hL1Hnsuo1zf5qHtCgBukS+UHJWoWpnSOMgT2iaBx2X3jIsev
ptPIqV5Alq8O+tnNOSDOwD2HH2drAdUBVH+gZ/yKMDbk9jfw9eZKqRw628U2P7rG
mD5CLzpmfj0mMAuqcOhhdqSQmPhjp2Y8Ju/8UAMSmcocvqjQBxvouNS3Tby9U5v+
R4CsuUwtnYpMyy5xyRrnwVsaE3le5fvneXCmCokk8FtQ8as/4ATd+cG+RROSa3vW
/AAjrB1HBQqyseE8lki4LFFmlr01Otc8u6pASy4/pxeuKMA9hSuLC3RuEfVetW6W
BcK4BCIJBy6chcsCBxv0F7FgI/y7/bpVyBmoTkW3Sb0I23xswXzTzbucn/dPJNdl
oqFj3KQygxw=
=vEuZ
-----END PGP SIGNATURE-----
Merge tag 'locking-urgent-2026-09-06' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
Pull locking fixes from Ingo Molnar:
- Fix a softirq processing delay bug in local_interrupt_disable(),
which should mostly only affect the Rust runtime (Boqun Feng)
- Remove the hardirq_disable_count() function which caused the
previous bug and is now unused & unnecessary (Boqun Feng)
- lockdep: Invalidate stale class_cache entries for zapped classes
(Eric Dumazet)
- Fix rt_mutex specific futex scheduling helpers
(Sebastian Andrzej Siewior)
- Fix rcuwait use-after-free race during futex requeue PI (Yao Kai)
* tag 'locking-urgent-2026-09-06' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip:
futex: Prevent rcuwait use-after-free during requeue PI
futex: Provide rt_mutex_.*_schedule() equivalents for futex scheduling
locking/lockdep: Invalidate stale class_cache entries for zapped classes
preempt: Remove hardirq_disable_count()
interrupt: Disable interrupt before modifying hardirq_disable counter
|
||
|
Tejun Heo
|
90f19b2816 |
sched_ext: Use @prev's scheduler for the keep decisions in dispatch_one()
dispatch_one() tests ops flags and bypass state against the root scheduler
in both places where it decides to keep running @prev: the early keep of a
@prev with slice left tests the root's bypass state, and the keep-last at
the end tests the root's SCX_OPS_ENQ_LAST and bypass state. Both are
properties of the scheduler @prev belongs to, and put_prev_task_scx(), which
acts on the outcome, reads them from that scheduler. When @prev belongs to a
sub-scheduler the two sides disagree.
The keep-last case is visible. The root set SCX_OPS_ENQ_LAST, so a lone
@prev of a sub-scheduler is not kept and is enqueued with SCX_ENQ_LAST to a
sub-scheduler that never opted in. This trips the WARN_ON_ONCE in
put_prev_task_scx() for the missing flag, and the sub-scheduler queues the
task like any other and triggers no follow-up scheduling event, which can
lead to stalls.
Test SCX_OPS_ENQ_LAST and bypass state on @prev's sched in both places and
charge SCX_EV_DISPATCH_KEEP_LAST to it. Read the sched at each decision, as
the dispatch in between can drop the rq lock.
Fixes:
|
||
|
Tejun Heo
|
3265ef0b67 |
sched_ext: Rename sch to root_sch in dispatch_one()
dispatch_one() uses the root scheduler for everything it does, including the two decisions to keep running @prev, which are wrong when @prev belongs to a sub-scheduler. The function has to deal with @prev's scheduler too. Rename the root's local from sch to root_sch for clarity and to make room for it. No functional change. Signed-off-by: Tejun Heo <tj@kernel.org> Reviewed-by: Andrea Righi <arighi@nvidia.com> |
||
|
Kees Cook
|
3a2c4d55e3 |
treewide: refresh kmalloc_obj() conversions
This is another run of the Coccinelle script for converting kmalloc() family of allocations to kmalloc_obj() via the existing rules in scripts/coccinelle/api/kmalloc_objs.cocci This catches both the set of kmalloc() uses added since the first kmalloc_obj() conversions in v7.0 and adds a large group missed in the first pass due to Coccinelle not interacting well with the cleanup.h scoped_...() family of macros[1]. I worked around this with spatch's "--macro-file" argument to a file with all the scoped_...() macros mapped to Coccinelle's YACFE_ITERATOR[2] as that was the closest viable control flow indicator I could find. Build tested allmodconfig on x86, arm64, arm, loongarch, mips, powerpc, riscv, and s390 with no new warnings. Link: https://lore.kernel.org/lkml/202609021314.8A9C0B8@keescook/ [1] Link: https://github.com/coccinelle/coccinelle/blob/master/standard.h [2] Signed-off-by: Kees Cook <kees+treewide@kernel.org> |
||
|
Sebastian Andrzej Siewior
|
912edebe85 |
futex: Provide rt_mutex_.*_schedule() equivalents for futex scheduling
There is rt_mutex_{pre|post}_schedule() around
rt_mutex_wait_proxy_lock() to ensure that sched_submit_work()/
sched_update_worker() is invoked before we schedule out and block on
rt_mutex while waiting for it become available.
The reason is that blocking on rt_mutex assigns a pi_waiter for the PI
chain and sched_submit_work() will also assign a pi_waiter if it blocks
on lock but a this point we already have a waiter assigned.
We can't skip sched_submit_work() entirely because I/O relies on the
fact that I/O queue is flushed while it blocks on a sleeping lock.
Therefore sched_submit_work() is moved before we block on the lock.
Sleeping lock in this context means mutex or rw_semaphore not spinlock_t
on PREEMPT_RT. Because the mutex abstraction on PREEMPT_RT uses the same
abstraction as the futex proxy lock, the futex code ended up using
rt_mutex_{pre|post}_schedule(), too.
Using it is/ was just to keep the task_struct::sched_rt_mutex assertion
happy. Futex proxy lock is used only in the syscall context of a task.
At this point it never got any I/O that needs to be flushed and it can't
be a workqueue that needs to notify that it will be scheduled out.
Therefore sched_submit_work() does nothing here.
By mistake futex_wait_requeue_pi() -> rt_mutex_wait_proxy_lock() did not
get the rt_mutex_{pre|post}_schedule() annotation. This was not noticed
because in this callchain the lock is (usually) not contended and so
rt_mutex_slowlock_block() does not schedule, triggering the assert.
Adding rt_mutex_pre_schedule() here looks wrong (as noted by PeterZ)
because at this point there is a pi_waiter recorded and invoking
sched_submit_work() with a possible lock contention would be wrong.
Add rt_mutex_futex_{pre|post}_schedule() which toggles the
sched_rt_mutex assert and does not involve sched_submit_work(). Add
asserts here to ensure that sched_submit_work() would do nothing. Use it
only in futex proxy lock case which is rt_mutex_wait_proxy_lock().
Remove it from futex_lock_pi().
Fixes:
|
||
|
Wanwu Li
|
0a85182723 |
sched_ext: Fix NULL sched deref in kfunc sub-sched error paths
When the root scheduler has sub-scheds attached, the COMPAT kfunc
wrappers scx_bpf_select_cpu_and() and scx_bpf_dsq_insert_vtime() refuse
the call and report to @p's scheduler:
scx_error(scx_task_sched(p), "... must be used");
The wrappers are reachable with tasks that have no scheduler.
scx_bpf_select_cpu_and() is in the select_cpu kfunc group, which
scx_kfunc_context_filter() opens to BPF_PROG_TYPE_SYSCALL programs;
scx_bpf_dsq_insert_vtime() is in the enqueue_dispatch group, which
ops.enqueue() and ops.dispatch() may call with any KF_RCU task -- the
group has no kf_tasks validation, and scx_dsq_insert_preamble() checks
task ownership with scx_task_on_sched() precisely because @p may be an
arbitrary task.
scx_task_sched(p) is p->scx.sched, which is NULL for tasks past
sched_ext_dead() -- which clears it via scx_disable_and_exit_task() on
exit -- and for idle tasks, which the enable paths skip as they are
never scheduled through SCX. It is also an rcu_dereference_protected()
that expects @p's pi_lock or rq lock, which neither wrapper holds.
Passing NULL to scx_error() reaches scx_vexit(), which dereferences
sch->exit_info, oopsing the kernel.
One concrete trigger exercised while developing the fix: a
BPF_PROG_TYPE_SYSCALL program calling the select_cpu_and wrapper on an
exited-but-not-reaped task while a sub-scheduler was attached (its pid
stays findable while the zombie is unreaped; faulting instruction is
the scx_vexit() prologue "mov r15,[rdi+0x398]" with RDI=NULL and 0x398
the offset of sch->exit_info):
sched_ext: BPF scheduler "kfunc_subsched_null" enabled
sched_ext: BPF sub-scheduler "kfunc_subsched_null" enabled
sched_ext: Unassociated program run_select_cpu_ (id 76)
BUG: kernel NULL pointer dereference, address: 0000000000000398
#PF: supervisor read access in kernel mode
#PF: error_code(0x0000) - not-present page
Oops: Oops: 0000 [#1] SMP NOPTI
CPU: 7 UID: 0 PID: 8201 Comm: kfunc_test_runn Tainted: G W
RIP: 0010:scx_vexit+0x25/0xa0
Code: ... <4c> 8b bf 98 03 00 00 ...
CR2: 0000000000000398
Call Trace:
<TASK>
__scx_exit+0x4f/0x70
scx_bpf_select_cpu_and+0xab/0xb0
bpf_prog_430ed61a7b66e03a_run_select_cpu_and+0x9c/0xe7
? __x64_sys_bpf+0x2c/0x40
bpf_prog_test_run_syscall+0x130/0x2f0
__sys_bpf+0x930/0x10d0
? __x64_sys_bpf+0x2c/0x40
__x64_sys_bpf+0x2c/0x40
do_syscall_64+0xbc/0x460
entry_SYSCALL_64_after_hwframe+0x76/0x7e
</TASK>
Read @p's scheduler under RCU instead, which the wrappers can do from
their guard(rcu)(): fault it when it can be determined, and when it
can't be determined -- @p is a task past sched_ext_dead() or an idle
task -- there is nothing obviously wrong to report, so just refuse the
call as before without faulting any scheduler.
These COMPAT wrappers are scheduled for eventual removal once the
deprecation grace period elapses, but until then -- and regardless of
their removal timeline -- they must not oops the kernel on a task they
are handed.
Cc: stable@vger.kernel.org # v7.1+
Fixes:
|
||
|
Tim Chen
|
f0d243a96f |
sched/fair: Avoid creating misfits during cache-aware balancing
Cache-aware load balancing biases tasks toward their preferred LLC. On asymmetric CPU capacity systems (e.g. big.LITTLE) the destination LLC may contain CPUs that are too small to run the task. Pulling the task there turns it into a misfit, trading a cache-locality gain for a capacity loss that's more detrimental to performance. Guard both cache-aware migration entry points against this: - can_migrate_llc_task(): forbid the LLC migration when the task fits its source CPU but would not fit the destination CPU. - alb_break_llc(): veto the active balance under the same condition so the runnable task is not pushed onto a CPU that cannot accommodate it. Both checks are gated with checks for hybrid processors, so symmetric systems are unaffected. Tasks that already do not fit their source CPU are left to the existing LLC policy, since the move cannot make their fitness worse (this also preserves misfit up-migration to bigger CPUs). Additionally, if there are misfit tasks found in the load balancing classification phase, prioritize misfit task migrations over LLC load aggregation on asymmetric systems. A better fitting CPU will boost performance more than better cache locality. Reviewed-by: Ricardo Neri <ricardo.neri-calderon@linux.intel.com> Tested-by: Ricardo Neri <ricardo.neri-calderon@linux.intel.com> Reviewed-by: Chen Yu <yu.c.chen@intel.com> Signed-off-by: Tim Chen <tim.c.chen@linux.intel.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Link: https://patch.msgid.link/edbb2503d554c63dc9b72e201fb4a17e1cb119e7.camel@linux.intel.com |
||
|
Wanwu Li
|
b038383526 |
sched/fair: Use cfs_rq->h_curr in distribute_cfs_runtime()
distribute_cfs_runtime() refreshes the rq clock and accounts elapsed runtime with update_curr() before redistributing bandwidth, but gates this on cfs_rq->curr. Since commit |
||
|
Wanwu Li
|
f8610c57f4 |
sched/fair: Use cfs_rq->h_curr in throttle_cfs_rq()
After commit |
||
|
Shubhang Kaushik (Ampere)
|
c6dcd97c8b |
sched/core: Skip rq->avg_idle update without a valid idle_stamp
Commit |
||
|
Seiji Nishikawa
|
dae5c02920 |
sched/rt,dl: Skip migrate-disabled tasks when picking a push candidate
A migrate_disable()'d RT task cannot be moved to another CPU, but the scheduler still keeps such a task on that CPU's pushable list (rq->rt.pushable_tasks) and still marks the runqueue RT-overloaded (rq->rt.overloaded = 1). So the RT balancer keeps treating this CPU as having a task to move away, and keeps trying to move the task, but the push can never succeed. When the head is pinned, push_rt_task() does not give up either. It falls back to pushing rq->curr instead, using the per-CPU stopper, as added by commit |