github-mirror/linux
1
0
Fork 0
mirror of https://github.com/torvalds/linux.git synced 2026-11-07 06:17:02 +01:00
Code Issues Actions Packages Projects Releases Wiki Activity Graph

sched_ext: Don't queue balance callbacks from a core-sched pick

Browse Source 
Commit ff0545e0c2 ("sched_ext: Don't leave balance callbacks queued
across a pick retry") queues the deferred work as a balance callback
only when do_pick_task_scx() returns a final pick. However, under core
scheduling, a pick that returns a task isn't final either:
pick_next_task() goes on to pick the SMT siblings, and their dispatch
can release the shared rq lock with the callback still queued,
triggering the rq_pin_lock() warning.

Moreover, dispatch_core_pick() returns early when another dispatch is
in flight on the rq, and do_pick_task_scx() then consumes
%SCX_RQ_BAL_CB_PENDING set by that dispatch while it's still running.

This can be reproduced running scx_qmap -I on SMT siblings with
coresched-tagged workloads that frequently sleep, mixed with untagged
CPU-bound tasks.

Fix this by always deferring the pending work to the irq_work path from
dispatch_core_pick() under core scheduling, for both the local and the
sibling picks, and by leaving %SCX_RQ_BAL_CB_PENDING alone in
do_pick_task_scx(), so that it's handled by the dispatch that set it.
Sample the core scheduling state once at the start of the pick, since
core scheduling can get enabled while a non-core dispatch has the rq lock
released, and the follow-up work must match the dispatch path that ran.

Fixes: 4c95380701 ("sched/ext: Fold balance_scx() into pick_task_scx()")
Cc: stable@vger.kernel.org # v6.19+
Signed-off-by: Andrea Righi <arighi@nvidia.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
This commit is contained in:
Andrea Righi 2026-10-17 22:05:14 +02:00 committed by Tejun Heo
parent ff0545e0c2
commit 680edc7f23
1 changed files with 24 additions and 11 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

35
kernel/sched/ext/ext.c
View File

@ -3367,15 +3367,20 @@ static enum scx_dsp_verdict dispatch_core_pick(struct rq *rq, struct rq_flags *r
verdict = dispatch_one(rq, prev);
if (cpu_of(rq) != smp_processor_id() &&
unlikely(rq->scx.flags & SCX_RQ_BAL_CB_PENDING)) {
/*
* Balance callbacks must run in the context that queued them,
* so they can't be queued on another CPU's rq. Run the deferred
* work directly instead.
*/
/*
* A pick that returns a task isn't final under core scheduling: the
* selection goes on to pick the siblings, and their dispatch can
* release the rq lock, which can't happen with balance callbacks
* queued. Don't queue the deferred work as a balance callback, for this
* CPU or a sibling, and run it from irq_work instead.
*
* do_pick_task_scx() leaves %SCX_RQ_BAL_CB_PENDING alone under core
* scheduling, so that the early return above leaves it to the dispatch
* in flight, which handles it here.
*/
if (unlikely(rq->scx.flags & SCX_RQ_BAL_CB_PENDING)) {
rq->scx.flags &= ~SCX_RQ_BAL_CB_PENDING;
run_deferred(rq);
schedule_deferred(rq);
}
rq_repin_lock(rq, rf);
@ -3398,6 +3403,7 @@ static struct task_struct *
do_pick_task_scx(struct rq *rq, struct rq_flags *rf, bool force_scx)
{
struct task_struct *prev = rq->curr;
bool core_sched = sched_core_enabled(rq);
enum scx_dsp_verdict verdict;
struct task_struct *p;
@ -3406,7 +3412,12 @@ do_pick_task_scx(struct rq *rq, struct rq_flags *rf, bool force_scx)
rq_modified_begin(rq, &ext_sched_class);
if (sched_core_enabled(rq))
/*
* Core scheduling can get enabled while dispatch_pick() has the rq lock
* released. Stick to @core_sched, so that the follow-up work below
* matches the dispatch that ran.
*/
if (core_sched)
verdict = dispatch_core_pick(rq, rf, prev);
else
verdict = dispatch_pick(rq, rf, prev);
@ -3472,7 +3483,9 @@ do_pick_task_scx(struct rq *rq, struct rq_flags *rf, bool force_scx)
* for the next pick.
*/
if (p != RETRY_TASK && (p || !force_scx)) {
maybe_queue_balance_callback(rq);
/* under core scheduling, see dispatch_core_pick() */
if (!core_sched)
maybe_queue_balance_callback(rq);
/*
* Defer to a balance callback which can drop rq lock and enable
@ -3485,7 +3498,7 @@ do_pick_task_scx(struct rq *rq, struct rq_flags *rf, bool force_scx)
queue_balance_callback(rq, &rq->scx.kick_sync_bal_cb,
kick_sync_wait_bal_cb);
}
} else if (rq->scx.flags & SCX_RQ_BAL_CB_PENDING) {
} else if (!core_sched && (rq->scx.flags & SCX_RQ_BAL_CB_PENDING)) {
rq->scx.flags &= ~SCX_RQ_BAL_CB_PENDING;
schedule_deferred(rq);
}