github-mirror/linux
1
0
Fork 0
mirror of https://github.com/torvalds/linux.git synced 2026-11-07 06:17:02 +01:00
Code Issues Actions Packages Projects Releases Wiki Activity Graph

sched_ext: Split curr|donor references properly

Browse Source 
With proxy execution, the task selected by the scheduler and the task
physically executing can differ. A blocked mutex waiter donates its
scheduling context to the lock owner:

  D -----------------> M -------------> O ----------------> T
  [donor] blocked on [mutex] owned by [owner] preempted by [task]
     \_________________________________^
          donates scheduling context

where:

  D = blocked donor
  M = mutex
  O = mutex owner
  T = competing runnable task

During a proxy execution switch, D supplies the scheduling class,
priority, and runtime budget, while O supplies the execution context: O
is the task whose code physically executes. T is a competing runnable
task which may preempt the D/O proxy execution.

Consider FAIR and EXT tasks with sched_ext running in partial mode. FAIR
can be replaced with a higher scheduling class such as RT or deadline
without changing the class interaction described here.

An active proxy pair can only have a donor in the same scheduling class
as the owner or in a higher-priority class. If the owner belongs to a
higher-priority class, pick_next_task() selects the owner directly
before considering the donor, so find_proxy_task() is not entered for
that donor. The blocked-on relationship may still exist, but it does
not result in proxy execution.

The relevant combinations are:

  1. D is EXT, O is EXT, T is EXT

     D can interrupt T according to BPF scheduling policy. O executes
     with D's EXT priority and runtime budget, while T waits in EXT.

  2. D is EXT, O is EXT, T is FAIR

     D is visible to the BPF scheduler, but cannot preempt T because
     EXT is below FAIR. Once T stops, BPF can dispatch D and O executes
     with D's EXT priority and runtime budget. If T becomes runnable
     again, it preempts the D/O proxy execution.

  3. D is FAIR, O is EXT, T is EXT

     D preempts T as the higher-class scheduling context. O executes
     with D's FAIR priority and runtime budget, while T waits in EXT.
     D is not visible to the BPF scheduler.

  4. D is FAIR, O is EXT, T is FAIR

     D competes with T according to its FAIR deadline. When D is
     selected, O executes with D's FAIR priority and runtime budget.
     D is not visible to the BPF scheduler.

  5. D is FAIR, O is FAIR, T is EXT

     The D/O proxy execution has a FAIR scheduling context, so T cannot
     preempt it from the lower EXT class.

  6. D is FAIR, O is FAIR, T is FAIR

     O, T, and D all have FAIR scheduling contexts. D remains runnable
     as a blocked proxy donor. When CFS selects D, O executes using D's
     FAIR scheduling context. When CFS selects O, O executes using its
     own FAIR context, and when CFS selects T, T executes normally. D
     is not visible to the BPF scheduler.

Thus, sched_ext policy and accounting must generally use rq->donor, the
scheduler-selected task which supplies the scheduling context, rather
than rq->curr, the task whose code physically executes. Without proxy
execution they are the same task.

On nohz_full CPUs, keep the tick running whenever the selected donor is
blocked. A selected blocked donor is necessarily awaiting proxy
execution, so this avoids relying on the transient relationship between
rq->curr and rq->donor during context switches. It also leaves ordinary
sched_ext context switches and non-nohz_full CPUs untouched. Allowing
proxy execution to run tickless remains a future improvement.

Moreover, extend scx_dump_state() to report both contexts. Each CPU
record now includes a donor= line. If an EXT donor differs from
rq->curr, also emit its detailed task record. The existing '*' marker
continues to identify rq->curr, while the donor= line identifies the
otherwise unmarked donor record.

Note that at this point in the series, CONFIG_SCHED_PROXY_EXEC still
depends on !CONFIG_SCHED_CLASS_EXT, so proxy execution and sched_ext
cannot be enabled together. The scheduling changes are therefore
preparatory. A later patch removes this restriction.

Co-developed-by: John Stultz <jstultz@google.com>
Signed-off-by: John Stultz <jstultz@google.com>
Signed-off-by: Andrea Righi <arighi@nvidia.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
This commit is contained in:
Andrea Righi 2026-09-22 18:51:50 +02:00 committed by Tejun Heo
parent 13ae184e4b
commit 3d6577ea93
4 changed files with 106 additions and 55 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

13
Documentation/scheduler/sched-ext.rst
View File

@ -503,6 +503,19 @@ and edge cases, to name a few examples:
class, in which case it will exit the tick-dispatch loop even though it is runnable class, in which case it will exit the tick-dispatch loop even though it is runnable
and has a non-zero slice. and has a non-zero slice.
* Under proxy execution, sched_ext continues to observe the donor as the
current scheduling context. Accordingly, ``ops.running()`` and
``ops.stopping()`` report when the donor's scheduling context becomes active
and inactive, even when the donor is blocked and a lock owner executes on its
behalf. The physical execution context is intentionally not reported through
these callbacks.
A blocked donor enters a running session only after proxy resolution finds
an execution context. The session remains active if only the physical
execution context changes while the donor remains the same. Running sessions
are tracked so that ``ops.running()`` and ``ops.stopping()`` remain paired
and are not emitted recursively.
See the "Scheduling Cycle" section for a more detailed description of how See the "Scheduling Cycle" section for a more detailed description of how
a freshly woken up task gets on a CPU. a freshly woken up task gets on a CPU.

132
kernel/sched/ext/ext.c
View File

@ -425,28 +425,28 @@ static bool rq_is_open(struct rq *rq, u64 enq_flags)
*/ */
/* /*
* If we're in the dispatch path holding rq lock, $curr may or may not * If we're in the dispatch path holding rq lock, $donor may or may not
* be ready depending on whether the on-going dispatch decides to extend * be ready depending on whether the on-going dispatch decides to extend
* $curr's slice. We say yes here and resolve it at the end of dispatch. * $donor's slice. We say yes here and resolve it at the end of dispatch.
* See dispatch_one(). * See dispatch_one().
*/ */
if (rq->scx.flags & SCX_RQ_IN_DISPATCH) if (rq->scx.flags & SCX_RQ_IN_DISPATCH)
return true; return true;
/* /*
* The preemption flags clear $curr's slice if on SCX and kick dispatch, * The preemption flags clear $donor's slice if on SCX and kick dispatch,
* so allow them to avoid spuriously triggering reenq on a combined * so allow it to avoid spuriously triggering reenq on a combined
* PREEMPT|IMMED insertion. * PREEMPT|IMMED insertion.
*/ */
if (enq_flags & (SCX_ENQ_PREEMPT | SCX_ENQ_PREEMPT_LAZY)) { if (enq_flags & (SCX_ENQ_PREEMPT | SCX_ENQ_PREEMPT_LAZY)) {
struct task_struct *curr = rq->curr; struct task_struct *donor = rq->donor;
/* /*
* A protected slice refuses the preemption and the cpu stays * A protected slice refuses the preemption and the cpu stays
* occupied. See rq_owned_post_enq(). * occupied. See rq_owned_post_enq().
*/ */
return curr->sched_class != &ext_sched_class || return donor->sched_class != &ext_sched_class ||
likely(!(curr->scx.flags & SCX_TASK_PROTECTED)); likely(!(donor->scx.flags & SCX_TASK_PROTECTED));
} }
/* /*
@ -1462,20 +1462,27 @@ static void apply_slice_vtime(struct task_struct *p, u64 slice, u64 vtime, u64 e
static void update_curr_scx(struct rq *rq) static void update_curr_scx(struct rq *rq)
{ {
struct task_struct *curr = rq->curr; struct task_struct *donor;
s64 delta_exec; s64 delta_exec;
/*
* update_curr_scx() is selected through rq->donor->sched_class, not
* rq->curr->sched_class, so @donor is always an EXT task here. If an EXT
* owner executes for a FAIR donor, FAIR's update_curr() runs instead.
*/
donor = rq->donor;
/* apply even on 0 delta_exec, callers may still act on the slice */ /* apply even on 0 delta_exec, callers may still act on the slice */
apply_task_slice_oob(rq, curr); apply_task_slice_oob(rq, donor);
delta_exec = update_curr_common(rq); delta_exec = update_curr_common(rq);
if (unlikely(delta_exec <= 0)) if (unlikely(delta_exec <= 0))
return; return;
if (curr->scx.slice != SCX_SLICE_INF) if (donor->scx.slice != SCX_SLICE_INF)
curr->scx.slice -= min_t(u64, curr->scx.slice, delta_exec); donor->scx.slice -= min_t(u64, donor->scx.slice, delta_exec);
if (unlikely(curr == scx_rescuee(rq))) if (unlikely(donor == scx_rescuee(rq)))
scx_rescue_charge(rq, delta_exec); scx_rescue_charge(rq, delta_exec);
dl_server_update(&rq->ext_server, delta_exec); dl_server_update(&rq->ext_server, delta_exec);
@ -1666,9 +1673,9 @@ static void rq_owned_post_enq(struct scx_sched *sch, struct rq *rq,
if (rq->scx.flags & SCX_RQ_IN_DISPATCH) if (rq->scx.flags & SCX_RQ_IN_DISPATCH)
return; return;
if ((enq_flags & (SCX_ENQ_PREEMPT | SCX_ENQ_PREEMPT_LAZY)) && p != rq->curr && if ((enq_flags & (SCX_ENQ_PREEMPT | SCX_ENQ_PREEMPT_LAZY)) &&
rq->curr->sched_class == &ext_sched_class) { p != rq->donor && rq->donor->sched_class == &ext_sched_class) {
if (likely(scx_set_task_slice(rq->curr, 0))) { if (likely(scx_set_task_slice(rq->donor, 0))) {
if (enq_flags & SCX_ENQ_PREEMPT) if (enq_flags & SCX_ENQ_PREEMPT)
resched_curr(rq); resched_curr(rq);
else else
@ -2273,13 +2280,14 @@ static void enqueue_task_scx(struct rq *rq, struct task_struct *p, int core_enq_
rq->scx.flags |= SCX_RQ_IN_WAKEUP; rq->scx.flags |= SCX_RQ_IN_WAKEUP;
/* /*
* Restoring a running task will be immediately followed by * Restoring the current scheduling context will be immediately followed
* set_next_task_scx() which expects the task to not be on the BPF * by set_next_task_scx() which expects the task to not be on the BPF
* scheduler as tasks can only start running through local DSQs. Force * scheduler as tasks can only start running through local DSQs. Force
* direct-dispatch into the local DSQ by setting the sticky_cpu. Mark * direct-dispatch into the local DSQ by setting the sticky_cpu. Mark
* IGNORE_CAPS to force entry into the local DSQ. * IGNORE_CAPS to force entry into the local DSQ.
*/ */
if (unlikely(enq_flags & ENQUEUE_RESTORE) && task_current(rq, p)) { if (unlikely(enq_flags & ENQUEUE_RESTORE) &&
task_current_donor(rq, p)) {
sticky_cpu = cpu_of(rq); sticky_cpu = cpu_of(rq);
enq_flags |= SCX_ENQ_IGNORE_CAPS; enq_flags |= SCX_ENQ_IGNORE_CAPS;
} }
@ -2446,7 +2454,7 @@ static bool dequeue_task_scx(struct rq *rq, struct task_struct *p, int core_deq_
p->scx.flags &= ~SCX_TASK_REENQ_REASON_MASK; p->scx.flags &= ~SCX_TASK_REENQ_REASON_MASK;
/* see scx_task_slice_ended() for the save/restore exception */ /* see scx_task_slice_ended() for the save/restore exception */
if (!((deq_flags & DEQUEUE_SAVE) && task_current(rq, p))) if (!((deq_flags & DEQUEUE_SAVE) && task_current_donor(rq, p)))
scx_task_slice_ended(rq, p); scx_task_slice_ended(rq, p);
clear_direct_dispatch(p); clear_direct_dispatch(p);
@ -3014,7 +3022,8 @@ static void dispatch_to_local_dsq(struct scx_sched *sch, struct rq *rq,
} }
/* if the destination CPU is idle, wake it up */ /* if the destination CPU is idle, wake it up */
if (!fallback && sched_class_above(p->sched_class, dst_rq->curr->sched_class)) if (!fallback && sched_class_above(p->sched_class,
dst_rq->donor->sched_class))
resched_curr(dst_rq); resched_curr(dst_rq);
} }
@ -3248,6 +3257,7 @@ static void scx_start_task_running(struct rq *rq, struct task_struct *p)
static void set_next_task_scx(struct rq *rq, struct task_struct *p, enum snt_e type) static void set_next_task_scx(struct rq *rq, struct task_struct *p, enum snt_e type)
{ {
bool first = type == SNT_PICK; bool first = type == SNT_PICK;
bool can_stop_tick;
if (type == SNT_REPICK) if (type == SNT_REPICK)
return; return;
@ -3283,6 +3293,7 @@ static void set_next_task_scx(struct rq *rq, struct task_struct *p, enum snt_e t
/* apply any pending out-of-band slice request before the tick decision */ /* apply any pending out-of-band slice request before the tick decision */
apply_task_slice_oob(rq, p); apply_task_slice_oob(rq, p);
can_stop_tick = p->scx.slice == SCX_SLICE_INF && !p->is_blocked;
/* /*
* @p is getting newly scheduled or got kicked after someone updated its * @p is getting newly scheduled or got kicked after someone updated its
@ -3293,7 +3304,7 @@ static void set_next_task_scx(struct rq *rq, struct task_struct *p, enum snt_e t
* nohz. In the future, we might want to add a mechanism to update * nohz. In the future, we might want to add a mechanism to update
* load_avgs periodically on tick-stopped CPUs. * load_avgs periodically on tick-stopped CPUs.
*/ */
if (p->scx.slice == SCX_SLICE_INF) { if (can_stop_tick) {
if (!(rq->scx.flags & SCX_RQ_CAN_STOP_TICK)) { if (!(rq->scx.flags & SCX_RQ_CAN_STOP_TICK)) {
/* /*
* Bypass mode always assigns finite slices, so @p * Bypass mode always assigns finite slices, so @p
@ -3314,7 +3325,8 @@ static void set_next_task_scx(struct rq *rq, struct task_struct *p, enum snt_e t
/* /*
* @rq still references the outgoing scheduling context. A finite * @rq still references the outgoing scheduling context. A finite
* slice is sufficient by itself to require the tick. * slice or a blocked proxy donor is sufficient by itself to require
* the tick.
*/ */
if (tick_nohz_full_cpu(cpu_of(rq))) if (tick_nohz_full_cpu(cpu_of(rq)))
tick_nohz_dep_set_cpu(cpu_of(rq), TICK_DEP_BIT_SCHED); tick_nohz_dep_set_cpu(cpu_of(rq), TICK_DEP_BIT_SCHED);
@ -3633,7 +3645,7 @@ static enum scx_dsp_verdict dispatch_core_pick(struct rq *rq, struct rq_flags *r
static struct task_struct * static struct task_struct *
do_pick_task_scx(struct rq *rq, struct rq_flags *rf, bool force_scx) do_pick_task_scx(struct rq *rq, struct rq_flags *rf, bool force_scx)
{ {
struct task_struct *prev = rq->curr; struct task_struct *prev = rq->donor;
enum scx_dsp_verdict verdict; enum scx_dsp_verdict verdict;
struct task_struct *p; struct task_struct *p;
@ -4079,9 +4091,9 @@ void scx_tick(struct rq *rq)
update_other_load_avgs(rq); update_other_load_avgs(rq);
} }
static void task_tick_scx(struct rq *rq, struct task_struct *curr, int queued) static void task_tick_scx(struct rq *rq, struct task_struct *donor, int queued)
{ {
struct scx_sched *sch = scx_task_sched(curr); struct scx_sched *sch = scx_task_sched(donor);
update_curr_scx(rq); update_curr_scx(rq);
@ -4090,13 +4102,14 @@ static void task_tick_scx(struct rq *rq, struct task_struct *curr, int queued)
* management. * management.
*/ */
if (scx_bypassing(sch, cpu_of(rq))) if (scx_bypassing(sch, cpu_of(rq)))
scx_set_task_slice(curr, 0); scx_set_task_slice(donor, 0);
else if (SCX_HAS_OP(sch, tick)) else if (SCX_HAS_OP(sch, tick))
SCX_CALL_OP_TASK(sch, tick, rq, curr); SCX_CALL_OP_TASK(sch, tick, rq, donor);
if (!curr->scx.slice) { if (!donor->scx.slice) {
/* the slice can't be trusted while bypassing */ /* the slice can't be trusted while bypassing */
if (READ_ONCE(curr->scx.lazy_resched) && !scx_bypassing(sch, cpu_of(rq))) if (READ_ONCE(donor->scx.lazy_resched) &&
!scx_bypassing(sch, cpu_of(rq)))
scx_resched_curr_lazy(rq); scx_resched_curr_lazy(rq);
else else
resched_curr(rq); resched_curr(rq);
@ -4789,16 +4802,16 @@ static u32 reenq_local(struct scx_sched *sch, struct rq *rq, u64 reenq_flags)
} }
/* /*
* The revoke that scheduled this scan may have raced the pick: curr * The revoke that scheduled this scan may have raced the pick: donor
* may be a now-capless task, either one that kept running or one * may be a now-capless task, either one that kept running or one
* promoted off the local DSQ between the ecaps sync and this scan. * promoted off the local DSQ between the ecaps sync and this scan.
* Zero the slice to evict it. The enqueue gate blocks new capless * Zero the slice to evict it. The enqueue gate blocks new capless
* inserts, so no later pick can slip through after the scan. * inserts, so no later pick can slip through after the scan.
*/ */
if ((reenq_flags & SCX_REENQ_CAP_REVOKE) && if ((reenq_flags & SCX_REENQ_CAP_REVOKE) &&
rq->curr->sched_class == &ext_sched_class && rq->donor->sched_class == &ext_sched_class &&
scx_task_reenq_on_cap_revoke(rq, rq->curr)) { scx_task_reenq_on_cap_revoke(rq, rq->donor)) {
scx_set_task_slice(rq->curr, 0); scx_set_task_slice(rq->donor, 0);
resched_curr(rq); resched_curr(rq);
} }
@ -5033,14 +5046,18 @@ static void run_deferred(struct rq *rq)
#ifdef CONFIG_NO_HZ_FULL #ifdef CONFIG_NO_HZ_FULL
bool scx_can_stop_tick(struct rq *rq) bool scx_can_stop_tick(struct rq *rq)
{ {
struct task_struct *p = rq->curr; struct task_struct *p = rq->donor;
struct scx_sched *sch = scx_task_sched(p); struct scx_sched *sch = scx_task_sched(p);
/* Keep the tick running while a blocked proxy donor is selected. */
if (p->is_blocked)
return false;
if (p->sched_class != &ext_sched_class) if (p->sched_class != &ext_sched_class)
return true; return true;
/* /*
* @rq->curr may still reference an outgoing EXT task after it has been * @rq->donor may still reference an outgoing EXT task after it has been
* dequeued. If no EXT tasks are accounted on @rq, ignore its stale * dequeued. If no EXT tasks are accounted on @rq, ignore its stale
* slice state. If another task is dispatched from a DSQ, * slice state. If another task is dispatched from a DSQ,
* set_next_task_scx() will update the dependency for the incoming task. * set_next_task_scx() will update the dependency for the incoming task.
@ -5061,7 +5078,8 @@ bool scx_can_stop_tick(struct rq *rq)
/* /*
* @rq can dispatch from different DSQs, so we can't tell whether it * @rq can dispatch from different DSQs, so we can't tell whether it
* needs the tick or not by looking at nr_running. Allow stopping ticks * needs the tick or not by looking at nr_running. Allow stopping ticks
* iff the BPF scheduler indicated so. See set_next_task_scx(). * iff set_next_task_scx() determined that the selected scheduling context
* can run tickless.
*/ */
return rq->scx.flags & SCX_RQ_CAN_STOP_TICK; return rq->scx.flags & SCX_RQ_CAN_STOP_TICK;
} }
@ -6576,9 +6594,9 @@ void scx_bypass(struct scx_sched *sch, bool bypass)
/* /*
* Bypass trumps protection. Cycling clears for queued * Bypass trumps protection. Cycling clears for queued
* tasks but current task needs explicit stripping. * tasks but the current donor needs explicit stripping.
*/ */
if (bypass && task_current(rq, p)) if (bypass && task_current_donor(rq, p))
scx_task_slice_ended(rq, p); scx_task_slice_ended(rq, p);
/* cycling deq/enq is enough, see the function comment */ /* cycling deq/enq is enough, see the function comment */
@ -7286,6 +7304,8 @@ static void scx_dump_cpu(struct scx_sched *sch, struct seq_buf *s,
scx_rescue_dump(&ns, rq); scx_rescue_dump(&ns, rq);
scx_dump_line(&ns, " curr=%s[%d] class=%ps", scx_dump_line(&ns, " curr=%s[%d] class=%ps",
rq->curr->comm, rq->curr->pid, rq->curr->sched_class); rq->curr->comm, rq->curr->pid, rq->curr->sched_class);
scx_dump_line(&ns, " donor=%s[%d] class=%ps",
rq->donor->comm, rq->donor->pid, rq->donor->sched_class);
if (!cpumask_empty(pcpu->cpus_to_kick)) if (!cpumask_empty(pcpu->cpus_to_kick))
scx_dump_line(&ns, " cpus_to_kick : %*pb", scx_dump_line(&ns, " cpus_to_kick : %*pb",
cpumask_pr_args(pcpu->cpus_to_kick)); cpumask_pr_args(pcpu->cpus_to_kick));
@ -7332,6 +7352,10 @@ static void scx_dump_cpu(struct scx_sched *sch, struct seq_buf *s,
if (rq->curr->sched_class == &ext_sched_class && if (rq->curr->sched_class == &ext_sched_class &&
(dump_all_tasks || scx_task_on_sched(sch, rq->curr))) (dump_all_tasks || scx_task_on_sched(sch, rq->curr)))
scx_dump_task(sch, s, dctx, rq, rq->curr, '*'); scx_dump_task(sch, s, dctx, rq, rq->curr, '*');
if (rq->donor != rq->curr &&
rq->donor->sched_class == &ext_sched_class &&
(dump_all_tasks || scx_task_on_sched(sch, rq->donor)))
scx_dump_task(sch, s, dctx, rq, rq->donor, ' ');
list_for_each_entry(p, &rq->scx.runnable_list, scx.runnable_node) list_for_each_entry(p, &rq->scx.runnable_list, scx.runnable_node)
if (dump_all_tasks || scx_task_on_sched(sch, p)) if (dump_all_tasks || scx_task_on_sched(sch, p))
@ -8890,7 +8914,7 @@ static bool kick_one_cpu(s32 cpu, struct scx_sched_pcpu *pcpu, struct rq *this_r
bool preempt, preempt_lazy, wait, immediate; bool preempt, preempt_lazy, wait, immediate;
rq_lock_irqsave(rq, &rf); rq_lock_irqsave(rq, &rf);
cur_class = rq->curr->sched_class; cur_class = rq->donor->sched_class;
preempt = cpumask_test_cpu(cpu, pcpu->cpus_to_preempt); preempt = cpumask_test_cpu(cpu, pcpu->cpus_to_preempt);
preempt_lazy = cpumask_test_cpu(cpu, pcpu->cpus_to_preempt_lazy); preempt_lazy = cpumask_test_cpu(cpu, pcpu->cpus_to_preempt_lazy);
wait = cpumask_test_cpu(cpu, pcpu->cpus_to_wait); wait = cpumask_test_cpu(cpu, pcpu->cpus_to_wait);
@ -8921,7 +8945,7 @@ static bool kick_one_cpu(s32 cpu, struct scx_sched_pcpu *pcpu, struct rq *this_r
__scx_add_event(pcpu->sch, SCX_EV_SUB_PREEMPT_DENIED, 1); __scx_add_event(pcpu->sch, SCX_EV_SUB_PREEMPT_DENIED, 1);
/* degrade to a plain, immediate kick */ /* degrade to a plain, immediate kick */
immediate = true; immediate = true;
} else if (unlikely(!scx_set_task_slice(rq->curr, 0))) { } else if (unlikely(!scx_set_task_slice(rq->donor, 0))) {
__scx_add_event(pcpu->sch, SCX_EV_SLICE_DENIED, 1); __scx_add_event(pcpu->sch, SCX_EV_SLICE_DENIED, 1);
} }
} }
@ -9924,8 +9948,10 @@ __bpf_kfunc bool scx_bpf_task_set_slice(struct task_struct *p, u64 slice,
return false; return false;
/* /*
* Directly write only when we hold the lock of the rq @p is queued or * Directly write only when we hold the lock of the rq @p is queued on or
* running on. See the write rules above. * provides the current scheduling context for. Under proxy execution,
* rq->donor owns and consumes the slice while rq->curr executes on its
* behalf. See the slice write rules above.
* *
* While @p is queued on a user DSQ or in the BPF scheduler, * While @p is queued on a user DSQ or in the BPF scheduler,
* synchronization is the scheduler's responsibility. This write can * synchronization is the scheduler's responsibility. This write can
@ -9939,7 +9965,7 @@ __bpf_kfunc bool scx_bpf_task_set_slice(struct task_struct *p, u64 slice,
locked_rq = scx_locked_rq(); locked_rq = scx_locked_rq();
if (!locked_rq || if (!locked_rq ||
(READ_ONCE(p->scx.runnable_cpu) != cpu_of(locked_rq) && (READ_ONCE(p->scx.runnable_cpu) != cpu_of(locked_rq) &&
!task_current(locked_rq, p))) { !task_current_donor(locked_rq, p))) {
set_task_slice_oob(sch, p, slice); set_task_slice_oob(sch, p, slice);
return true; return true;
} }
@ -10927,12 +10953,17 @@ __bpf_kfunc void scx_bpf_put_cpumask(const struct cpumask *cpumask)
} }
/** /**
* scx_bpf_task_running - Is task currently running? * scx_bpf_task_running - Is task the current scheduling context?
* @p: task of interest * @p: task of interest
*
* Under proxy execution, this reports the donor rather than the task whose
* code is physically executing. The physical execution context is intentionally
* not exposed to the BPF scheduler, which continues to observe the donor as the
* running scheduling context.
*/ */
__bpf_kfunc bool scx_bpf_task_running(const struct task_struct *p) __bpf_kfunc bool scx_bpf_task_running(const struct task_struct *p)
{ {
return task_rq(p)->curr == p; return rcu_access_pointer(task_rq(p)->donor) == p;
} }
/** /**
@ -10993,10 +11024,15 @@ __bpf_kfunc struct rq *scx_bpf_locked_rq(const struct bpf_prog_aux *aux)
} }
/** /**
* scx_bpf_cpu_curr - Return remote CPU's curr task * scx_bpf_cpu_curr - Return remote CPU's current scheduling context
* @cpu: CPU of interest * @cpu: CPU of interest
* @aux: implicit BPF argument to access bpf_prog_aux hidden from BPF progs * @aux: implicit BPF argument to access bpf_prog_aux hidden from BPF progs
* *
* Under proxy execution, this returns the donor, which supplies the scheduling
* policy and runtime budget, rather than the task whose code is physically
* executing. The physical execution context is intentionally not exposed to
* the BPF scheduler.
*
* Callers must hold RCU read lock (KF_RCU). * Callers must hold RCU read lock (KF_RCU).
*/ */
__bpf_kfunc struct task_struct *scx_bpf_cpu_curr(s32 cpu, const struct bpf_prog_aux *aux) __bpf_kfunc struct task_struct *scx_bpf_cpu_curr(s32 cpu, const struct bpf_prog_aux *aux)
@ -11012,7 +11048,7 @@ __bpf_kfunc struct task_struct *scx_bpf_cpu_curr(s32 cpu, const struct bpf_prog_
if (!scx_cpu_valid(sch, cpu, NULL)) if (!scx_cpu_valid(sch, cpu, NULL))
return NULL; return NULL;
return rcu_dereference(cpu_rq(cpu)->curr); return rcu_dereference(cpu_rq(cpu)->donor);
} }
/** /**
@ -11036,7 +11072,7 @@ __bpf_kfunc struct task_struct *scx_bpf_cid_curr(s32 cid, const struct bpf_prog_
cpu = scx_cid_to_cpu(sch, cid); cpu = scx_cid_to_cpu(sch, cid);
if (cpu < 0) if (cpu < 0)
return NULL; return NULL;
return rcu_dereference(cpu_rq(cpu)->curr); return rcu_dereference(cpu_rq(cpu)->donor);
} }
/** /**

5
kernel/sched/ext/sub.c
View File

@ -285,7 +285,8 @@ void scx_rescue_charge(struct rq *rq, s64 delta_exec)
rq->scx.rescue.budget -= delta_exec; rq->scx.rescue.budget -= delta_exec;
/* per-cpu usage average feeds the overload victim pick */ /* per-cpu usage average feeds the overload victim pick */
pcpu = per_cpu_ptr(scx_task_sched(rq->curr)->pcpu, cpu_of(rq)); pcpu = per_cpu_ptr(scx_task_sched(rq->scx.rescue.curr)->pcpu,
cpu_of(rq));
pcpu->rescue_avg = scx_rescue_decay_avg(pcpu) + delta_exec; pcpu->rescue_avg = scx_rescue_decay_avg(pcpu) + delta_exec;
if (!scx_rescue_slice_remaining(rq)) if (!scx_rescue_slice_remaining(rq))
@ -557,7 +558,7 @@ static void scx_rescue_timerfn(struct timer_list *timer)
scx_rescue_admit(rq, p, slice); scx_rescue_admit(rq, p, slice);
scx_move_local_task_to_local_dsq(scx_task_sched(p), p, scx_move_local_task_to_local_dsq(scx_task_sched(p), p,
SCX_ENQ_IGNORE_CAPS, rq); SCX_ENQ_IGNORE_CAPS, rq);
if (sched_class_above(&ext_sched_class, rq->curr->sched_class)) if (sched_class_above(&ext_sched_class, rq->donor->sched_class))
resched_curr(rq); resched_curr(rq);
} else if (p->scx.dsq && rq->scx.rescue.budget > 2 * scx_rescue_quantum_ns) { } else if (p->scx.dsq && rq->scx.rescue.budget > 2 * scx_rescue_quantum_ns) {
/* /*

11
kernel/sched/ext/sub.h
View File

@ -186,17 +186,18 @@ static inline u64 scx_caps_for_task(struct task_struct *p)
return SCX_CAP_ENQ; return SCX_CAP_ENQ;
} }
/* the cap @sch needs to preempt @rq's current task, 0 if none */ /* the cap @sch needs to preempt @rq's current scheduling context, 0 if none */
static inline u64 scx_caps_for_preempt(struct scx_sched *sch, struct rq *rq, u64 enq_flags) static inline u64 scx_caps_for_preempt(struct scx_sched *sch, struct rq *rq,
u64 enq_flags)
{ {
struct task_struct *curr = rq->curr; struct task_struct *donor = rq->donor;
/* a kernel-forced placement preempts regardless of caps */ /* a kernel-forced placement preempts regardless of caps */
if (unlikely(enq_flags & SCX_ENQ_IGNORE_CAPS)) if (unlikely(enq_flags & SCX_ENQ_IGNORE_CAPS))
return 0; return 0;
/* a non-ext task can't be preempted by ext, own-subtree needs no cap */ /* a non-ext task can't be preempted by ext, own-subtree needs no cap */
if (curr->sched_class != &ext_sched_class || if (donor->sched_class != &ext_sched_class ||
scx_is_descendant(scx_task_sched(curr), sch)) scx_is_descendant(scx_task_sched(donor), sch))
return 0; return 0;
return SCX_CAP_PREEMPT; return SCX_CAP_PREEMPT;
} }