cid-form ops.enable() now hands the task's cmask to the scheduler and
set_cmask() repeats it right after. Add a cid-form selftest that checks both
against p->cpus_ptr, that they match each other, that the initial
set_cmask() lands before set_weight() and before the task first becomes
runnable, and that set_cmask() never precedes enable(), across class-switch
enables, fork-path enables and live affinity changes.
v2: Mismatch details returned through a caller-local struct instead of
globals, alloc_words validated in the header check, loop bounded by nr_cids
directly (Andrea Righi).
Signed-off-by: Tejun Heo <tj@kernel.org>
Add a scheduler whose ops.dequeue() iterates the source user DSQ with
bpf_iter_scx_dsq. The iteration takes the DSQ's raw spinlock; on a
kernel that runs ops.dequeue() while the consume path still holds that
lock, the first task consumed self-deadlocks the CPU with IRQs
disabled. The watchdog cannot recover from that state, so on an
unfixed kernel this test wedges the system instead of failing cleanly.
On a fixed kernel the scheduler runs clean and the test passes.
Signed-off-by: fangqiurong <fangqiurong@kylinos.cn>
Signed-off-by: Tejun Heo <tj@kernel.org>
The ddsp_vtimelocal_fail and ddsp_bogus_dsq_fail tests skip calling
scx_bpf_dsq_insert_vtime() if scx_bpf_pick_idle_cpu() fails to find an idle
CPU (returns a negative error code). On loaded systems, this results in the
tests skipping the very assertions they are meant to verify.
Eliminate this flakiness by falling back to prev_cpu if no idle CPU is
found, ensuring the illegal dispatch operations are unconditionally
attempted and tested.
Fixes: a5db7817af ("sched_ext: Add selftests")
Signed-off-by: Michal Blaszczyk <michalblk@google.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
A CPU returned by scx_bpf_pick_idle_cpu_node() can be re-advertised as
idle by an idle-to-idle re-pick before the BPF program validates the
selection, and the scx_bpf_pick_any_cpu_node() fallback doesn't claim
the CPU at all. Asserting that the picked CPU is absent from the node's
idle cpumask is therefore inherently racy.
Follow the same approach as commit 12da4723b6 ("selftests/sched_ext:
Make allowed_cpus idle validation race-free") and validate a stable
local invariant instead: a CPU executing ops.select_cpu() in a non-idle
scheduling context must not be advertised as idle in its node's idle
cpumask. Keep the node-membership validation of the picked CPU, which
is stable.
Signed-off-by: Cheng-Yang Chou <yphbchou0911@gmail.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
Pull to receive:
c10b216a07 ("sched/core: Handle pick_task() releasing the rq lock")
f3629c63a4 ("sched/core: Make core-sched flips wait for in-flight selections")
ffaab58d21 ("sched_ext: Replace SCX_RQ_BAL_KEEP with a dispatch verdict return")
3dd52416e4 ("sched_ext: Fix this_rq() assumptions in dispatch kfuncs")
f2da958711 ("sched_ext: Count rq lock releases in rq->scx.lock_drop_seq")
d954004205 ("sched_ext: Fix rq->core_pick corruption under core scheduling")
for the pending core scheduling follow-ups and to resolve the conflicts
with the code reorganization and cap gate work on for-7.3.
ffaab58d21 converts scx_dispatch_sched() to a dispatch verdict return
which for-7.3 moved from ext.c into inlines.h. Resolved by applying the
conversion to the relocated copy and combining balance_one()'s verdict
returns with the scx_task_can_stay_on_cpu() gate from the cap work.
ffaab58d21 and 3dd52416e4 update scx_bpf_sub_dispatch() which
for-7.3 moved into sub.c. Resolved by applying the scx_locked_rq()
switch and the verdict test to the sub.c copy.
f2da958711 instruments the open-coded lock releases in
consume_remote_task() which for-7.3 folded into switch_rq_lock().
Resolved by keeping the accounting in switch_rq_lock() which covers all
its callers.
d954004205 widens the put_prev_task_scx() WARN suppression to all
core-sched rqs on the same condition that for-7.3 gated with
scx_task_can_stay_on_cpu(). Resolved by combining both.
Signed-off-by: Tejun Heo <tj@kernel.org>
exit.c does not check the return value of exit__open(). If it returns
NULL, which happens when the skeleton wrapper fails to allocate the
object or open the BPF ELF, the very next line dereferences it via
SCX_ENUM_INIT(). That macro expands to SCX_ENUM_SET(), which accesses
skel->rodata, causing a NULL pointer dereference.
Every other selftest guards this with SCX_FAIL_IF(!skel, ...); exit.c
is the only one missing it.
Fixes: a5db7817af ("sched_ext: Add selftests")
Signed-off-by: Liang Luo <luoliang@kylinos.cn>
Signed-off-by: Tejun Heo <tj@kernel.org>
A remotely selected CPU can be re-advertised as idle by an idle-to-idle
re-pick before the BPF program validates the selection. Checking that
the selected CPU remains absent from the idle mask is therefore
inherently racy.
Validate a stable local invariant instead: a CPU executing
ops.select_cpu() or ops.enqueue() in a non-idle scheduling context must
not be advertised as idle. Read the idle mask without modifying it and
also validate selected CPUs against the requested domain and task
affinity.
Suggested-by: Kuba Piecuch <jpiecuch@google.com>
Signed-off-by: Andrea Righi <arighi@nvidia.com>
Reviewed-by: Kuba Piecuch <jpiecuch@google.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
When a sleeping task's affinity is changed, task_cpu(p) can be outside
of p->cpus_ptr until after select_task_rq() selects a new runqueue for
the task during wakeup.
Thus, the task's NUMA node determined by numa_select_cpu() can be
completely outside of the task's cpumask, leading to
scx_pick_{idle,any}_cpu_node() failing to find an eligible CPU and
returning -EBUSY. This leads to the numa.bpf.c scheduler abnormally
exiting with the following message in dmesg:
sched_ext: numa: invalid CPU -16
scx_bpf_cpu_node+0x120/0x190
bpf_prog_0a34b8e0f515771f_numa_select_cpu+0x108/0x14e
bpf__sched_ext_ops_select_cpu+0x4f/0xb4
select_task_rq_scx+0xb0/0x210
select_task_rq+0xa0/0xd0
__try_to_wake_up+0x196/0x650
complete_all+0x76/0x100
migration_cpu_stop+0x22b/0x300
cpu_stopper_thread+0xc1/0x180
smpboot_thread_fn+0x16b/0x230
kthread+0x2d7/0x350
ret_from_fork+0x1c2/0x350
ret_from_fork_asm+0x1a/0x30
Make numa_select_cpu() robust against this case by returning @prev_cpu
if no CPU could be found in the selected NUMA node _and_ we have reason
to believe that the task's affinity was changed while it was sleeping.
Fixes: 5ae5161820 ("selftests/sched_ext: Add NUMA-aware scheduler test")
Signed-off-by: Kuba Piecuch <jpiecuch@google.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
Pull to receive:
db4e9defd2 ("sched_ext: Record an error on errno-only sub-enable failure")
49b3378a75 ("sched_ext: Fix premature ops->priv publication in scx_alloc_and_add_sched()")
e6979d05c6 ("tools/sched_ext: scx - Fix cmask_subset(), cmask_equal() and cmask_weight()")
for further sub-sched changes and to resolve the conflicts with the
sub-sched updates on for-7.3.
db4e9defd2 adds scx_error() to the sub-enable err_disable sink which
for-7.3 moved from ext.c into sub.c. Resolved by applying the fix to
scx_sub_enable_workfn() in sub.c.
49b3378a75 drops RCU_INIT_POINTER() from an scx_alloc_and_add_sched()
unwind label whose body changed with for-7.3's stall_cpus addition.
Resolved by dropping the line from the updated unwind.
Signed-off-by: Tejun Heo <tj@kernel.org>
In prog_run's run(), the bpf_link is attached early but only destroyed
on the success path. The three SCX_EQ assertions between attach and
destroy expand to a direct 'return SCX_TEST_FAIL', so if any of them
triggers, bpf_link__destroy() is never reached and the BPF scheduler
stays loaded. All subsequent tests then fail to attach because SCX is
not in the DISABLED state.
Convert those assertions to explicit checks that jump to a unified
'out' label which always runs the cleanup, matching the pattern used
in cyclic_kick_wait.c.
Fixes: a5db7817af ("sched_ext: Add selftests")
Signed-off-by: Liang Luo <luoliang@kylinos.cn>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
Finite-slice EXT tasks need the periodic scheduler tick to expire their
slices even when nohz_full is enabled.
Add a regression test that selects a nohz_full CPU and exercises both
infinite-to-finite and finite-to-finite slice transitions across an idle
interval. For each finite task, verify that its ops.tick() callback is
invoked.
Skip the test when an allowed nohz_full CPU and a separate housekeeping
CPU are not available.
Signed-off-by: Andrea Righi <arighi@nvidia.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
This depends on the BPF arena work in bpf-next and should be pulled after
it. for-7.1-fixes was merged in a few times to avoid conflicts with the
feature changes.
- Most of this continues the in-development sub-scheduler support, which
lets a root BPF scheduler delegate to nested sub-schedulers. The
dispatch-path building blocks landed in 7.1. A follow-up patchset in
development will complete enqueue-path support for hierarchical
scheduling. This cycle adds most of that infrastructure:
- Topological CPU IDs (cids): a dense, topology-ordered CPU numbering
where the CPUs of a core, LLC, or NUMA node form contiguous ranges,
so a topology unit becomes a (start, length) slice. Raw CPU numbers
are sparse and don't track topological closeness, which makes them
clumsy for sharding work across sub-schedulers and awkward in BPF.
- cmask: bitmaps windowed over a slice of cid space, so a sub-scheduler
can track, for example, the idle cids of its shard without a full
NR_CPUS cpumask.
- A struct_ops variant that cid-form sub-schedulers register with,
along with the cid-form kfuncs they call.
- BPF arena integration, which sub-scheduler support is built on. The
bpf-next additions let the kernel read and write the BPF scheduler's
arena directly, turning it into a real kernel/BPF shared-memory
channel. Shared state like the per-CPU cmask now lives there.
- scx_qmap is reworked to exercise the new arena and cid interfaces.
- Exit-dump improvements: dump the faulting CPU first, expose the exit
CPU to BPF and userspace, and normalize the dump header.
- Misc kfuncs and cleanups: a task-ID lookup kfunc, __printf checking on
the error and dump formatters, header reorganization, and assorted
fixes.
-----BEGIN PGP SIGNATURE-----
iIQEABYKACwWIQTfIjM1kS57o3GsC/uxYfJx3gVYGQUCajCCYw4cdGpAa2VybmVs
Lm9yZwAKCRCxYfJx3gVYGSnKAP4/v5Y5VCaB/C6a4RaWuhCKI5tR8hLd9zMwJY1h
0KrhQAEAtVVe8FoTQFFZhP12HEuebvNARqJ8E8bDQCb41B/FfQ4=
=Xrkh
-----END PGP SIGNATURE-----
Merge tag 'sched_ext-for-7.2' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/sched_ext
Pull sched_ext updates from Tejun Heo:
"Most of this continues the in-development sub-scheduler support, which
lets a root BPF scheduler delegate to nested sub-schedulers. The
dispatch-path building blocks landed in 7.1. A follow-up patchset in
development will complete enqueue-path support for hierarchical
scheduling. This cycle adds most of that infrastructure:
- Topological CPU IDs (cids): a dense, topology-ordered CPU numbering
where the CPUs of a core, LLC, or NUMA node form contiguous ranges,
so a topology unit becomes a (start, length) slice. Raw CPU numbers
are sparse and don't track topological closeness, which makes them
clumsy for sharding work across sub-schedulers and awkward in BPF.
- cmask: bitmaps windowed over a slice of cid space, so a
sub-scheduler can track, for example, the idle cids of its shard
without a full NR_CPUS cpumask.
- A struct_ops variant that cid-form sub-schedulers register with,
along with the cid-form kfuncs they call.
- BPF arena integration, which sub-scheduler support is built on. The
bpf-next additions let the kernel read and write the BPF
scheduler's arena directly, turning it into a real kernel/BPF
shared-memory channel. Shared state like the per-CPU cmask now
lives there.
- scx_qmap is reworked to exercise the new arena and cid interfaces.
Additionally:
- Exit-dump improvements: dump the faulting CPU first, expose the
exit CPU to BPF and userspace, and normalize the dump header.
- Misc kfuncs and cleanups: a task-ID lookup kfunc, __printf checking
on the error and dump formatters, header reorganization, and
assorted fixes"
* tag 'sched_ext-for-7.2' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/sched_ext: (59 commits)
sched_ext: Add scx_arena_to_kaddr() / scx_kaddr_to_arena()
sched_ext: Make scx_bpf_kick_cid() return s32
sched_ext: Add scx_cmask_test() and scx_cmask_for_each_cid()
tools/sched_ext: Order single-cid cmask helpers as (cid, mask)
sched_ext: Order single-cid cmask helpers as (cid, mask)
selftests/sched_ext: Fix dsq_move_to_local check
sched_ext: Guard BPF arena helper calls to fix 32-bit build
sched_ext: idle: Fix errno loss in scx_idle_init()
sched_ext: Convert ops.set_cmask() to arena-resident cmask
sched_ext: Sub-allocator over kernel-claimed BPF arena pages
sched_ext: Require an arena for cid-form schedulers
sched_ext: Add cmask mask ops
sched_ext: Track bits[] storage size in struct scx_cmask
sched_ext: Rename scx_cmask.nr_bits to nr_cids
tools/sched_ext: scx_qmap: Fix qa arena placement
sched_ext: Mark !CONFIG_EXT_SUB_SCHED dummy stubs static inline
sched_ext: Replace tryget_task_struct() with get_task_struct()
sched_ext: Add scx_task_iter_relock() and use it in scx_root_enable_workfn()
sched_ext: Fix ops_cid layout assert
sched_ext: Use offsetofend on both sides of the ops_cid layout assert
...
SMP load-balancing updates:
- A large series to introduce infrastructure for cache-aware load
balancing, with the goal of co-locating tasks that share data
within the same Last Level Cache (LLC) domain. By improving cache
locality, the scheduler can reduce cache bouncing and cache misses,
ultimately improving data access efficiency.
Implemented by Chen Yu and Tim Chen, based on early prototype work
by Peter Zijlstra, with fixes by Jianyong Wu, Peter Zijlstra and
Shrikanth Hegde.
- A series to simplify CONFIG_SCHED_SMT ifdef usage (Shrikanth Hegde)
Fair scheduler updates:
- A series to improve SD_ASYM_CPUCAPACITY scheduling by introducing
SMT awareness (Andrea Righi, K Prateek Nayak)
- A series to optimize cfs_rq and sched_entity allocation for
better data locality (Zecheng Li)
- A preparatory series to change fair/cgroup scheduling to a single
runqueue, without the final change (Peter Zijlstra)
- Auto-manage ext/fair dl_server bandwidth (Andrea Righi)
- Fix cpu_util runnable_avg arithmetic (Hongyan Xia)
- Optimize update_tg_load_avg()'s rate-limiting code
(Rik van Riel)
- Allow account_cfs_rq_runtime() to throttle current hierarchy
(K Prateek Nayak)
- Update util_est after updating util_avg during dequeue, to fix
the util signal update logic, which reduces signal noise
(Vincent Guittot)
Scheduler topology updates:
- Allow multiple domains to claim sched_domain_shared (K Prateek Nayak)
- Add parameter to split LLC (Peter Zijlstra)
Core scheduler updates:
- Use trace_call__<tp>() to save a static branch (Gabriele Monaco)
Scheduler statistics updates:
- Drop now-stale mul_u64_u64_div_u64() cputime over-approximation guard
(Nicolas Pitre)
Deadline scheduler updates:
- Reject debugfs dl_server writes for offline CPUs (Andrea Righi)
- Fix replenishment logic for non-deferred servers
(Yuri Andriaccio)
RT scheduling updates:
- Turn RT_PUSH_IPI default off for non PREEMPT_RT (Steven Rostedt)
- Update default bandwidth for real-time tasks to 1.0
(Yuri Andriaccio)
Proxy scheduling updates:
- A series to implement Optimized Donor Migration for Proxy Execution
(John Stultz, Peter Zijlstra)
- Various proxy scheduling cleanups and fixes
(Peter Zijlstra, K Prateek Nayak)
Misc fixes, improvements and cleanups by Aaron Lu, Andrea Righi,
Zenghui Yu, Chen Yu, Guanyou.Chen, John Stultz, Shrikanth Hegde,
Peter Zijlstra, Liang Luo and Yiyang Chen.
Signed-off-by: Ingo Molnar <mingo@kernel.org>
-----BEGIN PGP SIGNATURE-----
iQJFBAABCgAvFiEEBpT5eoXrXCwVQwEKEnMQ0APhK1gFAmouy9ERHG1pbmdvQGtl
cm5lbC5vcmcACgkQEnMQ0APhK1iLrxAApGz9fhzT8k8slnaSe9kzX5K3OtBLivi1
ZHPeyNHggonASdZ7+apwJq+sdtmAgEw/phMN69fHhteIIa24WCHEpFtLndDHAkAm
eUKRgBI/eAMrOBECldZ69QO1pfYmMV/2aZH/IyTDtgFvXYZS2+UtsE9fPCvx84V2
Uxa5r7a8NTrpbhsZ2YhncpLkrJx0SfaBQBtvckDVWQBbh5stvIZncCatYjX0M33x
yddLVkM7e5f6nenb5+rwW3TMayfzhkeaR+r9vUuZE1mt9ItwIqKfsi9PBFRFXNyr
zzTxGpN6iQCgtFtySJQpIYamjxJhMZYpWcirYXRkXbmjPgc3PIgA9fRcPcj9bZ0A
Z1sM9yBDFjvZ7Eds9TW7iEsmxSpbXDAXDs6SjQa34QU8bh7qJEOG9AQYDcULXZQM
5NK7+b6GEMvcS7P4Y66CKwvxYFFGVGzX3CrFEgngkOWNJeoWhz5EpgmYIocgl/uS
1TPIWASdFLyvvtnGxMx9iWcdb4mGbXvMLbDa3luSBDWAtYxQe0w9iLi87eqypSvU
irZBSbKX737OlrOzNy/d4DKk9sbNsk/In1IK3jfHCt+7iRpVB5aWBfekrg0DIREw
Jo83ln1nApdusK11DLQpbZcUerLHsvmGXRlGJa3IRGAXUS7MAnJ+XpYkKgmjW8Vm
rw4QLfkMbP4=
=V3Wc
-----END PGP SIGNATURE-----
Merge tag 'sched-core-2026-06-14' of gitolite.kernel.org:pub/scm/linux/kernel/git/tip/tip
Pull scheduler updates from Ingo Molnar:
"SMP load-balancing updates:
- A large series to introduce infrastructure for cache-aware load
balancing, with the goal of co-locating tasks that share data
within the same Last Level Cache (LLC) domain. By improving cache
locality, the scheduler can reduce cache bouncing and cache misses,
ultimately improving data access efficiency.
Implemented by Chen Yu and Tim Chen, based on early prototype work
by Peter Zijlstra, with fixes by Jianyong Wu, Peter Zijlstra and
Shrikanth Hegde.
- A series to simplify CONFIG_SCHED_SMT ifdef usage (Shrikanth Hegde)
Fair scheduler updates:
- A series to improve SD_ASYM_CPUCAPACITY scheduling by introducing
SMT awareness (Andrea Righi, K Prateek Nayak)
- A series to optimize cfs_rq and sched_entity allocation for better
data locality (Zecheng Li)
- A preparatory series to change fair/cgroup scheduling to a single
runqueue, without the final change (Peter Zijlstra)
- Auto-manage ext/fair dl_server bandwidth (Andrea Righi)
- Fix cpu_util runnable_avg arithmetic (Hongyan Xia)
- Optimize update_tg_load_avg()'s rate-limiting code (Rik van Riel)
- Allow account_cfs_rq_runtime() to throttle current hierarchy
(K Prateek Nayak)
- Update util_est after updating util_avg during dequeue, to fix the
util signal update logic, which reduces signal noise (Vincent
Guittot)
Scheduler topology updates:
- Allow multiple domains to claim sched_domain_shared (K Prateek
Nayak)
- Add parameter to split LLC (Peter Zijlstra)
Core scheduler updates:
- Use trace_call__<tp>() to save a static branch (Gabriele Monaco)
Scheduler statistics updates:
- Drop now-stale mul_u64_u64_div_u64() cputime over-approximation
guard (Nicolas Pitre)
Deadline scheduler updates:
- Reject debugfs dl_server writes for offline CPUs (Andrea Righi)
- Fix replenishment logic for non-deferred servers (Yuri Andriaccio)
RT scheduling updates:
- Turn RT_PUSH_IPI default off for non PREEMPT_RT (Steven Rostedt)
- Update default bandwidth for real-time tasks to 1.0 (Yuri
Andriaccio)
Proxy scheduling updates:
- A series to implement Optimized Donor Migration for Proxy Execution
(John Stultz, Peter Zijlstra)
- Various proxy scheduling cleanups and fixes (Peter Zijlstra,
K Prateek Nayak)
Misc fixes, improvements and cleanups by Aaron Lu, Andrea Righi,
Zenghui Yu, Chen Yu, Guanyou.Chen, John Stultz, Shrikanth Hegde,
Peter Zijlstra, Liang Luo and Yiyang Chen"
* tag 'sched-core-2026-06-14' of gitolite.kernel.org:pub/scm/linux/kernel/git/tip/tip: (91 commits)
sched/fair: Fix newidle vs core-sched
sched/deadline: Use task_on_rq_migrating() helper
sched/core: Combine separate 'else' and 'if' statements
sched/fair: Fix cpu_util runnable_avg arithmetic
sched/fair: Unify cfs_rq throttling via account_cfs_rq_runtime()
sched/fair: Move the throttled tasks to a local list in tg_unthrottle_up()
sched/fair: Call update_curr() before unthrottling the hierarchy
sched/fair: Use throttled_csd_list for local unthrottle
sched/fair: Convert cfs bandwidth throttling to use guards
sched/fair: Allocate cfs_tg_state with percpu allocator
sched/fair: Remove task_group->se pointer array
sched/fair: Co-locate cfs_rq and sched_entity in cfs_tg_state
sched: restore timer_slack_ns when resetting RT policy on fork
MAINTAINERS: Fix spelling mistake in Peter's name
sched: Simplify ttwu_runnable()
sched/proxy: Remove superfluous clear_task_blocked_in()
sched/proxy: Remove PROXY_WAKING
sched/proxy: Switch proxy to use p->is_blocked
sched/proxy: Only return migrate when needed
sched: Be more strict about p->is_blocked
...
scan_dsq_pool() checked == 0 against scx_bpf_dsq_move_to_local(),
which returns true on success. This inverted success and failure,
causing peek_dsq_dispatch() to double-dispatch on success and skip
the real_dsq fallback on failure.
Signed-off-by: Cheng-Yang Chou <yphbchou0911@gmail.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
Extend the total_bw selftest to validate the fair/ext dl_server
auto-attach/detach operations.
After the existing consistency checks, the test now doubles the
fair_server's runtime on every CPU via debugfs and verifies that:
1. total_bw grew after the customization (proves fair_server was
attached and apply_params() honored the dl_bw_attached flag),
2. with the minimal BPF scheduler loaded, total_bw drops back to the
baseline value (proves fair_server was detached and ext_server was
attached at its own default runtime),
3. after unload total_bw matches the doubled value from step 1 (proves
fair_server was re-attached with the runtime customization preserved
across the load/unload cycle).
Signed-off-by: Andrea Righi <arighi@nvidia.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Acked-by: Juri Lelli <juri.lelli@redhat.com>
Link: https://patch.msgid.link/20260526164420.638711-3-arighi@nvidia.com
Pull to receive:
9a415cc537 ("sched_ext: Avoid UAF in scx_root_enable_workfn() init failure path")
Conflicts with for-7.2's scx_task_iter_relock() rework. The fix moves
put_task_struct(p) past scx_error(); for-7.2 still has it at the old
position. Resolved by dropping the old one.
Signed-off-by: Tejun Heo <tj@kernel.org>
Building the dequeue selftest with newer compilers (e.g., gcc 16)
triggers the following error:
dequeue.c:28:22: error: variable 'sum' set but not used
The 'volatile' qualifier prevents the writes from being optimized away,
but does not silence the unused variable 'sum' is indeed only written
and never read.
Consume 'sum' via an empty asm() with a register input constraint. This
forces the compiler to keep the accumulated value (preserving the CPU
stress loop) and avoiding the build error.
Fixes: 658ad2259b ("selftests/sched_ext: Add test to validate ops.dequeue() semantics")
Signed-off-by: Andrea Righi <arighi@nvidia.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
If run() exits early via SCX_EQ/SCX_ASSERT (which calls return
directly), bpf_link__destroy() is never reached and the BPF
scheduler stays loaded. All subsequent tests then fail to attach
because SCX is not in the DISABLED state.
Move bpf_link into a context struct so cleanup() always destroys
it, regardless of how run() exits. Also skip waitpid() for children
where fork() returned -1, avoiding waitpid(-1,...) accidentally
reaping an unrelated child and triggering the early return path.
Signed-off-by: Cheng-Yang Chou <yphbchou0911@gmail.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
In scx-cid patchsets, sched_ext selftest failed to build with following
error:
non_scx_kfunc_deny.bpf.c:17:6: error: conflicting types for 'scx_bpf_kick_cpu'
17 | void scx_bpf_kick_cpu(s32 cpu, u64 flags) __ksym;
| ^
tools/testing/selftests/sched_ext/build/include/vmlinux.h:136300:13: note: previous declaration is here
136300 | extern void scx_bpf_kick_cpu(s32 cpu, u64 flags, const struct bpf_prog_aux *aux) __weak __ksym;
| ^
non_scx_kfunc_deny.bpf.c:26:23: error: too few arguments to function call, expected 3, have 2
26 | scx_bpf_kick_cpu(0, 0);
| ~~~~~~~~~~~~~~~~ ^
tools/testing/selftests/sched_ext/build/include/vmlinux.h:136300:13: note: 'scx_bpf_kick_cpu' declared here
136300 | extern void scx_bpf_kick_cpu(s32 cpu, u64 flags, const struct bpf_prog_aux *aux) __weak __ksym;
The root cause is on scx core part, but we can avoid this by including
common.bpf.h and remove scx_bpf_kick_cpu() to make it more robust, just
like the usage in other xx.bpf.c.
Link: https://lore.kernel.org/sched-ext/20260421071945.3110084-1-tj@kernel.org/
Signed-off-by: Zhao Mengmeng <zhaomengmeng@kylinos.cn>
Tested-by: Cheng-Yang Chou <yphbchou0911@gmail.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
Verify that the BPF verifier rejects a non-SCX struct_ops program
(tcp_congestion_ops) that attempts to call an SCX kfunc (scx_bpf_kick_cpu).
The test expects the load to fail with -EACCES from scx_kfunc_context_filter.
Signed-off-by: Cheng-Yang Chou <yphbchou0911@gmail.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
The error path after scx_bpf_create_dsq(real_dsq_id, ...) was reporting
test_dsq_id instead of real_dsq_id in the error message, which would
mislead debugging.
Signed-off-by: fangqiurong <fangqiurong@kylinos.cn>
Signed-off-by: Tejun Heo <tj@kernel.org>
Report an error for './runner foo' (positional arg instead of -t) and
for './runner -t foo' when the filter matches no tests. Previously both
cases produced no error output.
Pre-scan the test list before the main loop so the error is reported
immediately, avoiding spurious SKIP output from '-s' when no tests
match.
Signed-off-by: Cheng-Yang Chou <yphbchou0911@gmail.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
Conflict in kernel/sched/ext.c init_sched_ext_class() between:
415cb193bb ("sched_ext: Fix SCX_KICK_WAIT deadlock by deferring wait
to balance callback")
which adds cpus_to_sync cpumask allocation, and:
84b1a0ea0b ("sched_ext: Implement scx_bpf_dsq_reenq() for user DSQs")
8c1b9453fd ("sched_ext: Convert deferred_reenq_locals from llist to
regular list")
which add deferred_reenq init code at the same location. Both are
independent additions. Include both.
Signed-off-by: Tejun Heo <tj@kernel.org>
Add a test that creates a 3-CPU kick_wait cycle (A->B->C->A). A BPF
scheduler kicks the next CPU in the ring with SCX_KICK_WAIT on every
enqueue while userspace workers generate continuous scheduling churn via
sched_yield(). Without the preceding fix, this hangs the machine within seconds.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Christian Loehle <christian.loehle@arm.com>
Tested-by: Christian Loehle <christian.loehle@arm.com>
This reverts commit c50dcf5331.
The tests are superficial, likely AI-generated slop, and flaky. They
don't add actual value and just churn the selftests.
Signed-off-by: Tejun Heo <tj@kernel.org>
rt_stall tests the ext DL server which was introduced in commit
cd959a3562 ("sched_ext: Add a DL server for sched_ext tasks").
On older kernels that lack this feature, the test calls ksft_exit_fail()
internally which terminates the entire runner process, preventing
subsequent tests from running.
Add a guard in setup() that checks for the ext_server field in struct rq
via __COMPAT_struct_has_field() and returns SCX_TEST_SKIP if not present.
Also print the names of skipped tests in the results summary, mirroring
the existing behavior for failed tests.
Signed-off-by: Cheng-Yang Chou <yphbchou0911@gmail.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
Add three selftests covering features introduced in v7.1:
- dsq_reenq: Verify scx_bpf_dsq_reenq() on user DSQs triggers
ops.enqueue() with SCX_ENQ_REENQ and SCX_TASK_REENQ_KFUNC in
p->scx.flags.
- enq_immed: Verify SCX_OPS_ALWAYS_ENQ_IMMED slow path where tasks
dispatched to a busy CPU's local DSQ are re-enqueued through
ops.enqueue() with SCX_TASK_REENQ_IMMED.
- consume_immed: Verify SCX_ENQ_IMMED via the consume path using
scx_bpf_dsq_move_to_local___v2() with explicit SCX_ENQ_IMMED.
All three tests skip gracefully on kernels that predate the required
features by checking availability via __COMPAT_has_ksym() /
__COMPAT_read_enum() before loading.
Signed-off-by: zhidao su <suzhidao@xiaomi.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
runner.c always returned 0 regardless of test results. The kselftest
framework (tools/testing/selftests/kselftest/runner.sh) invokes the runner
binary and treats a non-zero exit code as a test failure; with the old
code, failed sched_ext tests were silently hidden from the parent harness
even though individual "not ok" TAP lines were emitted.
Return 1 when at least one test failed, 0 when all tests passed or were
skipped.
Signed-off-by: zhidao su <suzhidao@xiaomi.com>
Acked-by: Andrea Righi <arighi@nvidia.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
When tests fail, the runner only printed the failure count, making
it hard to tell which tests failed without scrolling through output.
Track failed test names in an array and print them after the summary
so failures are immediately visible at the end of the run.
Signed-off-by: Cheng-Yang Chou <yphbchou0911@gmail.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
ops.cpu_acquire/release() are deprecated by commit a3f5d48222
("sched_ext: Allow scx_bpf_reenqueue_local() to be called from
anywhere") in favor of handling CPU preemption via the sched_switch
tracepoint.
In the maximal selftest, replace the cpu_acquire/release stubs with a
minimal sched_switch TP program. Attach all non-struct_ops programs
(including the new TP) via maximal__attach() after disabling auto-attach
for the maximal_ops struct_ops map, which is managed manually in run().
Apply the same fix to reload_loop, which also uses the maximal skeleton.
Signed-off-by: Cheng-Yang Chou <yphbchou0911@gmail.com>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
Direct writes to p->scx.dsq_vtime are deprecated in favor of
scx_bpf_task_set_dsq_vtime(). Update scx_simple, scx_flatcg, and
select_cpu_vtime selftest to use the new kfunc with
scale_by_task_weight_inverse().
Signed-off-by: Cheng-Yang Chou <yphbchou0911@gmail.com>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
Fix two mismatched closing comments in header include guards:
- util.h: closing comment says __SCX_TEST_H__ but the guard is
__SCX_TEST_UTIL_H__
- exit_test.h: closing comment has a spurious '#' character before
the guard name
Signed-off-by: Cheng-Yang Chou <yphbchou0911@gmail.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
After commit 860683763e ("sched_ext: Add enq_flags to
scx_bpf_dsq_move_to_local()") some of the kselftests are failing to
build:
exit.bpf.c:44:34: error: too few arguments provided to function-like macro invocation
44 | scx_bpf_dsq_move_to_local(DSQ_ID);
Update the kselftests adding the new argument to
scx_bpf_dsq_move_to_local().
Fixes: 860683763e ("sched_ext: Add enq_flags to scx_bpf_dsq_move_to_local()")
Signed-off-by: Andrea Righi <arighi@nvidia.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
exit__load(skel) was called without checking its return value.
Every other test in the suite wraps the load call with
SCX_FAIL_IF(). Add the missing check to be consistent with the
rest of the test suite.
Fixes: a5db7817af ("sched_ext: Add selftests")
Signed-off-by: David Carlier <devnexen@gmail.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
Several demo schedulers and the selftest runner had usage strings
that omitted options which are actually supported:
- scx_central: add missing [-v]
- scx_pair: add missing [-v]
- scx_qmap: add missing [-S] and [-H]
- scx_userland: add missing [-v]
- scx_sdt: remove [-f] which no longer exists
- runner.c: add missing [-s], [-l], [-q]; drop [-h] which none of the
other sched_ext tools list in their usage lines
Suggested-by: Tejun Heo <tj@kernel.org>
Signed-off-by: Cheng-Yang Chou <yphbchou0911@gmail.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
Use %ld (not %lu) for signed long, and pass the actual string length
returned by sprintf() to write_text() instead of sizeof(buf).
Signed-off-by: Cheng-Yang Chou <yphbchou0911@gmail.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
When compiling sched_ext selftests using clang 17.0.6, it raised
compiler crash and build error:
Error at line 68: Unsupport signed division for DAG: 0x55b2f9a60240:
i64 = sdiv 0x55b2f9a609b0, Constant:i64<100>, peek_dsq.bpf.c:68:25 @[
peek_dsq.bpf.c:95:4 @[ peek_dsq.bpf.c:169:8 @[ peek
_dsq.bpf.c:140:6 ] ] ]Please convert to unsigned div/mod
After digging, it's not a compiler error, clang supported Signed division
only when using -mcpu=v4, while we use -mcpu=v3 currently, the better way
is to use unsigned div, see [1] for details.
[1] https://github.com/llvm/llvm-project/issues/70433
Signed-off-by: Zhao Mengmeng <zhaomengmeng@kylinos.cn>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
Similar to commit 835a507535 ("selftests/bpf: Add -fms-extensions to
bpf build flags") and commit 639f58a0f4 ("bpftool: Fix build warnings
due to MS extensions")
Fix "declaration does not declare anything" warning by using
-fms-extensions and -Wno-microsoft-anon-tag flags to build bpf programs
that #include "vmlinux.h"
Signed-off-by: Zhao Mengmeng <zhaomengmeng@kylinos.cn>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
Add a new kselftest to validate that the new ops.dequeue() semantics
work correctly for all task lifecycle scenarios, including the
distinction between terminal DSQs (where BPF scheduler is done with the
task), user DSQs (where BPF scheduler manages the task lifecycle) and
BPF data structures, regardless of which event performs the dispatch.
The test validates the following scenarios:
- From ops.select_cpu():
- scenario 0 (local DSQ): tasks dispatched to the local DSQ bypass
the BPF scheduler entirely; they never enter BPF custody, so
ops.dequeue() is not called,
- scenario 1 (global DSQ): tasks dispatched to SCX_DSQ_GLOBAL also
bypass the BPF scheduler, like the local DSQ; ops.dequeue() is
not called,
- scenario 2 (user DSQ): tasks dispatched to user DSQs from
ops.select_cpu(): tasks enter BPF scheduler's custody with full
enqueue/dequeue lifecycle tracking and state machine validation,
expects 1:1 enqueue/dequeue pairing,
- From ops.enqueue():
- scenario 3 (local DSQ): same behavior as scenario 0,
- scenario 4 (global DSQ): same behavior as scenario 1,
- scenario 5 (user DSQ): same behavior as scenario 2,
- scenario 6 (BPF internal queue): tasks are stored in a BPF queue
from ops.enqueue() and consumed from ops.dispatch(); similarly to
scenario 5, tasks enter BPF scheduler's custody with full
lifecycle tracking and 1:1 enqueue/dequeue validation.
This verifies that:
- terminal DSQ dispatch (local, global) don't trigger ops.dequeue(),
- tasks dispatched to user DSQs, either from ops.select_cpu() or
ops.enqueue(), enter BPF scheduler's custody and have exact 1:1
enqueue/dequeue pairing,
- tasks stored to internal BPF data structures from ops.enqueue() enter
BPF scheduler's custody and have exact 1:1 enqueue/dequeue pairing,
- dispatch dequeues have no flags (normal workflow),
- property change dequeues have the %SCX_DEQ_SCHED_CHANGE flag set,
- no duplicate enqueues or invalid state transitions are happening.
Cc: Tejun Heo <tj@kernel.org>
Cc: Emil Tsalapatis <emil@etsalapatis.com>
Cc: Kuba Piecuch <jpiecuch@google.com>
Reviewed-by: Daniel Jordan <daniel.m.jordan@oracle.com>
Signed-off-by: Andrea Righi <arighi@nvidia.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
The header <unistd.h> is included twice in rt_stall.c. Remove the
redundant inclusion to clean up the code.
Signed-off-by: Cheng-Yang Chou <yphbchou0911@gmail.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
The read() call in run_test() triggers a warn_unused_result compiler
warning, which breaks the build under -Werror.
Check the return value of read() and exit the child process on failure to
satisfy the compiler and handle pipe read errors.
Reviewed-by: Andrea Righi <arighi@nvidia.com>
Signed-off-by: Cheng-Yang Chou <yphbchou0911@gmail.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
The runner sets exit_req on SIGINT/SIGTERM but ignores it during the
main loop. This prevents users from cleanly interrupting a test run.
Check exit_req each iteration to safely break out on exit signals.
Signed-off-by: Cheng-Yang Chou <yphbchou0911@gmail.com>
Acked-by: Andrea Righi <arighi@nvidia.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
The rt_stall test measures the runtime ratio between an EXT and an RT
task pinned to the same CPU, verifying that the deadline server prevents
RT tasks from starving SCHED_EXT tasks. It expects the EXT task to get
at least 4% of CPU time.
The test is flaky because sched_stress_test() calls sleep(RUN_TIME)
immediately after fork(), without waiting for the RT child to complete
its setup (set_affinity + set_sched). If the RT child experiences
scheduling latency before completing setup, that delay eats into the
measurement window: the RT child runs for less than RUN_TIME seconds,
and the EXT task's measured ratio drops below the 4% threshold.
For example, in the failing CI run [1]:
EXT=0.140s RT=4.750s total=4.890s (expected ~5.0s)
ratio=2.86% < 4% → FAIL
The 110ms gap (5.0 - 4.89) corresponds to the RT child's setup time
being counted inside the measurement window, during which fewer
deadline server ticks fire for the EXT task.
Fix by using pipes to synchronize: each child signals the parent after
completing its setup, and the parent waits for both signals before
starting sleep(RUN_TIME). This ensures the measurement window only
counts time when both tasks are fully configured and competing.
[1] https://github.com/kernel-patches/bpf/actions/runs/21961895809/job/63442490449
Fixes: be621a7634 ("selftests/sched_ext: Add test for sched_ext dl_server")
Assisted-by: claude-opus-4-6-v1
Signed-off-by: Ihor Solodrai <ihor.solodrai@linux.dev>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
- Move C example schedulers back from the external scx repo to
tools/sched_ext as the authoritative source. scx_userland and scx_pair
are returning while scx_sdt (BPF arena-based task data management) is
new. These schedulers will be dropped from the external repo.
- Improve error reporting by adding scx_bpf_error() calls when DSQ
creation fails across all in-tree schedulers.
- Avoid redundant irq_work_queue() calls in destroy_dsq() by only
queueing when llist_add() indicates an empty list.
- Fix flaky init_enable_count selftest by properly synchronizing
pre-forked children using a pipe instead of sleep().
-----BEGIN PGP SIGNATURE-----
iIQEABYKACwWIQTfIjM1kS57o3GsC/uxYfJx3gVYGQUCaYo1pQ4cdGpAa2VybmVs
Lm9yZwAKCRCxYfJx3gVYGa5VAP9udiGksQ4bBFQrUD+0yhuvjsSuXzssfdfxHgT6
Hj66wgEAjgbnSyxfcGrB+w7DWUxNLaZlXepibVMPcfvAaSieSgU=
=UvLY
-----END PGP SIGNATURE-----
Merge tag 'sched_ext-for-6.20' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/sched_ext
Pull sched_ext updates from Tejun Heo:
- Move C example schedulers back from the external scx repo to
tools/sched_ext as the authoritative source. scx_userland and
scx_pair are returning while scx_sdt (BPF arena-based task data
management) is new. These schedulers will be dropped from the
external repo.
- Improve error reporting by adding scx_bpf_error() calls when DSQ
creation fails across all in-tree schedulers
- Avoid redundant irq_work_queue() calls in destroy_dsq() by only
queueing when llist_add() indicates an empty list
- Fix flaky init_enable_count selftest by properly synchronizing
pre-forked children using a pipe instead of sleep()
* tag 'sched_ext-for-6.20' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/sched_ext:
selftests/sched_ext: Fix init_enable_count flakiness
tools/sched_ext: Fix data header access during free in scx_sdt
tools/sched_ext: Add error logging for dsq creation failures in remaining schedulers
tools/sched_ext: add arena based scheduler
tools/sched_ext: add scx_pair scheduler
tools/sched_ext: add scx_userland scheduler
sched_ext: Add error logging for dsq creation failures
sched_ext: Avoid multiple irq_work_queue() calls in destroy_dsq()
Add a new kselftest to verify that the total_bw value in
/sys/kernel/debug/sched/debug remains consistent across all CPUs
under different sched_ext BPF program states:
1. Before a BPF scheduler is loaded
2. While a BPF scheduler is loaded and active
3. After a BPF scheduler is unloaded
The test runs CPU stress threads to ensure DL server bandwidth
values stabilize before checking consistency. This helps catch
potential issues with DL server bandwidth accounting during
sched_ext transitions.
Co-developed-by: Andrea Righi <arighi@nvidia.com>
Signed-off-by: Andrea Righi <arighi@nvidia.com>
Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Tested-by: Christian Loehle <christian.loehle@arm.com>
Link: https://patch.msgid.link/20260126100050.3854740-8-arighi@nvidia.com
Add a selftest to validate the correct behavior of the deadline server
for the ext_sched_class.
Co-developed-by: Joel Fernandes <joelagnelf@nvidia.com>
Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com>
Signed-off-by: Andrea Righi <arighi@nvidia.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Emil Tsalapatis <emil@etsalapatis.com>
Tested-by: Christian Loehle <christian.loehle@arm.com>
Link: https://patch.msgid.link/20260126100050.3854740-7-arighi@nvidia.com
The init_enable_count test is flaky. The test forks 1024 children before
attaching the scheduler to verify that existing tasks get ops.init_task()
called. The children were using sleep(1) before exiting.
7900aa699c ("sched_ext: Fix cgroup exit ordering by moving sched_ext_free()
to finish_task_switch()") changed when tasks are removed from scx_tasks -
previously when the task_struct was freed, now immediately in
finish_task_switch() when the task dies.
Before the commit, pre-forked children would linger on scx_tasks until freed
regardless of when they exited, so the scheduler would always see them during
iteration. The sleep(1) was unnecessary. After the commit, children are
removed as soon as they die. The sleep(1) masks the problem in most cases but
the test becomes flaky depending on timing.
Fix by synchronizing properly using a pipe. All children block on read() and
the parent signals them to exit by closing the write end after attaching the
scheduler. The children are auto-reaped so there's no need to wait on them.
Reported-by: Ihor Solodrai <ihor.solodrai@linux.dev>
Cc: David Vernet <void@manifault.com>
Cc: Andrea Righi <arighi@nvidia.com>
Cc: Changwoo Min <changwoo@igalia.com>
Cc: Emil Tsalapatis <emil@etsalapatis.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
The sched_ext selftests runner runs each test in the same process,
with each test possibly forking multiple times. When the main runner
has not flushed its stdout, the children inherit the buffered output
for previous tests and emit it during exit. This causes log spam.
Make sure stdout/stderr is fully flushed before each test.
Cc: Ihor Solodrai <ihor.solodrai@linux.dev>
Signed-off-by: Emil Tsalapatis <emil@etsalapatis.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
This commit adds two tests. The first is the most basic unit test:
make sure an empty queue peeks as empty, and when we put one element
in the queue, make sure peek returns that element.
However, even this simple test is a little complicated by the different
behavior of scx_bpf_dsq_insert in different calling contexts:
- insert is for direct dispatch in enqueue
- insert is delayed when called from select_cpu
In this case we split the insert and the peek that verifies the
result between enqueue/dispatch.
Note: An alternative would be to call `scx_bpf_dsq_move_to_local` on an
empty queue, which in turn calls `flush_dispatch_buf`, in order to flush
the buffered insert. Unfortunately, this is not viable within the
enqueue path, as it attempts a voluntary context switch within an RCU
read-side critical section.
The second test is a stress test that performs many peeks on all DSQs
and records the observed tasks.
Signed-off-by: Ryan Newton <newton@meta.com>
Reviewed-by: Christian Loehle <christian.loehle@arm.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
