github-mirror/linux
1
0
Fork 0
mirror of https://github.com/torvalds/linux.git synced 2026-10-22 23:01:49 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity Graph

Scheduler updates:

Browse Source 
Kconfig updates:
 
  - Simplify PREEMPT_DYNAMIC (Mark Rutland)
 
 Fair scheduler updates:
 
  - Remove redundant RUN_TO_PARITY check (Elif Topuz)
 
  - Declare augmented callbacks per field, fix rb_add_augmented_cached()
    descent (Yiwei Lin, Peter Zijlstra)
 
  - Restart fair hrtick after same-task repicks (Shubhang Kaushik)
 
  - Improve the scheduling latency of short slice tasks (Vincent Guittot)
 
 Deadline/RT scheduler updates:
 
  - Check start_dl_timer expiry with ktime_before()
    (Liang Hao)
 
  - irq_work: CPU-hotplug improvements on PREEMPT_RT (Sebastian Andrzej
    Siewior)
 
 Energy-aware scheduling updates:
 
  - Implement the steal_governor, introduce preferred CPUs and
    steal-driven vCPU backoff (Shrikanth Hegde)
 
 Multi-processor load-balancing updates:
 
  - Enable preferred SMT siblings on NVIDIA Olympus (Andrea Righi)
 
  - Randomize equally shallow idle CPU picks (Christian Loehle)
 
  - Remove redundant core_sched_seq (Hui Su)
 
 Cache-aware scheduling updates:
 
  - Avoid publishing partial LLC utilization (Jemmy Wong)
 
 NUMA-balancing updates:
 
  - Reset NUMA fault locality after scan period update (Eric Kim)
 
 Proxy execution updates:
 
  - Ongoing work to implement sleeping owner handling for proxy
    execution (John Stultz, Vasily Gorbik)
 
  - Preparatory steps to make sched_ext compatible with proxy
    execution (Andrea Righi)
 
 Scheduler infrastructure updates & fixes:
 
  - Remove the RCU wait from PSI trigger destruction (Shakeel Butt)
 
  - Avoid modifying wait-entry on failed priority registration
    (Hui Su)
 
  - Use the new static key API for sched_feat (Hongyan Xia)
 
  - Convert paravirt_steal to new static key APIs (Hongyan Xia)
 
  - cputime: Don't account idle time twice after dyntick-idle
    (Stian Halseth)
 
 Scheduler debugging updates:
 
  - Set TIF_NEED_RESCHED before calling __trace_set_need_resched()
    (Sechang Lim)
 
  - Fix context analysis errors in non-preferred CPU push (Shrikanth
    Hegde)
 
 Misc cleanups and fixes by Christian Loehle, Shubhang Kaushik,
 Thomas Huth and Vladimir Zapolskiy.
 
 Signed-off-by: Ingo Molnar <mingo@kernel.org>
 -----BEGIN PGP SIGNATURE-----
 
 iQJFBAABCgAvFiEEBpT5eoXrXCwVQwEKEnMQ0APhK1gFAmrUomIRHG1pbmdvQGtl
 cm5lbC5vcmcACgkQEnMQ0APhK1ik+Q/+Jx0tXmwOm/wk1ba50zz/tkUL17APJzD1
 0TdHSf0sqssyVnQVmGBNWoKfp30zzrnd8Yf46PH5kUmyIxX0LqazAXsPA/0lxAY6
 jB2b5ZGosyQQPqyjM1q48K/BMB+GGO9xDKlFxxkBsICi+vS+6diBzXM70xla9M4O
 CcAO5DHb50otN/TljfnkmbxG7vgssCVZyS7XG6bSyhd6tUrcr4DVD7gYOMbSVcaL
 8Cgu+b54rOAcrcWU1454YKHI5etUOTjblCbFgISrdw7lwf4dKVtTAKSanFX9n522
 q9sQ1vzuRz6JfSak9nxFkCy5SMjuHJ53DkVNR4eUXNmM0tHdAcVd8jOexmQVBrug
 Yjloo/WydUqGT2TYYEVQPi4J+jMqMRyDIp3TL9UuQjwcjHRnUpd8EVmoNme0TOi+
 e3SoS+xOSVBLgbMFwrjJR/tQuqDPjgBrgIyg8wL+ed5cY6Equ+J3467wp8lO7RBN
 uGwChvxQAOaMlkh5tIR38Zjq9kzs79y85vz3uD2SNk17TcvijVw9UsmQKfAkpdZo
 5FUDM2AdyOhOtZkzMNLHfjjBLSMVCBRo726c6WbELibhuWN7xTAJSJx8o8JNOCrC
 vOBcNNAHpU7BwqA0gFIA3lL1uMXvKn9npKKjnB2C4WxhFGf23NF6uPapyA4sMZ00
 YwRGvOeDvu4=
 =cPCx
 -----END PGP SIGNATURE-----

Merge tag 'sched-core-2026-10-18' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip

Pull scheduler updates from Ingo Molnar:
 "Kconfig updates:

   - Simplify PREEMPT_DYNAMIC (Mark Rutland)

  Fair scheduler updates:

   - Remove redundant RUN_TO_PARITY check (Elif Topuz)

   - Declare augmented callbacks per field, fix
     rb_add_augmented_cached() descent (Yiwei Lin, Peter Zijlstra)

   - Restart fair hrtick after same-task repicks (Shubhang Kaushik)

   - Improve the scheduling latency of short slice tasks (Vincent
     Guittot)

  Deadline/RT scheduler updates:

   - Check start_dl_timer expiry with ktime_before() (Liang Hao)

   - irq_work: CPU-hotplug improvements on PREEMPT_RT (Sebastian Andrzej
     Siewior)

  Energy-aware scheduling updates:

   - Implement the steal_governor, introduce preferred CPUs and
     steal-driven vCPU backoff (Shrikanth Hegde)

  Multi-processor load-balancing updates:

   - Enable preferred SMT siblings on NVIDIA Olympus (Andrea Righi)

   - Randomize equally shallow idle CPU picks (Christian Loehle)

   - Remove redundant core_sched_seq (Hui Su)

  Cache-aware scheduling updates:

   - Avoid publishing partial LLC utilization (Jemmy Wong)

  NUMA-balancing updates:

   - Reset NUMA fault locality after scan period update (Eric Kim)

  Proxy execution updates:

   - Ongoing work to implement sleeping owner handling for proxy
     execution (John Stultz, Vasily Gorbik)

   - Preparatory steps to make sched_ext compatible with proxy execution
     (Andrea Righi)

  Scheduler infrastructure updates & fixes:

   - Remove the RCU wait from PSI trigger destruction (Shakeel Butt)

   - Avoid modifying wait-entry on failed priority registration
     (Hui Su)

   - Use the new static key API for sched_feat (Hongyan Xia)

   - Convert paravirt_steal to new static key APIs (Hongyan Xia)

   - cputime: Don't account idle time twice after dyntick-idle
     (Stian Halseth)

  Scheduler debugging updates:

   - Set TIF_NEED_RESCHED before calling __trace_set_need_resched()
     (Sechang Lim)

   - Fix context analysis errors in non-preferred CPU push (Shrikanth
     Hegde)

  Misc cleanups and fixes by Christian Loehle, Shubhang Kaushik,
  Thomas Huth and Vladimir Zapolskiy"

* tag 'sched-core-2026-10-18' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip: (64 commits)
  sched: Replace nr_iowait_cpu() with a bool accessor
  Revert "sched/fair: Remove dead code on enqueue_task_fair()"
  sched/cache: Avoid publishing partial LLC utilization
  sched/cputime: Don't account idle time twice after dyntick-idle
  sched/psi: Remove the RCU wait from trigger destruction
  sched/eevdf: Remove redundant RUN_TO_PARITY check
  sched/wait: Avoid modifying entry on failed priority registration
  rbtree_test: Use rb_add_augmented_cached() for the cached augmented test
  rbtree: Update augmented data on the way down in rb_add_augmented_cached()
  rbtree: Declare augmented callbacks per field with RB_AUG()
  rbtree_test: Use rb_add() and rb_add_cached() for the basic tests
  smpboot: Don't park the thread if work is pending
  irq_work: Flush lazy work CPU down on PREEMPT_RT
  irq_work: Update a comment regarding CPU hotplug invocation
  sched/topology: Add asymmetric SMT packing override
  sched/fair: Honor asymmetric SMT priority in idle selection
  sched: Set TIF_NEED_RESCHED before calling __trace_set_need_resched()
  sched/core: Fix context analysis errors in non-preferred CPU push
  virt/steal_governor: Enable the driver
  virt/steal_governor: Implement steal_governor policy loop
  ...
This commit is contained in:
Linus Torvalds 2026-10-20 12:35:35 -10:00
commit cb95904b47
69 changed files with 1514 additions and 816 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

14
Documentation/ABI/testing/sysfs-devices-system-cpu
View File

@ -812,3 +812,17 @@ Date: Nov 2022
Contact: Linux kernel mailing list <linux-kernel@vger.kernel.org>
Description:
(RO) the list of CPUs that can be brought online.
What: /sys/devices/system/cpu/preferred
Date: Sep 2026
Contact: Linux kernel mailing list <linux-kernel@vger.kernel.org>
Description:
(RO) the list of preferred CPUs applicable in
paravirtualized environments.
The steal governor driver dynamically adjusts this mask
based on observed steal time. Scheduling tasks on
CPUs outside of this list may lead to performance
degradations due to underlying physical CPU contention.
See Documentation/scheduler/sched-paravirt.rst for more details.

12
Documentation/admin-guide/kernel-parameters.txt
View File

@ -6801,6 +6801,18 @@ Kernel parameters
solution to mutex-based priority inversion.
Format: <bool>
sched_smt_asym_packing= [KNL,SMP]
Format: on
Force asymmetric packing at the SMT scheduling domain.
Idle CPU selection prefers siblings with a higher
arch_asym_cpu_priority(). The default implementation
prefers lower-numbered logical CPUs, but architecture
overrides remain authoritative. Equal priorities do
not establish a sibling preference. For example, x86
with CONFIG_SCHED_MC_PRIO normally assigns the same
core priority to both SMT siblings, so this option
does not favor either sibling.
sched_verbose [KNL,EARLY] Enables verbose scheduler debug messages.
schedstats= [KNL,X86] Enable or disable scheduled statistics.

27
Documentation/core-api/rbtree.rst
View File

@ -257,8 +257,14 @@ When erasing a node, the user must call rb_erase_augmented() instead of
rb_erase(). rb_erase_augmented() calls back into user provided functions
to update the augmented information on affected subtrees.
In both cases, the callbacks are provided through struct rb_augment_callbacks.
3 callbacks must be defined:
Alternatively, rb_add_augmented_cached() performs the whole insertion into
a cached tree: it walks down to the insertion point, merges the new node's
augmented information into every node on that path, links the node and
rebalances. The new node's augmented information must already describe
the node alone when it is called.
In all cases, the callbacks are provided through struct rb_augment_callbacks.
3 callbacks must be defined, plus a fourth one for rb_add_augmented_cached():
- A propagation callback, which updates the augmented value for a given
node and its ancestors, up to a given stop point (or NULL to update
@ -271,6 +277,10 @@ In both cases, the callbacks are provided through struct rb_augment_callbacks.
subtree to a newly assigned subtree root AND recomputes the augmented
information for the former subtree root.
- A merge callback, which folds the augmented value of a node being inserted
into the augmented value of one of its future ancestors. It is only used
by rb_add_augmented_cached().
The compiled code for rb_erase_augmented() may inline the propagation and
copy callbacks, which results in a large function, so each augmented rbtree
user should have a single rb_erase_augmented() call site in order to limit
@ -395,8 +405,19 @@ Insertion/removal are defined using the following augmented callbacks::
old->__subtree_last = compute_subtree_last(old);
}
static void augment_merge(struct rb_node *rb, struct rb_node *rb_new)
{
struct interval_tree_node *node =
rb_entry(rb, struct interval_tree_node, rb);
struct interval_tree_node *new =
rb_entry(rb_new, struct interval_tree_node, rb);
if (node->__subtree_last < new->__subtree_last)
node->__subtree_last = new->__subtree_last;
}
static const struct rb_augment_callbacks augment_callbacks = {
augment_propagate, augment_copy, augment_rotate
augment_propagate, augment_copy, augment_rotate, augment_merge
};
void interval_tree_insert(struct interval_tree_node *node,

1
Documentation/driver-api/index.rst
View File

@ -139,6 +139,7 @@ Subsystem-specific APIs
sm501
soundwire/index
spi
steal-governor
surface_aggregator/index
switchtec
sync_file

151
Documentation/driver-api/steal-governor.rst Normal file
View File

@ -0,0 +1,151 @@
.. SPDX-License-Identifier: GPL-2.0
Steal Governor
==============
:Author: Shrikanth Hegde <sshegde@linux.ibm.com>
Introduction
============
The steal governor is aimed at mitigating the Noisy Neighbour problem
which occurs in paravirtualized environments with CPU overcommit.
The performance of a workload running in one VM gets degraded by
the activity of other VMs on the same host. As a result, all VMs
collectively make slower forward progress.
In such systems, high utilization in all VMs causes the hypervisor to
frequently preempt vCPUs. This vCPU preemption is expensive.
To mitigate this, the kernel aims to restrict workloads to a subset of
Preferred CPUs to reduce physical CPU contention.
A detailed explanation of Preferred CPUs is available in
``Documentation/scheduler/sched-paravirt.rst``.
The steal governor selects ``CONFIG_PREFERRED_CPU=y`` which enables the
scheduler core infrastructure to move the tasks to Preferred CPUs where
possible. The driver controls the policy decisions regarding the state of
preferred CPUs. That is, this driver decides which CPUs are preferred
and which CPUs are non-preferred.
The driver code is available at ``drivers/virt/steal_governor.c``.
Core idea
=========
steal time is an indication available today in Guest which shows contention
for underlying physical CPU. Use it as a hint in the guest to fold the
workload to a reduced set of vCPUs. When there is contention, steal time
will show up in all the guests. When each guest honors the hint and folds
the workload to a smaller set of vCPUs (Preferred CPUs), it reduces the
contention and thereby reduces vCPU preemption.
This is achieved without any cross-guest communication.
Steal governor driver effectively does:
1. Periodically computes the steal ratio using accumulated steal time
across possible CPUs, normalized by the number of active CPUs.
2. If steal ratio is greater than high threshold, reduce the number of
preferred CPUs by 1 core. Ensure at least one core is left always.
Skip changing the state of offline CPUs in that core.
3. If steal ratio is less than or equal to low threshold, increase the
number of preferred CPUs by 1 core. If preferred is same as active,
nothing to be done. Skip changing the state of offline CPUs.
This helps to handle cases where few CPUs are offline in a core and
those offline CPUs will not be marked as preferred.
4. Ensure preferred CPUs is always subset of active CPUs.
On feature disable it is same as active CPUs.
This feature works best only when all the VMs enable the feature as
it is a co-operative scheme. If a specific VM doesn't enable this feature
it may end up with more CPUs than others, still should lead to better
performance when seen from system view. Those who enable this driver must
ensure it is enabled in all VMs.
Note that this driver is strictly intended for actual guests; for example,
loading this module in a privileged VM like Xen Dom0 is blocked.
Workload considerations
=======================
The steal governor is useful for workloads where vCPU preemption has
costs beyond the lost CPU time, such as lock-holder preemption, critical
sections, communicating threads, and cache or TLB disruption.
Pure CPU-time workloads with independent workers may not benefit and
could see a small regression due to additional guest scheduling overhead.
Module Parameters
=================
interval_ms
-----------
How often steal governor checks for steal time.
Default: 1000 i.e. 1 second. Value should be in between 100ms to 100sec.
This controls how fast steal governor driver reacts to changes to the
contention of physical CPUs. Since it does a fair amount of work, setting
too low may have overhead. Setting it too high might render it ineffective.
low_threshold
-------------
lower threshold value in percentage * 100.
Default: 200, i.e. 2% steal is considered as low threshold.
Can't be higher than high_threshold.
This determines what values should be considered as nil/no steal values.
When steal governor sees steal ratio is less than or equal to this value,
it will increase the preferred CPUs by 1 core.
Using zero might cause oscillations.
high_threshold
--------------
higher threshold value in percentage * 100
Default: 500, i.e. 5% steal is considered as high threshold.
Can't be lower than low_threshold. Must be less than 10000.
This determines what values should be considered as high steal values.
When steal governor sees steal ratio is higher than this value, it will
reduce the preferred CPUs by 1 core.
Limitations of default values
-----------------------------
Because of the vast diversity in VM configurations and different
architectures, the default thresholds may not be optimal for all systems.
Users may need to tune these parameters based on the system under
test to achieve the best results.
The governor sums the steal time across all possible CPUs, which ensures
the accumulated steal time remains a monotonically increasing value.
However, to calculate the effective steal ratio, it divides this sum
by the number of active CPUs. Because only active CPUs contribute to
the steal time delta, this prevents threshold dilution on sparsely
populated systems.
The driver reduces/increases preferred CPUs by core-level. This could provide
faster convergence for hypervisors such as powerVM. But on KVM and Xen
convergence could be slower depending on the configuration.
Using a smaller interval_ms could help one to expedite it.
Reasons for CONFIG_STEAL_GOVERNOR=m
===================================
Selecting this driver makes CONFIG_PREFERRED_CPU=y. That makes configs
driven by user preference. Though one can have CONFIG_STEAL_GOVERNOR=y,
It is recommended to build CONFIG_STEAL_GOVERNOR=m due to below reasons:
1. Doing periodic work has additional overheads. Enabling this driver
in systems where steal time cannot happen is of no use. There is no
benefit with additional overheads in such systems.
2. This works well when all VMs work in a co-operative manner. When an
administrative user enables it in one VM, he/she will likely enable
it in all VMs.
3. User can tweak the module parameters by reloading the module.

1
Documentation/scheduler/index.rst
View File

@ -23,5 +23,6 @@ Scheduler
sched-stats
sched-ext
sched-debug
sched-paravirt
text_files

67
Documentation/scheduler/sched-paravirt.rst Normal file
View File

@ -0,0 +1,67 @@
.. SPDX-License-Identifier: GPL-2.0
.. _sched-paravirt:
Preferred CPUs
==============
In paravirtualized environments CPU overcommit is a common scenario.
i.e. the sum of virtual CPUs (vCPUs) of all VMs is greater than number of
physical CPUs (pCPUs). Under such conditions when all or many VMs have
high utilization, hypervisor won't be able to satisfy the CPU requirement
and has to context switch within or across VMs. The hypervisor needs to
preempt one vCPU to run another. This is called vCPU preemption.
This is more expensive compared to task context switch within a vCPU, since
hypervisor lacks vCPU context and could preempt a critical section which
slows forward progress.
In such cases it is better that combined vCPU demand from all VMs is reduced
by not using some of the vCPUs in each VM. vCPUs where workload can be safely
scheduled which won't increase any contention for pCPU are called
"Preferred CPUs".
One of the main design constructs is that preferred CPUs are always
a subset of active CPUs. In most cases preferred CPUs will be same as
active CPUs. When there is pCPU contention, Preferred CPUs will reduce
based on the steal time. When the pCPU contention goes away as indicated
by steal time, Preferred CPUs could become same as active CPUs again.
The policy decisions are to be taken by driver.
For example, steal_governor. Look at its documentation for more
details. (``drivers/virt/steal_governor.c``)
Scheduling decisions such as wakeup, pushing the task etc, need this
CPU state info. This is maintained in ``cpu_preferred_mask``.
vCPUs which are not in ``cpu_preferred_mask`` should be treated as vCPUs which
should not be used at this moment provided it doesn't break user affinity.
This is achieved by:
1. Selecting a preferred CPU at wakeup using fallback mechanism.
2. Pushing the task away from non-preferred CPU at tick.
3. Selecting only preferred CPUs for load balance.
``/sys/devices/system/cpu/preferred`` prints the current ``cpu_preferred_mask``
in cpulist format.
Notes:
1. This feature is available under ``CONFIG_PREFERRED_CPU``. Driver which
makes decisions should enable it. For example, steal_governor driver
(``CONFIG_STEAL_GOVERNOR``). On enabling the driver, CPU preferred state
can change based on steal time. Without the driver, preferred CPUs is
same as active CPUs.
2. This feature works for the FAIR class only.
3. A pinned task, which can't be moved to preferred CPUs will continue
to run based on its affinity. But no load balancing happens if it is affined
only on non-preferred CPUs.
4. Decision to change the preferred CPU state is driven by the kernel.
Hence it shouldn't break user affinities. One of the main reasons why
CPU hotplug or Isolated cpuset partitions was not a solution.
5. This feature works best only when all the Guest VMs enable the feature as
it is a co-operative scheme. If a specific VM doesn't enable this feature
it may end up with more CPUs than others, still should lead to better
performance when seen from system view.
Users who enable this driver must ensure it is enabled in all Guest VMs.

9
MAINTAINERS
View File

@ -26256,6 +26256,15 @@ F: rust/helpers/jump_label.c
F: rust/kernel/generated_arch_static_branch_asm.rs.S
F: rust/kernel/jump_label.rs
STEAL GOVERNOR DRIVER
M: Shrikanth Hegde <sshegde@linux.ibm.com>
R: Yury Norov <yury.norov@gmail.com>
L: linux-kernel@vger.kernel.org
S: Maintained
T: git git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git sched/core
F: Documentation/driver-api/steal-governor.rst
F: drivers/virt/steal_governor.c
STI AUDIO (ASoC) DRIVERS
M: Arnaud Pouliquen <arnaud.pouliquen@foss.st.com>
L: linux-sound@vger.kernel.org

38
arch/Kconfig
View File

@ -1712,44 +1712,6 @@ config HAVE_STATIC_CALL_INLINE
depends on HAVE_STATIC_CALL
select OBJTOOL
config HAVE_PREEMPT_DYNAMIC
bool
config HAVE_PREEMPT_DYNAMIC_CALL
bool
depends on HAVE_STATIC_CALL
select HAVE_PREEMPT_DYNAMIC
help
An architecture should select this if it can handle the preemption
model being selected at boot time using static calls.
Where an architecture selects HAVE_STATIC_CALL_INLINE, any call to a
preemption function will be patched directly.
Where an architecture does not select HAVE_STATIC_CALL_INLINE, any
call to a preemption function will go through a trampoline, and the
trampoline will be patched.
It is strongly advised to support inline static call to avoid any
overhead.
config HAVE_PREEMPT_DYNAMIC_KEY
bool
depends on HAVE_ARCH_JUMP_LABEL
select HAVE_PREEMPT_DYNAMIC
help
An architecture should select this if it can handle the preemption
model being selected at boot time using static keys.
Each preemption function will be given an early return based on a
static key. This should have slightly lower overhead than non-inline
static calls, as this effectively inlines each trampoline into the
start of its callee. This may avoid redundant work, and may
integrate better with CFI schemes.
This will have greater overhead than using inline static calls as
the call to the preemption function cannot be entirely elided.
config ARCH_WANT_LD_ORPHAN_WARN
bool
help

1
arch/arm64/Kconfig
View File

@ -218,7 +218,6 @@ config ARM64
select HAVE_PERF_EVENTS_NMI if ARM64_PSEUDO_NMI
select HAVE_PERF_REGS
select HAVE_PERF_USER_STACK_DUMP
select HAVE_PREEMPT_DYNAMIC_KEY
select HAVE_REGS_AND_STACK_ACCESS_API
select HAVE_RELIABLE_STACKTRACE
select HAVE_POSIX_CPU_TIMERS_TASK_WORK

10
arch/arm64/include/asm/preempt.h
View File

@ -97,19 +97,9 @@ static inline bool __preempt_count_dec_and_test(void)
void preempt_schedule(void);
void preempt_schedule_notrace(void);
#ifdef CONFIG_PREEMPT_DYNAMIC
void dynamic_preempt_schedule(void);
#define __preempt_schedule() dynamic_preempt_schedule()
void dynamic_preempt_schedule_notrace(void);
#define __preempt_schedule_notrace() dynamic_preempt_schedule_notrace()
#else /* CONFIG_PREEMPT_DYNAMIC */
#define __preempt_schedule() preempt_schedule()
#define __preempt_schedule_notrace() preempt_schedule_notrace()
#endif /* CONFIG_PREEMPT_DYNAMIC */
#endif /* CONFIG_PREEMPTION */
#endif /* __ASM_PREEMPT_H */

4
arch/arm64/kernel/paravirt.c
View File

@ -157,9 +157,9 @@ int __init pv_time_init(void)
static_call_update(pv_steal_clock, para_steal_clock);
static_key_slow_inc(&paravirt_steal_enabled);
static_branch_inc(&paravirt_steal_enabled);
if (steal_acc)
static_key_slow_inc(&paravirt_steal_rq_enabled);
static_branch_inc(&paravirt_steal_rq_enabled);
pr_info("using stolen time PV\n");

1
arch/loongarch/Kconfig
View File

@ -170,7 +170,6 @@ config LOONGARCH
select HAVE_PERF_REGS
select HAVE_PERF_USER_STACK_DUMP
select HAVE_POSIX_CPU_TIMERS_TASK_WORK
select HAVE_PREEMPT_DYNAMIC_KEY
select HAVE_REGS_AND_STACK_ACCESS_API
select HAVE_RELIABLE_STACKTRACE if UNWINDER_ORC
select HAVE_RETHOOK

4
arch/loongarch/kernel/paravirt.c
View File

@ -308,10 +308,10 @@ int __init pv_time_init(void)
static_call_update(pv_steal_clock, paravt_steal_clock);
static_key_slow_inc(&paravirt_steal_enabled);
static_branch_inc(&paravirt_steal_enabled);
#ifdef CONFIG_PARAVIRT_TIME_ACCOUNTING
if (steal_acc)
static_key_slow_inc(&paravirt_steal_rq_enabled);
static_branch_inc(&paravirt_steal_rq_enabled);
#endif
if (static_key_enabled(&virt_preempt_key))

1
arch/powerpc/Kconfig
View File

@ -278,7 +278,6 @@ config PPC
select HAVE_PERF_EVENTS_NMI if PPC64
select HAVE_PERF_REGS
select HAVE_PERF_USER_STACK_DUMP
select HAVE_PREEMPT_DYNAMIC_KEY
select HAVE_POSIX_CPU_TIMERS_TASK_WORK
select HAVE_RETHOOK if KPROBES
select HAVE_REGS_AND_STACK_ACCESS_API

4
arch/powerpc/platforms/pseries/setup.c
View File

@ -876,9 +876,9 @@ static void __init pSeries_setup_arch(void)
static_branch_enable(&shared_processor);
pv_spinlocks_init();
#ifdef CONFIG_PARAVIRT_TIME_ACCOUNTING
static_key_slow_inc(&paravirt_steal_enabled);
static_branch_inc(&paravirt_steal_enabled);
if (steal_acc)
static_key_slow_inc(&paravirt_steal_rq_enabled);
static_branch_inc(&paravirt_steal_rq_enabled);
#endif
}

1
arch/riscv/Kconfig
View File

@ -195,7 +195,6 @@ config RISCV
select HAVE_PERF_REGS
select HAVE_PERF_USER_STACK_DUMP
select HAVE_POSIX_CPU_TIMERS_TASK_WORK
select HAVE_PREEMPT_DYNAMIC_KEY
select HAVE_REGS_AND_STACK_ACCESS_API
select HAVE_RETHOOK
select HAVE_RSEQ

4
arch/riscv/kernel/paravirt.c
View File

@ -116,9 +116,9 @@ int __init pv_time_init(void)
static_call_update(pv_steal_clock, pv_time_steal_clock);
static_key_slow_inc(&paravirt_steal_enabled);
static_branch_inc(&paravirt_steal_enabled);
if (steal_acc)
static_key_slow_inc(&paravirt_steal_rq_enabled);
static_branch_inc(&paravirt_steal_rq_enabled);
pr_info("Computing paravirt steal-time\n");

1
arch/s390/Kconfig
View File

@ -246,7 +246,6 @@ config S390
select HAVE_PERF_REGS
select HAVE_PERF_USER_STACK_DUMP
select HAVE_POSIX_CPU_TIMERS_TASK_WORK
select HAVE_PREEMPT_DYNAMIC_KEY
select HAVE_REGS_AND_STACK_ACCESS_API
select HAVE_RELIABLE_STACKTRACE
select HAVE_RETHOOK

11
arch/s390/include/asm/preempt.h
View File

@ -155,20 +155,9 @@ static __always_inline int __preempt_count_sub_return(int val)
void preempt_schedule(void);
void preempt_schedule_notrace(void);
#ifdef CONFIG_PREEMPT_DYNAMIC
void dynamic_preempt_schedule(void);
void dynamic_preempt_schedule_notrace(void);
#define __preempt_schedule() dynamic_preempt_schedule()
#define __preempt_schedule_notrace() dynamic_preempt_schedule_notrace()
#else /* CONFIG_PREEMPT_DYNAMIC */
#define __preempt_schedule() preempt_schedule()
#define __preempt_schedule_notrace() preempt_schedule_notrace()
#endif /* CONFIG_PREEMPT_DYNAMIC */
#endif /* CONFIG_PREEMPTION */
#endif /* __ASM_PREEMPT_H */

10
arch/s390/kernel/hiperdispatch.c
View File

@ -207,16 +207,12 @@ static unsigned long hd_calculate_steal_percentage(void)
{
unsigned long time_delta, steal_delta, steal, percentage;
static ktime_t prev;
int cpus, cpu;
int cpus;
ktime_t now;
cpus = 0;
steal = 0;
percentage = 0;
for_each_cpu(cpu, &hd_vmvl_cpumask) {
steal += kcpustat_cpu(cpu).cpustat[CPUTIME_STEAL];
cpus++;
}
steal = kcpustat_field_total(CPUTIME_STEAL, &hd_vmvl_cpumask);
cpus = cpumask_weight(&hd_vmvl_cpumask);
/*
* If there is no vertical medium and low CPUs steal time
* is 0 as vertical high CPUs shouldn't experience steal time.

4
arch/s390/kernel/idle.c
View File

@ -82,13 +82,13 @@ static u64 arch_cpu_idle_time(int cpu, enum cpu_usage_stat idx, bool compute_del
u64 arch_kcpustat_field_idle(int cpu)
{
return arch_cpu_idle_time(cpu, CPUTIME_IDLE, !nr_iowait_cpu(cpu));
return arch_cpu_idle_time(cpu, CPUTIME_IDLE, !cpu_has_iowait(cpu));
}
EXPORT_SYMBOL_GPL(arch_kcpustat_field_idle);
u64 arch_kcpustat_field_iowait(int cpu)
{
return arch_cpu_idle_time(cpu, CPUTIME_IOWAIT, nr_iowait_cpu(cpu));
return arch_cpu_idle_time(cpu, CPUTIME_IOWAIT, cpu_has_iowait(cpu));
}
EXPORT_SYMBOL_GPL(arch_kcpustat_field_iowait);

1
arch/x86/Kconfig
View File

@ -297,7 +297,6 @@ config X86
select HAVE_STACK_VALIDATION if HAVE_OBJTOOL
select HAVE_STATIC_CALL
select HAVE_STATIC_CALL_INLINE if HAVE_OBJTOOL
select HAVE_PREEMPT_DYNAMIC_CALL
select HAVE_RSEQ
select HAVE_RUST if X86_64
select HAVE_SYSCALL_TRACEPOINTS

28
arch/x86/include/asm/preempt.h
View File

@ -137,43 +137,15 @@ static __always_inline bool should_resched(int preempt_offset)
extern asmlinkage void preempt_schedule(void);
extern asmlinkage void preempt_schedule_thunk(void);
#define preempt_schedule_dynamic_enabled preempt_schedule_thunk
#define preempt_schedule_dynamic_disabled NULL
extern asmlinkage void preempt_schedule_notrace(void);
extern asmlinkage void preempt_schedule_notrace_thunk(void);
#define preempt_schedule_notrace_dynamic_enabled preempt_schedule_notrace_thunk
#define preempt_schedule_notrace_dynamic_disabled NULL
#ifdef CONFIG_PREEMPT_DYNAMIC
DECLARE_STATIC_CALL(preempt_schedule, preempt_schedule_dynamic_enabled);
#define __preempt_schedule() \
do { \
__STATIC_CALL_MOD_ADDRESSABLE(preempt_schedule); \
asm volatile ("call " STATIC_CALL_TRAMP_STR(preempt_schedule) : ASM_CALL_CONSTRAINT); \
} while (0)
DECLARE_STATIC_CALL(preempt_schedule_notrace, preempt_schedule_notrace_dynamic_enabled);
#define __preempt_schedule_notrace() \
do { \
__STATIC_CALL_MOD_ADDRESSABLE(preempt_schedule_notrace); \
asm volatile ("call " STATIC_CALL_TRAMP_STR(preempt_schedule_notrace) : ASM_CALL_CONSTRAINT); \
} while (0)
#else /* PREEMPT_DYNAMIC */
#define __preempt_schedule() \
asm volatile ("call preempt_schedule_thunk" : ASM_CALL_CONSTRAINT);
#define __preempt_schedule_notrace() \
asm volatile ("call preempt_schedule_notrace_thunk" : ASM_CALL_CONSTRAINT);
#endif /* PREEMPT_DYNAMIC */
#endif /* PREEMPTION */
#undef __pc_op

4
arch/x86/kernel/cpu/vmware.c
View File

@ -328,9 +328,9 @@ static int vmware_cpu_down_prepare(unsigned int cpu)
static __init int activate_jump_labels(void)
{
if (has_steal_clock) {
static_key_slow_inc(&paravirt_steal_enabled);
static_branch_inc(&paravirt_steal_enabled);
if (steal_acc)
static_key_slow_inc(&paravirt_steal_rq_enabled);
static_branch_inc(&paravirt_steal_rq_enabled);
}
return 0;

4
arch/x86/kernel/kvm.c
View File

@ -1080,9 +1080,9 @@ const __initconst struct hypervisor_x86 x86_hyper_kvm = {
static __init int activate_jump_labels(void)
{
if (has_steal_clock) {
static_key_slow_inc(&paravirt_steal_enabled);
static_branch_inc(&paravirt_steal_enabled);
if (steal_acc)
static_key_slow_inc(&paravirt_steal_rq_enabled);
static_branch_inc(&paravirt_steal_rq_enabled);
}
return 0;

12
drivers/base/cpu.c
View File

@ -391,6 +391,15 @@ static int cpu_uevent(const struct device *dev, struct kobj_uevent_env *env)
}
#endif
#ifdef CONFIG_PREFERRED_CPU
static ssize_t preferred_show(struct device *dev,
struct device_attribute *attr, char *buf)
{
return sysfs_emit(buf, "%*pbl\n", cpumask_pr_args(cpu_preferred_mask));
}
static DEVICE_ATTR_RO(preferred);
#endif
const struct bus_type cpu_subsys = {
.name = "cpu",
.dev_name = "cpu",
@ -531,6 +540,9 @@ static struct attribute *cpu_root_attrs[] = {
#endif
#ifdef CONFIG_GENERIC_CPU_AUTOPROBE
&dev_attr_modalias.attr,
#endif
#ifdef CONFIG_PREFERRED_CPU
&dev_attr_preferred.attr,
#endif
NULL
};

17
drivers/virt/Kconfig
View File

@ -41,6 +41,23 @@ config FSL_HV_MANAGER
4) A kernel interface for receiving callbacks when a managed
partition shuts down.
config STEAL_GOVERNOR
tristate "Dynamic vCPU management based on steal time"
depends on PARAVIRT && SMP
select PREFERRED_CPU
default m
help
This driver helps to reduce the steal time in paravirtualized
environments, thereby reducing vCPU preemption costs.
By default preferred CPUs will be same as active CPUs. Depending
on the steal time when steal_governor driver is enabled,
preferred CPUs could become subset of active CPUs.
More details are at: Documentation/driver-api/steal-governor.rst
It is recommended to build it as module and load the module
to enable it.
source "drivers/virt/vboxguest/Kconfig"
source "drivers/virt/nitro_enclaves/Kconfig"

1
drivers/virt/Makefile
View File

@ -5,6 +5,7 @@
obj-$(CONFIG_FSL_HV_MANAGER) += fsl_hypervisor.o
obj-$(CONFIG_VMGENID) += vmgenid.o
obj-$(CONFIG_STEAL_GOVERNOR) += steal_governor.o
obj-y += vboxguest/
obj-$(CONFIG_NITRO_ENCLAVES) += nitro_enclaves/

296
drivers/virt/steal_governor.c Normal file
View File

@ -0,0 +1,296 @@
// SPDX-License-Identifier: GPL-2.0-only
/*
* Steal time governor driver periodically computes steal time.
* Based on the thresholds it either reduce/increase the preferred
* CPUs which can be used by the workload to avoid vCPU preemption
* to an extent possible in paravirtualized environment.
*
* Available with CONFIG_STEAL_GOVERNOR
*
* Copyright (C) 2026 IBM
* Author: Shrikanth Hegde <sshegde@linux.ibm.com>
*/
#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
#include <linux/cleanup.h>
#include <linux/cpuhplock.h>
#include <linux/cpumask.h>
#include <linux/init.h>
#include <linux/kernel.h>
#include <linux/kernel_stat.h>
#include <linux/kconfig.h>
#include <linux/ktime.h>
#include <linux/math64.h>
#include <linux/module.h>
#include <linux/sched/isolation.h>
#include <linux/topology.h>
#include <linux/types.h>
#include <linux/workqueue.h>
#ifdef CONFIG_XEN
#include <xen/xen.h>
#endif
#if !IS_ENABLED(CONFIG_PREFERRED_CPU)
#error "Steal Governor requires CONFIG_PREFERRED_CPU"
#endif
struct steal_governor {
ktime_t time;
u64 steal;
unsigned long delay;
unsigned int interval_ms;
unsigned int high_threshold;
unsigned int low_threshold;
struct delayed_work work;
};
static struct steal_governor sg_ctx = {
.interval_ms = 1000, /* 1 second */
.high_threshold = 500, /* 5% */
.low_threshold = 200, /* 2% */
};
static void restore_preferred_to_active(void)
{
int cpu;
guard(cpus_read_lock)();
for_each_cpu(cpu, cpu_active_mask)
set_cpu_preferred(cpu, true);
}
static int param_set_interval_ms(const char *val, const struct kernel_param *kp)
{
unsigned int interval;
int ret;
ret = kstrtouint(val, 0, &interval);
if (ret)
return ret;
if (interval < 100 || interval > 100000) {
pr_err("interval_ms must be between 100 and 100000\n");
return -EINVAL;
}
return param_set_uint(val, kp);
}
static const struct kernel_param_ops interval_ms_ops = {
.set = param_set_interval_ms,
.get = param_get_uint,
};
module_param_cb(interval_ms, &interval_ms_ops, &sg_ctx.interval_ms, 0444);
MODULE_PARM_DESC(interval_ms,
"Sampling frequency in milliseconds. default: 1000");
static int param_set_high_threshold(const char *val, const struct kernel_param *kp)
{
unsigned int threshold;
int ret;
ret = kstrtouint(val, 0, &threshold);
if (ret)
return ret;
if (threshold >= 100 * 100) {
pr_err("high_threshold (%u) can't be more than 99.99%%\n", threshold);
return -EINVAL;
}
return param_set_uint(val, kp);
}
static const struct kernel_param_ops high_threshold_ops = {
.set = param_set_high_threshold,
.get = param_get_uint,
};
module_param_cb(high_threshold, &high_threshold_ops, &sg_ctx.high_threshold, 0444);
MODULE_PARM_DESC(high_threshold,
"High steal threshold. default: 500 i.e 5%. Must be > low_threshold");
module_param_named(low_threshold, sg_ctx.low_threshold, uint, 0444);
MODULE_PARM_DESC(low_threshold,
"Low steal threshold. default: 200 i.e 2%. Must be < high_threshold");
/* Return collective steal time across system. */
static u64 get_system_steal_time(void)
{
return kcpustat_field_total(CPUTIME_STEAL, cpu_possible_mask);
}
/* Return number of CPUs to consider for steal ratio. */
static unsigned int get_system_cpus(void)
{
return num_active_cpus();
}
/*
* Called when the steal governor detects high physical CPU contention.
* It finds the last active core in the preferred mask and mark those
* CPUs as non-preferred.
*
* Must ensure:
* - at least one core is always kept as preferred
* - preferred is always subset of active.
*/
static void decrease_preferred_cpus(void)
{
const struct cpumask *first_hk_core;
int target_cpu = nr_cpu_ids;
int cpu;
guard(cpus_read_lock)();
cpu = cpumask_first_and(housekeeping_cpumask(HK_TYPE_KERNEL_NOISE),
cpu_preferred_mask);
if (cpu >= nr_cpu_ids)
return;
/* Always leave first housekeeping core as preferred. */
first_hk_core = topology_sibling_cpumask(cpu);
cpu = cpumask_last(cpu_preferred_mask);
if (cpu >= nr_cpu_ids)
return;
/* Find the last CPU which doesn't belong to that first hk_core. */
if (!cpumask_test_cpu(cpu, first_hk_core)) {
target_cpu = cpu;
} else {
for_each_cpu_andnot(cpu, cpu_preferred_mask, first_hk_core)
target_cpu = cpu;
}
/* Only the first housekeeping core remains */
if (target_cpu >= nr_cpu_ids)
return;
for_each_cpu_and(cpu, topology_sibling_cpumask(target_cpu),
cpu_preferred_mask)
set_cpu_preferred(cpu, false);
}
/*
* Called when the steal governor detects no/low physical CPU contention.
* It finds the first active core outside of preferred mask and mark
* those CPUs as preferred.
*
* Must ensure preferred is subset of active.
*/
static void increase_preferred_cpus(void)
{
int first_cpu, cpu;
guard(cpus_read_lock)();
first_cpu = cpumask_first_andnot(cpu_active_mask, cpu_preferred_mask);
/* All CPUs are preferred. Nothing to increase further */
if (first_cpu >= nr_cpu_ids)
return;
for_each_cpu_and(cpu, topology_sibling_cpumask(first_cpu),
cpu_active_mask)
set_cpu_preferred(cpu, true);
}
static bool preferred_cpus_valid(void)
{
if (cpumask_empty(cpu_preferred_mask)) {
pr_err("empty preferred mask. stopping\n");
return false;
}
if (!cpumask_subset(cpu_preferred_mask, cpu_active_mask)) {
pr_err("preferred: %*pbl is not subset of active: %*pbl, stopping\n",
cpumask_pr_args(cpu_preferred_mask),
cpumask_pr_args(cpu_active_mask));
return false;
}
return true;
}
static void steal_governor_loop(struct work_struct *work)
{
u64 curr_steal, delta_steal, delta_ns, steal_ratio;
ktime_t now;
now = ktime_get();
delta_ns = ktime_to_ns(ktime_sub(now, sg_ctx.time));
if (unlikely(delta_ns < NSEC_PER_MSEC)) {
pr_err_ratelimited("work scheduled too soon delta_ns: %llu\n", delta_ns);
goto requeue_work;
}
curr_steal = get_system_steal_time();
delta_steal = curr_steal > sg_ctx.steal ? curr_steal - sg_ctx.steal : 0;
sg_ctx.steal = curr_steal;
sg_ctx.time = now;
/*
* steal_ratio = (delta_steal * 100*100)/(delta_ns * num_cpus())
* To avoid possible overflow, divide the denominator early.
* Note minimum interval is 100ms.
*/
delta_ns = max_t(u64, div_u64(delta_ns * get_system_cpus(), 10000), 1);
steal_ratio = div64_u64(delta_steal, delta_ns);
if (steal_ratio > sg_ctx.high_threshold)
decrease_preferred_cpus();
else if (steal_ratio <= sg_ctx.low_threshold)
increase_preferred_cpus();
/*
* else: steal ratio is within bounds. Still do design checks so that
* module restores to active if CPU hotplug breaks those assumptions.
*/
if (!preferred_cpus_valid()) {
restore_preferred_to_active();
return;
}
requeue_work:
schedule_delayed_work(&sg_ctx.work, sg_ctx.delay);
}
static int __init steal_governor_init(void)
{
#ifdef CONFIG_XEN
if (xen_initial_domain()) {
pr_err("Cannot load in Xen Dom0 (Host OS). Driver is for guests only.\n");
return -ENODEV;
}
#endif
if (sg_ctx.low_threshold >= sg_ctx.high_threshold) {
pr_err("low_threshold (%u) must be less than high_threshold (%u)\n",
sg_ctx.low_threshold, sg_ctx.high_threshold);
return -EINVAL;
}
sg_ctx.delay = msecs_to_jiffies(sg_ctx.interval_ms);
INIT_DELAYED_WORK(&sg_ctx.work, steal_governor_loop);
sg_ctx.steal = get_system_steal_time();
sg_ctx.time = ktime_get();
schedule_delayed_work(&sg_ctx.work, sg_ctx.delay);
pr_info("enabled. interval: %ums, high_threshold: %u, low_threshold: %u\n",
sg_ctx.interval_ms, sg_ctx.high_threshold, sg_ctx.low_threshold);
return 0;
}
static void __exit steal_governor_exit(void)
{
disable_delayed_work_sync(&sg_ctx.work);
restore_preferred_to_active();
pr_info("disabled\n");
}
module_init(steal_governor_init);
module_exit(steal_governor_exit);
MODULE_LICENSE("GPL");
MODULE_AUTHOR("IBM Corporation");
MODULE_DESCRIPTION("Virtualization Steal Time Governor");

4
drivers/xen/time.c
View File

@ -169,7 +169,7 @@ void __init xen_time_setup_guest(void)
static_call_update(pv_steal_clock, xen_steal_clock);
static_key_slow_inc(&paravirt_steal_enabled);
static_branch_inc(&paravirt_steal_enabled);
if (xen_runstate_remote)
static_key_slow_inc(&paravirt_steal_rq_enabled);
static_branch_inc(&paravirt_steal_rq_enabled);
}

6
fs/proc/uptime.c
View File

@ -15,12 +15,8 @@ static int uptime_proc_show(struct seq_file *m, void *v)
struct timespec64 idle;
u64 idle_nsec;
u32 rem;
int i;
idle_nsec = 0;
for_each_possible_cpu(i)
idle_nsec += kcpustat_field(CPUTIME_IDLE, i);
idle_nsec = kcpustat_field_total(CPUTIME_IDLE, cpu_possible_mask);
ktime_get_boottime_ts64(&uptime);
timens_add_boottime(&uptime);

10
include/asm-generic/preempt.h
View File

@ -96,19 +96,9 @@ static __always_inline bool should_resched(int preempt_offset)
extern asmlinkage void preempt_schedule(void);
extern asmlinkage void preempt_schedule_notrace(void);
#if defined(CONFIG_PREEMPT_DYNAMIC) && defined(CONFIG_HAVE_PREEMPT_DYNAMIC_KEY)
void dynamic_preempt_schedule(void);
void dynamic_preempt_schedule_notrace(void);
#define __preempt_schedule() dynamic_preempt_schedule()
#define __preempt_schedule_notrace() dynamic_preempt_schedule_notrace()
#else /* !CONFIG_PREEMPT_DYNAMIC || !CONFIG_HAVE_PREEMPT_DYNAMIC_KEY*/
#define __preempt_schedule() preempt_schedule()
#define __preempt_schedule_notrace() preempt_schedule_notrace()
#endif /* CONFIG_PREEMPT_DYNAMIC && CONFIG_HAVE_PREEMPT_DYNAMIC_KEY*/
#endif /* CONFIG_PREEMPTION */
#endif /* __ASM_PREEMPT_H */

14
include/linux/bitmap.h
View File

@ -52,6 +52,7 @@ struct device;
* bitmap_complement(dst, src, nbits) *dst = ~(*src)
* bitmap_equal(src1, src2, nbits) Are *src1 and *src2 equal?
* bitmap_intersects(src1, src2, nbits) Do *src1 and *src2 overlap?
* bitmap_intersects_and(src1, src2, src3, nbits) Do *src1, *src2 and *src3 overlap?
* bitmap_subset(src1, src2, nbits) Is *src1 a subset of *src2?
* bitmap_empty(src, nbits) Are all bits zero in *src?
* bitmap_full(src, nbits) Are all bits set in *src?
@ -181,6 +182,9 @@ void __bitmap_replace(unsigned long *dst,
const unsigned long *mask, unsigned int nbits);
bool __bitmap_intersects(const unsigned long *bitmap1,
const unsigned long *bitmap2, unsigned int nbits);
bool __bitmap_intersects_and(const unsigned long *bitmap1,
const unsigned long *bitmap2,
const unsigned long *bitmap3, unsigned int nbits);
bool __bitmap_subset(const unsigned long *bitmap1,
const unsigned long *bitmap2, unsigned int nbits);
unsigned int __bitmap_weight(const unsigned long *bitmap, unsigned int nbits);
@ -445,6 +449,16 @@ bool bitmap_intersects(const unsigned long *src1, const unsigned long *src2, uns
return __bitmap_intersects(src1, src2, nbits);
}
static __always_inline
bool bitmap_intersects_and(const unsigned long *src1, const unsigned long *src2,
const unsigned long *src3, unsigned int nbits)
{
if (small_const_nbits(nbits))
return ((*src1 & *src2 & *src3) & BITMAP_LAST_WORD_MASK(nbits)) != 0;
else
return __bitmap_intersects_and(src1, src2, src3, nbits);
}
static __always_inline
bool bitmap_subset(const unsigned long *src1, const unsigned long *src2, unsigned int nbits)
{

42
include/linux/cpumask.h
View File

@ -121,12 +121,20 @@ extern struct cpumask __cpu_enabled_mask;
extern struct cpumask __cpu_present_mask;
extern struct cpumask __cpu_active_mask;
extern struct cpumask __cpu_dying_mask;
#ifdef CONFIG_PREFERRED_CPU
extern struct cpumask __cpu_preferred_mask;
#else
#define __cpu_preferred_mask __cpu_active_mask
#endif
#define cpu_possible_mask ((const struct cpumask *)&__cpu_possible_mask)
#define cpu_online_mask ((const struct cpumask *)&__cpu_online_mask)
#define cpu_enabled_mask ((const struct cpumask *)&__cpu_enabled_mask)
#define cpu_present_mask ((const struct cpumask *)&__cpu_present_mask)
#define cpu_active_mask ((const struct cpumask *)&__cpu_active_mask)
#define cpu_dying_mask ((const struct cpumask *)&__cpu_dying_mask)
#define cpu_preferred_mask ((const struct cpumask *)&__cpu_preferred_mask)
extern atomic_t __num_online_cpus;
extern unsigned int __num_possible_cpus;
@ -824,6 +832,24 @@ bool cpumask_intersects(const struct cpumask *src1p, const struct cpumask *src2p
small_cpumask_bits);
}
/**
* cpumask_intersects_and - (*src1p & *src2p & *src3p) != 0
* @src1p: the first input
* @src2p: the second input
* @src3p: the third input
*
* Return: true if AND of the three cpumasks is non-empty,
* otherwise false
*/
static __always_inline
bool cpumask_intersects_and(const struct cpumask *src1p,
const struct cpumask *src2p,
const struct cpumask *src3p)
{
return bitmap_intersects_and(cpumask_bits(src1p), cpumask_bits(src2p),
cpumask_bits(src3p), small_cpumask_bits);
}
/**
* cpumask_subset - (*src1p & ~*src2p) == 0
* @src1p: the first input
@ -1163,6 +1189,12 @@ void init_cpu_possible(const struct cpumask *src);
#define set_cpu_active(cpu, active) assign_cpu((cpu), &__cpu_active_mask, (active))
#define set_cpu_dying(cpu, dying) assign_cpu((cpu), &__cpu_dying_mask, (dying))
#ifdef CONFIG_PREFERRED_CPU
#define set_cpu_preferred(cpu, preferred) assign_cpu((cpu), &__cpu_preferred_mask, (preferred))
#else
#define set_cpu_preferred(cpu, preferred) do { } while (0)
#endif
void set_cpu_online(unsigned int cpu, bool online);
void set_cpu_possible(unsigned int cpu, bool possible);
@ -1257,6 +1289,11 @@ static __always_inline bool cpu_dying(unsigned int cpu)
return cpumask_test_cpu(cpu, cpu_dying_mask);
}
static __always_inline bool cpu_preferred(unsigned int cpu)
{
return cpumask_test_cpu(cpu, cpu_preferred_mask);
}
#else
#define num_online_cpus() 1U
@ -1295,6 +1332,11 @@ static __always_inline bool cpu_dying(unsigned int cpu)
return false;
}
static __always_inline bool cpu_preferred(unsigned int cpu)
{
return cpu == 0;
}
#endif /* NR_CPUS > 1 */
#define cpu_is_offline(cpu) unlikely(!cpu_online(cpu))

17
include/linux/irq-entry-common.h
View File

@ -346,22 +346,7 @@ typedef struct irqentry_state {
*
* Conditional reschedule with additional sanity checks.
*/
void raw_irqentry_exit_cond_resched(void);
#ifdef CONFIG_PREEMPT_DYNAMIC
#if defined(CONFIG_HAVE_PREEMPT_DYNAMIC_CALL)
#define irqentry_exit_cond_resched_dynamic_enabled raw_irqentry_exit_cond_resched
#define irqentry_exit_cond_resched_dynamic_disabled NULL
DECLARE_STATIC_CALL(irqentry_exit_cond_resched, raw_irqentry_exit_cond_resched);
#define irqentry_exit_cond_resched() static_call(irqentry_exit_cond_resched)()
#elif defined(CONFIG_HAVE_PREEMPT_DYNAMIC_KEY)
DECLARE_STATIC_KEY_TRUE(sk_dynamic_irqentry_exit_cond_resched);
void dynamic_irqentry_exit_cond_resched(void);
#define irqentry_exit_cond_resched() dynamic_irqentry_exit_cond_resched()
#endif
#else /* CONFIG_PREEMPT_DYNAMIC */
#define irqentry_exit_cond_resched() raw_irqentry_exit_cond_resched()
#endif /* CONFIG_PREEMPT_DYNAMIC */
void irqentry_exit_cond_resched(void);
/**
* irqentry_enter_from_kernel_mode - Establish state before invoking the irq handler

2
include/linux/irq_work.h
View File

@ -59,6 +59,7 @@ void irq_work_sync(struct irq_work *work);
#include <asm/irq_work.h>
void irq_work_run(void);
void irq_work_run_cpu(unsigned int cpu);
bool irq_work_needs_cpu(void);
void irq_work_single(void *arg);
@ -67,6 +68,7 @@ void arch_irq_work_raise(void);
#else
static inline bool irq_work_needs_cpu(void) { return false; }
static inline void irq_work_run(void) { }
static inline void irq_work_run_cpu(unsigned int cpu) { }
static inline void irq_work_single(void *arg) { }
#endif

20
include/linux/kernel.h
View File

@ -43,30 +43,10 @@ struct completion;
struct user;
#ifdef CONFIG_PREEMPT_VOLUNTARY_BUILD
extern int __cond_resched(void);
# define might_resched() __cond_resched()
#elif defined(CONFIG_PREEMPT_DYNAMIC) && defined(CONFIG_HAVE_PREEMPT_DYNAMIC_CALL)
extern int __cond_resched(void);
DECLARE_STATIC_CALL(might_resched, __cond_resched);
static __always_inline void might_resched(void)
{
static_call_mod(might_resched)();
}
#elif defined(CONFIG_PREEMPT_DYNAMIC) && defined(CONFIG_HAVE_PREEMPT_DYNAMIC_KEY)
extern int dynamic_might_resched(void);
# define might_resched() dynamic_might_resched()
#else
# define might_resched() do { } while (0)
#endif /* CONFIG_PREEMPT_* */
#ifdef CONFIG_DEBUG_ATOMIC_SLEEP

18
include/linux/kernel_stat.h
View File

@ -40,6 +40,9 @@ struct kernel_cpustat {
seqcount_t idle_sleeptime_seq;
u64 idle_entrytime;
u64 idle_stealtime[2];
u64 idle_dyntick_entry;
u64 idle_tick_period;
u64 idle_tick_overlap;
#endif
u64 cpustat[NR_STATS];
};
@ -111,7 +114,7 @@ static inline unsigned long kstat_cpu_irqs_sum(unsigned int cpu)
#ifdef CONFIG_HAVE_VIRT_CPU_ACCOUNTING_IDLE
static inline void kcpustat_dyntick_start(u64 now) { }
static inline void kcpustat_dyntick_stop(u64 now) { }
static inline void kcpustat_dyntick_stop(u64 now, u64 tick_start) { }
static inline void kcpustat_irq_enter(u64 now) { }
static inline void kcpustat_irq_exit(u64 now) { }
static inline bool kcpustat_idle_dyntick(void) { return false; }
@ -132,7 +135,7 @@ static inline u64 kcpustat_field_iowait(int cpu)
#else /* !CONFIG_HAVE_VIRT_CPU_ACCOUNTING_IDLE */
extern void kcpustat_dyntick_start(u64 now);
extern void kcpustat_dyntick_stop(u64 now);
extern void kcpustat_dyntick_stop(u64 now, u64 tick_start);
extern void kcpustat_irq_enter(u64 now);
extern void kcpustat_irq_exit(u64 now);
extern u64 kcpustat_field_idle(int cpu);
@ -196,6 +199,17 @@ static inline void kcpustat_cpu_fetch(struct kernel_cpustat *dst, int cpu)
}
#endif /* !CONFIG_VIRT_CPU_ACCOUNTING_GEN */
static inline u64 kcpustat_field_total(enum cpu_usage_stat usage, const struct cpumask *cpus)
{
u64 total = 0;
int cpu;
for_each_cpu(cpu, cpus)
total += kcpustat_field(usage, cpu);
return total;
}
extern void account_user_time(struct task_struct *, u64);
extern void account_guest_time(struct task_struct *, u64);
extern void account_system_time(struct task_struct *, int, u64);

20
include/linux/preempt.h
View File

@ -498,21 +498,11 @@ DEFINE_LOCK_GUARD_0(preempt_notrace, preempt_disable_notrace(), preempt_enable_n
#ifdef CONFIG_PREEMPT_DYNAMIC
extern bool preempt_model_none(void);
extern bool preempt_model_voluntary(void);
extern bool preempt_model_full(void);
extern bool preempt_model_lazy(void);
#else
static inline bool preempt_model_none(void)
{
return IS_ENABLED(CONFIG_PREEMPT_NONE);
}
static inline bool preempt_model_voluntary(void)
{
return IS_ENABLED(CONFIG_PREEMPT_VOLUNTARY);
}
static inline bool preempt_model_full(void)
{
return IS_ENABLED(CONFIG_PREEMPT);
@ -525,6 +515,16 @@ static inline bool preempt_model_lazy(void)
#endif
static inline bool preempt_model_none(void)
{
return IS_ENABLED(CONFIG_PREEMPT_NONE);
}
static inline bool preempt_model_voluntary(void)
{
return IS_ENABLED(CONFIG_PREEMPT_VOLUNTARY);
}
static inline bool preempt_model_rt(void)
{
return IS_ENABLED(CONFIG_PREEMPT_RT);

201
include/linux/rbtree_augmented.h
View File

@ -15,6 +15,8 @@
#include <linux/compiler.h>
#include <linux/rbtree.h>
#include <linux/rcupdate.h>
#include <linux/args.h>
#include <linux/minmax.h>
/*
* Please note - only struct rb_augment_callbacks and the prototypes for
@ -28,6 +30,7 @@ struct rb_augment_callbacks {
void (*propagate)(struct rb_node *node, struct rb_node *stop);
void (*copy)(struct rb_node *old, struct rb_node *new);
void (*rotate)(struct rb_node *old, struct rb_node *new);
void (*merge)(struct rb_node *node, struct rb_node *new);
};
extern void __rb_insert_augmented(struct rb_node *node, struct rb_root *root,
@ -60,6 +63,12 @@ rb_insert_augmented_cached(struct rb_node *node,
rb_insert_augmented(node, &root->rb_root, augment);
}
/*
* Insert @node into the leftmost cached augmented tree @tree.
*
* The augmented data of @node must already describe @node alone; it is
* merged into every ancestor on the way down through augment->merge().
*/
static __always_inline struct rb_node *
rb_add_augmented_cached(struct rb_node *node, struct rb_root_cached *tree,
bool (*less)(struct rb_node *, const struct rb_node *),
@ -71,6 +80,7 @@ rb_add_augmented_cached(struct rb_node *node, struct rb_root_cached *tree,
while (*link) {
parent = *link;
augment->merge(parent, node);
if (less(node, parent)) {
link = &parent->rb_left;
} else {
@ -80,12 +90,108 @@ rb_add_augmented_cached(struct rb_node *node, struct rb_root_cached *tree,
}
rb_link_node(node, parent, link);
augment->propagate(parent, NULL); /* suboptimal */
rb_insert_augmented_cached(node, tree, leftmost, augment);
return leftmost ? node : NULL;
}
#define RB_FOR_EACH_1(what, RBNAME, RBSTRUCT, RBFIELD, x) \
what(1, RBNAME, RBSTRUCT, RBFIELD, x)
#define RB_FOR_EACH_2(what, RBNAME, RBSTRUCT, RBFIELD, x, ...) \
what(2, RBNAME, RBSTRUCT, RBFIELD, x) \
RB_FOR_EACH_1(what, RBNAME, RBSTRUCT, RBFIELD, __VA_ARGS__)
#define RB_FOR_EACH_3(what, RBNAME, RBSTRUCT, RBFIELD, x, ...) \
what(3, RBNAME, RBSTRUCT, RBFIELD, x) \
RB_FOR_EACH_2(what, RBNAME, RBSTRUCT, RBFIELD, __VA_ARGS__)
#define RB_FOR_EACH_4(what, RBNAME, RBSTRUCT, RBFIELD, x, ...) \
what(4, RBNAME, RBSTRUCT, RBFIELD, x) \
RB_FOR_EACH_3(what, RBNAME, RBSTRUCT, RBFIELD, __VA_ARGS__)
#define RB_FOR_EACH_5(what, RBNAME, RBSTRUCT, RBFIELD, x, ...) \
what(5, RBNAME, RBSTRUCT, RBFIELD, x) \
RB_FOR_EACH_4(what, RBNAME, RBSTRUCT, RBFIELD, __VA_ARGS__)
#define RB_FOR_EACH_6(what, RBNAME, RBSTRUCT, RBFIELD, x, ...) \
what(6, RBNAME, RBSTRUCT, RBFIELD, x) \
RB_FOR_EACH_5(what, RBNAME, RBSTRUCT, RBFIELD, __VA_ARGS__)
#define RB_FOR_EACH_7(what, RBNAME, RBSTRUCT, RBFIELD, x, ...) \
what(7, RBNAME, RBSTRUCT, RBFIELD, x) \
RB_FOR_EACH_6(what, RBNAME, RBSTRUCT, RBFIELD, __VA_ARGS__)
#define RB_FOR_EACH_8(what, RBNAME, RBSTRUCT, RBFIELD, x, ...) \
what(8, RBNAME, RBSTRUCT, RBFIELD, x) \
RB_FOR_EACH_7(what, RBNAME, RBSTRUCT, RBFIELD, __VA_ARGS__)
#define RB_FOR_EACH(action, RBNAME, RBSTRUCT, RBFIELD, ...) \
CONCATENATE(RB_FOR_EACH_, COUNT_ARGS(__VA_ARGS__)) \
(action, RBNAME, RBSTRUCT, RBFIELD, __VA_ARGS__)
/*
* One augmented field: @val is the node's own contribution (a member for
* RB_AUG(), a function of the node for RB_AUG_FUNC()), @aug the member
* holding the aggregate over the subtree, and @fold(a, b) combines two
* aggregates: min, max, a sum, ... It must be commutative and associative.
*/
#define RB_AUG_FUNC(val, aug, fold) (val(s), aug, fold)
#define RB_AUG(val, aug, fold) (s->val, aug, fold)
#define RB_UNPACK(...) __VA_ARGS__
#define __RB_INST(n, RBNAME, RBSTRUCT, RBFIELD, val, aug, fold) \
static inline void \
RBNAME ## _copy_ ## n(RBSTRUCT *old, RBSTRUCT *new) \
{ \
new->aug = old->aug; \
} \
static inline bool \
RBNAME ## _compute_ ## n(RBSTRUCT *s, bool exit) \
{ \
TYPEOF_UNQUAL(s->aug) _old_aug = s->aug; \
TYPEOF_UNQUAL(s->aug) _val = val; \
struct rb_node *_node = &s->RBFIELD; \
if (_node->rb_right) { \
RBSTRUCT *_c = container_of(_node->rb_right, typeof(*s), RBFIELD); \
_val = fold(_val, _c->aug); \
} \
if (_node->rb_left) { \
RBSTRUCT *_c = container_of(_node->rb_left, typeof(*s), RBFIELD); \
_val = fold(_val, _c->aug); \
} \
if (exit && _old_aug == _val) \
return true; \
s->aug = _val; \
return false; \
}
#define _RB_INST(n, RBNAME, RBSTRUCT, RBFIELD, args) \
__RB_INST(n, RBNAME, RBSTRUCT, RBFIELD, args)
#define RB_INST(n, RBNAME, RBSTRUCT, RBFIELD, x) \
_RB_INST(n, RBNAME, RBSTRUCT, RBFIELD, RB_UNPACK x)
#define __RB_COPY(n, RBNAME, RBSTRUCT, RBFIELD, val, aug, fold) \
RBNAME ## _copy_ ## n(old, new);
#define _RB_COPY(n, RBNAME, RBSTRUCT, RBFIELD, args) \
__RB_COPY(n, RBNAME, RBSTRUCT, RBFIELD, args)
#define RB_COPY(n, RBNAME, RBSTRUCT, RBFIELD, x) \
_RB_COPY(n, RBNAME, RBSTRUCT, RBFIELD, RB_UNPACK x)
#define __RB_COMPUTE(n, RBNAME, RBSTRUCT, RBFIELD, val, aug, fold) \
ret &= RBNAME ## _compute_ ## n(node, exit);
#define _RB_COMPUTE(n, RBNAME, RBSTRUCT, RBFIELD, args) \
__RB_COMPUTE(n, RBNAME, RBSTRUCT, RBFIELD, args)
#define RB_COMPUTE(n, RBNAME, RBSTRUCT, RBFIELD, x) \
_RB_COMPUTE(n, RBNAME, RBSTRUCT, RBFIELD, RB_UNPACK x)
/* fold @new, about to become a descendant of @node, into @node */
#define __RB_MERGE(n, RBNAME, RBSTRUCT, RBFIELD, val, aug, fold) \
node->aug = fold(node->aug, new->aug);
#define _RB_MERGE(n, RBNAME, RBSTRUCT, RBFIELD, args) \
__RB_MERGE(n, RBNAME, RBSTRUCT, RBFIELD, args)
#define RB_MERGE(n, RBNAME, RBSTRUCT, RBFIELD, x) \
_RB_MERGE(n, RBNAME, RBSTRUCT, RBFIELD, RB_UNPACK x)
/*
* Template for declaring augmented rbtree callbacks (generic multi fields)
*
@ -93,18 +199,29 @@ rb_add_augmented_cached(struct rb_node *node, struct rb_root_cached *tree,
* RBNAME: name of the rb_augment_callbacks structure
* RBSTRUCT: struct type of the tree nodes
* RBFIELD: name of struct rb_node field within RBSTRUCT
* RBCOPY: name of function that copies the RBAUGMENTED datas
* RBCOMPUTE: name of function that recomputes the RBAUGMENTED datas
* RBAUG...: list of RB_AUG() describing the augmented data
*/
#define RB_DECLARE_CALLBACKS_MULTI(RBSTATIC, RBNAME, \
RBSTRUCT, RBFIELD, RBCOPY, RBCOMPUTE) \
#define RB_DECLARE_CALLBACKS(RBSTATIC, RBNAME, \
RBSTRUCT, RBFIELD, RBAUG...) \
RB_FOR_EACH(RB_INST, RBNAME, RBSTRUCT, RBFIELD, RBAUG) \
static inline void \
RBNAME ## __copy(RBSTRUCT *old, RBSTRUCT *new) \
{ \
RB_FOR_EACH(RB_COPY, RBNAME, RBSTRUCT, RBFIELD, RBAUG); \
} \
static inline bool \
RBNAME ## __compute(RBSTRUCT *node, bool exit) \
{ \
bool ret = true; \
RB_FOR_EACH(RB_COMPUTE, RBNAME, RBSTRUCT, RBFIELD, RBAUG); \
return ret; \
} \
static inline void \
RBNAME ## _propagate(struct rb_node *rb, struct rb_node *stop) \
{ \
while (rb != stop) { \
RBSTRUCT *node = rb_entry(rb, RBSTRUCT, RBFIELD); \
if (RBCOMPUTE(node, true)) \
if (RBNAME ## __compute(node, true)) \
break; \
rb = rb_parent(&node->RBFIELD); \
} \
@ -114,43 +231,30 @@ RBNAME ## _copy(struct rb_node *rb_old, struct rb_node *rb_new) \
{ \
RBSTRUCT *old = rb_entry(rb_old, RBSTRUCT, RBFIELD); \
RBSTRUCT *new = rb_entry(rb_new, RBSTRUCT, RBFIELD); \
RBCOPY(new, old); \
RBNAME ## __copy(old, new); \
} \
static void \
RBNAME ## _rotate(struct rb_node *rb_old, struct rb_node *rb_new) \
{ \
RBSTRUCT *old = rb_entry(rb_old, RBSTRUCT, RBFIELD); \
RBSTRUCT *new = rb_entry(rb_new, RBSTRUCT, RBFIELD); \
RBCOPY(new, old); \
RBCOMPUTE(old, false); \
RBNAME ## __copy(old, new); \
RBNAME ## __compute(old, false); \
} \
static inline void \
RBNAME ## _merge(struct rb_node *rb, struct rb_node *rb_new) \
{ \
RBSTRUCT *node = rb_entry(rb, RBSTRUCT, RBFIELD); \
RBSTRUCT *new = rb_entry(rb_new, RBSTRUCT, RBFIELD); \
RB_FOR_EACH(RB_MERGE, RBNAME, RBSTRUCT, RBFIELD, RBAUG); \
} \
RBSTATIC const struct rb_augment_callbacks RBNAME = { \
.propagate = RBNAME ## _propagate, \
.copy = RBNAME ## _copy, \
.rotate = RBNAME ## _rotate \
.rotate = RBNAME ## _rotate, \
.merge = RBNAME ## _merge \
};
/*
* Template for declaring augmented rbtree callbacks (generic single field)
*
* RBSTATIC: 'static' or empty
* RBNAME: name of the rb_augment_callbacks structure
* RBSTRUCT: struct type of the tree nodes
* RBFIELD: name of struct rb_node field within RBSTRUCT
* RBAUGMENTED: name of field within RBSTRUCT holding data for subtree
* RBCOMPUTE: name of function that recomputes the RBAUGMENTED data
*/
#define RB_DECLARE_CALLBACKS(RBSTATIC, RBNAME, \
RBSTRUCT, RBFIELD, RBAUGMENTED, RBCOMPUTE) \
static inline void \
RBNAME ## _copy_single(RBSTRUCT *new, RBSTRUCT *old) \
{ \
new->RBAUGMENTED = old->RBAUGMENTED; \
} \
RB_DECLARE_CALLBACKS_MULTI(RBSTATIC, RBNAME, \
RBSTRUCT, RBFIELD, RBNAME ## _copy_single, RBCOMPUTE)
/*
* Template for declaring augmented rbtree callbacks,
* computing RBAUGMENTED scalar as max(RBCOMPUTE(node)) for all subtree nodes.
@ -159,34 +263,15 @@ RB_DECLARE_CALLBACKS_MULTI(RBSTATIC, RBNAME, \
* RBNAME: name of the rb_augment_callbacks structure
* RBSTRUCT: struct type of the tree nodes
* RBFIELD: name of struct rb_node field within RBSTRUCT
* RBTYPE: type of the RBAUGMENTED field
* RBAUGMENTED: name of RBTYPE field within RBSTRUCT holding data for subtree
* RBCOMPUTE: name of function that returns the per-node RBTYPE scalar
* RBTYPE: type of the RBAUGMENTED field -- unused, assumed typeof(RBAUGMENTED)
* RBAUGMENTED: name of field within RBSTRUCT holding data for subtree
* RBVALUE: name of function that returns the per-node RBTYPE scalar
*/
#define RB_DECLARE_CALLBACKS_MAX(RBSTATIC, RBNAME, RBSTRUCT, RBFIELD, \
RBTYPE, RBAUGMENTED, RBCOMPUTE) \
static inline bool RBNAME ## _compute_max(RBSTRUCT *node, bool exit) \
{ \
RBSTRUCT *child; \
RBTYPE max = RBCOMPUTE(node); \
if (node->RBFIELD.rb_left) { \
child = rb_entry(node->RBFIELD.rb_left, RBSTRUCT, RBFIELD); \
if (child->RBAUGMENTED > max) \
max = child->RBAUGMENTED; \
} \
if (node->RBFIELD.rb_right) { \
child = rb_entry(node->RBFIELD.rb_right, RBSTRUCT, RBFIELD); \
if (child->RBAUGMENTED > max) \
max = child->RBAUGMENTED; \
} \
if (exit && node->RBAUGMENTED == max) \
return true; \
node->RBAUGMENTED = max; \
return false; \
} \
RB_DECLARE_CALLBACKS(RBSTATIC, RBNAME, \
RBSTRUCT, RBFIELD, RBAUGMENTED, RBNAME ## _compute_max)
#define RB_DECLARE_CALLBACKS_MAX(RBSTATIC, RBNAME, RBSTRUCT, RBFIELD, \
RBTYPE, RBAUGMENTED, RBVALUE) \
RB_DECLARE_CALLBACKS(RBSTATIC, RBNAME, RBSTRUCT, RBFIELD, \
RB_AUG_FUNC(RBVALUE, RBAUGMENTED, max))
#define RB_RED 0

37
include/linux/sched.h
View File

@ -554,6 +554,7 @@ struct sched_statistics {
u64 nr_failed_migrations_running;
u64 nr_failed_migrations_hot;
u64 nr_forced_migrations;
u64 nr_migrations_cpu_non_preferred;
u64 nr_wakeups;
u64 nr_wakeups_sync;
@ -2105,10 +2106,9 @@ static inline int test_tsk_thread_flag(struct task_struct *tsk, int flag)
static inline void set_tsk_need_resched(struct task_struct *tsk)
{
if (tracepoint_enabled(sched_set_need_resched_tp) &&
!test_tsk_thread_flag(tsk, TIF_NEED_RESCHED))
if (!test_and_set_tsk_thread_flag(tsk, TIF_NEED_RESCHED) &&
tracepoint_enabled(sched_set_need_resched_tp))
__trace_set_need_resched(tsk, TIF_NEED_RESCHED);
set_tsk_thread_flag(tsk,TIF_NEED_RESCHED);
}
static inline void clear_tsk_need_resched(struct task_struct *tsk)
@ -2135,44 +2135,19 @@ static inline void set_need_resched_current(void)
* value indicates whether a reschedule was done in fact.
* cond_resched_lock() will drop the spinlock before scheduling,
*/
#if !defined(CONFIG_PREEMPTION) || defined(CONFIG_PREEMPT_DYNAMIC)
#if !defined(CONFIG_PREEMPTION)
extern int __cond_resched(void);
#if defined(CONFIG_PREEMPT_DYNAMIC) && defined(CONFIG_HAVE_PREEMPT_DYNAMIC_CALL)
DECLARE_STATIC_CALL(cond_resched, __cond_resched);
static __always_inline int _cond_resched(void)
{
return static_call_mod(cond_resched)();
}
#elif defined(CONFIG_PREEMPT_DYNAMIC) && defined(CONFIG_HAVE_PREEMPT_DYNAMIC_KEY)
extern int dynamic_cond_resched(void);
static __always_inline int _cond_resched(void)
{
return dynamic_cond_resched();
}
#else /* !CONFIG_PREEMPTION */
static inline int _cond_resched(void)
{
return __cond_resched();
}
#endif /* PREEMPT_DYNAMIC && CONFIG_HAVE_PREEMPT_DYNAMIC_CALL */
#else /* CONFIG_PREEMPTION && !CONFIG_PREEMPT_DYNAMIC */
#else
static inline int _cond_resched(void)
{
return 0;
}
#endif /* !CONFIG_PREEMPTION || CONFIG_PREEMPT_DYNAMIC */
#endif
#define cond_resched() ({ \
__might_resched(__FILE__, __LINE__, 0); \

6
include/linux/sched/cputime.h
View File

@ -182,9 +182,9 @@ extern unsigned long long
task_sched_runtime(struct task_struct *task);
#ifdef CONFIG_PARAVIRT
struct static_key;
extern struct static_key paravirt_steal_enabled;
extern struct static_key paravirt_steal_rq_enabled;
#include <linux/jump_label.h>
DECLARE_STATIC_KEY_FALSE(paravirt_steal_enabled);
DECLARE_STATIC_KEY_FALSE(paravirt_steal_rq_enabled);
#ifdef CONFIG_HAVE_PV_STEAL_CLOCK_GEN
u64 dummy_steal_clock(int cpu);

2
include/linux/sched/stat.h
View File

@ -20,7 +20,7 @@ extern int nr_processes(void);
extern unsigned int nr_running(void);
extern bool single_task_running(void);
extern unsigned int nr_iowait(void);
extern unsigned int nr_iowait_cpu(int cpu);
extern bool cpu_has_iowait(int cpu);
static inline int sched_info_on(void)
{

2
include/uapi/linux/sched.h
View File

@ -53,7 +53,7 @@
*/
#define UNSHARE_EMPTY_MNTNS 0x00100000 /* Unshare an empty mount namespace. */
#ifndef __ASSEMBLY__
#ifndef __ASSEMBLER__
/**
* struct clone_args - arguments for the clone3 syscall
* @flags: Flags for the new process as listed above.

13
kernel/Kconfig.preempt
View File

@ -132,10 +132,9 @@ config PREEMPTION
config PREEMPT_DYNAMIC
bool "Preemption behaviour defined on boot"
depends on HAVE_PREEMPT_DYNAMIC
select JUMP_LABEL if HAVE_PREEMPT_DYNAMIC_KEY
depends on ARCH_HAS_PREEMPT_LAZY
select PREEMPT_BUILD
default y if HAVE_PREEMPT_DYNAMIC_CALL
default y
help
This option allows to define the preemption model on the kernel
command line parameter and thus override the default preemption
@ -145,9 +144,7 @@ config PREEMPT_DYNAMIC
provide a pre-built kernel binary to reduce the number of kernel
flavors they offer while still offering different usecases.
The runtime overhead is negligible with HAVE_STATIC_CALL_INLINE enabled
but if runtime patching is not available for the specific architecture
then the potential overhead should be considered.
The runtime overhead is negligible.
Interesting if you want the same pre-built kernel should be used for
both Server and Desktop workloads.
@ -197,3 +194,7 @@ config SCHED_CLASS_EXT
For more information:
Documentation/scheduler/sched-ext.rst
https://github.com/sched-ext/scx
config PREFERRED_CPU
bool
depends on SMP && PARAVIRT

6
kernel/cpu.c
View File

@ -3103,6 +3103,11 @@ EXPORT_SYMBOL(__cpu_dying_mask);
atomic_t __num_online_cpus __read_mostly;
EXPORT_SYMBOL(__num_online_cpus);
#ifdef CONFIG_PREFERRED_CPU
struct cpumask __cpu_preferred_mask __read_mostly;
EXPORT_SYMBOL_GPL(__cpu_preferred_mask);
#endif
void init_cpu_present(const struct cpumask *src)
{
cpumask_copy(&__cpu_present_mask, src);
@ -3160,6 +3165,7 @@ void __init boot_cpu_init(void)
/* Mark the boot cpu "present", "online" etc for SMP and UP case */
set_cpu_online(cpu, true);
set_cpu_active(cpu, true);
set_cpu_preferred(cpu, true);
set_cpu_present(cpu, true);
set_cpu_possible(cpu, true);

17
kernel/entry/common.c
View File

@ -123,7 +123,7 @@ noinstr irqentry_state_t irqentry_enter(struct pt_regs *regs)
/**
* arch_irqentry_exit_need_resched - Architecture specific need resched function
*
* Invoked from raw_irqentry_exit_cond_resched() to check if resched is needed.
* Invoked from irqentry_exit_cond_resched() to check if resched is needed.
* Defaults return true.
*
* The main purpose is to permit arch to avoid preemption of a task from an IRQ.
@ -134,7 +134,7 @@ static inline bool arch_irqentry_exit_need_resched(void);
static inline bool arch_irqentry_exit_need_resched(void) { return true; }
#endif
void raw_irqentry_exit_cond_resched(void)
void irqentry_exit_cond_resched(void)
{
if (!preempt_count()) {
/* Sanity check RCU and thread stack */
@ -145,19 +145,6 @@ void raw_irqentry_exit_cond_resched(void)
preempt_schedule_irq();
}
}
#ifdef CONFIG_PREEMPT_DYNAMIC
#if defined(CONFIG_HAVE_PREEMPT_DYNAMIC_CALL)
DEFINE_STATIC_CALL(irqentry_exit_cond_resched, raw_irqentry_exit_cond_resched);
#elif defined(CONFIG_HAVE_PREEMPT_DYNAMIC_KEY)
DEFINE_STATIC_KEY_TRUE(sk_dynamic_irqentry_exit_cond_resched);
void dynamic_irqentry_exit_cond_resched(void)
{
if (!static_branch_unlikely(&sk_dynamic_irqentry_exit_cond_resched))
return;
raw_irqentry_exit_cond_resched();
}
#endif
#endif
noinstr void irqentry_exit(struct pt_regs *regs, irqentry_state_t state)
{

16
kernel/irq_work.c
View File

@ -252,10 +252,7 @@ static void irq_work_run_list(struct llist_head *list)
irq_work_single(work);
}
/*
* hotplug calls this through:
* hotplug_cfd() -> flush_smp_call_function_queue()
*/
/* CPU hotplug calls this through smpcfd_dying_cpu() */
void irq_work_run(void)
{
irq_work_run_list(this_cpu_ptr(&raised_list));
@ -266,6 +263,17 @@ void irq_work_run(void)
}
EXPORT_SYMBOL_GPL(irq_work_run);
void irq_work_run_cpu(unsigned int cpu)
{
if (WARN_ON_ONCE(!cpumask_test_cpu(cpu, cpu_dying_mask)))
return;
if (!IS_ENABLED(CONFIG_PREEMPT_RT))
return;
irq_work_run_list(per_cpu_ptr(&lazy_list, cpu));
}
void irq_work_tick(void)
{
struct llist_head *raised = this_cpu_ptr(&raised_list);

464
kernel/sched/core.c
View File

@ -805,7 +805,7 @@ struct rq *_task_rq_lock(struct task_struct *p, struct rq_flags *rf)
/* Use CONFIG_PARAVIRT as this will avoid more #ifdef in arch code. */
#ifdef CONFIG_PARAVIRT
struct static_key paravirt_steal_rq_enabled;
DEFINE_STATIC_KEY_FALSE(paravirt_steal_rq_enabled);
#endif
static void update_rq_clock_task(struct rq *rq, s64 delta)
@ -844,7 +844,7 @@ static void update_rq_clock_task(struct rq *rq, s64 delta)
}
#endif
#ifdef CONFIG_PARAVIRT_TIME_ACCOUNTING
if (static_key_false((&paravirt_steal_rq_enabled))) {
if (static_branch_unlikely(&paravirt_steal_rq_enabled)) {
u64 prev_steal;
steal = prev_steal = paravirt_steal_clock(cpu_of(rq));
@ -1196,6 +1196,7 @@ static void __resched_curr(struct rq *rq, int tif)
{
struct task_struct *curr = rq->curr;
struct thread_info *cti = task_thread_info(curr);
bool need_ipi;
int cpu;
lockdep_assert_rq_held(rq);
@ -1212,15 +1213,17 @@ static void __resched_curr(struct rq *rq, int tif)
cpu = cpu_of(rq);
trace_sched_set_need_resched_tp(curr, cpu, tif);
if (cpu == smp_processor_id()) {
set_ti_thread_flag(cti, tif);
if (tif == TIF_NEED_RESCHED)
set_preempt_need_resched();
trace_sched_set_need_resched_tp(curr, cpu, tif);
return;
}
if (set_nr_and_not_polling(cti, tif)) {
need_ipi = set_nr_and_not_polling(cti, tif);
trace_sched_set_need_resched_tp(curr, cpu, tif);
if (need_ipi) {
if (tif == TIF_NEED_RESCHED)
smp_send_reschedule(cpu);
} else {
@ -2252,7 +2255,8 @@ void deactivate_task(struct rq *rq, struct task_struct *p, int flags)
dequeue_task(rq, p, flags);
}
static void block_task(struct rq *rq, struct task_struct *p, unsigned long task_state)
static bool dequeue_block_task(struct rq *rq, struct task_struct *p,
unsigned long task_state)
{
int flags = DEQUEUE_NOCLOCK;
@ -2273,9 +2277,15 @@ static void block_task(struct rq *rq, struct task_struct *p, unsigned long task_
*
* Where __schedule() and ttwu() have matching control dependencies.
*
* After this, schedule() must not care about p->state any more.
* Once the caller invokes __block_task(), schedule() must not care about
* p->state any more.
*/
if (dequeue_task(rq, p, DEQUEUE_SLEEP | flags))
return dequeue_task(rq, p, DEQUEUE_SLEEP | flags);
}
static void block_task(struct rq *rq, struct task_struct *p, unsigned long task_state)
{
if (dequeue_block_task(rq, p, task_state))
__block_task(rq, p);
}
@ -2504,6 +2514,24 @@ static inline bool rq_has_pinned_tasks(struct rq *rq)
return rq->nr_pinned;
}
static inline bool task_can_migrate_to_preferred(struct task_struct *p, int cpu)
{
/* No need to migrate from a preferred CPU */
if (cpu_preferred(cpu))
return false;
/* Only FAIR tasks honor preferred CPU state */
if (unlikely(p->sched_class != &fair_sched_class))
return false;
/* Ignore preferred state if task affinity is changing */
if (unlikely(!cpumask_test_cpu(task_cpu(p), p->cpus_ptr)))
return false;
return cpumask_intersects_and(p->cpus_ptr, cpu_preferred_mask,
task_cpu_possible_mask(p));
}
/*
* Per-CPU kthreads are allowed to run on !active && online CPUs, see
* __set_cpus_allowed_ptr() and select_fallback_rq().
@ -2519,8 +2547,12 @@ static inline bool is_cpu_allowed(struct task_struct *p, int cpu)
return cpu_online(cpu);
/* Non kernel threads are not allowed during either online or offline. */
if (!(p->flags & PF_KTHREAD))
if (!(p->flags & PF_KTHREAD)) {
/* Try to use preferred CPU if task's affinity allows */
if (task_can_migrate_to_preferred(p, cpu))
return false;
return cpu_active(cpu);
}
/* KTHREAD_IS_PER_CPU is always allowed. */
if (kthread_is_per_cpu(p))
@ -2530,7 +2562,11 @@ static inline bool is_cpu_allowed(struct task_struct *p, int cpu)
if (cpu_dying(cpu))
return false;
/* But are allowed during online. */
/* Try to keep unbound kthreads on a preferred CPU if possible. */
if (task_can_migrate_to_preferred(p, cpu))
return false;
/* Otherwise, they are allowed to run on online CPU. */
return cpu_online(cpu);
}
@ -3773,6 +3809,7 @@ static inline void proxy_reset_donor(struct rq *rq)
WARN_ON_ONCE(rq->donor == rq->curr);
put_prev_set_next_task(rq, rq->donor, rq->curr);
rq->next_class = rq->curr->sched_class;
rq_set_donor(rq, rq->curr);
zap_balance_callbacks(rq);
resched_curr(rq);
@ -3787,6 +3824,8 @@ static inline void proxy_reset_donor(struct rq *rq)
*/
static inline bool proxy_needs_return(struct rq *rq, struct task_struct *p)
{
bool dequeued;
/*
* Typically per __set_task_cpu(), task_cpu(p) == p->wake_cpu.
*
@ -3809,12 +3848,23 @@ static inline bool proxy_needs_return(struct rq *rq, struct task_struct *p)
/* If already current, don't need to return migrate */
if (task_current(rq, p))
return false;
/* If we're return migrating the rq->donor, switch it out for idle */
if (task_current_donor(rq, p))
proxy_reset_donor(rq);
}
block_task(rq, p, TASK_WAKING);
dequeued = dequeue_block_task(rq, p, TASK_WAKING);
/*
* Dequeue @p from its scheduling class before resetting rq->donor.
* In particular, sched_ext needs to end the donor's running session
* and clear SCX_TASK_QUEUED before put_prev_task_scx() is called by
* proxy_reset_donor(); otherwise it would reenqueue the blocked donor.
*
* Keep on_rq set until all donor references have been replaced.
*/
if (task_current_donor(rq, p))
proxy_reset_donor(rq);
if (dequeued)
__block_task(rq, p);
return true;
}
#else /* !CONFIG_SCHED_PROXY_EXEC */
@ -3905,7 +3955,7 @@ static int ttwu_runnable(struct task_struct *p, int wake_flags)
* When on_rq && !on_cpu the task is preempted, see if
* it should preempt the task that is current now.
*/
wakeup_preempt(rq, p, wake_flags);
wakeup_preempt(rq, p, wake_flags | WF_TTWU_RQ);
}
ttwu_do_wakeup(p);
return 1;
@ -5149,7 +5199,7 @@ static void do_balance_callbacks(struct rq *rq, struct balance_callback *head)
lockdep_assert_rq_held(rq);
while (head) {
func = (void (*)(struct rq *))head->func;
func = head->func;
next = head->next;
head->next = NULL;
head = next;
@ -5588,15 +5638,12 @@ unsigned long long nr_context_switches(void)
}
/*
* Consumers of these two interfaces, like for example the cpuidle menu
* governor, are using nonsensical data. Preferring shallow idle state selection
* for a CPU that has IO-wait which might not even end up running the task when
* it does become runnable.
* Return whether any tasks are accounted as I/O waiters on this CPU.
* They may wake up on a different CPU.
*/
unsigned int nr_iowait_cpu(int cpu)
bool cpu_has_iowait(int cpu)
{
return atomic_read(&cpu_rq(cpu)->nr_iowait);
return atomic_read(&cpu_rq(cpu)->nr_iowait) != 0;
}
/*
@ -5634,7 +5681,7 @@ unsigned int nr_iowait(void)
unsigned int i, sum = 0;
for_each_possible_cpu(i)
sum += nr_iowait_cpu(i);
sum += atomic_read(&cpu_rq(i)->nr_iowait);
return sum;
}
@ -5789,6 +5836,9 @@ void sched_tick(void)
unsigned long hw_pressure;
u64 resched_latency;
if (!cpu_preferred(cpu))
sched_push_current_non_preferred_cpu(rq);
if (housekeeping_cpu(cpu, HK_TYPE_KERNEL_NOISE))
arch_scale_freq_tick();
@ -6283,10 +6333,7 @@ pick_next_task(struct rq *rq, struct rq_flags *rf)
* selection. In this case, do a core-wide selection.
*/
if (rq->core->core_pick_seq == rq->core->core_task_seq &&
rq->core->core_pick_seq != rq->core_sched_seq &&
rq->core_pick) {
WRITE_ONCE(rq->core_sched_seq, rq->core->core_pick_seq);
next = rq->core_pick;
rq->dl_server = rq->core_dl_server;
rq->core_pick = NULL;
@ -6318,11 +6365,13 @@ pick_next_task(struct rq *rq, struct rq_flags *rf)
}
/*
* core->core_task_seq, core->core_pick_seq, rq->core_sched_seq
* core->core_task_seq, core->core_pick_seq
*
* @task_seq guards the task state ({en,de}queues)
* @pick_seq is the @task_seq we did a selection on
* @sched_seq is the @pick_seq we scheduled
*
* Once a core-wide selection is committed, a non-NULL core_pick denotes
* a pick which still needs to be consumed on this CPU.
*
* However, preemptions can cause multiple picks on the same task set.
* 'Fix' this by also increasing @task_seq for every pick.
@ -6429,7 +6478,6 @@ pick_next_task(struct rq *rq, struct rq_flags *rf)
rq->core->core_pick_seq = rq->core->core_task_seq;
next = rq->core_pick;
rq->core_sched_seq = rq->core->core_pick_seq;
/* Something should have been selected for current CPU */
WARN_ON_ONCE(!next);
@ -6517,7 +6565,10 @@ static bool try_steal_cookie(int this, int that)
return false;
do {
if (p == src->core_pick || p == src->curr)
if (p == src->core_pick || p == src->curr || p == src->donor)
goto next;
if (task_is_blocked(p))
goto next;
if (!is_cpu_allowed(p, this))
@ -6820,6 +6871,34 @@ static void proxy_deactivate(struct rq *rq, struct task_struct *donor)
block_task(rq, donor, state);
}
/*
* Remove a retained proxy donor before changing its scheduler ownership.
* The caller holds p->pi_lock, so p cannot wake and migrate if block_task()
* drops it from the runqueue. If DELAY_DEQUEUE keeps a blocked fair task
* queued, switching_from_fair() completes the dequeue in the immediately
* following sched_change_begin().
*/
void sched_proxy_block_task(struct rq *rq, struct task_struct *p)
{
unsigned long state = READ_ONCE(p->__state);
lockdep_assert_held(&p->pi_lock);
lockdep_assert_rq_held(rq);
if (!p->is_blocked || !task_on_rq_queued(p))
return;
if (WARN_ON_ONCE(state == TASK_RUNNING))
return;
if (task_current_donor(rq, p))
proxy_reset_donor(rq);
if (!p->se.sched_delayed)
block_task(rq, p, state);
WARN_ON_ONCE(task_on_rq_queued(p) && !p->se.sched_delayed);
}
static inline void proxy_release_rq_lock(struct rq *rq, struct rq_flags *rf)
__releases(__rq_lockp(rq))
{
@ -6865,9 +6944,9 @@ static void proxy_migrate_task(struct rq *rq, struct rq_flags *rf,
__must_hold(__rq_lockp(rq))
{
struct rq *target_rq = cpu_rq(target_cpu);
LIST_HEAD(migrate_list);
lockdep_assert_rq_held(rq);
WARN_ON(p == rq->curr);
/*
* Since we are migrating a blocked donor, it could be rq->donor,
* and we want to make sure there aren't any references from this
@ -6880,13 +6959,20 @@ static void proxy_migrate_task(struct rq *rq, struct rq_flags *rf,
* before we release the lock.
*/
proxy_resched_idle(rq);
deactivate_task(rq, p, DEQUEUE_NOCLOCK);
proxy_set_task_cpu(p, target_cpu);
for (; p; p = p->blocked_donor) {
WARN_ON(p == rq->curr);
deactivate_task(rq, p, DEQUEUE_NOCLOCK);
proxy_set_task_cpu(p, target_cpu);
/*
* We can re-use se.group_node to migrate the thing,
* because @p is deactivated (won't be balanced) and
* we hold the rq_lock.
*/
list_add(&p->se.group_node, &migrate_list);
}
proxy_release_rq_lock(rq, rf);
attach_one_task(target_rq, p);
__attach_tasks(target_rq, &migrate_list);
proxy_reacquire_rq_lock(rq, rf);
}
@ -6979,7 +7065,7 @@ find_proxy_task(struct rq *rq, struct task_struct *donor, struct rq_flags *rf)
if (!READ_ONCE(owner->on_rq) || owner->se.sched_delayed) {
/* XXX Don't handle blocked owners/delayed dequeue yet */
if (curr_in_chain)
return proxy_resched_idle(rq);
goto resched_idle;
__clear_task_blocked_on(p, NULL);
goto deactivate;
}
@ -6991,7 +7077,7 @@ find_proxy_task(struct rq *rq, struct task_struct *donor, struct rq_flags *rf)
* and leave that CPU to sort things out.
*/
if (curr_in_chain)
return proxy_resched_idle(rq);
goto resched_idle;
goto migrate_task;
}
@ -7004,7 +7090,7 @@ find_proxy_task(struct rq *rq, struct task_struct *donor, struct rq_flags *rf)
* case we should end up back in find_proxy_task(), this time
* hopefully with all relevant tasks already enqueued.
*/
return proxy_resched_idle(rq);
goto resched_idle;
}
/*
@ -7041,7 +7127,7 @@ find_proxy_task(struct rq *rq, struct task_struct *donor, struct rq_flags *rf)
* So schedule rq->idle so that ttwu_runnable() can get the rq
* lock and mark owner as running.
*/
return proxy_resched_idle(rq);
goto resched_idle;
}
/*
* OK, now we're absolutely sure @owner is on this
@ -7051,8 +7137,18 @@ find_proxy_task(struct rq *rq, struct task_struct *donor, struct rq_flags *rf)
owner->blocked_donor = p;
}
WARN_ON_ONCE(owner && !owner->on_rq);
if (owner && !sched_cpu_cookie_match(rq, owner)) {
if (curr_in_chain)
return proxy_resched_idle(rq);
p = donor; /* Deactivate the donor, not the runnable owner */
clear_task_blocked_on(p, NULL);
goto deactivate;
}
return owner;
resched_idle:
return proxy_resched_idle(rq);
deactivate:
proxy_deactivate(rq, p);
return NULL;
@ -7184,13 +7280,12 @@ static void __sched notrace __schedule(int sched_mode)
}
} else if (!preempt && prev_state) {
/*
* We pass task_is_blocked() as the should_block arg
* in order to keep mutex-blocked tasks on the runqueue
* for slection with proxy-exec (without proxy-exec
* task_is_blocked() will always be false).
* Keep mutex-blocked tasks on the runqueue for proxy execution
* only when their scheduling class allows it. Without proxy
* execution, task_is_blocked() always returns false.
*/
try_to_block_task(rq, prev, &prev_state,
!task_is_blocked(prev));
!task_is_blocked(prev) || !scx_allow_proxy_exec(prev));
switch_count = &prev->nvcsw;
}
@ -7211,6 +7306,7 @@ static void __sched notrace __schedule(int sched_mode)
}
if (next == rq->idle) {
zap_balance_callbacks(rq);
scx_proxy_reenqueue_retry(rq, next);
goto keep_resched;
}
}
@ -7229,8 +7325,10 @@ static void __sched notrace __schedule(int sched_mode)
* on_cpu.
*/
donor->sched_class->put_prev_task(rq, donor, donor);
donor->sched_class->set_next_task(rq, donor, true);
donor->sched_class->set_next_task(rq, donor, SNT_PICK);
}
scx_proxy_donor_start(rq);
scx_proxy_reenqueue_retry(rq, next);
} else {
rq_set_donor(rq, next);
}
@ -7489,27 +7587,6 @@ asmlinkage __visible void __sched notrace preempt_schedule(void)
NOKPROBE_SYMBOL(preempt_schedule);
EXPORT_SYMBOL(preempt_schedule);
#ifdef CONFIG_PREEMPT_DYNAMIC
# ifdef CONFIG_HAVE_PREEMPT_DYNAMIC_CALL
# ifndef preempt_schedule_dynamic_enabled
# define preempt_schedule_dynamic_enabled preempt_schedule
# define preempt_schedule_dynamic_disabled NULL
# endif
DEFINE_STATIC_CALL(preempt_schedule, preempt_schedule_dynamic_enabled);
EXPORT_STATIC_CALL_TRAMP(preempt_schedule);
# elif defined(CONFIG_HAVE_PREEMPT_DYNAMIC_KEY)
static DEFINE_STATIC_KEY_TRUE(sk_dynamic_preempt_schedule);
void __sched notrace dynamic_preempt_schedule(void)
{
if (!static_branch_unlikely(&sk_dynamic_preempt_schedule))
return;
preempt_schedule();
}
NOKPROBE_SYMBOL(dynamic_preempt_schedule);
EXPORT_SYMBOL(dynamic_preempt_schedule);
# endif
#endif /* CONFIG_PREEMPT_DYNAMIC */
/**
* preempt_schedule_notrace - preempt_schedule called by tracing
*
@ -7562,27 +7639,6 @@ asmlinkage __visible void __sched notrace preempt_schedule_notrace(void)
}
EXPORT_SYMBOL_GPL(preempt_schedule_notrace);
#ifdef CONFIG_PREEMPT_DYNAMIC
# if defined(CONFIG_HAVE_PREEMPT_DYNAMIC_CALL)
# ifndef preempt_schedule_notrace_dynamic_enabled
# define preempt_schedule_notrace_dynamic_enabled preempt_schedule_notrace
# define preempt_schedule_notrace_dynamic_disabled NULL
# endif
DEFINE_STATIC_CALL(preempt_schedule_notrace, preempt_schedule_notrace_dynamic_enabled);
EXPORT_STATIC_CALL_TRAMP(preempt_schedule_notrace);
# elif defined(CONFIG_HAVE_PREEMPT_DYNAMIC_KEY)
static DEFINE_STATIC_KEY_TRUE(sk_dynamic_preempt_schedule_notrace);
void __sched notrace dynamic_preempt_schedule_notrace(void)
{
if (!static_branch_unlikely(&sk_dynamic_preempt_schedule_notrace))
return;
preempt_schedule_notrace();
}
NOKPROBE_SYMBOL(dynamic_preempt_schedule_notrace);
EXPORT_SYMBOL(dynamic_preempt_schedule_notrace);
# endif
#endif
#endif /* CONFIG_PREEMPTION */
/*
@ -7799,7 +7855,7 @@ void rt_mutex_setprio(struct task_struct *p, struct task_struct *pi_task)
}
#endif /* CONFIG_RT_MUTEXES */
#if !defined(CONFIG_PREEMPTION) || defined(CONFIG_PREEMPT_DYNAMIC)
#if !defined(CONFIG_PREEMPTION)
int __sched __cond_resched(void)
{
if (should_resched(0) && !irqs_disabled()) {
@ -7827,38 +7883,6 @@ int __sched __cond_resched(void)
EXPORT_SYMBOL(__cond_resched);
#endif
#ifdef CONFIG_PREEMPT_DYNAMIC
# ifdef CONFIG_HAVE_PREEMPT_DYNAMIC_CALL
# define cond_resched_dynamic_enabled __cond_resched
# define cond_resched_dynamic_disabled ((void *)&__static_call_return0)
DEFINE_STATIC_CALL_RET0(cond_resched, __cond_resched);
EXPORT_STATIC_CALL_TRAMP(cond_resched);
# define might_resched_dynamic_enabled __cond_resched
# define might_resched_dynamic_disabled ((void *)&__static_call_return0)
DEFINE_STATIC_CALL_RET0(might_resched, __cond_resched);
EXPORT_STATIC_CALL_TRAMP(might_resched);
# elif defined(CONFIG_HAVE_PREEMPT_DYNAMIC_KEY)
static DEFINE_STATIC_KEY_FALSE(sk_dynamic_cond_resched);
int __sched dynamic_cond_resched(void)
{
if (!static_branch_unlikely(&sk_dynamic_cond_resched))
return 0;
return __cond_resched();
}
EXPORT_SYMBOL(dynamic_cond_resched);
static DEFINE_STATIC_KEY_FALSE(sk_dynamic_might_resched);
int __sched dynamic_might_resched(void)
{
if (!static_branch_unlikely(&sk_dynamic_might_resched))
return 0;
return __cond_resched();
}
EXPORT_SYMBOL(dynamic_might_resched);
# endif
#endif /* CONFIG_PREEMPT_DYNAMIC */
/*
* __cond_resched_lock() - if a reschedule is pending, drop the given lock,
* call schedule, and on return reacquire the lock.
@ -7928,50 +7952,21 @@ EXPORT_SYMBOL(__cond_resched_rwlock_write);
# endif
/*
* SC:cond_resched
* SC:might_resched
* SC:preempt_schedule
* SC:preempt_schedule_notrace
* SC:irqentry_exit_cond_resched
*
*
* NONE:
* cond_resched <- __cond_resched
* might_resched <- RET0
* preempt_schedule <- NOP
* preempt_schedule_notrace <- NOP
* irqentry_exit_cond_resched <- NOP
* dynamic_preempt_lazy <- false
* (unselectable)
*
* VOLUNTARY:
* cond_resched <- __cond_resched
* might_resched <- __cond_resched
* preempt_schedule <- NOP
* preempt_schedule_notrace <- NOP
* irqentry_exit_cond_resched <- NOP
* dynamic_preempt_lazy <- false
* (unselectable)
*
* FULL:
* cond_resched <- RET0
* might_resched <- RET0
* preempt_schedule <- preempt_schedule
* preempt_schedule_notrace <- preempt_schedule_notrace
* irqentry_exit_cond_resched <- irqentry_exit_cond_resched
* dynamic_preempt_lazy <- false
*
* LAZY:
* cond_resched <- RET0
* might_resched <- RET0
* preempt_schedule <- preempt_schedule
* preempt_schedule_notrace <- preempt_schedule_notrace
* irqentry_exit_cond_resched <- irqentry_exit_cond_resched
* dynamic_preempt_lazy <- true
*/
enum {
preempt_dynamic_undefined = -1,
preempt_dynamic_none,
preempt_dynamic_voluntary,
preempt_dynamic_full,
preempt_dynamic_lazy,
};
@ -7980,21 +7975,11 @@ int preempt_dynamic_mode = preempt_dynamic_undefined;
int sched_dynamic_mode(const char *str)
{
# if !(defined(CONFIG_PREEMPT_RT) || defined(CONFIG_ARCH_HAS_PREEMPT_LAZY))
if (!strcmp(str, "none"))
return preempt_dynamic_none;
if (!strcmp(str, "voluntary"))
return preempt_dynamic_voluntary;
# endif
if (!strcmp(str, "full"))
return preempt_dynamic_full;
# ifdef CONFIG_ARCH_HAS_PREEMPT_LAZY
if (!strcmp(str, "lazy"))
return preempt_dynamic_lazy;
# endif
return -EINVAL;
}
@ -8002,71 +7987,18 @@ int sched_dynamic_mode(const char *str)
# define preempt_dynamic_key_enable(f) static_key_enable(&sk_dynamic_##f.key)
# define preempt_dynamic_key_disable(f) static_key_disable(&sk_dynamic_##f.key)
# if defined(CONFIG_HAVE_PREEMPT_DYNAMIC_CALL)
# define preempt_dynamic_enable(f) static_call_update(f, f##_dynamic_enabled)
# define preempt_dynamic_disable(f) static_call_update(f, f##_dynamic_disabled)
# elif defined(CONFIG_HAVE_PREEMPT_DYNAMIC_KEY)
# define preempt_dynamic_enable(f) preempt_dynamic_key_enable(f)
# define preempt_dynamic_disable(f) preempt_dynamic_key_disable(f)
# else
# error "Unsupported PREEMPT_DYNAMIC mechanism"
# endif
static DEFINE_MUTEX(sched_dynamic_mutex);
static void __sched_dynamic_update(int mode)
{
/*
* Avoid {NONE,VOLUNTARY} -> FULL transitions from ever ending up in
* the ZERO state, which is invalid.
*/
preempt_dynamic_enable(cond_resched);
preempt_dynamic_enable(might_resched);
preempt_dynamic_enable(preempt_schedule);
preempt_dynamic_enable(preempt_schedule_notrace);
preempt_dynamic_enable(irqentry_exit_cond_resched);
preempt_dynamic_key_disable(preempt_lazy);
switch (mode) {
case preempt_dynamic_none:
preempt_dynamic_enable(cond_resched);
preempt_dynamic_disable(might_resched);
preempt_dynamic_disable(preempt_schedule);
preempt_dynamic_disable(preempt_schedule_notrace);
preempt_dynamic_disable(irqentry_exit_cond_resched);
preempt_dynamic_key_disable(preempt_lazy);
if (mode != preempt_dynamic_mode)
pr_info("Dynamic Preempt: none\n");
break;
case preempt_dynamic_voluntary:
preempt_dynamic_enable(cond_resched);
preempt_dynamic_enable(might_resched);
preempt_dynamic_disable(preempt_schedule);
preempt_dynamic_disable(preempt_schedule_notrace);
preempt_dynamic_disable(irqentry_exit_cond_resched);
preempt_dynamic_key_disable(preempt_lazy);
if (mode != preempt_dynamic_mode)
pr_info("Dynamic Preempt: voluntary\n");
break;
case preempt_dynamic_full:
preempt_dynamic_disable(cond_resched);
preempt_dynamic_disable(might_resched);
preempt_dynamic_enable(preempt_schedule);
preempt_dynamic_enable(preempt_schedule_notrace);
preempt_dynamic_enable(irqentry_exit_cond_resched);
preempt_dynamic_key_disable(preempt_lazy);
if (mode != preempt_dynamic_mode)
pr_info("Dynamic Preempt: full\n");
break;
case preempt_dynamic_lazy:
preempt_dynamic_disable(cond_resched);
preempt_dynamic_disable(might_resched);
preempt_dynamic_enable(preempt_schedule);
preempt_dynamic_enable(preempt_schedule_notrace);
preempt_dynamic_enable(irqentry_exit_cond_resched);
preempt_dynamic_key_enable(preempt_lazy);
if (mode != preempt_dynamic_mode)
pr_info("Dynamic Preempt: lazy\n");
@ -8099,11 +8031,7 @@ __setup("preempt=", setup_preempt_mode);
static void __init preempt_dynamic_init(void)
{
if (preempt_dynamic_mode == preempt_dynamic_undefined) {
if (IS_ENABLED(CONFIG_PREEMPT_NONE)) {
sched_dynamic_update(preempt_dynamic_none);
} else if (IS_ENABLED(CONFIG_PREEMPT_VOLUNTARY)) {
sched_dynamic_update(preempt_dynamic_voluntary);
} else if (IS_ENABLED(CONFIG_PREEMPT_LAZY)) {
if (IS_ENABLED(CONFIG_PREEMPT_LAZY)) {
sched_dynamic_update(preempt_dynamic_lazy);
} else {
/* Default static call setting, nothing to do */
@ -8123,8 +8051,6 @@ static void __init preempt_dynamic_init(void)
} \
EXPORT_SYMBOL_GPL(preempt_model_##mode)
PREEMPT_MODEL_ACCESSOR(none);
PREEMPT_MODEL_ACCESSOR(voluntary);
PREEMPT_MODEL_ACCESSOR(full);
PREEMPT_MODEL_ACCESSOR(lazy);
@ -8137,7 +8063,7 @@ static inline void preempt_dynamic_init(void) { }
#endif /* CONFIG_PREEMPT_DYNAMIC */
const char *preempt_modes[] = {
"none", "voluntary", "full", "lazy", NULL,
"full", "lazy", NULL,
};
const char *preempt_model_str(void)
@ -8759,6 +8685,9 @@ int sched_cpu_activate(unsigned int cpu)
*/
sched_set_rq_online(rq, cpu);
/* preferred is subset of active and follows its state */
set_cpu_preferred(cpu, true);
return 0;
}
@ -8772,6 +8701,8 @@ int sched_cpu_deactivate(unsigned int cpu)
if (ret)
return ret;
set_cpu_preferred(cpu, false);
/*
* Remove CPU from nohz.idle_cpus_mask to prevent participating in
* load balancing when not active
@ -11348,3 +11279,88 @@ void sched_change_end(struct sched_change_ctx *ctx)
p->sched_class->prio_changed(rq, p, ctx->prio);
}
}
#ifdef CONFIG_PREFERRED_CPU
static DEFINE_PER_CPU(struct cpu_stop_work, npc_push_task_work);
static int sched_non_preferred_cpu_push_stop(void *arg)
{
struct task_struct *p = arg;
struct rq *rq = this_rq();
struct rq_flags rf;
int cpu;
if (cpu_preferred(rq->cpu)) {
scoped_guard(rq_lock_irqsave, rq)
rq->npc_push_work_pending = false;
put_task_struct(p);
return 0;
}
scoped_guard (raw_spinlock_irq, &p->pi_lock) {
/*
* select_fallback_rq() may acquire the rq lock in case of
* fallback. So call it before grabbing rq lock. If the task
* migrates to another CPU before the rq lock is acquired,
* subsequent validation of task's current rq will help to
* safely bail out.
*/
cpu = select_fallback_rq(rq->cpu, p);
context_unsafe_alias(rq);
rq_lock(rq, &rf);
rq->npc_push_work_pending = false;
update_rq_clock(rq);
if (task_rq(p) == rq && task_on_rq_queued(p)) {
struct rq *dest_rq = __migrate_task(rq, &rf, p, cpu);
if (rq != dest_rq)
schedstat_inc(p->stats.nr_migrations_cpu_non_preferred);
rq = dest_rq;
}
rq_unlock(rq, &rf);
}
put_task_struct(p);
return 0;
}
/*
* Push the current task running on non-preferred CPU(npc).
* Using this non preferred CPU will lead to more contention
* in the host. So it is better not to use this CPU.
*
* Since task is running, call a stopper to push the task out. This is
* similar to how task moves during hotplug. In select_fallback_rq() a
* preferred CPU will be chosen and henceforth task shouldn't come back to
* this CPU again.
*
* Works for FAIR class only.
*
* If task is affined only on non-preferred CPUs, no point in moving it out.
*/
void sched_push_current_non_preferred_cpu(struct rq *rq)
{
struct task_struct *push_task = rq->curr;
scoped_guard(rq_lock, rq) {
/* Push the task if its explicit affinity allows */
if (!task_can_migrate_to_preferred(push_task, rq->cpu))
return;
/* There is already a stopper thread. Don't race with it. */
if (rq->npc_push_work_pending)
return;
if (is_migration_disabled(push_task))
return;
rq->npc_push_work_pending = true;
}
/* sched_tick runs with interrupts disabled. */
get_task_struct(push_task);
stop_one_cpu_nowait(rq->cpu, sched_non_preferred_cpu_push_stop,
push_task, this_cpu_ptr(&npc_push_task_work));
}
#endif

63
kernel/sched/cputime.c
View File

@ -226,9 +226,8 @@ void account_steal_time(u64 cputime)
void account_idle_time(u64 cputime)
{
u64 *cpustat = kcpustat_this_cpu->cpustat;
struct rq *rq = this_rq();
if (atomic_read(&rq->nr_iowait) > 0)
if (cpu_has_iowait(smp_processor_id()))
cpustat[CPUTIME_IOWAIT] += cputime;
else
cpustat[CPUTIME_IDLE] += cputime;
@ -255,7 +254,7 @@ void __account_forceidle_time(struct task_struct *p, u64 delta)
* occasion account more time than the calling functions think elapsed.
*/
#ifdef CONFIG_PARAVIRT
struct static_key paravirt_steal_enabled;
DEFINE_STATIC_KEY_FALSE(paravirt_steal_enabled);
#ifdef CONFIG_HAVE_PV_STEAL_CLOCK_GEN
static u64 native_steal_clock(int cpu)
@ -270,7 +269,7 @@ DEFINE_STATIC_CALL(pv_steal_clock, native_steal_clock);
static __always_inline u64 steal_account_process_time(u64 maxtime)
{
#ifdef CONFIG_PARAVIRT
if (static_key_false(&paravirt_steal_enabled)) {
if (static_branch_unlikely(&paravirt_steal_enabled)) {
u64 steal;
steal = paravirt_steal_clock(smp_processor_id());
@ -381,9 +380,9 @@ void thread_group_cputime(struct task_struct *tsk, struct task_cputime *times)
* softirq as those do not count in task exec_runtime any more.
*/
static void irqtime_account_process_tick(struct task_struct *p, int user_tick,
int ticks)
u64 cputime)
{
u64 other, cputime = TICK_NSEC * ticks;
u64 other;
/*
* When returning from idle, many ticks can get accounted at
@ -418,7 +417,7 @@ static void irqtime_account_process_tick(struct task_struct *p, int user_tick,
#else /* !CONFIG_IRQ_TIME_ACCOUNTING: */
static inline void irqtime_account_process_tick(struct task_struct *p, int user_tick,
int nr_ticks) { }
u64 cputime) { }
#endif /* !CONFIG_IRQ_TIME_ACCOUNTING */
#if defined(CONFIG_NO_HZ_COMMON) && !defined(CONFIG_HAVE_VIRT_CPU_ACCOUNTING_IDLE)
@ -431,7 +430,7 @@ static void kcpustat_idle_stop(struct kernel_cpustat *kc, u64 now)
if (!kc->idle_elapse)
return;
iowait = nr_iowait_cpu(smp_processor_id()) > 0;
iowait = cpu_has_iowait(smp_processor_id());
delta = now - kc->idle_entrytime;
steal = steal_account_process_time(delta);
@ -468,18 +467,38 @@ static void kcpustat_idle_start(struct kernel_cpustat *kc, u64 now)
write_seqcount_end(&kc->idle_sleeptime_seq);
}
void kcpustat_dyntick_stop(u64 now)
void kcpustat_dyntick_stop(u64 now, u64 tick_start)
{
struct kernel_cpustat *kc = kcpustat_this_cpu;
if (!vtime_generic_enabled_this_cpu()) {
WARN_ON_ONCE(!kc->idle_dyntick);
if (kc->idle_tick_period != tick_start) {
kc->idle_tick_period = tick_start;
kc->idle_tick_overlap = 0;
}
tick_start = max(tick_start, kc->idle_dyntick_entry);
if (!WARN_ON_ONCE(now < tick_start))
kc->idle_tick_overlap += now - tick_start;
kcpustat_idle_stop(kc, now);
kc->idle_dyntick = false;
vtime_dyntick_stop();
}
}
/*
* The first tick after dyntick-idle covers a period that the dyntick-idle
* accounting may already have accounted in part.
*/
static inline u64 kcpustat_tick_overlap(void)
{
struct kernel_cpustat *kc = kcpustat_this_cpu;
u64 overlap = kc->idle_tick_overlap;
kc->idle_tick_overlap = 0;
return overlap;
}
void kcpustat_dyntick_start(u64 now)
{
struct kernel_cpustat *kc = kcpustat_this_cpu;
@ -487,6 +506,7 @@ void kcpustat_dyntick_start(u64 now)
if (!vtime_generic_enabled_this_cpu()) {
vtime_dyntick_start();
kc->idle_dyntick = true;
kc->idle_dyntick_entry = now;
kcpustat_idle_start(kc, now);
}
}
@ -544,17 +564,22 @@ static u64 kcpustat_field_dyntick(int cpu, enum cpu_usage_stat idx,
u64 kcpustat_field_idle(int cpu)
{
return kcpustat_field_dyntick(cpu, CPUTIME_IDLE,
!nr_iowait_cpu(cpu), ktime_get());
!cpu_has_iowait(cpu), ktime_get());
}
EXPORT_SYMBOL_GPL(kcpustat_field_idle);
u64 kcpustat_field_iowait(int cpu)
{
return kcpustat_field_dyntick(cpu, CPUTIME_IOWAIT,
nr_iowait_cpu(cpu), ktime_get());
cpu_has_iowait(cpu), ktime_get());
}
EXPORT_SYMBOL_GPL(kcpustat_field_iowait);
#else
static inline u64 kcpustat_tick_overlap(void)
{
return 0;
}
static u64 kcpustat_field_dyntick(int cpu, enum cpu_usage_stat idx,
bool compute_delta, ktime_t now)
{
@ -601,7 +626,7 @@ static u64 get_cpu_sleep_time_us(int cpu, enum cpu_usage_stat idx,
u64 get_cpu_idle_time_us(int cpu, u64 *last_update_time)
{
return get_cpu_sleep_time_us(cpu, CPUTIME_IDLE,
!nr_iowait_cpu(cpu), last_update_time);
!cpu_has_iowait(cpu), last_update_time);
}
EXPORT_SYMBOL_GPL(get_cpu_idle_time_us);
@ -625,7 +650,7 @@ EXPORT_SYMBOL_GPL(get_cpu_idle_time_us);
u64 get_cpu_iowait_time_us(int cpu, u64 *last_update_time)
{
return get_cpu_sleep_time_us(cpu, CPUTIME_IOWAIT,
nr_iowait_cpu(cpu), last_update_time);
cpu_has_iowait(cpu), last_update_time);
}
EXPORT_SYMBOL_GPL(get_cpu_iowait_time_us);
@ -695,12 +720,13 @@ void account_process_tick(struct task_struct *p, int user_tick)
if (kcpustat_idle_dyntick())
return;
cputime = TICK_NSEC - kcpustat_tick_overlap();
if (irqtime_enabled()) {
irqtime_account_process_tick(p, user_tick, 1);
irqtime_account_process_tick(p, user_tick, cputime);
return;
}
cputime = TICK_NSEC;
steal = steal_account_process_time(ULONG_MAX);
if (steal >= cputime)
@ -1109,7 +1135,6 @@ static int kcpustat_field_vtime(u64 *cpustat,
int cpu, u64 *val)
{
struct vtime *vtime = &tsk->vtime;
struct rq *rq = cpu_rq(cpu);
unsigned int seq;
do {
@ -1152,11 +1177,11 @@ static int kcpustat_field_vtime(u64 *cpustat,
*val += vtime->gtime + vtime_delta(vtime);
break;
case CPUTIME_IDLE:
if (state == VTIME_IDLE && !atomic_read(&rq->nr_iowait))
if (state == VTIME_IDLE && !cpu_has_iowait(cpu))
*val += vtime_delta(vtime);
break;
case CPUTIME_IOWAIT:
if (state == VTIME_IDLE && atomic_read(&rq->nr_iowait) > 0)
if (state == VTIME_IDLE && cpu_has_iowait(cpu))
*val += vtime_delta(vtime);
break;
default:
@ -1251,7 +1276,7 @@ static int kcpustat_cpu_fetch_vtime(struct kernel_cpustat *dst,
}
break;
case VTIME_IDLE:
if (atomic_read(&cpu_rq(cpu)->nr_iowait) > 0)
if (cpu_has_iowait(cpu))
cpustat[CPUTIME_IOWAIT] += delta;
else
cpustat[CPUTIME_IDLE] += delta;

9
kernel/sched/deadline.c
View File

@ -1097,7 +1097,7 @@ static int start_dl_timer(struct sched_dl_entity *dl_se)
* chosen as the deadline is too small, don't even try to
* start the timer in the past!
*/
if (ktime_us_delta(act, now) < 0)
if (ktime_before(act, now))
return 0;
/*
@ -2773,11 +2773,14 @@ static void start_hrtick_dl(struct rq *rq, struct sched_dl_entity *dl_se)
* DL keeps current in tree, because ->deadline is not typically changed while
* a task is runnable.
*/
static void set_next_task_dl(struct rq *rq, struct task_struct *p, bool first)
static void set_next_task_dl(struct rq *rq, struct task_struct *p, enum snt_e type)
{
struct sched_dl_entity *dl_se = &p->dl;
struct dl_rq *dl_rq = &rq->dl;
if (type == SNT_REPICK)
return;
p->se.exec_start = rq_clock_task(rq);
if (on_dl_rq(&p->dl))
update_stats_wait_end_dl(dl_rq, dl_se);
@ -2788,7 +2791,7 @@ static void set_next_task_dl(struct rq *rq, struct task_struct *p, bool first)
WARN_ON_ONCE(dl_rq->curr);
dl_rq->curr = dl_se;
if (!first)
if (type != SNT_PICK)
return;
if (rq->donor->sched_class != &dl_sched_class)

21
kernel/sched/debug.c
View File

@ -73,13 +73,13 @@ static int sched_feat_show(struct seq_file *m, void *v)
#ifdef CONFIG_JUMP_LABEL
#define jump_label_key__true STATIC_KEY_INIT_TRUE
#define jump_label_key__false STATIC_KEY_INIT_FALSE
#define jump_label_key__true { .key_true = STATIC_KEY_TRUE_INIT }
#define jump_label_key__false { .key_false = STATIC_KEY_FALSE_INIT }
#define SCHED_FEAT(name, enabled) \
jump_label_key__##enabled ,
struct static_key sched_feat_keys[__SCHED_FEAT_NR] = {
union sched_feat_key sched_feat_keys[__SCHED_FEAT_NR] = {
#include "features.h"
};
@ -87,12 +87,12 @@ struct static_key sched_feat_keys[__SCHED_FEAT_NR] = {
static void sched_feat_disable(int i)
{
static_key_disable_cpuslocked(&sched_feat_keys[i]);
static_branch_disable_cpuslocked(&sched_feat_keys[i].key_true);
}
static void sched_feat_enable(int i)
{
static_key_enable_cpuslocked(&sched_feat_keys[i]);
static_branch_enable_cpuslocked(&sched_feat_keys[i].key_false);
}
#else /* !CONFIG_JUMP_LABEL: */
static void sched_feat_disable(int i) { };
@ -280,16 +280,10 @@ static ssize_t sched_dynamic_write(struct file *filp, const char __user *ubuf,
static int sched_dynamic_show(struct seq_file *m, void *v)
{
int i = (IS_ENABLED(CONFIG_PREEMPT_RT) || IS_ENABLED(CONFIG_ARCH_HAS_PREEMPT_LAZY)) * 2;
int mode = READ_ONCE(preempt_dynamic_mode);
int j;
/* Count entries in NULL terminated preempt_modes */
for (j = 0; preempt_modes[j]; j++)
;
j -= !IS_ENABLED(CONFIG_ARCH_HAS_PREEMPT_LAZY);
for (; i < j; i++) {
/* Stop at NULL terminator */
for (int i = 0; preempt_modes[i]; i++) {
if (mode == i)
seq_puts(m, "(");
seq_puts(m, preempt_modes[i]);
@ -1446,6 +1440,7 @@ void proc_sched_show_task(struct task_struct *p, struct pid_namespace *ns,
P_SCHEDSTAT(nr_failed_migrations_running);
P_SCHEDSTAT(nr_failed_migrations_hot);
P_SCHEDSTAT(nr_forced_migrations);
P_SCHEDSTAT(nr_migrations_cpu_non_preferred);
P_SCHEDSTAT(nr_wakeups);
P_SCHEDSTAT(nr_wakeups_sync);
P_SCHEDSTAT(nr_wakeups_migrate);

18
kernel/sched/ext/ext.c
View File

@ -24,6 +24,11 @@
DEFINE_RAW_SPINLOCK(scx_sched_lock);
bool scx_allow_proxy_exec(const struct task_struct *p)
{
return true;
}
/*
* NOTE: sched_ext is in the process of growing multiple scheduler support and
* scx_root usage is in a transitional state. Naked dereferences are safe if the
@ -1087,6 +1092,10 @@ static void schedule_deferred_locked(struct rq *rq)
schedule_deferred(rq);
}
void scx_proxy_reenqueue_retry(struct rq *rq, struct task_struct *next)
{
}
void schedule_dsq_reenq(struct scx_sched *sch, struct scx_dispatch_q *dsq,
u64 reenq_flags, struct rq *locked_rq)
{
@ -3044,10 +3053,13 @@ static enum scx_dsp_verdict dispatch_one(struct rq *rq, struct task_struct *prev
return verdict;
}
static void set_next_task_scx(struct rq *rq, struct task_struct *p, bool first)
static void set_next_task_scx(struct rq *rq, struct task_struct *p, enum snt_e type)
{
struct scx_sched *sch = scx_task_sched(p);
if (type == SNT_REPICK)
return;
if (p->scx.flags & SCX_TASK_QUEUED) {
/*
* Core-sched might decide to execute @p before it is
@ -3105,6 +3117,10 @@ static void set_next_task_scx(struct rq *rq, struct task_struct *p, bool first)
}
}
void scx_proxy_donor_start(struct rq *rq)
{
}
static enum scx_cpu_preempt_reason
preempt_reason_from_class(const struct sched_class *class)
{

7
kernel/sched/ext/ext.h
View File

@ -20,6 +20,9 @@ void scx_rq_deactivate(struct rq *rq);
int scx_check_setscheduler(struct task_struct *p, int policy);
bool task_should_scx(int policy);
bool scx_allow_ttwu_queue(const struct task_struct *p);
bool scx_allow_proxy_exec(const struct task_struct *p);
void scx_proxy_donor_start(struct rq *rq);
void scx_proxy_reenqueue_retry(struct rq *rq, struct task_struct *next);
void init_sched_ext_class(void);
static inline u32 scx_cpuperf_target(s32 cpu)
@ -54,6 +57,10 @@ static inline void scx_rq_deactivate(struct rq *rq) {}
static inline int scx_check_setscheduler(struct task_struct *p, int policy) { return 0; }
static inline bool task_on_scx(const struct task_struct *p) { return false; }
static inline bool scx_allow_ttwu_queue(const struct task_struct *p) { return true; }
static inline bool scx_allow_proxy_exec(const struct task_struct *p) { return true; }
static inline void scx_proxy_donor_start(struct rq *rq) {}
static inline void scx_proxy_reenqueue_retry(struct rq *rq,
struct task_struct *next) {}
static inline void init_sched_ext_class(void) {}
#endif /* CONFIG_SCHED_CLASS_EXT */

281
kernel/sched/fair.c
View File

@ -24,6 +24,7 @@
#include <linux/mmap_lock.h>
#include <linux/hugetlb_inline.h>
#include <linux/jiffies.h>
#include <linux/math.h>
#include <linux/mm_api.h>
#include <linux/highmem.h>
#include <linux/hrtimer.h>
@ -820,12 +821,6 @@ static u64 ineligible_vruntime(struct cfs_rq *cfs_rq)
if (curr && !curr->on_rq)
curr = NULL;
/*
* This is called from set_next_task_fair(.first=true) /
* set_protect_slice() so curr had better be set and on_rq.
*/
WARN_ON_ONCE(!curr);
if (weight) {
s64 runtime = cfs_rq->sum_w_vruntime;
@ -1004,71 +999,17 @@ static inline bool __entity_less(struct rb_node *a, const struct rb_node *b)
return entity_before(__node_2_se(a), __node_2_se(b));
}
static inline void __min_vruntime_update(struct sched_entity *se, struct rb_node *node)
/* min() for wrapping vruntimes */
static inline u64 __min_vruntime(u64 a, u64 b)
{
if (node) {
struct sched_entity *rse = __node_2_se(node);
if (vruntime_cmp(se->min_vruntime, ">", rse->min_vruntime))
se->min_vruntime = rse->min_vruntime;
}
return vruntime_cmp(a, "<", b) ? a : b;
}
static inline void __min_slice_update(struct sched_entity *se, struct rb_node *node)
{
if (node) {
struct sched_entity *rse = __node_2_se(node);
if (rse->min_slice < se->min_slice)
se->min_slice = rse->min_slice;
}
}
static inline void __max_slice_update(struct sched_entity *se, struct rb_node *node)
{
if (node) {
struct sched_entity *rse = __node_2_se(node);
if (rse->max_slice > se->max_slice)
se->max_slice = rse->max_slice;
}
}
static inline void min_vruntime_copy(struct sched_entity *new, struct sched_entity *old)
{
new->min_vruntime = old->min_vruntime;
new->min_slice = old->min_slice;
new->max_slice = old->max_slice;
}
/*
* se->min_vruntime = min(se->vruntime, {left,right}->min_vruntime)
*/
static inline bool min_vruntime_update(struct sched_entity *se, bool exit)
{
u64 old_min_vruntime = se->min_vruntime;
u64 old_min_slice = se->min_slice;
u64 old_max_slice = se->max_slice;
struct rb_node *node = &se->run_node;
se->min_vruntime = se->vruntime;
__min_vruntime_update(se, node->rb_right);
__min_vruntime_update(se, node->rb_left);
se->min_slice = se->slice;
__min_slice_update(se, node->rb_right);
__min_slice_update(se, node->rb_left);
se->max_slice = se->slice;
__max_slice_update(se, node->rb_right);
__max_slice_update(se, node->rb_left);
return se->min_vruntime == old_min_vruntime &&
se->min_slice == old_min_slice &&
se->max_slice == old_max_slice;
}
RB_DECLARE_CALLBACKS_MULTI(static, min_vruntime_cb, struct sched_entity,
run_node, min_vruntime_copy, min_vruntime_update);
RB_DECLARE_CALLBACKS(static, min_vruntime_cb,
struct sched_entity, run_node,
RB_AUG(vruntime, min_vruntime, __min_vruntime),
RB_AUG(slice, min_slice, min),
RB_AUG(slice, max_slice, max));
/*
* Enqueue an entity into the rb-tree:
@ -1137,10 +1078,9 @@ static inline void set_protect_slice(struct cfs_rq *cfs_rq, struct sched_entity
/* If there are shorter slices than se's one */
if (slice != se->slice) {
vprot = min_vruntime(vprot, se->vruntime + calc_delta_fair(slice, se));
if (sched_feat(PREEMPT_SHORT))
vprot = min_vruntime(vprot, ineligible_vruntime(cfs_rq));
else
vprot = min_vruntime(vprot, se->vruntime + calc_delta_fair(slice, se));
}
se->vprot = vprot;
@ -1148,10 +1088,14 @@ static inline void set_protect_slice(struct cfs_rq *cfs_rq, struct sched_entity
static inline void update_protect_slice(struct cfs_rq *cfs_rq, struct sched_entity *se)
{
u64 slice = cfs_rq_min_slice(cfs_rq);
u64 vruntime = min_vruntime(se->vruntime, avg_vruntime(cfs_rq));
u64 slice = cfs_rq_min_slice(cfs_rq);
u64 vprot = min_vruntime(se->vprot, vruntime + calc_delta_fair(slice, se));
se->vprot = min_vruntime(se->vprot, vruntime + calc_delta_fair(slice, se));
if (sched_feat(PREEMPT_SHORT) && slice != se->slice)
vprot = min_vruntime(vprot, ineligible_vruntime(cfs_rq));
se->vprot = vprot;
}
static inline bool protect_slice(struct sched_entity *se)
@ -3713,7 +3657,7 @@ static void update_task_scan_period(struct task_struct *p,
p->mm->numa_next_scan = jiffies +
msecs_to_jiffies(p->numa_scan_period);
return;
goto out;
}
/*
@ -3757,7 +3701,10 @@ static void update_task_scan_period(struct task_struct *p,
p->numa_scan_period = clamp(p->numa_scan_period + diff,
task_scan_min(p), task_scan_max(p));
memset(p->numa_faults_locality, 0, sizeof(p->numa_faults_locality));
out:
memset(p->numa_faults_locality, 0,
sizeof(p->numa_faults_locality));
}
/*
@ -8246,10 +8193,12 @@ enqueue_task_fair(struct rq *rq, struct task_struct *p, int flags)
if (p->in_iowait)
cpufreq_update_util(rq, SCHED_CPUFREQ_IOWAIT);
/*
* XXX comment on the curr thing
*/
curr = (cfs_rq->curr == se);
/*
* This can actually happen with CFS_BANDWIDTH, a task can be throttled
* and dequeued, unthrottled from another CPU and then enqueued again
* before schedule().
*/
if (curr)
place_entity(cfs_rq, se, flags);
@ -8682,8 +8631,8 @@ static int
sched_balance_find_dst_group_cpu(struct sched_group *group, struct task_struct *p, int this_cpu)
{
unsigned long load, min_load = ULONG_MAX;
unsigned int min_exit_latency = UINT_MAX;
u64 latest_idle_timestamp = 0;
u64 min_exit_latency = U64_MAX;
unsigned int nr_candidates = 0;
int least_loaded_cpu = this_cpu;
int shallowest_idle_cpu = -1;
int i;
@ -8704,24 +8653,16 @@ sched_balance_find_dst_group_cpu(struct sched_group *group, struct task_struct *
if (available_idle_cpu(i)) {
struct cpuidle_state *idle = idle_get_state(rq);
if (idle && idle->exit_latency < min_exit_latency) {
/*
* We give priority to a CPU whose idle state
* has the smallest exit latency irrespective
* of any idle timestamp.
*/
min_exit_latency = idle->exit_latency;
latest_idle_timestamp = rq->idle_stamp;
shallowest_idle_cpu = i;
} else if ((!idle || idle->exit_latency == min_exit_latency) &&
rq->idle_stamp > latest_idle_timestamp) {
/*
* If equal or no active idle state, then
* the most recently idled CPU might have
* a warmer cache.
*/
latest_idle_timestamp = rq->idle_stamp;
u64 exit_latency = idle ? idle->exit_latency : U64_MAX;
if (shallowest_idle_cpu == -1 || exit_latency < min_exit_latency) {
min_exit_latency = exit_latency;
shallowest_idle_cpu = i;
nr_candidates = 1;
} else if (exit_latency == min_exit_latency) {
nr_candidates++;
if (!reciprocal_scale(sched_rng(), nr_candidates))
shallowest_idle_cpu = i;
}
} else if (shallowest_idle_cpu == -1) {
load = cpu_load(cpu_rq(i));
@ -8819,6 +8760,35 @@ static inline bool test_idle_cores(int cpu)
return false;
}
/*
* Redirect a CPU to a higher-priority available sibling in its SMT domain,
* subject to task affinity.
*/
static inline int select_idle_smt_cpu(struct task_struct *p, int cpu)
{
struct sched_domain *sd;
int best = cpu;
int sibling;
if (!sched_smt_active())
return cpu;
sd = rcu_dereference_all(cpu_rq(cpu)->sd);
if (!sd || !(sd->flags & SD_SHARE_CPUCAPACITY) ||
!(sd->flags & SD_ASYM_PACKING))
return cpu;
for_each_cpu_and(sibling, sched_domain_span(sd), p->cpus_ptr) {
if (sibling == best || !choose_idle_cpu(sibling, p))
continue;
if (sched_asym_prefer(sibling, best))
best = sibling;
}
return best;
}
/*
* Scans the local SMT mask to see if the entire core is idle, and records this
* information in sd_balance_shared->has_idle_cores.
@ -9203,7 +9173,7 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target)
if (choose_idle_cpu(target, p) &&
asym_fits_cpu(task_util, util_min, util_max, target))
return target;
goto select_smt_priority;
/*
* If the previous CPU is cache affine and idle, don't be stupid:
@ -9213,8 +9183,10 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target)
asym_fits_cpu(task_util, util_min, util_max, prev)) {
if (!static_branch_unlikely(&sched_cluster_active) ||
cpus_share_resources(prev, target))
return prev;
cpus_share_resources(prev, target)) {
target = prev;
goto select_smt_priority;
}
prev_aff = prev;
}
@ -9232,7 +9204,8 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target)
prev == smp_processor_id() &&
this_rq()->nr_running <= 1 &&
asym_fits_cpu(task_util, util_min, util_max, prev)) {
return prev;
target = prev;
goto select_smt_priority;
}
/* Check a recently used CPU as a potential idle candidate: */
@ -9246,8 +9219,10 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target)
asym_fits_cpu(task_util, util_min, util_max, recent_used_cpu)) {
if (!static_branch_unlikely(&sched_cluster_active) ||
cpus_share_resources(recent_used_cpu, target))
return recent_used_cpu;
cpus_share_resources(recent_used_cpu, target)) {
target = recent_used_cpu;
goto select_smt_priority;
}
} else {
recent_used_cpu = -1;
@ -9269,7 +9244,11 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target)
*/
if (sd) {
i = select_idle_capacity(p, sd, target);
return ((unsigned)i < nr_cpumask_bits) ? i : target;
if ((unsigned int)i < nr_cpumask_bits) {
target = i;
goto select_smt_priority;
}
return target;
}
}
@ -9282,14 +9261,18 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target)
if (!has_idle_core && cpus_share_cache(prev, target)) {
i = select_idle_smt(p, sd, prev);
if ((unsigned int)i < nr_cpumask_bits)
return i;
if ((unsigned int)i < nr_cpumask_bits) {
target = i;
goto select_smt_priority;
}
}
}
i = select_idle_cpu(p, sd, has_idle_core, target);
if ((unsigned)i < nr_cpumask_bits)
return i;
if ((unsigned int)i < nr_cpumask_bits) {
target = i;
goto select_smt_priority;
}
/*
* For cluster machines which have lower sharing cache like L2 or
@ -9297,12 +9280,19 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target)
* first. But prev_cpu or recent_used_cpu may also be a good candidate,
* use them if possible when no idle CPU found in select_idle_cpu().
*/
if ((unsigned int)prev_aff < nr_cpumask_bits)
return prev_aff;
if ((unsigned int)recent_used_cpu < nr_cpumask_bits)
return recent_used_cpu;
if ((unsigned int)prev_aff < nr_cpumask_bits) {
target = prev_aff;
goto select_smt_priority;
}
if ((unsigned int)recent_used_cpu < nr_cpumask_bits) {
target = recent_used_cpu;
goto select_smt_priority;
}
return target;
select_smt_priority:
return select_idle_smt_cpu(p, target);
}
/**
@ -9979,8 +9969,10 @@ select_task_rq_fair(struct task_struct *p, int prev_cpu, int wake_flags)
}
/* Slow path */
if (unlikely(sd))
return sched_balance_find_dst_cpu(sd, p, cpu, prev_cpu, sd_flag);
if (unlikely(sd)) {
new_cpu = sched_balance_find_dst_cpu(sd, p, cpu, prev_cpu, sd_flag);
return select_idle_smt_cpu(p, new_cpu);
}
/* Fast path */
if (wake_flags & WF_TTWU)
@ -10080,8 +10072,14 @@ static inline bool set_preempt_buddy(struct cfs_rq *cfs_rq, struct sched_entity
static inline bool set_short_buddy(struct cfs_rq *cfs_rq, struct sched_entity *pse)
{
if (cfs_rq->next && cfs_rq->next->slice < pse->slice)
return false;
if (cfs_rq->next) {
if (cfs_rq->next->slice < pse->slice)
return false;
if (cfs_rq->next->slice == pse->slice &&
entity_before(cfs_rq->next, pse))
return false;
}
set_next_buddy(cfs_rq, pse);
return true;
@ -11447,21 +11445,7 @@ static int detach_tasks(struct lb_env *env)
*/
static void attach_tasks(struct lb_env *env)
{
struct list_head *tasks = &env->tasks;
struct task_struct *p;
struct rq_flags rf;
rq_lock(env->dst_rq, &rf);
update_rq_clock(env->dst_rq);
while (!list_empty(tasks)) {
p = list_first_entry(tasks, struct task_struct, se.group_node);
list_del_init(&p->se.group_node);
attach_task(env->dst_rq, p);
}
rq_unlock(env->dst_rq, &rf);
__attach_tasks(env->dst_rq, &env->tasks);
}
#ifdef CONFIG_NO_HZ_COMMON
@ -12119,6 +12103,14 @@ static void record_sg_llc_stats(struct lb_env *env,
if (env->sd->child != rcu_dereference_all(per_cpu(sd_llc, env->dst_cpu)))
return;
/*
* A redo repeats stats published on the first pass. Hotplug can
* still leave env->cpus partial, so util would not match capacity.
*/
if ((env->flags & LBF_ALL_PINNED) ||
!cpumask_subset(sched_group_span(group), env->cpus))
return;
/*
* At this point we know this group spans a LLC domain.
* Record the statistic of this group in its corresponding
@ -13754,7 +13746,7 @@ static int sched_balance_rq(int this_cpu, struct rq *this_rq,
};
bool need_unlock = false;
cpumask_and(cpus, sched_domain_span(sd), cpu_active_mask);
cpumask_and(cpus, sched_domain_span(sd), cpu_preferred_mask);
schedstat_inc(sd->lb_count[idle]);
@ -14879,10 +14871,8 @@ static int sched_balance_newidle(struct rq *this_rq, struct rq_flags *rf)
*/
this_rq->idle_stamp = rq_clock(this_rq);
/*
* Do not pull tasks towards !active CPUs...
*/
if (!cpu_active(this_cpu))
/* Do not pull tasks towards !preferred CPUs */
if (!cpu_preferred(this_cpu))
return 0;
/*
@ -15522,14 +15512,18 @@ static void switched_to_fair(struct rq *rq, struct task_struct *p)
}
}
static void set_next_task_fair(struct rq *rq, struct task_struct *p, bool first)
static void set_next_task_fair(struct rq *rq, struct task_struct *p, enum snt_e type)
{
struct sched_entity *se = &p->se;
bool throttled = false;
struct cfs_rq *cfs_rq = &rq->cfs;
unsigned long weight = NICE_0_LOAD;
bool first = type == SNT_PICK;
bool throttled = false;
bool on_rq = se->on_rq;
if (type == SNT_REPICK)
goto repick;
clear_buddies(cfs_rq, se);
if (on_rq)
@ -15573,11 +15567,18 @@ static void set_next_task_fair(struct rq *rq, struct task_struct *p, bool first)
WARN_ON_ONCE(se->sched_delayed);
if (hrtick_enabled_fair(rq))
hrtick_start_fair(rq, p);
update_misfit_status(p, rq);
sched_fair_update_stop_tick(rq, p);
repick:
/*
* A same-task repick skips put_prev_task_fair(), but
* pick_task_fair() refreshed the entity hrtick_start_fair() reads
* before selecting it again. rq->cfs.curr identifies that entity,
* including with group scheduling.
*/
if (hrtick_enabled_fair(rq))
hrtick_start_fair(rq, p);
}
void init_cfs_rq(struct cfs_rq *cfs_rq)

5
kernel/sched/idle.c
View File

@ -487,8 +487,11 @@ static void put_prev_task_idle(struct rq *rq, struct task_struct *prev, struct t
update_rq_avg_idle(rq);
}
static void set_next_task_idle(struct rq *rq, struct task_struct *next, bool first)
static void set_next_task_idle(struct rq *rq, struct task_struct *next, enum snt_e type)
{
if (type == SNT_REPICK)
return;
update_idle_core(rq);
scx_update_idle(rq, true, true);
schedstat_inc(rq->sched_goidle);

25
kernel/sched/psi.c
View File

@ -626,8 +626,6 @@ static void init_rtpoll_triggers(struct psi_group *group, u64 now)
static void psi_schedule_rtpoll_work(struct psi_group *group, unsigned long delay,
bool force)
{
struct task_struct *task;
/*
* atomic_xchg should be called even when !force to provide a
* full memory barrier (see the comment inside psi_rtpoll_work).
@ -635,19 +633,16 @@ static void psi_schedule_rtpoll_work(struct psi_group *group, unsigned long dela
if (atomic_xchg(&group->rtpoll_scheduled, 1) && !force)
return;
rcu_read_lock();
task = rcu_dereference(group->rtpoll_task);
/*
* kworker might be NULL in case psi_trigger_destroy races with
* psi_task_change (hotpath) which can't use locks
* Only test whether a worker is installed; do not dereference the task.
* A racing trigger destruction may leave the timer armed.
* psi_cgroup_free() shuts it down before freeing the group.
* The system PSI group is static.
*/
if (likely(task))
if (likely(rcu_access_pointer(group->rtpoll_task)))
mod_timer(&group->rtpoll_timer, jiffies + delay);
else
atomic_set(&group->rtpoll_scheduled, 0);
rcu_read_unlock();
}
static void psi_rtpoll_work(struct psi_group *group)
@ -1488,22 +1483,12 @@ void psi_trigger_destroy(struct psi_trigger *t)
mutex_unlock(&group->rtpoll_trigger_lock);
}
/*
* Wait for psi_schedule_rtpoll_work RCU to complete its read-side
* critical section before destroying the trigger and optionally the
* rtpoll_task.
*/
synchronize_rcu();
/*
* Stop kthread 'psimon' after releasing rtpoll_trigger_lock to prevent
* a deadlock while waiting for psi_rtpoll_work to acquire
* rtpoll_trigger_lock
*/
if (task_to_destroy) {
/*
* After the RCU grace period has expired, the worker
* can no longer be found through group->rtpoll_task.
*/
kthread_stop(task_to_destroy);
atomic_set(&group->rtpoll_scheduled, 0);
}

7
kernel/sched/rt.c
View File

@ -1654,11 +1654,14 @@ static void wakeup_preempt_rt(struct rq *rq, struct task_struct *p, int flags)
check_preempt_equal_prio(rq, p);
}
static inline void set_next_task_rt(struct rq *rq, struct task_struct *p, bool first)
static inline void set_next_task_rt(struct rq *rq, struct task_struct *p, enum snt_e type)
{
struct sched_rt_entity *rt_se = &p->rt;
struct rt_rq *rt_rq = &rq->rt;
if (type == SNT_REPICK)
return;
p->se.exec_start = rq_clock_task(rq);
if (on_rt_rq(&p->rt))
update_stats_wait_end_rt(rt_rq, rt_se);
@ -1666,7 +1669,7 @@ static inline void set_next_task_rt(struct rq *rq, struct task_struct *p, bool f
/* The running task is never eligible for pushing */
dequeue_pushable_task(rq, p);
if (!first)
if (type != SNT_PICK)
return;
/*

80
kernel/sched/sched.h
View File

@ -1325,6 +1325,9 @@ struct rq {
#ifdef CONFIG_PARAVIRT_TIME_ACCOUNTING
u64 prev_steal_time_rq;
#endif
#ifdef CONFIG_PREFERRED_CPU
bool npc_push_work_pending;
#endif
/* calc_load related fields */
unsigned long calc_load_update;
@ -1370,7 +1373,6 @@ struct rq {
struct task_struct *core_pick;
struct sched_dl_entity *core_dl_server;
unsigned int core_enabled;
unsigned int core_sched_seq;
struct rb_root core_tree;
/* shared state -- careful with sched_core_cpu_deactivate() */
@ -2446,16 +2448,25 @@ extern __read_mostly unsigned int sysctl_sched_features;
#ifdef CONFIG_JUMP_LABEL
#define SCHED_FEAT(name, enabled) \
static __always_inline bool static_branch_##name(struct static_key *key) \
{ \
return static_key_##enabled(key); \
union sched_feat_key {
struct static_key_true key_true;
struct static_key_false key_false;
};
#define sched_feat_branch_true(key) static_branch_likely(&(key)->key_true)
#define sched_feat_branch_false(key) static_branch_unlikely(&(key)->key_false)
#define SCHED_FEAT(name, enabled) \
static __always_inline bool \
static_branch_##name(union sched_feat_key *key) \
{ \
return sched_feat_branch_##enabled(key); \
}
#include "features.h"
#undef SCHED_FEAT
extern struct static_key sched_feat_keys[__SCHED_FEAT_NR];
extern union sched_feat_key sched_feat_keys[__SCHED_FEAT_NR];
#define sched_feat(x) (static_branch_##x(&sched_feat_keys[__SCHED_FEAT_##x]))
#else /* !CONFIG_JUMP_LABEL: */
@ -2507,6 +2518,12 @@ static inline bool task_is_blocked(struct task_struct *p)
return !!p->blocked_on;
}
#ifdef CONFIG_SCHED_PROXY_EXEC
void sched_proxy_block_task(struct rq *rq, struct task_struct *p);
#else
static inline void sched_proxy_block_task(struct rq *rq, struct task_struct *p) {}
#endif
static inline int task_on_cpu(struct rq *rq, struct task_struct *p)
{
return p->on_cpu;
@ -2526,11 +2543,17 @@ static inline int task_on_rq_migrating(struct task_struct *p)
#define WF_EXEC 0x02 /* Wakeup after exec; maps to SD_BALANCE_EXEC */
#define WF_FORK 0x04 /* Wakeup after fork; maps to SD_BALANCE_FORK */
#define WF_TTWU 0x08 /* Wakeup; maps to SD_BALANCE_WAKE */
#define WF_SYNC 0x10 /* Waker goes to sleep after wakeup */
/*
* Hint that the caller expects the waker to sleep soon.
* Scheduler classes may use it for placement or preemption.
* Callers must not rely on it to prevent migration,
* preserve CPU locality or make the wakee run next.
*/
#define WF_SYNC 0x10
#define WF_MIGRATED 0x20 /* Internal use, task got migrated */
#define WF_CURRENT_CPU 0x40 /* Prefer to move the wakee to the current CPU. */
#define WF_RQ_SELECTED 0x80 /* ->select_task_rq() was called */
#define WF_TTWU_RQ 0x100 /* Wakeup completed through ttwu_runnable() */
static_assert(WF_EXEC == SD_BALANCE_EXEC);
static_assert(WF_FORK == SD_BALANCE_FORK);
@ -2620,6 +2643,12 @@ struct affinity_context {
extern s64 update_curr_common(struct rq *rq);
enum snt_e {
SNT_NORMAL, /* set_next_task() */
SNT_PICK, /* put_prev_set_next_task(): prev != next */
SNT_REPICK, /* put_prev_set_next_task(): prev == next */
};
struct sched_class {
#ifdef CONFIG_UCLAMP_TASK
@ -2677,7 +2706,7 @@ struct sched_class {
* __schedule: rq->lock
*/
void (*put_prev_task)(struct rq *rq, struct task_struct *p, struct task_struct *next);
void (*set_next_task)(struct rq *rq, struct task_struct *p, bool first);
void (*set_next_task)(struct rq *rq, struct task_struct *p, enum snt_e type);
/*
* select_task_rq: p->pi_lock
@ -2780,7 +2809,7 @@ static inline void put_prev_task(struct rq *rq, struct task_struct *prev)
static inline void set_next_task(struct rq *rq, struct task_struct *next)
{
next->sched_class->set_next_task(rq, next, false);
next->sched_class->set_next_task(rq, next, SNT_NORMAL);
}
static inline void
@ -2801,11 +2830,13 @@ static inline void put_prev_set_next_task(struct rq *rq,
__put_prev_set_next_dl_server(rq, prev, next);
if (next == prev)
if (next == prev) {
next->sched_class->set_next_task(rq, next, SNT_REPICK);
return;
}
prev->sched_class->put_prev_task(rq, prev, next);
next->sched_class->set_next_task(rq, next, true);
next->sched_class->set_next_task(rq, next, SNT_PICK);
}
/*
@ -3138,6 +3169,25 @@ static inline void attach_one_task(struct rq *rq, struct task_struct *p)
attach_task(rq, p);
}
/*
* __attach_tasks() - attaches a list of tasks (using se.group_node) to
* the new rq
*/
static inline void __attach_tasks(struct rq *rq, struct list_head *tasks)
{
guard(rq_lock)(rq);
update_rq_clock(rq);
while (!list_empty(tasks)) {
struct task_struct *p;
p = list_first_entry(tasks, struct task_struct, se.group_node);
list_del_init(&p->se.group_node);
attach_task(rq, p);
}
}
#ifdef CONFIG_PREEMPT_RT
# define SCHED_NR_MIGRATE_BREAK 8
#else
@ -4252,4 +4302,10 @@ DEFINE_CLASS_IS_UNCONDITIONAL(sched_change)
#include "ext/ext.h"
#ifdef CONFIG_PREFERRED_CPU
void sched_push_current_non_preferred_cpu(struct rq *rq);
#else /* !CONFIG_PREFERRED_CPU */
static inline void sched_push_current_non_preferred_cpu(struct rq *rq) { }
#endif
#endif /* _KERNEL_SCHED_SCHED_H */

5
kernel/sched/stop_task.c
View File

@ -27,8 +27,11 @@ wakeup_preempt_stop(struct rq *rq, struct task_struct *p, int flags)
/* we're never preempted */
}
static void set_next_task_stop(struct rq *rq, struct task_struct *stop, bool first)
static void set_next_task_stop(struct rq *rq, struct task_struct *stop, enum snt_e type)
{
if (type == SNT_REPICK)
return;
stop->se.exec_start = rq_clock_task(rq);
}

18
kernel/sched/topology.c
View File

@ -32,6 +32,20 @@ static int __init sched_debug_setup(char *str)
}
early_param("sched_verbose", sched_debug_setup);
#ifdef CONFIG_SCHED_SMT
static bool sched_smt_asym_packing __read_mostly;
static int __init setup_sched_smt_asym_packing(char *str)
{
if (strcmp(str, "on"))
return 0;
sched_smt_asym_packing = true;
return 1;
}
__setup("sched_smt_asym_packing=", setup_sched_smt_asym_packing);
#endif
static inline bool sched_debug(void)
{
return sched_debug_verbose;
@ -1954,6 +1968,10 @@ sd_init(struct sched_domain_topology_level *tl,
if (WARN_ONCE(sd_flags & ~TOPOLOGY_SD_FLAGS,
"wrong sd_flags in topology description\n"))
sd_flags &= TOPOLOGY_SD_FLAGS;
#ifdef CONFIG_SCHED_SMT
if (sched_smt_asym_packing && (sd_flags & SD_SHARE_CPUCAPACITY))
sd_flags |= SD_ASYM_PACKING;
#endif
sd_flags |= asym_cpu_capacity_classify(sd_span, cpu_map);
*sd = (struct sched_domain){

25
kernel/sched/wait.c
View File

@ -54,14 +54,13 @@ int add_wait_queue_priority_exclusive(struct wait_queue_head *wq_head,
{
struct list_head *head = &wq_head->head;
wq_entry->flags |= WQ_FLAG_EXCLUSIVE | WQ_FLAG_PRIORITY;
guard(spinlock_irqsave)(&wq_head->lock);
if (!list_empty(head) &&
(list_first_entry(head, typeof(*wq_entry), entry)->flags & WQ_FLAG_PRIORITY))
return -EBUSY;
wq_entry->flags |= WQ_FLAG_EXCLUSIVE | WQ_FLAG_PRIORITY;
list_add(&wq_entry->entry, head);
return 0;
}
@ -174,15 +173,11 @@ EXPORT_SYMBOL_GPL(__wake_up_locked_key);
* @mode: which threads
* @key: opaque value to be passed to wakeup targets
*
* The sync wakeup differs that the waker knows that it will schedule
* away soon, so while the target thread will be woken up, it will not
* be migrated to another CPU - ie. the two threads are 'synchronized'
* with each other. This can prevent needless bouncing between CPUs.
* Passes WF_SYNC to waitqueue wake functions. The default wake function
* forwards it to the scheduler; see WF_SYNC for the hint's semantics.
*
* On UP it can prevent extra preemption.
*
* If this function wakes up a task, it executes a full memory barrier before
* accessing the task state.
* If this function wakes up a task, it executes a full memory barrier
* before accessing the task state.
*/
void __wake_up_sync_key(struct wait_queue_head *wq_head, unsigned int mode,
void *key)
@ -200,15 +195,7 @@ EXPORT_SYMBOL_GPL(__wake_up_sync_key);
* @mode: which threads
* @key: opaque value to be passed to wakeup targets
*
* The sync wakeup differs in that the waker knows that it will schedule
* away soon, so while the target thread will be woken up, it will not
* be migrated to another CPU - ie. the two threads are 'synchronized'
* with each other. This can prevent needless bouncing between CPUs.
*
* On UP it can prevent extra preemption.
*
* If this function wakes up a task, it executes a full memory barrier before
* accessing the task state.
* Same as __wake_up_sync_key(), but called with @wq_head->lock held.
*/
void __wake_up_locked_sync_key(struct wait_queue_head *wq_head,
unsigned int mode, void *key)

1
kernel/smp.c
View File

@ -87,6 +87,7 @@ int smpcfd_dead_cpu(unsigned int cpu)
free_cpumask_var(cfd->cpumask);
free_cpumask_var(cfd->cpumask_ipi);
irq_work_run_cpu(cpu);
return 0;
}

6
kernel/smpboot.c
View File

@ -103,6 +103,7 @@ static int smpboot_thread_fn(void *data)
{
struct smpboot_thread_data *td = data;
struct smp_hotplug_thread *ht = td->ht;
bool should_run;
while (1) {
set_current_state(TASK_INTERRUPTIBLE);
@ -117,7 +118,8 @@ static int smpboot_thread_fn(void *data)
return 0;
}
if (kthread_should_park()) {
should_run = td->status == HP_THREAD_ACTIVE && ht->thread_should_run(td->cpu);
if (kthread_should_park() && !should_run) {
__set_current_state(TASK_RUNNING);
preempt_enable();
if (ht->park && td->status == HP_THREAD_ACTIVE) {
@ -151,7 +153,7 @@ static int smpboot_thread_fn(void *data)
continue;
}
if (!ht->thread_should_run(td->cpu)) {
if (!should_run) {
preempt_enable_no_resched();
schedule();
} else {

22
kernel/time/tick-sched.c
View File

@ -763,12 +763,18 @@ static ktime_t tick_forward_now(ktime_t expires, ktime_t now)
return expires + TICK_NSEC;
}
static void tick_nohz_forward_last_tick(struct tick_sched *ts, ktime_t now)
{
if (now >= ts->last_tick)
ts->last_tick = tick_forward_now(ts->last_tick, now);
}
static void tick_nohz_restart(struct tick_sched *ts, ktime_t now)
{
ktime_t expires = ts->last_tick;
ktime_t expires;
if (now >= expires)
expires = tick_forward_now(expires, now);
tick_nohz_forward_last_tick(ts, now);
expires = ts->last_tick;
if (tick_sched_flag_test(ts, TS_FLAG_HIGHRES)) {
hrtimer_start(&ts->sched_timer, expires, HRTIMER_MODE_ABS_PINNED_HARD);
@ -1329,6 +1335,12 @@ unsigned long tick_nohz_get_idle_calls_cpu(int cpu)
return ts->idle_calls;
}
static void tick_nohz_dyntick_stop(struct tick_sched *ts, ktime_t now)
{
tick_nohz_forward_last_tick(ts, now);
kcpustat_dyntick_stop(now, ts->last_tick - TICK_NSEC);
}
void tick_nohz_idle_restart_tick(void)
{
struct tick_sched *ts = this_cpu_ptr(&tick_cpu_sched);
@ -1341,7 +1353,7 @@ void tick_nohz_idle_restart_tick(void)
* no tiny amount of idle time is accounted twice.
*/
ts->idle_entrytime = ktime_get();
kcpustat_dyntick_stop(ts->idle_entrytime);
tick_nohz_dyntick_stop(ts, ts->idle_entrytime);
tick_nohz_restart_sched_tick(ts, ts->idle_entrytime);
}
}
@ -1385,7 +1397,7 @@ void tick_nohz_idle_exit(void)
if (tick_sched_flag_test(ts, TS_FLAG_STOPPED)) {
now = ktime_get();
kcpustat_dyntick_stop(now);
tick_nohz_dyntick_stop(ts, now);
tick_nohz_idle_update_tick(ts, now);
}

17
lib/bitmap.c
View File

@ -308,6 +308,23 @@ bool __bitmap_intersects(const unsigned long *bitmap1,
}
EXPORT_SYMBOL(__bitmap_intersects);
bool __bitmap_intersects_and(const unsigned long *bitmap1,
const unsigned long *bitmap2,
const unsigned long *bitmap3, unsigned int bits)
{
unsigned int k, lim = bits / BITS_PER_LONG;
for (k = 0; k < lim; ++k)
if (bitmap1[k] & bitmap2[k] & bitmap3[k])
return true;
if (bits % BITS_PER_LONG)
if ((bitmap1[k] & bitmap2[k] & bitmap3[k]) & BITMAP_LAST_WORD_MASK(bits))
return true;
return false;
}
EXPORT_SYMBOL(__bitmap_intersects_and);
bool __bitmap_subset(const unsigned long *bitmap1,
const unsigned long *bitmap2, unsigned int bits)
{

62
lib/rbtree_test.c
View File

@ -30,41 +30,20 @@ static struct test_node *nodes = NULL;
static struct rnd_state rnd;
static inline bool less(struct rb_node *a, const struct rb_node *b)
{
return rb_entry(a, struct test_node, rb)->key <
rb_entry(b, struct test_node, rb)->key;
}
static void insert(struct test_node *node, struct rb_root_cached *root)
{
struct rb_node **new = &root->rb_root.rb_node, *parent = NULL;
u32 key = node->key;
while (*new) {
parent = *new;
if (key < rb_entry(parent, struct test_node, rb)->key)
new = &parent->rb_left;
else
new = &parent->rb_right;
}
rb_link_node(&node->rb, parent, new);
rb_insert_color(&node->rb, &root->rb_root);
rb_add(&node->rb, &root->rb_root, less);
}
static void insert_cached(struct test_node *node, struct rb_root_cached *root)
{
struct rb_node **new = &root->rb_root.rb_node, *parent = NULL;
u32 key = node->key;
bool leftmost = true;
while (*new) {
parent = *new;
if (key < rb_entry(parent, struct test_node, rb)->key)
new = &parent->rb_left;
else {
new = &parent->rb_right;
leftmost = false;
}
}
rb_link_node(&node->rb, parent, new);
rb_insert_color_cached(&node->rb, root, leftmost);
rb_add_cached(&node->rb, root, less);
}
static inline void erase(struct test_node *node, struct rb_root_cached *root)
@ -110,29 +89,8 @@ static void insert_augmented(struct test_node *node,
static void insert_augmented_cached(struct test_node *node,
struct rb_root_cached *root)
{
struct rb_node **new = &root->rb_root.rb_node, *rb_parent = NULL;
u32 key = node->key;
u32 val = node->val;
struct test_node *parent;
bool leftmost = true;
while (*new) {
rb_parent = *new;
parent = rb_entry(rb_parent, struct test_node, rb);
if (parent->augmented < val)
parent->augmented = val;
if (key < parent->key)
new = &parent->rb.rb_left;
else {
new = &parent->rb.rb_right;
leftmost = false;
}
}
node->augmented = val;
rb_link_node(&node->rb, rb_parent, new);
rb_insert_augmented_cached(&node->rb, root,
leftmost, &augment_callbacks);
node->augmented = node->val;
rb_add_augmented_cached(&node->rb, root, less, &augment_callbacks);
}

2
tools/perf/trace/beauty/include/uapi/linux/sched.h
View File

@ -53,7 +53,7 @@
*/
#define UNSHARE_EMPTY_MNTNS 0x00100000 /* Unshare an empty mount namespace. */
#ifndef __ASSEMBLY__
#ifndef __ASSEMBLER__
/**
* struct clone_args - arguments for the clone3 syscall
* @flags: Flags for the new process as listed above.