github-mirror/linux
1
0
Fork 0
mirror of https://github.com/torvalds/linux.git synced 2026-10-23 23:51:02 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity Graph

sched/core: Try to use a preferred CPU in is_cpu_allowed

Browse Source 
When possible, try to choose a preferred CPU.

This is essential to maintain user affinities when preferred
CPUs change. A task pinned on a non-preferred CPU should continue
to run there, since this is a non-user triggered event.

If a CPU is non-preferred and the task can run on other CPUs which are
currently preferred, then choose a preferred CPU instead.
This is decided by checking if cpus_ptr, cpu_preferred_mask and
task possible CPUs intersect or not. If yes, then the task has
other preferred CPUs. This takes care of tasks with architecture-specific
CPU masks (e.g., 32-bit tasks on arm64).

The push task mechanism uses a stopper thread which calls
select_fallback_rq() and uses this mechanism to pick a preferred CPU.

This takes care of the wakeup path for FAIR tasks too.
is_cpu_allowed() is called to ensure wakeups happen on preferred CPUs.
With that, additional checks in available_idle_cpu() are not necessary.

Ignore the preferred CPU state if a task's affinity is changing and
its new mask no longer includes the CPU it is currently running on.
This ensures migration_cpu_stop() does not abort, preventing the task
from being stranded outside its allowed affinity.

For the majority of cases, this would still keep select_fallback_rq()
as O(N). cpumask_intersects_and(), which is O(N), is called only if
!cpu_preferred. The task running there is expected to move out.
Subsequently, it should run on a preferred CPU. This becomes O(N**2)
only for tasks pinned solely to non-preferred CPUs. That is a rare case.

Overhead is minimal when the CPU is preferred.

Signed-off-by: Shrikanth Hegde <sshegde@linux.ibm.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/20260928053728.797539-7-sshegde@linux.ibm.com
This commit is contained in:
Shrikanth Hegde 2026-09-28 11:07:21 +05:30 committed by Peter Zijlstra
parent 6208245165
commit d8a3da0de8
1 changed files with 28 additions and 2 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

30
kernel/sched/core.c
View File

@ -2511,6 +2511,24 @@ static inline bool rq_has_pinned_tasks(struct rq *rq)
return rq->nr_pinned;
}
static inline bool task_can_migrate_to_preferred(struct task_struct *p, int cpu)
{
/* No need to migrate from a preferred CPU */
if (cpu_preferred(cpu))
return false;
/* Only FAIR tasks honor preferred CPU state */
if (unlikely(p->sched_class != &fair_sched_class))
return false;
/* Ignore preferred state if task affinity is changing */
if (unlikely(!cpumask_test_cpu(task_cpu(p), p->cpus_ptr)))
return false;
return cpumask_intersects_and(p->cpus_ptr, cpu_preferred_mask,
task_cpu_possible_mask(p));
}
/*
* Per-CPU kthreads are allowed to run on !active && online CPUs, see
* __set_cpus_allowed_ptr() and select_fallback_rq().
@ -2526,8 +2544,12 @@ static inline bool is_cpu_allowed(struct task_struct *p, int cpu)
return cpu_online(cpu);
/* Non kernel threads are not allowed during either online or offline. */
if (!(p->flags & PF_KTHREAD))
if (!(p->flags & PF_KTHREAD)) {
/* Try to use preferred CPU if task's affinity allows */
if (task_can_migrate_to_preferred(p, cpu))
return false;
return cpu_active(cpu);
}
/* KTHREAD_IS_PER_CPU is always allowed. */
if (kthread_is_per_cpu(p))
@ -2537,7 +2559,11 @@ static inline bool is_cpu_allowed(struct task_struct *p, int cpu)
if (cpu_dying(cpu))
return false;
/* But are allowed during online. */
/* Try to keep unbound kthreads on a preferred CPU if possible. */
if (task_can_migrate_to_preferred(p, cpu))
return false;
/* Otherwise, they are allowed to run on online CPU. */
return cpu_online(cpu);
}