There's three loops to find a task to pull.
In the first iteration, we look for a task that is not running,
which fits a strict criteria.
In the second iteration, we look for a task that is not running,
which fits a lax criteria.
If we have still not found a valid task to pull, we intended to pull the
running task servicing an active load migration.
However, there is some leftover code, wherein a task that is not running
will end up being detached and pulled. This was not intended, and leads
to incorrect behavior. Essentially, this is negating the effects of the
first two iterations.
Change-Id: Ia225d89043c76ec6e661771a39d058498346920d
Signed-off-by: Shaleen Agrawal <quic_shalagra@quicinc.com>
Currently, it is observed that multiple calls to active_load_balance can
take place.
For example, say two Big threads are running on Golds, and
become misfits at the same time. Then, Prime which is currently say
halted, will end up pulling both threads. This is because first, as
Prime is unhalted, the CPU is kicked to enable it to pull tasks.
However, while Prime is kicked, it goes back to idle briefly, causing
the upstream newidle_balance path to come into play.
As a consequence, both threads are queued on Prime, and eventually, the
one sitting in the runqueue is put back on a Gold. This behavior is
unoptimal and unnecessary.
Ensure that walt_newidle_balance only takes when the CPU is not marked
as reserved.
Change-Id: I00feb8df3c4433333a356f2e855fb7a3ee848789
Signed-off-by: Shaleen Agrawal <quic_shalagra@quicinc.com>
During CTS testing, the user-space code can get an affinity
that includes a halted cpu, and immediately use that to set
the affinity. Currently sched_setaffinity will cause an error
if a halted cpu is selected.
Port the change that was in use prior to the introduction of
pause, correcting the CTS test case.
Change-Id: I9bfc819e4a6904494c199931e76f2702b7990758
Signed-off-by: Stephen Dickey <quic_dickey@quicinc.com>
Use nested spin lock api to avoid deadlock warning
Change-Id: Ia2aed8f025bb209876ff87ba5a9e32b2b80718d8
Signed-off-by: Ashay Jaiswal <quic_ashayj@quicinc.com>
Signed-off-by: Sai Harshini Nimmala <quic_snimmala@quicinc.com>
Currently, wg_cpu is used, however, this is incorrect, as the driving
CPU should be used when evaluating region3 conditions.
Change-Id: I4f86ec6021b718e35d84f589dc9be68a53ab9523
Signed-off-by: Shaleen Agrawal <quic_shalagra@quicinc.com>
Currently, WALT holds rq lock for both the cpus when handling task
migration. Upstream code however (PELT) holds rq locks consecutively for
migrations. This results in cumbersome tricks in WALT to hold them at
the same time - rq locks need to be acquired in order of cpu numbers.
Besides that, holding two rq locks for every migration results in
contentions in busy situations.
Split the migration accounting in two steps - subtracting time from src
cpu and adding time to the dest cpu. Subtract when we know the task is
moving i.e. the migration hook and add on the dest cpu during subsequent
enqueue.
Upstream code guarantees the rq lock for src cpu is held during
migration hook for queued or running task. For a waking task grab the
src cpu lock. No need to grab rq lock for dest cpu as enqueue hook is
always run with that lock held.
The basic idea of walt migration accounting, non group tasks move their
contributions only for inter cluster migration while group tasks do that
for every migration, stays.
Change-Id: I5d319e26bcbc088448954fd83bbcd1981ee7fb5c
Signed-off-by: Shaleen Agrawal <quic_shalagra@quicinc.com>
Signed-off-by: Sai Harshini Nimmala <quic_snimmala@quicinc.com>
Introduce awareness for task window start and identify if task
window_start doesn't match that of the CPUs just before accounting.
Change-Id: I58d806e2a75bc647dc28b1f95d00352d29c5c447
Signed-off-by: Shaleen Agrawal <quic_shalagra@quicinc.com>
The upstream code has a WARN when wakelist is being enforced on the same
cpu where the waker runs. Although WALT tries its best to skip the
current cpu during many_wakeups situation, it may have to put the task
on it during highload/offline or 32 bit situations.
Simply disable queue_wakelist() if we wakeup the task on the waker cpu.
Change-Id: Ie13f1156bf0389570dbe0fbac70e271db2b9b211
Signed-off-by: Abhijeet Dharmapurikar <quic_adharmap@quicinc.com>
Signed-off-by: Shaleen Agrawal <quic_shalagra@quicinc.com>
This change is for general scheduler improvement.
Change-Id: I2b95dc15c0dbe88966c90acfa4dca6aebf1a3f95
Signed-off-by: Shaleen Agrawal <quic_shalagra@quicinc.com>
Currently, there are no protections under conditions where CPUs may be
halted or unhalted twice, which could lead to undesired behavior. Track
reasons for refcounts per CPU to ensure proper accounting.
Change-Id: Ic85a13f3536e3287fc76a02004d2b4601ada7a1a
Signed-off-by: Shaleen Agrawal <quic_shalagra@quicinc.com>
Due to restructuring of upstream logic, it is necessary to properly
deduce start and end times of irqs to maintain correct accounting.
Change-Id: I8f0ff9b0f9137283315e5b67cdf2cfb1dede033f
Signed-off-by: Abhijeet Dharmapurikar <quic_adharmap@quicinc.com>
Signed-off-by: Shaleen Agrawal <quic_shalagra@quicinc.com>
A misfit task can be pulled to a CPU within the same cluster in the
newly-idle balance path.
Since that just increases migration count and does not show any
significant benefits, prevent pulling of a task if it is going to be a
misfit on the destination CPU. Only pull if there are many such tasks and
the source CPU needs help from a newly idle CPU.
Change-Id: Ief55e3916d5902e473c0eaa101e8aefa01c04398
Signed-off-by: Sai Harshini Nimmala <quic_snimmala@quicinc.com>
Currently, the flags and reasons are defined based off left shifts,
which requires manual translation when printed in traces. Change those
definitions to be directly in hex format, maintaining congruence.
Additionally, in the event that a boost_util is 0, ensure that a reason
is not inappropriately assigned.
Change-Id: Ia8e9c1628373f3355a3fcd9707a5950a9757e42b
Signed-off-by: Shaleen Agrawal <quic_shalagra@quicinc.com>
In the event that thermal throttling is under way, and user has not
enabled update_cpu_capacity trace point, add a backup way of quickly
determining scheduler under thermal constraints by displaying the
thermal pressure in the sched_cpu_util trace point.
Change-Id: I36b8f23e37b4e9b42770e1535feabb9d513fb730
Signed-off-by: Shaleen Agrawal <quic_shalagra@quicinc.com>
This change is for general scheduler improvement.
Change-Id: I88cf8678b918aed530f3af122edea2bca5b60503
Signed-off-by: Shaleen Agrawal <quic_shalagra@quicinc.com>
Present newidle balance design doesn't allow farthest cluster to help
during newidle balance (silver's newidle balance cannot help prime and
prime cannot help silver).
For Example:
If silver is entering idle and it find's prime core busy then newidle
balance will try to find and kick an idle gold core but it will not
participate in pulling task from prime to silver.
Update the logic to allow silver cluster to help and pull task during
newidle balance if none of the cores in the nearest cluster (Gold here)
are idle.
Change-Id: I0908058d55f344467f559b50209a6bf8bb18b0ef
Signed-off-by: Ashay Jaiswal <quic_ashayj@quicinc.com>
Under low-util conditions, small tasks can be packed onto
a single dedicated cpu per cluster, the first cpu in the
cluster. This can improve performance by ensuring fewer
exits from wfi or deep-idle, and reduce power by keeping
more of the other cpus in wfi or deep-idle, for longer.
Add the functionality to perform choosing of the packing cpu
and incorporate this into the select_rq path for fair and rt
in walt hooks.
Change-Id: I02675ceb6dd2e5fb40f4b14308159cc1fdde245e
Signed-off-by: Stephen Dickey <quic_dickey@quicinc.com>
Very small tasks can end up spread across all cpus
in a cluster, and cause excessive wfi and lpm exit
latencies. This can be reduced significantly by combining
these tasks on to a single cpu.
Create the functionality to be used in cfs/rt that can
make the decision as to whether a packing cpu should
be chosen. Add supporting sysctl nodes to tune the
thresholds for this.
Change-Id: I68723b8dc461efffeba3069bfd5d41724f071bd2
Signed-off-by: Stephen Dickey <quic_dickey@quicinc.com>
Previously it was possible to update core control with new
statistics and those statistics would be unavailable to
any other decisions that need to be made.
Update sched_avg to cache the statistics locally and simply
pass a reference to that information back to core control.
Provide a cluster util function to provide window based load.
Create a sysctl node and threshold to use to compare against
cluster util.
Change-Id: I8cc40603e49e29e7f8a343f6ff667ce9abe09cb5
Signed-off-by: Stephen Dickey <quic_dickey@quicinc.com>
When a cpu is unhalted currently, it will attempt to perform
a balance through the normal ipi kick interface. However, this
can fail because a balance might have happened in the current
4mS tick, while the cpu was still halted. The next kick as a
result of unhalting, fails to execute (min 1 tick between kicks).
Instead, use walt_newidle_balance() to ensure that this cpu can
take a task. Perform an smp async function call to the target
cpu, invoke newidle balance, and allow the cpu to pull tasks
immediately after unhalting.
Change-Id: Ide1bb51f80dc8c4eba079588ffb6a7ed341e8628
Signed-off-by: Stephen Dickey <quic_dickey@quicinc.com>
Currently, the code has a flow where a window rollover is not issued
during migration. This could lead to a situation where no future window
rollovers are handled.
For example in a 4 + 3 + 1 topology, A B C indicate the window boundaries
while X indicates the rq->window_start of each cpu and "GLB" indicates
the timestamp of the global window tracker (walt_irq_work_lastq_ws)
1.
A B C
CPU0----X-----------------------------------------------------------------
CPU1----X-----------------------------------------------------------------
CPU2----X-----------------------------------------------------------------
CPU3----X-----------------------------------------------------------------
CPU4----X-----------------------------------------------------------------
CPU5----X-----------------------------------------------------------------
CPU6----X-----------------------------------------------------------------
CPU7----X-----------------------------------------------------------------
GLB X
Now CPU1 gets an event(e) right before C, in which case it advances his
rq1->window_start to B, also advances GLB to B and issues a rollover irq
work, which will advance window_starts for all other cpus.
2.
A B C
CPU0----X-----------------------------------------------------------------
CPU1----------------------------X---------------------e-------------------
CPU2----X-----------------------------------------------------------------
CPU3----X-----------------------------------------------------------------
CPU4----X-----------------------------------------------------------------
CPU5----X-----------------------------------------------------------------
CPU6----X-----------------------------------------------------------------
CPU7----X-----------------------------------------------------------------
GLB X
Since the event was very close to C, the irq_work(i) could run right after
C, which advances all the window_starts to C, including that of cpu1.
3.
A B C
CPU0----------------------------------------------------X-----------------
CPU1--------------------------------------------------e-X-i---------------
CPU2----------------------------------------------------X-----------------
CPU3----------------------------------------------------X-----------------
CPU4----------------------------------------------------X-----------------
CPU5----------------------------------------------------X-----------------
CPU6----------------------------------------------------X-----------------
CPU7----------------------------------------------------X-----------------
GLB X
This advancement of window_start on cpu1 causes it to advance G (and
reissue another irq_work, which is not relevant to this discussion) like
below. CPU1 will advance the global because it matches its window_start
(atomic_cmpxchg)
4.
A B C
CPU0----------------------------------------------------X-----------------
CPU1--------------------------------------------------e-X-i---------------
CPU2----------------------------------------------------X-----------------
CPU3----------------------------------------------------X-----------------
CPU4----------------------------------------------------X-----------------
CPU5----------------------------------------------------X-----------------
CPU6----------------------------------------------------X-----------------
CPU7----------------------------------------------------X-----------------
GLB X
However we could run in to a situation where the irq work (i) is held up
because cpu0's rq lock isn't available (BTW a window rollover holds all the
rqlocks) for some other reason. And while the irq work is held there,
the rq locks of cpus1-7 are available, an inter cluster migration between
cpu1 and cpu7 could happen causing cpu1,7 to advance their window_start.
Redrawing the 3rd picture
3a.
A B C
CPU0----X-----------------------------------------------------------------
CPU1--------------------------------------------------e-X---m-----i-------
CPU2----X-----------------------------------------------------------------
CPU3----X-----------------------------------------------------------------
CPU4----X-----------------------------------------------------------------
CPU5----X-----------------------------------------------------------------
CPU6----X-----------------------------------------------------------------
CPU7----------------------------------------------------X---m-------------
GLB X
Eventually when the irq work gets around to hold the locks on all the cpus,
it will move the rest of the cpu's window_start to C. However the Global is
left at B and cpu1 that should have advanced the Global, won't do it
because of an unchecked advancement by the migration with cpu7. After this
point the global is old and will never match any cpu's window_start and
WALT will never do a window_rollover
4a.
A B C
CPU0----------------------------------------------------X-----------------
CPU1--------------------------------------------------e-X---m-----i-------
CPU2----------------------------------------------------X-----------------
CPU3----------------------------------------------------X-----------------
CPU4----------------------------------------------------X-----------------
CPU5----------------------------------------------------X-----------------
CPU6----------------------------------------------------X-----------------
CPU7----------------------------------------------------X---m-------------
GLB X
Fix this by issuing migration walt_irq_work when an intercluster
migration ends up rolling over the window in the dest rq.
Change-Id: Ib3985a1cdf72342305398a9c648159195ca99658
Signed-off-by: Shaleen Agrawal <quic_shalagra@quicinc.com>
When __migrate_task() is given a new cpu to run a 32bit
task, it may be migrating to a halted cpu. is_cpu_allowed
tracehook currently rejects halted cpus, forcing
__migrate_task() to leave the task on its current cpu.
The current cpu may not be 32 bit capable.
If the halted state of the task is changed just prior
to the migrate operation, the migration is rejected and
the prev_cpu is chosen. If the prev_cpu is not 32 bit
capable, then the task will be migrated to an improper
cpu. This can only happen while a 32bit task is being
execv'd.
Detect the situation where a halted cpu is about to be
rejected. If the task is in execve, and it's a 32 bit
task, allow it to run on a halted cpu.
Change-Id: I0a0ca79efeab08aed8136f92cc86dcc6b4179aab
Signed-off-by: Stephen Dickey <quic_dickey@quicinc.com>
It is possible that all 32bit cpus are halted. In this case
a 32 bit task can be woken, and the scheduler will not find
a cpu for it to run on that is 32 bit capable.
Prevent the least-significant 32bit capable cpu from ever
being halted.
Change-Id: I23f616a5988d7a7eaa6b0d7dd7ef82dbe6f5ec9b
Signed-off-by: Stephen Dickey <quic_dickey@quicinc.com>
Add a parameter in UTRA and UTRA_mini traces to help identify the global
window_start.
Change-Id: I1eb7432f47fe2f622ba1efc78047551b97f08952
Signed-off-by: Shaleen Agrawal <quic_shalagra@quicinc.com>
set_window_start is a deprecated function that is never used. Remove it.
Change-Id: I636a2a37e55c4d812606ea1f52c4ce80b70d44d2
Signed-off-by: Shaleen Agrawal <quic_shalagra@quicinc.com>
If there are no notifications pending for performing migration
work, there is no migration work to be done because the work
must have already been handled.
Optimize the walt_irq_work execution for migrations. If no
cpus got locked, then no notifications were handled, and
the irq work function should exit.
Change-Id: I57730503ebb8f440a288666c79c4cb4f31ed1023
Signed-off-by: Stephen Dickey <quic_dickey@quicinc.com>
Tracking latest_clock from walt run queue structure.
Change-Id: I50322b4efc91edbeaf61b72dd031e2a7e96edf28
Signed-off-by: Stephen Dickey <quic_dickey@quicinc.com>
Since we are called from the main tick, rq clock task must
have been updated very recently. Use it directly, instead of
update_rq_clock_task() to avoid warnings.
Change-Id: I8ea644cb28844b9ded6e45703aa72e72c21486f5
Signed-off-by: Stephen Dickey <quic_dickey@quicinc.com>
This change is for general scheduler improvements.
Change-Id: I611eb9aa071e4fac1fad962fb98139f9aff79dcd
Signed-off-by: Satya Durga Srinivasu Prabhala <quic_satyap@quicinc.com>
Signed-off-by: Sai Harshini Nimmala <quic_snimmala@quicinc.com>
There is mvp task running on rq if num_mvp_task not equal 0,
so use num_mvp_tasks check if rq have mvp task.
Change-Id: Ia653b8d1dbbaa642ccc06296338617252bf1225b
Signed-off-by: Tengfei Fan <quic_tengfan@quicinc.com>
Don't requeue mvp task if only have one mvp task on rq, because
only have one mvp task on rq, current mvp task will be select for
running if this mvp task runtime still less than limit.
Change-Id: Idaf343e7471665c01fe711e86a3d7747e501ed76
Signed-off-by: Tengfei Fan <quic_tengfan@quicinc.com>
Add num_mvp_tasks for get the number of mvp task in walt rq.
Change-Id: I4367723cd61f7ee5fd63c2afcfefd6eb3258c118
Signed-off-by: Tengfei Fan <quic_tengfan@quicinc.com>
uclamp_latency_sensitive() function will use struct
task_struct's member of cgroups, but rcu lock should
be got before use it.
Change-Id: I1988ed7fe836f9f1ba99d59c5d46f26f3418b51e
Signed-off-by: Tengfei Fan <quic_tengfan@quicinc.com>
Current task's mvp prio get from walt_get_mvp_task_prio will still be
there even if the task is not in the mvp list. For example task removed
from mvp_list due to runtime slice limition.
Change-Id: Iae5d6e420213111acd0b0e71095e0a6caffbe0aa
Signed-off-by: Tengfei Fan <quic_tengfan@quicinc.com>
As stopper task's scheduling policy is set to FIFO and task may run
long time depends on the number of tasks to be migrated etc., current
long running RT task detection feature crashes system due to task's
arrivial time stamp is taken when stopper task scheduled in during
CPU hot-plug out operation and used it later when CPU gets hot-plugged
in back.
Fix this false positive issue by skipping update of arrivial time stamp
for stopper tasks.
Fixes: b429bee50fe2c8f ("sched/walt/rt: limit long running RT task detection to FIFO")
Change-Id: I714494fee49b6e8e5d50885bd47b38ad0d535579
Signed-off-by: Satya Durga Srinivasu Prabhala <quic_satyap@quicinc.com>
In an earlier fix, we intended to clear lock CPUs with a
given cluster if and only all CPUs belonging to the
cluster did not have a notification pending.
However, the fix was incomplete, as we were continuing to do the
opposite, resulting in undesired behavior.
Fixes: 95d58bdfe8d ("sched/walt: Fix lock_cpus mask setting during IC migrations")
Change-Id: Ia3c0d886a2a27b8f4cf8d6f60673aeb7213d5fb9
Signed-off-by: Shaleen Agrawal <quic_shalagra@quicinc.com>
This change is for general scheduler improvement.
Change-Id: Iebc41134b83b1c83d02f433d92ec2b44ed972511
Signed-off-by: Shaleen Agrawal <quic_shalagra@quicinc.com>
Now that our window history size is a power of 2, we can take
advantage of this to replace division operations in a critical code
path by using bit-shifts instead.
Change-Id: Idc0ed37a87814783fcab2f4d5204b79fb9ce9996
Signed-off-by: Shaleen Agrawal <quic_shalagra@quicinc.com>
rt_task_arrival_time is being defined in a header as well as in
a source file which is incorrect, fix it by changing definition
to declaration in header.
While at it, clean-up related code and make sure long running RT
detection happens for FIFO sched class only.
Change-Id: Ia9f227ad3af0059d354eb0e4df3ab719c1348ac5
Signed-off-by: Satya Durga Srinivasu Prabhala <quic_satyap@quicinc.com>
When a task is in execve, it can be assigned to any cpu regardless
of its status as a 32bit task. However, once execve is done, the
task needs to be assigned to a 32 bit cpu.
Capture the case where a 32 bit task, not in execve, is assigned
to an incorrect cpu.
Change-Id: I7880521716f6ed16d81aad13e42c0ad4374bab15
Signed-off-by: Stephen Dickey <quic_dickey@quicinc.com>
complete_all() being called from android_rvh_tick_entry()
trace hook implementation to wake up task (kworker) waiting
on completion variable.
While task waking if scheduler chooses to place the task on
same CPU where complete_all() is invoked (from scheduler_tick()),
system would run into deadlock as wake up path tries to acquire
rq lock of the CPU from ttwu_queue() which was already acquired
in scheduler_tick().
Fix the deadlock by moving related code chunk to
android_vh_scheduler_tick() trace hook implementation.
Call statck for info:
[<ffffffda02613964>] queued_spin_lock_slowpath+0x88
[<ffffffda025c94c0>] try_to_wake_up[jt]+0x25c
[<ffffffda025fbe0c>] complete_all+0xb4
[<ffffffd9fdb7635c>] android_rvh_tick_entry[sched_walt]+0x5c
[<ffffffda025d1bdc>] scheduler_tick+0x4b0
Fixes: 4e3e7cc6133d2f7 ("sched: Align window start with tick boundary")
Change-Id: I1e1310c6c5513f38c4fd95b933ff407d740c1112
Signed-off-by: Satya Durga Srinivasu Prabhala <quic_satyap@quicinc.com>
When a RT task is marked migration_disabled, i.e. its p->cpus_ptr
points to a single cpu, walt_newidle_balance() could pull the task
to an un-allowed cpu.
To prevent that check p->cpus_ptr in addition to
pick_highest_pushable_task(), as pick_highest_pushable_task()
and pick_rt_task() only check for p->cpu_mask,
which will be inaccurate in migration_disabled situation.
Change-Id: If7b3cfcb357af7a3ff15de486e70b7b304e4f6ec
Signed-off-by: Abhijeet Dharmapurikar <quic_adharmap@quicinc.com>
Signed-off-by: Shaleen Agrawal <quic_shalagra@quicinc.com>
Upstream tracks affinity mask in two ways, p->cpus_ptr and p->cpus_mask.
However, we are currently using these two masks interchangeably.
Currently cpus_ptr points to cpus_mask member in task_struct for most of
the time. One exception being when the task is marked
migration_disabled, during which cpus_ptr could point to a different
mask of single_cpu.
Streamline this to ensure WALT only relies on p->cpus_ptr to determine
CPU affinity, which would be more accurate in migrate_disable situations.
Change-Id: I437ffe531d7417eada71304414654341ce4a4ad9
Signed-off-by: Shaleen Agrawal <quic_shalagra@quicinc.com>
In the existing code, we set the notifier pending flag only on the
source and destination CPU. But, in irq_work_restrict_to_mig_clusters(),
it clears the lock_cpus mask upon visiting non-involved CPUs in the
cluster. This ends up not locking any CPUs, essentially not updating CPU
frequencies upon inter cluster migrations.
Fix this by only clearly lock_cpus if none of the CPUs in the cluster
are involved in a migration.
Change-Id: I486e6c62bf306de8fddbd3eab0537c419aedaff2
Signed-off-by: Shaleen Agrawal <quic_shalagra@quicinc.com>
re-arrange the code to avoid following issues:
1. Mark walt_drain_thread as RT thread with highest priority to avoid
getting preempted by other tasks in the system.
2. Handle kthread_should_stop() in walt_drain_thread.
Change-Id: I404674477948626503742a66fce657070b89b9e6
Signed-off-by: Satya Durga Srinivasu Prabhala <quic_satyap@quicinc.com>
Signed-off-by: Stephen Dickey <quic_dickey@quicinc.com>
