uclamp_latency_sensitive() function will use struct
task_struct's member of cgroups, but rcu lock should
be got before use it.
Change-Id: I1988ed7fe836f9f1ba99d59c5d46f26f3418b51e
Signed-off-by: Tengfei Fan <quic_tengfan@quicinc.com>
Current task's mvp prio get from walt_get_mvp_task_prio will still be
there even if the task is not in the mvp list. For example task removed
from mvp_list due to runtime slice limition.
Change-Id: Iae5d6e420213111acd0b0e71095e0a6caffbe0aa
Signed-off-by: Tengfei Fan <quic_tengfan@quicinc.com>
As stopper task's scheduling policy is set to FIFO and task may run
long time depends on the number of tasks to be migrated etc., current
long running RT task detection feature crashes system due to task's
arrivial time stamp is taken when stopper task scheduled in during
CPU hot-plug out operation and used it later when CPU gets hot-plugged
in back.
Fix this false positive issue by skipping update of arrivial time stamp
for stopper tasks.
Fixes: b429bee50fe2c8f ("sched/walt/rt: limit long running RT task detection to FIFO")
Change-Id: I714494fee49b6e8e5d50885bd47b38ad0d535579
Signed-off-by: Satya Durga Srinivasu Prabhala <quic_satyap@quicinc.com>
In an earlier fix, we intended to clear lock CPUs with a
given cluster if and only all CPUs belonging to the
cluster did not have a notification pending.
However, the fix was incomplete, as we were continuing to do the
opposite, resulting in undesired behavior.
Fixes: 95d58bdfe8d ("sched/walt: Fix lock_cpus mask setting during IC migrations")
Change-Id: Ia3c0d886a2a27b8f4cf8d6f60673aeb7213d5fb9
Signed-off-by: Shaleen Agrawal <quic_shalagra@quicinc.com>
This change is for general scheduler improvement.
Change-Id: Iebc41134b83b1c83d02f433d92ec2b44ed972511
Signed-off-by: Shaleen Agrawal <quic_shalagra@quicinc.com>
Now that our window history size is a power of 2, we can take
advantage of this to replace division operations in a critical code
path by using bit-shifts instead.
Change-Id: Idc0ed37a87814783fcab2f4d5204b79fb9ce9996
Signed-off-by: Shaleen Agrawal <quic_shalagra@quicinc.com>
rt_task_arrival_time is being defined in a header as well as in
a source file which is incorrect, fix it by changing definition
to declaration in header.
While at it, clean-up related code and make sure long running RT
detection happens for FIFO sched class only.
Change-Id: Ia9f227ad3af0059d354eb0e4df3ab719c1348ac5
Signed-off-by: Satya Durga Srinivasu Prabhala <quic_satyap@quicinc.com>
When a task is in execve, it can be assigned to any cpu regardless
of its status as a 32bit task. However, once execve is done, the
task needs to be assigned to a 32 bit cpu.
Capture the case where a 32 bit task, not in execve, is assigned
to an incorrect cpu.
Change-Id: I7880521716f6ed16d81aad13e42c0ad4374bab15
Signed-off-by: Stephen Dickey <quic_dickey@quicinc.com>
complete_all() being called from android_rvh_tick_entry()
trace hook implementation to wake up task (kworker) waiting
on completion variable.
While task waking if scheduler chooses to place the task on
same CPU where complete_all() is invoked (from scheduler_tick()),
system would run into deadlock as wake up path tries to acquire
rq lock of the CPU from ttwu_queue() which was already acquired
in scheduler_tick().
Fix the deadlock by moving related code chunk to
android_vh_scheduler_tick() trace hook implementation.
Call statck for info:
[<ffffffda02613964>] queued_spin_lock_slowpath+0x88
[<ffffffda025c94c0>] try_to_wake_up[jt]+0x25c
[<ffffffda025fbe0c>] complete_all+0xb4
[<ffffffd9fdb7635c>] android_rvh_tick_entry[sched_walt]+0x5c
[<ffffffda025d1bdc>] scheduler_tick+0x4b0
Fixes: 4e3e7cc6133d2f7 ("sched: Align window start with tick boundary")
Change-Id: I1e1310c6c5513f38c4fd95b933ff407d740c1112
Signed-off-by: Satya Durga Srinivasu Prabhala <quic_satyap@quicinc.com>
When a RT task is marked migration_disabled, i.e. its p->cpus_ptr
points to a single cpu, walt_newidle_balance() could pull the task
to an un-allowed cpu.
To prevent that check p->cpus_ptr in addition to
pick_highest_pushable_task(), as pick_highest_pushable_task()
and pick_rt_task() only check for p->cpu_mask,
which will be inaccurate in migration_disabled situation.
Change-Id: If7b3cfcb357af7a3ff15de486e70b7b304e4f6ec
Signed-off-by: Abhijeet Dharmapurikar <quic_adharmap@quicinc.com>
Signed-off-by: Shaleen Agrawal <quic_shalagra@quicinc.com>
Upstream tracks affinity mask in two ways, p->cpus_ptr and p->cpus_mask.
However, we are currently using these two masks interchangeably.
Currently cpus_ptr points to cpus_mask member in task_struct for most of
the time. One exception being when the task is marked
migration_disabled, during which cpus_ptr could point to a different
mask of single_cpu.
Streamline this to ensure WALT only relies on p->cpus_ptr to determine
CPU affinity, which would be more accurate in migrate_disable situations.
Change-Id: I437ffe531d7417eada71304414654341ce4a4ad9
Signed-off-by: Shaleen Agrawal <quic_shalagra@quicinc.com>
In the existing code, we set the notifier pending flag only on the
source and destination CPU. But, in irq_work_restrict_to_mig_clusters(),
it clears the lock_cpus mask upon visiting non-involved CPUs in the
cluster. This ends up not locking any CPUs, essentially not updating CPU
frequencies upon inter cluster migrations.
Fix this by only clearly lock_cpus if none of the CPUs in the cluster
are involved in a migration.
Change-Id: I486e6c62bf306de8fddbd3eab0537c419aedaff2
Signed-off-by: Shaleen Agrawal <quic_shalagra@quicinc.com>
re-arrange the code to avoid following issues:
1. Mark walt_drain_thread as RT thread with highest priority to avoid
getting preempted by other tasks in the system.
2. Handle kthread_should_stop() in walt_drain_thread.
Change-Id: I404674477948626503742a66fce657070b89b9e6
Signed-off-by: Satya Durga Srinivasu Prabhala <quic_satyap@quicinc.com>
Signed-off-by: Stephen Dickey <quic_dickey@quicinc.com>
Reset the arrival time of the RT task when task yields.
This way, time spent during and after yield is not conidered towards RT
task runtime.
Change-Id: I8f961bca1ee7bac67dafa0238594fc4a444fdcd1
Signed-off-by: Sai Harshini Nimmala <quic_snimmala@quicinc.com>
need_all_cpus() was put in place in past products, so that silver
isolation could be enabled and disabled based upon the window
size, as a work around for lack of user-space controls for
disabling/enabling core_control. This blocks the ability to
isolate silver cores for lower window sizes, which is a valid
case that should be allowed for most products.
Cleanup the code for need all cpus, such that silver core control
may be used.
Change-Id: I0e029a6c0df5eb1207c549cf843614c360e91b8e
Signed-off-by: Stephen Dickey <quic_dickey@quicinc.com>
Currently, cpufreq traces contain a "flags" parameter, which helps
understand why cpufreq updates were triggered.
In this patch, a "reasons" parameter is introduced to provide an accurate
guidance towards *why* the cpufreq chosen was selected, and the specific
reason behind any applied value adds that have potentially modified the
true load.
Change-Id: I88a62569e0a1ec1555a89161aa532db6c72df334
Signed-off-by: Stephen Dickey <quic_dickey@quicinc.com>
Signed-off-by: Shaleen Agrawal <quic_shalagra@quicinc.com>
As a side effect of the rq_clock optimization patch, the window_start
initialized in the stop handler is no longer aligned with the tick
boundary, which causes undesired effects as observed in APT usecases.
Change-Id: I82a0c03fcdcdc18b47e382c47c94fd84680e978a
Signed-off-by: Shaleen Agrawal <quic_shalagra@quicinc.com>
While performing a move of a task from one CPU to another, there's
a possibility of a change in task's affinity occurring between detaching
the task from previous CPU and setting new CPU of task.
As a result, the new affinity of the task may not support the move.
Account for this scenario by adding appropriate checks in the detaching
task path and changing the destination of the move, if necessary.
Change-Id: I25c456f4ed0131eabae48ab1fc82cc4a29ce5e7a
Signed-off-by: Abhijeet Dharmapurikar <quic_adharmap@quicinc.com>
Signed-off-by: Sai Harshini Nimmala <quic_snimmala@quicinc.com>
Right now, it is possible to detect the long running RT tasks false
positively in scenarios where a DL or RR task could be running as well.
The intention behind long running RT task detection is to catch
any long running FIFO tasks. Update check to make sure RT arrival time
is updated only for FIFO tasks to limit long running RT task detection.
Change-Id: Ide59791b40acd5e9529af23217c16350278c5f60
Signed-off-by: Satya Durga Srinivasu Prabhala <quic_satyap@quicinc.com>
The history samples were recently increased from 5 to 8. Reflect the
same in the trace outputs to print out all 8 history samples.
Change-Id: Ib79dbd193a643bf65c056d2f1cc0ba357ce3cdaa
Signed-off-by: Shaleen Agrawal <quic_shalagra@quicinc.com>
A stopper thread has priority 0, and therefore, it can be classified as
an RT thread. However, in certain usecases, we may be executing for a
prolonged duration under stopper context, and as such would hit long
running RT task notifier unnecessarily.
Change-Id: Ibe5c7aa447292a6a577d405f6e5ae8cf9cd2842c
Signed-off-by: Shaleen Agrawal <quic_shalagra@quicinc.com>
When the window is advanced to the next boundary, the CPU busy time
trackers need to be advanced and top-task trackers should be flipped.
This rollover needs to be done just once for a cpu per window
advancement, while it could be subject to numerous update_task_ravg
(called utra henceforth) with various tasks spanning window cross over
point. The current code achieves this one-time-rollover only when it
sees a utra called for the currently running task and its time has
crossed over in the new window.
As a result of this rollover being done only
when utra is called for current task, it became mandatory to call
utra on current task before calling utra on a non-current task.
This ensured a proper window rollover i.e. the trackers reset/flipped
if needed prior to calling utra on non-current task which may have
entered its execution in a new window.
Explaining pictorially
old new
ws ms ws wc
| | | |
V V V V
-----|----------------|--------
prev curr
In the above example, a non current task wc may have entered the new
window but prev and curr are still set as per the old window start. The
time from new ws to wc cannot be accounted because curr(_runnable_sum)
hasnt advanced i.e hasn't rolled over. Hence utra is called with the
current task which causes a rollover i.e. it advances prev,
curr(_runnable_sum) pointers amongst other things
old new
ws ms ws wc
| | | |
V V V V
-----|----------------|--------
prev curr
We see utra on current being called,
when a task 'p' is woken up, two update_task_ravg() calls are needed.
update_task_ravg(rq->curr, rq, TASK_UPDATE, wallclock, 0);
update_task_ravg(p, rq, TASK_WAKE, wallclock, 0);
when a task 'p' is migrating, three update_task_ravg() calls are needed.
update_task_ravg(task_rq(p)->curr, task_rq(p), TASK_UPDATE, wallclock, 0);
update_task_ravg(dest_rq->curr, dest_rq, TASK_UPDATE, wallclock, 0);
update_task_ravg(p, task_rq(p), TASK_MIGRATE, wallclock, 0);
when a task 'p' is tranferring in/out of rtg
update_task_ravg(rq->curr, rq, TASK_UPDATE, wallclock, 0);
update_task_ravg(p, rq, TASK_UPDATE, wallclock, 0);
We can optimize the number of update_task_ravg() calls by allowing to
rollover regardless of utra on current task. The best place to do it is
in update_window_start() right along the code which advances the window
start.
Since the update is not happening on the running task i.e current, the
CPU utilization i.e prev_runnable_sum does not reflect the current task
utilization immediately after the window is rolled over. However
update_task_ravg() is called on the current before presenting the
utilization to the governor. So there should not be any impact on
the final frequency selection.
A TASK_WAKE of a iowaited task needs to terminate the accumulation of
iowait time on the cpu where the task had slept. The mark_start of the
idle task is advanced and the "advanced" time is accounted as cpu busy
time. For this situation alone call utra on idle task if an iowaited
task is waking up.
Change-Id: Ifb771b41bd62f3c748035bf1e5e5ad7912b400f4
Signed-off-by: Pavankumar Kondeti <quic_pkondeti@quicinc.com>
Signed-off-by: Abhijeet Dharmapurikar <quic_adharmap@quicinc.com>
Signed-off-by: Shaleen Agrawal <quic_shalagra@quicinc.com>
The current code requires an execution of the core
control kthread in order to perform any pause/unpause
operations. Because the mechanism to pause or unpause
a cpu (halt) is only a bit-flip operation, this can
cause unnecessary delay in the unpause path.
Use spinlocks in halt, and call the main core control
functionality immediately from the walt_irq_work
context, eliminating this potential delay.
Change-Id: I74d82c445097c1074ad106501b101a06875da38e
Signed-off-by: Stephen Dickey <quic_dickey@quicinc.com>
cpu_drain_rq() will use stop_machine to schedule the draining of
a cpu. This must not be called within an atomic context, and
therefore constrains us as to where a halt operation can be
initiated.
To prepare for initiating halt/resume operations directly from
the core_control_check/walt_irq_work context, use a kthread to
create a context that will allow scheduling, and perform the
drain in that context.
Change-Id: Ic2d9232bebbebf2173cb77be43cf5ba90ce875d5
Signed-off-by: Stephen Dickey <quic_dickey@quicinc.com>
Currently the code uses walt_load_reported_window (wlrw in short) to
mark the beginning of the new window. All the updates to cpufreq use the
latest wlrw.
With 'commit acd27bd371dd2 ("sched/walt: walt irq work reduce locking")'
we could hit BUG(curr_ws < last_ws) in cpufreq_walt.c because of using a
stale wlrw. An example of the race.
cpu0 ED tick cpu1 ED tick cpu2 tick utra G->P migration
rq_lock(0) rq_lock(2)
cpu_util_freq_walt() detects rollover sets walt_irq_work_for_mgrtn
read wlrw =760 walt_irq_work_lastq_ws =776 rq_lock 3,4,5,6,7
rq_unlock(2) UPDATE wlrw = 776
walt_irq_work_for_rlvr ...
EL2/EL3 EXCEPTION ...waits for all rq lock ...
OR SVM rq_unlock 3,4,5,6,7
rq_lock(1)
RUNS cpu_util_freq_walt()
read wlrw =776
write load->ws =776
waltgov_calc_avg_cap()
HERE silver_policy->last_ws =776
rq_unlock(1)
write load->ws=760
waltgov_calc_avg_cap()
BUG(walt_load->ws 760 <
silver_policy->last_ws 776)
To fix this ensure that wlrw is updated ONLY when rq locks of all the
cpus are held.
Change-Id: I9a6649cbccdfdb5a70a0256703624dde47868a15
Signed-off-by: Sai Harshini Nimmala <quic_snimmala@quicinc.com>
Currently, every time a task migrates while running, the enqueue resets
the slice time accounted thus far, i.e. enqueue resets
sum_exec_snapshot. There is a potential for a frequently active
migrating task to get infinite slice.
Fix this by ensuring total_exec gets updated regardless of SLICE
threshold is crossed. Having a non zero total_exec will make
enqueue skip taking a snapshot.
Also we need to track two snapshots of sum_exec_runtime, one for total
and one for slice. The total snapshot will be taken upon task's enqueue
after it wakes up from sleep while the slice snapshot advances everytime
the SLICE threshold is crossed. This scheme helps us track the total
time the task ran since wakeup and the total time it ran in the slice.
The snapshot however is the origin for accounting time towards SLICE,
hence it should not be updated as long runtime is within SLICE. Once
SLICE is crossed, snapshot should be updated for tracking the runtime
towards the next SLICE period.
Change-Id: I3cf82570c3224b2c9a40b278d2c9baa059216bc3
Signed-off-by: Abhijeet Dharmapurikar <quic_adharmap@quicinc.com>
Signed-off-by: Sai Harshini Nimmala <quic_snimmala@quicinc.com>
Ensure that long running RT task detection is set to a minimum
of 800ms.
Change-Id: I378c6b51ef9d417f9143e19e2938b625d1dd6a65
Signed-off-by: Sai Harshini Nimmala <quic_snimmala@quicinc.com>
Disable detection of long running RT tasks by default.
Change-Id: Ia0224ebdfef5ee59523aeea80b25429d5b99f2be
Signed-off-by: Sai Harshini Nimmala <quic_snimmala@quicinc.com>
While deprecating the RT throttling mechanism, associated sysctl node
remained to be removed.
Remove it now.
Change-Id: Iac5a3da157d71bc878e1406f1f64f57e0ccb7103
Signed-off-by: Sai Harshini Nimmala <quic_snimmala@quicinc.com>
Correct copyright issues and update copyrights for proper year.
Change-Id: I2a8a407814c7124a6f0b7a257c1e26b82f018606
Signed-off-by: Stephen Dickey <quic_dickey@quicinc.com>
Currently, walt maintains cra and adds demand when enqueue
task happens and subtracts from it when dequeue happens.
However, when an rt task is throttled, walt is not informed
of the dequeue. This leads to a residual demand when the
idle task switches in post throttling leading to
walt accounting error.
Remove that specific accounting check since it is hard to
inform WALT of the throttle dequeue and unthrottle enqueue,
until rt can be fixed to support walt accounting in the
throttling case.
Change-Id: If1416a61fbe3979253a42cb82990b7b13ad2c4eb
Signed-off-by: Stephen Dickey <quic_dickey@quicinc.com>
When a cpu is unhalted it has no tasks enqueued on it. This means
that nothing will be immediately scheduled on that cpu.
Kick the cpu that has been unhalted, to ensure it can load-balance
and help other cpus.
Change-Id: I23fd12bd9967d6aabdbd13b868e0dc316f118342
Signed-off-by: Stephen Dickey <quic_dickey@quicinc.com>
Currently the tunable to set RTG Boost Frequency is set on the basis of
a hardcoded policy->cpu. However, this is incorrect on two fronts -
first, CPU4 is no longer the default CPU managing the Gold CPU policy
(it is now CPU3).
Secondly, the "typical" CPU managing the cluster's policy cpu may be
offline, in which case, another CPU from the cluster would be managing
the policy, and this implementation would break.
Change-Id: I0b9d4b882c93f48ea73f4cfa20e547d4216e142c
Signed-off-by: Shaleen Agrawal <quic_shalagra@quicinc.com>
When cpu capacities are updated, due to either thermal or frequency
limits, print trace output to understand why the change happened.
Change-Id: I5c09ddbcbfae5b26a23d6d2ca07b38c89159f878
Signed-off-by: Shaleen Agrawal <quic_shalagra@quicinc.com>
Currently, there is a issue in the system where the rt capacity could be
greater than the thermal capacity, and as a consequence we could end up
with a negative capacity in calculations.
Change-Id: If81be87878ae7948b0278e30ac4a8618d434d070
Signed-off-by: Shaleen Agrawal <quic_shalagra@quicinc.com>
Prior to the introduction of the PREEMPT_RT, migrate_disable()
was introduced to support PREEMPT_RT. Many places throughout the
code avoid using migrate_disable() if PREEMPT_RT is not set.
However, bpf related code makes no such precautions, which is
enabled.
migrate_tasks() will forcefully migrate tasks off of this cpu
without regard for migration_disabled state of the task. This
causes a warning when attempting to migrate a migration_disabled
task, and may have other unforseen consequences.
Prevent migrate_tasks from moving tasks for which migration is
disabled.
Change-Id: I04aef81611c8d4df8e2abbd7833b8b12ec146548
Signed-off-by: Stephen Dickey <quic_dickey@quicinc.com>
Sevearl hooks for halt were present in the main walt file, when
it should be in the halt file. Move, and change initialization
appropriately.
Change-Id: Ibab5012a3728b70caebab24b19e25d5eb7771cb0
Signed-off-by: Stephen Dickey <quic_dickey@quicinc.com>
Firstly, we change the bucket size to 16. This has two effects: firstly,
it helps improve power by predicting less aggressive midpoint values.
Secondly, it allows us optimize multiplication/division operations by
using bitshifts.
To enable these further overhead optimizations, translate pred_demand
to a 1024 unit value.
Change-Id: Ieb00f5bbce8aedefc5a023ce1e354dbfb0055543
Signed-off-by: Shaleen Agrawal <quic_shalagra@quicinc.com>
In get_pred_busy, we attempt to find the closing bucket, which is
nothing but the first bucket which is filled after the start bucket.
We can optimize this computation by using a bitmask of buckets.
Change-Id: I377be499fdd4876d5d3a43e8ebb34a489e6ae17b
Signed-off-by: Shaleen Agrawal <quic_shalagra@quicinc.com>
When determining predictive demand, we currently look at the history
samples in our past 8 windows that fall under the bucket range for which
we are evaluating. However, this level of precision is not required, and
can be dropped in favor of reduction of overheads.
Change-Id: Icef5d23b1574c625ef1a6d6228603d2b18aa8a40
Signed-off-by: Shaleen Agrawal <quic_shalagra@quicinc.com>
Reduce the calls to clock hardware by utilizing cached rq_clock() values.
Also track the latest clock seen on runqueue to workaround stale
rq_clock() snapshots.
Change-Id: Id4d905c36325ca1031231b55c697514dc4505658
Signed-off-by: Shaleen Agrawal <quic_shalagra@quicinc.com>
Halted cpus can still get assigned a timer through
get_nohz_timer_target, as long as they are not isolcpus.
Remove halted cpus from selection by get_nohz_timer_target,
while maintaining the preference for idle cpus, and in-domain
cpus.
Change-Id: I2619b6cedc831c64857ed71461067b66873cb61a
Signed-off-by: Stephen Dickey <quic_dickey@quicinc.com>
is_cpu_allowed is critical to select_fallback_rq, select_task_rq,
and __migrate_task, to ensure that only valid cpus are chosen for
the execution of a task.
Register for the trace hook, and restrict cpus selected by is_cpu_allowed
to non-halted cpus.
Change-Id: I72b2e90230603a8f97ce1da7014d72c0ebdee97e
Signed-off-by: Stephen Dickey <quic_dickey@quicinc.com>
This reverts commit 89fc117b538f9687ace7b399e12934cdfe601d66.
WALT was missing set_next_entity() calls on mvp task hierarchy. Now that
is addressed, let us revert the disabling of mvp feature.
Change-Id: I50763e354fb1b885c3186078fa6db499b4a975e1
Signed-off-by: Sai Harshini Nimmala <quic_snimmala@quicinc.com>
Currently, in the simple case when replace_next_task_fair() hook picks
a task it does not call set_next_entity on the newly picked task.
Add that.
Change-Id: I77074cf9a38ce1d2b659d8845cbaae362bd33441
Signed-off-by: Abhijeet Dharmapurikar <quic_adharmap@quicinc.com>
It is possible that the fallback destination cpu is the same
as the current cpu. In this case, this is an inappropriate
migration request, causing a double lock unbalance error in
walt_detach_task.
Prevent this case by preventing a task that is not going to be
migrated from being passed to __migrate_task, and instead
detach the task and add it to the list to be added back later.
Change-Id: I26f57bf52ca9a7bfe221d02d9ccb353d8c8d27ff
Signed-off-by: Stephen Dickey <quic_dickey@quicinc.com>
Deprecate RT thottling mechanism in lieu of new feature that handles
long running RT tasks.
Change-Id: I1137c6deefc61bc311deac66788a06201e64146e
Signed-off-by: Sai Harshini Nimmala <quic_snimmala@quicinc.com>
Detect long running RT tasks by tapping into the scheduler_tick
tracepoint, so that task can be detected while it is actively
running on the CPU and cause panic on the same CPU to get relevant
stack information for debug purposes.
Change-Id: I205a5512ba61e1a92d64659b723e661e11c39dd1
Signed-off-by: Sai Harshini Nimmala <quic_snimmala@quicinc.com>
