When the window is advanced to the next boundary, the CPU busy time
trackers need to be advanced and top-task trackers should be flipped.
This rollover needs to be done just once for a cpu per window
advancement, while it could be subject to numerous update_task_ravg
(called utra henceforth) with various tasks spanning window cross over
point. The current code achieves this one-time-rollover only when it
sees a utra called for the currently running task and its time has
crossed over in the new window.
As a result of this rollover being done only
when utra is called for current task, it became mandatory to call
utra on current task before calling utra on a non-current task.
This ensured a proper window rollover i.e. the trackers reset/flipped
if needed prior to calling utra on non-current task which may have
entered its execution in a new window.
Explaining pictorially
old new
ws ms ws wc
| | | |
V V V V
-----|----------------|--------
prev curr
In the above example, a non current task wc may have entered the new
window but prev and curr are still set as per the old window start. The
time from new ws to wc cannot be accounted because curr(_runnable_sum)
hasnt advanced i.e hasn't rolled over. Hence utra is called with the
current task which causes a rollover i.e. it advances prev,
curr(_runnable_sum) pointers amongst other things
old new
ws ms ws wc
| | | |
V V V V
-----|----------------|--------
prev curr
We see utra on current being called,
when a task 'p' is woken up, two update_task_ravg() calls are needed.
update_task_ravg(rq->curr, rq, TASK_UPDATE, wallclock, 0);
update_task_ravg(p, rq, TASK_WAKE, wallclock, 0);
when a task 'p' is migrating, three update_task_ravg() calls are needed.
update_task_ravg(task_rq(p)->curr, task_rq(p), TASK_UPDATE, wallclock, 0);
update_task_ravg(dest_rq->curr, dest_rq, TASK_UPDATE, wallclock, 0);
update_task_ravg(p, task_rq(p), TASK_MIGRATE, wallclock, 0);
when a task 'p' is tranferring in/out of rtg
update_task_ravg(rq->curr, rq, TASK_UPDATE, wallclock, 0);
update_task_ravg(p, rq, TASK_UPDATE, wallclock, 0);
We can optimize the number of update_task_ravg() calls by allowing to
rollover regardless of utra on current task. The best place to do it is
in update_window_start() right along the code which advances the window
start.
Since the update is not happening on the running task i.e current, the
CPU utilization i.e prev_runnable_sum does not reflect the current task
utilization immediately after the window is rolled over. However
update_task_ravg() is called on the current before presenting the
utilization to the governor. So there should not be any impact on
the final frequency selection.
A TASK_WAKE of a iowaited task needs to terminate the accumulation of
iowait time on the cpu where the task had slept. The mark_start of the
idle task is advanced and the "advanced" time is accounted as cpu busy
time. For this situation alone call utra on idle task if an iowaited
task is waking up.
Change-Id: Ifb771b41bd62f3c748035bf1e5e5ad7912b400f4
Signed-off-by: Pavankumar Kondeti <quic_pkondeti@quicinc.com>
Signed-off-by: Abhijeet Dharmapurikar <quic_adharmap@quicinc.com>
Signed-off-by: Shaleen Agrawal <quic_shalagra@quicinc.com>
The current code requires an execution of the core
control kthread in order to perform any pause/unpause
operations. Because the mechanism to pause or unpause
a cpu (halt) is only a bit-flip operation, this can
cause unnecessary delay in the unpause path.
Use spinlocks in halt, and call the main core control
functionality immediately from the walt_irq_work
context, eliminating this potential delay.
Change-Id: I74d82c445097c1074ad106501b101a06875da38e
Signed-off-by: Stephen Dickey <quic_dickey@quicinc.com>
cpu_drain_rq() will use stop_machine to schedule the draining of
a cpu. This must not be called within an atomic context, and
therefore constrains us as to where a halt operation can be
initiated.
To prepare for initiating halt/resume operations directly from
the core_control_check/walt_irq_work context, use a kthread to
create a context that will allow scheduling, and perform the
drain in that context.
Change-Id: Ic2d9232bebbebf2173cb77be43cf5ba90ce875d5
Signed-off-by: Stephen Dickey <quic_dickey@quicinc.com>
Currently the code uses walt_load_reported_window (wlrw in short) to
mark the beginning of the new window. All the updates to cpufreq use the
latest wlrw.
With 'commit acd27bd371dd2 ("sched/walt: walt irq work reduce locking")'
we could hit BUG(curr_ws < last_ws) in cpufreq_walt.c because of using a
stale wlrw. An example of the race.
cpu0 ED tick cpu1 ED tick cpu2 tick utra G->P migration
rq_lock(0) rq_lock(2)
cpu_util_freq_walt() detects rollover sets walt_irq_work_for_mgrtn
read wlrw =760 walt_irq_work_lastq_ws =776 rq_lock 3,4,5,6,7
rq_unlock(2) UPDATE wlrw = 776
walt_irq_work_for_rlvr ...
EL2/EL3 EXCEPTION ...waits for all rq lock ...
OR SVM rq_unlock 3,4,5,6,7
rq_lock(1)
RUNS cpu_util_freq_walt()
read wlrw =776
write load->ws =776
waltgov_calc_avg_cap()
HERE silver_policy->last_ws =776
rq_unlock(1)
write load->ws=760
waltgov_calc_avg_cap()
BUG(walt_load->ws 760 <
silver_policy->last_ws 776)
To fix this ensure that wlrw is updated ONLY when rq locks of all the
cpus are held.
Change-Id: I9a6649cbccdfdb5a70a0256703624dde47868a15
Signed-off-by: Sai Harshini Nimmala <quic_snimmala@quicinc.com>
Currently, every time a task migrates while running, the enqueue resets
the slice time accounted thus far, i.e. enqueue resets
sum_exec_snapshot. There is a potential for a frequently active
migrating task to get infinite slice.
Fix this by ensuring total_exec gets updated regardless of SLICE
threshold is crossed. Having a non zero total_exec will make
enqueue skip taking a snapshot.
Also we need to track two snapshots of sum_exec_runtime, one for total
and one for slice. The total snapshot will be taken upon task's enqueue
after it wakes up from sleep while the slice snapshot advances everytime
the SLICE threshold is crossed. This scheme helps us track the total
time the task ran since wakeup and the total time it ran in the slice.
The snapshot however is the origin for accounting time towards SLICE,
hence it should not be updated as long runtime is within SLICE. Once
SLICE is crossed, snapshot should be updated for tracking the runtime
towards the next SLICE period.
Change-Id: I3cf82570c3224b2c9a40b278d2c9baa059216bc3
Signed-off-by: Abhijeet Dharmapurikar <quic_adharmap@quicinc.com>
Signed-off-by: Sai Harshini Nimmala <quic_snimmala@quicinc.com>
Ensure that long running RT task detection is set to a minimum
of 800ms.
Change-Id: I378c6b51ef9d417f9143e19e2938b625d1dd6a65
Signed-off-by: Sai Harshini Nimmala <quic_snimmala@quicinc.com>
Disable detection of long running RT tasks by default.
Change-Id: Ia0224ebdfef5ee59523aeea80b25429d5b99f2be
Signed-off-by: Sai Harshini Nimmala <quic_snimmala@quicinc.com>
While deprecating the RT throttling mechanism, associated sysctl node
remained to be removed.
Remove it now.
Change-Id: Iac5a3da157d71bc878e1406f1f64f57e0ccb7103
Signed-off-by: Sai Harshini Nimmala <quic_snimmala@quicinc.com>
Correct copyright issues and update copyrights for proper year.
Change-Id: I2a8a407814c7124a6f0b7a257c1e26b82f018606
Signed-off-by: Stephen Dickey <quic_dickey@quicinc.com>
Currently, walt maintains cra and adds demand when enqueue
task happens and subtracts from it when dequeue happens.
However, when an rt task is throttled, walt is not informed
of the dequeue. This leads to a residual demand when the
idle task switches in post throttling leading to
walt accounting error.
Remove that specific accounting check since it is hard to
inform WALT of the throttle dequeue and unthrottle enqueue,
until rt can be fixed to support walt accounting in the
throttling case.
Change-Id: If1416a61fbe3979253a42cb82990b7b13ad2c4eb
Signed-off-by: Stephen Dickey <quic_dickey@quicinc.com>
When a cpu is unhalted it has no tasks enqueued on it. This means
that nothing will be immediately scheduled on that cpu.
Kick the cpu that has been unhalted, to ensure it can load-balance
and help other cpus.
Change-Id: I23fd12bd9967d6aabdbd13b868e0dc316f118342
Signed-off-by: Stephen Dickey <quic_dickey@quicinc.com>
Currently the tunable to set RTG Boost Frequency is set on the basis of
a hardcoded policy->cpu. However, this is incorrect on two fronts -
first, CPU4 is no longer the default CPU managing the Gold CPU policy
(it is now CPU3).
Secondly, the "typical" CPU managing the cluster's policy cpu may be
offline, in which case, another CPU from the cluster would be managing
the policy, and this implementation would break.
Change-Id: I0b9d4b882c93f48ea73f4cfa20e547d4216e142c
Signed-off-by: Shaleen Agrawal <quic_shalagra@quicinc.com>
When cpu capacities are updated, due to either thermal or frequency
limits, print trace output to understand why the change happened.
Change-Id: I5c09ddbcbfae5b26a23d6d2ca07b38c89159f878
Signed-off-by: Shaleen Agrawal <quic_shalagra@quicinc.com>
Currently, there is a issue in the system where the rt capacity could be
greater than the thermal capacity, and as a consequence we could end up
with a negative capacity in calculations.
Change-Id: If81be87878ae7948b0278e30ac4a8618d434d070
Signed-off-by: Shaleen Agrawal <quic_shalagra@quicinc.com>
Prior to the introduction of the PREEMPT_RT, migrate_disable()
was introduced to support PREEMPT_RT. Many places throughout the
code avoid using migrate_disable() if PREEMPT_RT is not set.
However, bpf related code makes no such precautions, which is
enabled.
migrate_tasks() will forcefully migrate tasks off of this cpu
without regard for migration_disabled state of the task. This
causes a warning when attempting to migrate a migration_disabled
task, and may have other unforseen consequences.
Prevent migrate_tasks from moving tasks for which migration is
disabled.
Change-Id: I04aef81611c8d4df8e2abbd7833b8b12ec146548
Signed-off-by: Stephen Dickey <quic_dickey@quicinc.com>
Sevearl hooks for halt were present in the main walt file, when
it should be in the halt file. Move, and change initialization
appropriately.
Change-Id: Ibab5012a3728b70caebab24b19e25d5eb7771cb0
Signed-off-by: Stephen Dickey <quic_dickey@quicinc.com>
Firstly, we change the bucket size to 16. This has two effects: firstly,
it helps improve power by predicting less aggressive midpoint values.
Secondly, it allows us optimize multiplication/division operations by
using bitshifts.
To enable these further overhead optimizations, translate pred_demand
to a 1024 unit value.
Change-Id: Ieb00f5bbce8aedefc5a023ce1e354dbfb0055543
Signed-off-by: Shaleen Agrawal <quic_shalagra@quicinc.com>
In get_pred_busy, we attempt to find the closing bucket, which is
nothing but the first bucket which is filled after the start bucket.
We can optimize this computation by using a bitmask of buckets.
Change-Id: I377be499fdd4876d5d3a43e8ebb34a489e6ae17b
Signed-off-by: Shaleen Agrawal <quic_shalagra@quicinc.com>
When determining predictive demand, we currently look at the history
samples in our past 8 windows that fall under the bucket range for which
we are evaluating. However, this level of precision is not required, and
can be dropped in favor of reduction of overheads.
Change-Id: Icef5d23b1574c625ef1a6d6228603d2b18aa8a40
Signed-off-by: Shaleen Agrawal <quic_shalagra@quicinc.com>
Reduce the calls to clock hardware by utilizing cached rq_clock() values.
Also track the latest clock seen on runqueue to workaround stale
rq_clock() snapshots.
Change-Id: Id4d905c36325ca1031231b55c697514dc4505658
Signed-off-by: Shaleen Agrawal <quic_shalagra@quicinc.com>
Halted cpus can still get assigned a timer through
get_nohz_timer_target, as long as they are not isolcpus.
Remove halted cpus from selection by get_nohz_timer_target,
while maintaining the preference for idle cpus, and in-domain
cpus.
Change-Id: I2619b6cedc831c64857ed71461067b66873cb61a
Signed-off-by: Stephen Dickey <quic_dickey@quicinc.com>
is_cpu_allowed is critical to select_fallback_rq, select_task_rq,
and __migrate_task, to ensure that only valid cpus are chosen for
the execution of a task.
Register for the trace hook, and restrict cpus selected by is_cpu_allowed
to non-halted cpus.
Change-Id: I72b2e90230603a8f97ce1da7014d72c0ebdee97e
Signed-off-by: Stephen Dickey <quic_dickey@quicinc.com>
This reverts commit 89fc117b538f9687ace7b399e12934cdfe601d66.
WALT was missing set_next_entity() calls on mvp task hierarchy. Now that
is addressed, let us revert the disabling of mvp feature.
Change-Id: I50763e354fb1b885c3186078fa6db499b4a975e1
Signed-off-by: Sai Harshini Nimmala <quic_snimmala@quicinc.com>
Currently, in the simple case when replace_next_task_fair() hook picks
a task it does not call set_next_entity on the newly picked task.
Add that.
Change-Id: I77074cf9a38ce1d2b659d8845cbaae362bd33441
Signed-off-by: Abhijeet Dharmapurikar <quic_adharmap@quicinc.com>
It is possible that the fallback destination cpu is the same
as the current cpu. In this case, this is an inappropriate
migration request, causing a double lock unbalance error in
walt_detach_task.
Prevent this case by preventing a task that is not going to be
migrated from being passed to __migrate_task, and instead
detach the task and add it to the list to be added back later.
Change-Id: I26f57bf52ca9a7bfe221d02d9ccb353d8c8d27ff
Signed-off-by: Stephen Dickey <quic_dickey@quicinc.com>
Deprecate RT thottling mechanism in lieu of new feature that handles
long running RT tasks.
Change-Id: I1137c6deefc61bc311deac66788a06201e64146e
Signed-off-by: Sai Harshini Nimmala <quic_snimmala@quicinc.com>
Detect long running RT tasks by tapping into the scheduler_tick
tracepoint, so that task can be detected while it is actively
running on the CPU and cause panic on the same CPU to get relevant
stack information for debug purposes.
Change-Id: I205a5512ba61e1a92d64659b723e661e11c39dd1
Signed-off-by: Sai Harshini Nimmala <quic_snimmala@quicinc.com>
Cleanup to give units to sched_get_cpu_util.
Change-Id: If04a3b38d7ff0c6c0a53edb0ed6014e7698b5195
Signed-off-by: Stephen Dickey <quic_dickey@quicinc.com>
It is possible that walt_find_energy_efficient_cpu will return
an error value, or a negative number for the cpu. In walt_lb_tick
this was called without checking the return value.
Update walt_find_energy_efficient_cpu to guarantee that it never
returns a bad value, and that in all cases it returns prev_cpu
if it cannot find an energy efficient cpu.
Change-Id: Ifc08bdf42b8d54923f8a4ecc3ca8b2c0dba11f4f
Signed-off-by: Stephen Dickey <quic_dickey@quicinc.com>
In case of fully loaded system with spare capacity 0 on all cores,
placement logic selects previous cpu of the task for its placement.
Update the logic to select cpu which has minimum number of runnables.
Also, remove "active_candidate" as we always fallback to least runnable
cpu.
Change-Id: Ib698969d38dbe7fc7b08451f3c3637aba19a72bc
Signed-off-by: Ashay Jaiswal <quic_ashayj@quicinc.com>
Currently, to compare cluster IDs of new and old CPUs,
new cpu's runqueue is being accessed without checking
if new cpu is valid.
Fix this by checking new cpu's validity before it is used.
Change-Id: I6a374de9690ede3d1deedb076959229b1e79b025
Signed-off-by: Sai Harshini Nimmala <quic_snimmala@quicinc.com>
It is causing rb_tree corruption - disable it for now.
Change-Id: I4bf8b47fd9ed0edf11a81706cbb2c0de5b817eb1
Signed-off-by: Abhijeet Dharmapurikar <quic_adharmap@quicinc.com>
When a task's cpu affinity is changed, do_set_cpus_allowed()
dequeues the task, updates the task's cpus_mask with new
affinity but enqueues the task back on the current cpu, even
if the current_cpu is not in the updated mask. This causes
the affinity check in enqueue path to fail.
Instead add this affinity check to set_task_cpu, this will
catch any selection of an cpu outside allowed cpus.
Change-Id: I6c1eaf77951f20aea961da463c1e8de0189d36e1
Signed-off-by: Stephen Dickey <quic_dickey@quicinc.com>
Support the ability to check whether the current time is within
a threshold since the last time the cpu was halted. If it's
within that threshold, pass the check.
Use that check to catch tasks being enqueued on halted cpus.
Change-Id: I1716aefc970fc3dea7474f6321971bf00c3b76f4
Signed-off-by: Stephen Dickey <quic_dickey@quicinc.com>
Allow for finer grain control over the walt-bug capabilities,
such that different walt-bug types can be treated differently,
either causing a panic, or simply printing out content.
echo 0x4544DEXY > /proc/sys/walt/panic_on_walt_bug
Y in the above represents the bitmask for the walt debug class
that is enabled. If set, the device will panic on that class
of issue.
X in the above represents the bitmask for the walt debug class
that is enabled, for printing to console. If set, device will
printk on that class of issue.
X and Y are independent, and maybe configured separately as
desired. However, nothing will happen if the most significant
3 bytes of panic_on_walt_bug are not properly set to the SENTINEL
value.
Change-Id: I01382fdb5717625f42f45eac4151d47578f0b0c6
Signed-off-by: Stephen Dickey <quic_dickey@quicinc.com>
Currently, we make certain decisions based on capacities of individual
CPUs. However, we may be interested in grouping certain cpus of
different capacities together. In this case, we will need to use their
cluster IDs to differentiate between two CPUs.
Change-Id: I20d9c473a98e55dfdd658b96fc9285f0a0f51652
Signed-off-by: Shaleen Agrawal <quic_shalagra@quicinc.com>
Update variable type to ensure all the valid values can be
accomodated in variable.
Change-Id: Ieaf0301d6ad524bfb87ff019aec41063ebc4c4c4
Signed-off-by: Ashay Jaiswal <quic_ashayj@quicinc.com>
walt_rt_energy_aware_wake_cpu could return -1, which makes
walt_select_task_rq_rt use the task's prev cpu (the backup cpu).
However, the backup_cpu could be halted.
Address this by replacing the chosen cpu with the lowest order non-halted
cpu in the system.
Change-Id: I366bd2e9093b945a093ae4542a77983b0bc57229
Signed-off-by: Stephen Dickey <quic_dickey@quicinc.com>
When the android vendor hook is not registered yet, there
is already tasks. And if those kind of tasks being killed
without any wakeup after vendor hook registered, it will
be a NULL referenced data.
Change-Id: Ied2eb3805dd01740a01707e2016c24a4648b1b25
Signed-off-by: Maria Yu <quic_aiquny@quicinc.com>
The sliding window logic in update_history is unnecessary complicated
and involves moving each window entry into a new position. Instead, we
can simply replace the older entries in place.
Change-Id: I49a80184241393219f421421f07514e01b8a674e
Signed-off-by: Shaleen Agrawal <quic_shalagra@quicinc.com>
After reviewing, it was determind that the can_halt_mask is no
longer needed. Remove it and cleanup code as appropriate.
Change-Id: I5430e6f3eda505e0f8fe1cedb9fa7b20425f6900
Signed-off-by: Stephen Dickey <quic_dickey@quicinc.com>
cpu_active mask is a super-set of cpu_online. Any cpus that are
active are by definition online, so it is redundant to check both.
Cleanup the usage of cpu_active and cpu_online, to eliminate
redundancies and ensure usage of the super set (active) when
needed.
Change-Id: I085cb94a609819b6786c81d8abc784aae8b4ff54
Signed-off-by: Stephen Dickey <quic_dickey@quicinc.com>
If a task is enqueued on a CPU for which it is not allowed
as dictated by the cpus_ptr of the task then report the issue.
Change-Id: Ib049e9ae07be1830d9ed6166c47aa78aff7e1b12
Signed-off-by: Stephen Dickey <quic_dickey@quicinc.com>
Minor cleanup including renaming of the drain functions and
commenting the update_halt_cpus function and usage.
Change-Id: I44502cde850af8fc6124e7b8ae014c562b21d0d9
Signed-off-by: Stephen Dickey <quic_dickey@quicinc.com>
The number history samples should be raised to a higher value so that an
active history of 100ms is considered in accounting task demand. Raise
the history size from 5 to 8 given the default windows size is 8ms. This
will capture 64ms of history for 8ms windows, and 128ms of history for
16ms windows, granting a compromise to prevent further code bloat.
Change-Id: I63ad5b2a20c9a32d3a65448a78d6aec98afe6ad5
Signed-off-by: Shaleen Agrawal <quic_shalagra@quicinc.com>
Adapt the min/max capacity cpu helper functions to use cluster IDs to
guide decisions rather than capacities.
Change-Id: Ic204f286c8eb95742ab64d366ce30d5ee780c340
Signed-off-by: Shaleen Agrawal <quic_shalagra@quicinc.com>
This change is for general scheduler improvement.
Change-Id: I6235ca8fce3d0c2ab3bcab1ecee0d97c6d0a9942
Signed-off-by: Abhijeet Dharmapurikar <adharmap@codeaurora.org>
Signed-off-by: Shaleen Agrawal <shalagra@codeaurora.org>
In walt_find_energy_efficient_cpu, the trace for compute energy
was being called just prior to an update to best energy cpu. The
goal of the trace is to print the selected state of the routine,
not an intermediate state.
Change-Id: I8132b42e65b9b3bada9316c0ba738fa3baac9f49
Signed-off-by: Stephen Dickey <quic_dickey@quicinc.com>
