When determining predictive demand, we currently look at the history
samples in our past 8 windows that fall under the bucket range for which
we are evaluating. However, this level of precision is not required, and
can be dropped in favor of reduction of overheads.
Change-Id: Icef5d23b1574c625ef1a6d6228603d2b18aa8a40
Signed-off-by: Shaleen Agrawal <quic_shalagra@quicinc.com>
Reduce the calls to clock hardware by utilizing cached rq_clock() values.
Also track the latest clock seen on runqueue to workaround stale
rq_clock() snapshots.
Change-Id: Id4d905c36325ca1031231b55c697514dc4505658
Signed-off-by: Shaleen Agrawal <quic_shalagra@quicinc.com>
Halted cpus can still get assigned a timer through
get_nohz_timer_target, as long as they are not isolcpus.
Remove halted cpus from selection by get_nohz_timer_target,
while maintaining the preference for idle cpus, and in-domain
cpus.
Change-Id: I2619b6cedc831c64857ed71461067b66873cb61a
Signed-off-by: Stephen Dickey <quic_dickey@quicinc.com>
is_cpu_allowed is critical to select_fallback_rq, select_task_rq,
and __migrate_task, to ensure that only valid cpus are chosen for
the execution of a task.
Register for the trace hook, and restrict cpus selected by is_cpu_allowed
to non-halted cpus.
Change-Id: I72b2e90230603a8f97ce1da7014d72c0ebdee97e
Signed-off-by: Stephen Dickey <quic_dickey@quicinc.com>
This reverts commit 89fc117b538f9687ace7b399e12934cdfe601d66.
WALT was missing set_next_entity() calls on mvp task hierarchy. Now that
is addressed, let us revert the disabling of mvp feature.
Change-Id: I50763e354fb1b885c3186078fa6db499b4a975e1
Signed-off-by: Sai Harshini Nimmala <quic_snimmala@quicinc.com>
Currently, in the simple case when replace_next_task_fair() hook picks
a task it does not call set_next_entity on the newly picked task.
Add that.
Change-Id: I77074cf9a38ce1d2b659d8845cbaae362bd33441
Signed-off-by: Abhijeet Dharmapurikar <quic_adharmap@quicinc.com>
It is possible that the fallback destination cpu is the same
as the current cpu. In this case, this is an inappropriate
migration request, causing a double lock unbalance error in
walt_detach_task.
Prevent this case by preventing a task that is not going to be
migrated from being passed to __migrate_task, and instead
detach the task and add it to the list to be added back later.
Change-Id: I26f57bf52ca9a7bfe221d02d9ccb353d8c8d27ff
Signed-off-by: Stephen Dickey <quic_dickey@quicinc.com>
Deprecate RT thottling mechanism in lieu of new feature that handles
long running RT tasks.
Change-Id: I1137c6deefc61bc311deac66788a06201e64146e
Signed-off-by: Sai Harshini Nimmala <quic_snimmala@quicinc.com>
Detect long running RT tasks by tapping into the scheduler_tick
tracepoint, so that task can be detected while it is actively
running on the CPU and cause panic on the same CPU to get relevant
stack information for debug purposes.
Change-Id: I205a5512ba61e1a92d64659b723e661e11c39dd1
Signed-off-by: Sai Harshini Nimmala <quic_snimmala@quicinc.com>
Cleanup to give units to sched_get_cpu_util.
Change-Id: If04a3b38d7ff0c6c0a53edb0ed6014e7698b5195
Signed-off-by: Stephen Dickey <quic_dickey@quicinc.com>
It is possible that walt_find_energy_efficient_cpu will return
an error value, or a negative number for the cpu. In walt_lb_tick
this was called without checking the return value.
Update walt_find_energy_efficient_cpu to guarantee that it never
returns a bad value, and that in all cases it returns prev_cpu
if it cannot find an energy efficient cpu.
Change-Id: Ifc08bdf42b8d54923f8a4ecc3ca8b2c0dba11f4f
Signed-off-by: Stephen Dickey <quic_dickey@quicinc.com>
In case of fully loaded system with spare capacity 0 on all cores,
placement logic selects previous cpu of the task for its placement.
Update the logic to select cpu which has minimum number of runnables.
Also, remove "active_candidate" as we always fallback to least runnable
cpu.
Change-Id: Ib698969d38dbe7fc7b08451f3c3637aba19a72bc
Signed-off-by: Ashay Jaiswal <quic_ashayj@quicinc.com>
Currently, to compare cluster IDs of new and old CPUs,
new cpu's runqueue is being accessed without checking
if new cpu is valid.
Fix this by checking new cpu's validity before it is used.
Change-Id: I6a374de9690ede3d1deedb076959229b1e79b025
Signed-off-by: Sai Harshini Nimmala <quic_snimmala@quicinc.com>
It is causing rb_tree corruption - disable it for now.
Change-Id: I4bf8b47fd9ed0edf11a81706cbb2c0de5b817eb1
Signed-off-by: Abhijeet Dharmapurikar <quic_adharmap@quicinc.com>
When a task's cpu affinity is changed, do_set_cpus_allowed()
dequeues the task, updates the task's cpus_mask with new
affinity but enqueues the task back on the current cpu, even
if the current_cpu is not in the updated mask. This causes
the affinity check in enqueue path to fail.
Instead add this affinity check to set_task_cpu, this will
catch any selection of an cpu outside allowed cpus.
Change-Id: I6c1eaf77951f20aea961da463c1e8de0189d36e1
Signed-off-by: Stephen Dickey <quic_dickey@quicinc.com>
Support the ability to check whether the current time is within
a threshold since the last time the cpu was halted. If it's
within that threshold, pass the check.
Use that check to catch tasks being enqueued on halted cpus.
Change-Id: I1716aefc970fc3dea7474f6321971bf00c3b76f4
Signed-off-by: Stephen Dickey <quic_dickey@quicinc.com>
Allow for finer grain control over the walt-bug capabilities,
such that different walt-bug types can be treated differently,
either causing a panic, or simply printing out content.
echo 0x4544DEXY > /proc/sys/walt/panic_on_walt_bug
Y in the above represents the bitmask for the walt debug class
that is enabled. If set, the device will panic on that class
of issue.
X in the above represents the bitmask for the walt debug class
that is enabled, for printing to console. If set, device will
printk on that class of issue.
X and Y are independent, and maybe configured separately as
desired. However, nothing will happen if the most significant
3 bytes of panic_on_walt_bug are not properly set to the SENTINEL
value.
Change-Id: I01382fdb5717625f42f45eac4151d47578f0b0c6
Signed-off-by: Stephen Dickey <quic_dickey@quicinc.com>
Currently, we make certain decisions based on capacities of individual
CPUs. However, we may be interested in grouping certain cpus of
different capacities together. In this case, we will need to use their
cluster IDs to differentiate between two CPUs.
Change-Id: I20d9c473a98e55dfdd658b96fc9285f0a0f51652
Signed-off-by: Shaleen Agrawal <quic_shalagra@quicinc.com>
Update variable type to ensure all the valid values can be
accomodated in variable.
Change-Id: Ieaf0301d6ad524bfb87ff019aec41063ebc4c4c4
Signed-off-by: Ashay Jaiswal <quic_ashayj@quicinc.com>
walt_rt_energy_aware_wake_cpu could return -1, which makes
walt_select_task_rq_rt use the task's prev cpu (the backup cpu).
However, the backup_cpu could be halted.
Address this by replacing the chosen cpu with the lowest order non-halted
cpu in the system.
Change-Id: I366bd2e9093b945a093ae4542a77983b0bc57229
Signed-off-by: Stephen Dickey <quic_dickey@quicinc.com>
When the android vendor hook is not registered yet, there
is already tasks. And if those kind of tasks being killed
without any wakeup after vendor hook registered, it will
be a NULL referenced data.
Change-Id: Ied2eb3805dd01740a01707e2016c24a4648b1b25
Signed-off-by: Maria Yu <quic_aiquny@quicinc.com>
The sliding window logic in update_history is unnecessary complicated
and involves moving each window entry into a new position. Instead, we
can simply replace the older entries in place.
Change-Id: I49a80184241393219f421421f07514e01b8a674e
Signed-off-by: Shaleen Agrawal <quic_shalagra@quicinc.com>
After reviewing, it was determind that the can_halt_mask is no
longer needed. Remove it and cleanup code as appropriate.
Change-Id: I5430e6f3eda505e0f8fe1cedb9fa7b20425f6900
Signed-off-by: Stephen Dickey <quic_dickey@quicinc.com>
cpu_active mask is a super-set of cpu_online. Any cpus that are
active are by definition online, so it is redundant to check both.
Cleanup the usage of cpu_active and cpu_online, to eliminate
redundancies and ensure usage of the super set (active) when
needed.
Change-Id: I085cb94a609819b6786c81d8abc784aae8b4ff54
Signed-off-by: Stephen Dickey <quic_dickey@quicinc.com>
If a task is enqueued on a CPU for which it is not allowed
as dictated by the cpus_ptr of the task then report the issue.
Change-Id: Ib049e9ae07be1830d9ed6166c47aa78aff7e1b12
Signed-off-by: Stephen Dickey <quic_dickey@quicinc.com>
Minor cleanup including renaming of the drain functions and
commenting the update_halt_cpus function and usage.
Change-Id: I44502cde850af8fc6124e7b8ae014c562b21d0d9
Signed-off-by: Stephen Dickey <quic_dickey@quicinc.com>
The number history samples should be raised to a higher value so that an
active history of 100ms is considered in accounting task demand. Raise
the history size from 5 to 8 given the default windows size is 8ms. This
will capture 64ms of history for 8ms windows, and 128ms of history for
16ms windows, granting a compromise to prevent further code bloat.
Change-Id: I63ad5b2a20c9a32d3a65448a78d6aec98afe6ad5
Signed-off-by: Shaleen Agrawal <quic_shalagra@quicinc.com>
Adapt the min/max capacity cpu helper functions to use cluster IDs to
guide decisions rather than capacities.
Change-Id: Ic204f286c8eb95742ab64d366ce30d5ee780c340
Signed-off-by: Shaleen Agrawal <quic_shalagra@quicinc.com>
This change is for general scheduler improvement.
Change-Id: I6235ca8fce3d0c2ab3bcab1ecee0d97c6d0a9942
Signed-off-by: Abhijeet Dharmapurikar <adharmap@codeaurora.org>
Signed-off-by: Shaleen Agrawal <shalagra@codeaurora.org>
In walt_find_energy_efficient_cpu, the trace for compute energy
was being called just prior to an update to best energy cpu. The
goal of the trace is to print the selected state of the routine,
not an intermediate state.
Change-Id: I8132b42e65b9b3bada9316c0ba738fa3baac9f49
Signed-off-by: Stephen Dickey <quic_dickey@quicinc.com>
This change is for general scheduler improvement.
Change-Id: I997827d143a370e49e43ecba8494ee8e69652b57
Signed-off-by: Abhijeet Dharmapurikar <adharmap@codeaurora.org>
Signed-off-by: Shaleen Agrawal <shalagra@codeaurora.org>
select_task_rq_fair invokes the select_task_rq_fair hook, which
ultimately invokes walt_find_energy_aware_cpu for cfs tasks. This
can return a cpu that is halted.
Update the one remaining case that can cause feec to return a
halted cpu.
Change-Id: Ic97345764ad8be1060868414869dc0e455f4bc54
Signed-off-by: Stephen Dickey <quic_dickey@quicinc.com>
select_task_rq_rt() and push_rt_task() need to select
a cpu, and ultimately attempt to get a good match on
a cpu through find_lowest_rq. Walt hooks into this
routine to make an adjustment such that the best energy
aware cpu is found.
It is possible that walt_rt_energy_aware_wake_cpu will be
unable to find a non-halted cpu. find_lowest_rq at that
point will use the lowest_mask, initialized from the
task cpus_ptr, to find a CPU. At this point, find_lowest_rq
might return a halted CPU.
Instead, in the case where walt cannot find a best cpu to
choose, force the lowest_mask to no longer include halted
cpus, which will push find_lowest_rq to no longer choose a
halted cpu. In the unlikely event that find_lowest_rq still
fails to find a cpu it will pick anything from the
lowest_mask. In the unlikely event that lowest_mask is empty
after removing all of the halted cpus, find_lowest_rq will
fail and the cpu chosen will be unconstrained by halt.
Change-Id: I398bc804eb474396f6f06e2c3d4308fea71dafe8
Signed-off-by: Stephen Dickey <quic_dickey@quicinc.com>
rto_next_cpu can select any rd->rto_mask cpu, which may be
halted. restrict rto_next_cpu to unhalted CPUs.
Change-Id: I1c41c4c72b630d00cca080ed73609f6fdfb7143d
Signed-off-by: Stephen Dickey <quic_dickey@quicinc.com>
When set_cpus_allowed_ptr_locked runs, it chooses a new destination
cpu for the task based upon the new mask for the task. Unfortunately
this means that halted cpus can be included.
Register for a trace hook such that dest_cpu can be recomputed, and
a dest_cpu that is not halted, chosen.
Change-Id: I0ab072eb395bece8c0ab896d7a1008f9cf55f468
Signed-off-by: Stephen Dickey <quic_dickey@quicinc.com>
rebalance_domains will call load_balance, and ultimately will
call detach_tasks, walt_migrated_queued_task, and set_task_cpu, choosing
a cpu that is halted.
can_migrate_task will be used to validate whether a particular migration
is allowed to happen. Reject migrations for tasks that are being
migrated to a halted cpu.
Change-Id: I0b2f5d3857a2b1db9a2fa5017d5238f143a85ed9
Signed-off-by: Stephen Dickey <quic_dickey@quicinc.com>
The halted flag must be checked when performing a wakeup of
an rt task, to prevent tasks from running on halted CPUs.
Change-Id: Ib4a5cff48611abb0d1f71fc10be45439d2b7dd64
Signed-off-by: Stephen Dickey <quic_dickey@quicinc.com>
Currently walt_find_best_target can return a halted cpu.
Prevent the PREV_CPU_FASTPATH for cpus that are currently
halted, and reject them in general from being selected.
Change-Id: Iced61bf9b1355703c6403d1dc9f90d6e64aefdc6
Signed-off-by: Stephen Dickey <quic_dickey@quicinc.com>
If a halted CPU goes idle, don't try to pull tasks into
the CPU.
Change-Id: I50383f736ae03992b8bb43f5aa12cc2803068f6f
Signed-off-by: Stephen Dickey <quic_dickey@quicinc.com>
Make the minimum change for core_ctl to work properly with halt
instead of pause.
Change-Id: I7d2a2b50daaffc96846c86b8fd9dedf7f604b0b6
Signed-off-by: Stephen Dickey <quic_dickey@quicinc.com>
halt/start tracepoints need a little enhancement, to ensure
a measurement can be made from the start of an operation.
Also to debug RT task placements, add the lowest_mask
cpu mask which contains the cpus available for the
RT task placement algorithm to trace_sched_cpu_util, and
update the trace to show the halted state.
Change-Id: Idd5ab03d8ca5af21d42415dce790993c8c5df1d5
Signed-off-by: Stephen Dickey <quic_dickey@quicinc.com>
References to pause have been removed and replaced with
halt. Remove walt pause functionality from the build.
Change-Id: I07c412cfba4d9f13aff298aed43a0e208db9f046
Signed-off-by: Stephen Dickey <quic_dickey@quicinc.com>
Provide apis to the system in support of halting
cpus, as a replacement for pause/isolation.
Track the requests to halt a cpu, and only
start the cpu again when all requests have been
recanted.
Perform a migration of tasks off of a halted cpu,
if that cpu is online and not idle.
Change-Id: I99340b12e01ff53499b4107d843b7d48e6d6dff1
Signed-off-by: Stephen Dickey <quic_dickey@quicinc.com>
Optimize walt_irq_work to reduce locking for migrations.
Reduce scheduler overheads by reducing the locks needed
for the walt irq work migrations. Precisely track the
cpus locked, and unlock them at conclusion of work function.
Change-Id: I6ea6d24dc69e9876387253b489e91f82e45ed6d0
Signed-off-by: Stephen Dickey <quic_dickey@quicinc.com>
When a task yields, it relinquishes the cpu and scheduler
is tasked to find another task.
However he MVP implementation could return the same task
leading to a loop where the yielded task gets to run back.
To fix this, drop the MVP status of tasks that yield
themselves. The MVP status is restored the next time they
actually get enqueued - -likely because of a wakeup from
sleep or task migration.
Change-Id: Ia2a2074f18c4de56e0695ee2c5f1daf8a4eb5980
Signed-off-by: Tengfei Fan <tengfeif@codeaurora.org>
Signed-off-by: Sai Harshini Nimmala <quic_snimmala@quicinc.com>
As a side effect of commit f64fafbec582dd46169fba528cad2341c0c2d906
the check to skip frequency update if the same frequency as
the previous frequency is being applied, got removed at the point of
update.
Restore this check at the appropriate place.
Change-Id: I7aee7f84f0a64747b01ac31c2fdcbb4440908deb
Signed-off-by: Sai Harshini Nimmala <quic_snimmala@quicinc.com>
Restore obj file in Makefile that was removed in
commit 3908a213bc56 ("pause: port the users of pause").
Change-Id: I2014b049ca7cb5e4188b07087cc22fa9ee40cf85
Signed-off-by: Sai Harshini Nimmala <quic_snimmala@quicinc.com>
This change is for general scheduler improvement.
Change-Id: Iffd4ae221581aaa4aeb244a0cddd40a8b6aac74d
Signed-off-by: Abhijeet Dharmapurikar <adharmap@codeaurora.org>
