Make task_boost_type declarations public, as it is being used by other
modules, by moving to appropriate header.
Change-Id: I24baca67b9fdba883d7393247ca774ce9a664039
Signed-off-by: Shaleen Agrawal <shalagra@codeaurora.org>
Currently the code tries to pull RT tasks from balance_rt()
i.e when previous task is a RT task and it is de-activated.
To cover more cases, attempt this from newly idle load balance
which gets called before a CPU goes to idle.
Change-Id: Id3c9a68c19d2b5709f9953ac2383c14f5134b7de
Signed-off-by: Pavankumar Kondeti <pkondeti@codeaurora.org>
Add more debugging information to walt_newidle_balance trace point
to better understand newidle balance behavior.
Change-Id: I946d1d89644cc4fffb026e00686eedf805b06c01
Signed-off-by: Pavankumar Kondeti <pkondeti@codeaurora.org>
SCHED_BUG_ON is called from update_task_rq_cpu_cycles() when
current time is less than the previous recorded time for the
task. SCHED_BUG_ON isn't providing the precise details of
the time delta.
Improve the debugging such that the previous time, current, time
pid, and CPU are all printed before the scheduler dump.
Change-Id: I2156310be7db792e5579f8f56a0b653ed1ace225
Signed-off-by: Stephen Dickey <dickey@codeaurora.org>
It is possible that a cpu paused with thermal might be
unpaused by core-ctl, as well as the reverse. This means
that a cpu that has been paused for thermal reasons
can be unpaused when there is need.
Ensure that cpus paused by more than one entity are sticky,
such that only the last unpause request will actually
perform the operation.
Change-Id: Ieb2f836af7348d48a43ae1b22e11d97046ef9dea
Signed-off-by: Stephen Dickey <dickey@codeaurora.org>
Signed-off-by: Sai Harshini Nimmala <snimmala@codeaurora.org>
The first CPU that supports AArch32 in EL0 on asymmetric AArch32
systems can't be paused. Track such CPUs in core_ctl to not to
request pause_cpus().
Change-Id: I4a9132884c3e322e9a6bd7d1f06a82bd9113eec8
Signed-off-by: Pavankumar Kondeti <pkondeti@codeaurora.org>
When a CPU is not in idle state, the idle exit latency should be 0.
However walt_get_idle_exit_latency() returns UINT_MAX. Fix this.
Without this, the idle CPU that is serving an interrupt will not
win the battle against a CPU that is in a deep c-state.
Change-Id: Ic9ca36add5cd97cc7174c8e25b269ae40aadd451
Signed-off-by: Pavankumar Kondeti <pkondeti@codeaurora.org>
The cum_window_demand tracking is not supported in WALT. Remove the
left over code and use the PELT signal based util_avg as replacement
in RT task placement. The CFS task placement is already using the
same.
Change-Id: I078353ed33729b7bf4eab324c7ec861982a3bc57
Signed-off-by: Pavankumar Kondeti <pkondeti@codeaurora.org>
The tick path load balance is meant for fair tasks. Add a check
for the same before considering the task for migration. The
fair_policy() check alone does not work as it returns true
for idle task and also for RT boosted tasks. Add a wrapper
based on priority check.
Change-Id: I05cc996cb15051e4f6bb1a73005234e4d5188d19
Signed-off-by: Pavankumar Kondeti <pkondeti@codeaurora.org>
WALT core module register for android_rvh_update_misfit_status
restricted vendor hook. Remove the duplicate registration from
WALT CFS module. The later registered function has a compilation
issue when FAIR_GROUP_SCHED is enabled.
Change-Id: I2d89efa92ca704941f12263ee4994dc389202d5f
Signed-off-by: Pavankumar Kondeti <pkondeti@codeaurora.org>
The balance_rt vendor hook is implemented to pull a runnable RT
task from the other CPU.
Change-Id: I288fc9a8858cb29ae9c6993b398a349b4a8c7475
Signed-off-by: Pavankumar Kondeti <pkondeti@codeaurora.org>
Signed-off-by: Sai Harshini Nimmala <snimmala@codeaurora.org>
FIFO/RT tasks are supposed to run for short amount of time. So
when a CPU is busy with 1 RT and 1 CFS task, the other idle CPU
won't offer help thinking the CFS task will run immediately.
However, results indicate that realaxing this policy and pulling
the CFS task helps in reducing the runnables. This policy is
limited to the same capacity cluster load balance as we want
strict prefer idle behavior for it.
Change-Id: I4df6e00233503e096b1bb87fd17a784ea53fd9be
Signed-off-by: Pavankumar Kondeti <pkondeti@codeaurora.org>
The utilization return by cpu_util() is bounded by the original
capacity of the CPU. Load balancer uses the utilization to
break the ties. Use unbounded cpu utilization so that the most
loaded CPU gets help.
Change-Id: Id16cea205e6eb07a42a1e86fda533cd825df6908
Signed-off-by: Pavankumar Kondeti <pkondeti@codeaurora.org>
Active migration of running tasks is allowed only onto idle CPUs.
Add a check for it in walt_lb_find_busiest_lower_cap_cpu().
Change-Id: I195aadc8b5822d4f231edd2f57d9df5ef1a6e350
Signed-off-by: Pavankumar Kondeti <pkondeti@codeaurora.org>
Currently we are using task_fits_capacity() for misfit task
accounting. This does not consider boost scenarios. We also
end up accounting 100% tasks running on the highest capacity
CPU as misfit. A task on the highest capacity CPU should never
be treated as misfit.
rq->misfit_task_load should use task_util() not task_load(). The
later is task demand scaled to sched_ravg_window in nano seconds.
Change-Id: I1f923e33530d2d04d7bfca9dd42920f799794864
Signed-off-by: Pavankumar Kondeti <pkondeti@codeaurora.org>
walt_lb_find_busiest_lower_cap_cpu() is meant to be called when
the destination CPU has more capacity than the source CPU group.
However, the condition for calling this function is inverted.
Change-Id: I03dc62f42795479d21f676f8e27459955229dc82
Signed-off-by: Pavankumar Kondeti <pkondeti@codeaurora.org>
Implement select_task_rq_rt trace hook to be able to find suitable
CPU for the RT tasks from wake-up path.
Change-Id: Id1e8a4529d7391bf39a2a069ae37c146fb8a55c9
Signed-off-by: Satya Durga Srinivasu Prabhala <satyap@codeaurora.org>
Signed-off-by: Sai Harshini Nimmala <snimmala@codeaurora.org>
Currently there is no trace support for indicating
whether an application is 32 or 64 bit compatible.
Since SOCs going forward will have a mix of support
for 32 bit vs 64 bit applications, this is needed
information.
Update the trace_sched_enq_deq output such that the
task's compatibility (32 or 64) is printed.
Change-Id: I780b6b290a77a8405dd268b45a1e7affd035036b
Signed-off-by: Stephen Dickey <dickey@codeaurora.org>
The core kernel side has race conditions which lead to memory faults.
Disable this feature temporarily.
Change-Id: I0013753d571f1e7265ce385a1130f2671c574ab0
Signed-off-by: Shaleen Agrawal <shalagra@codeaurora.org>
The clean up including removing unused variables and functions,
converting global variables to static.
Change-Id: Ib5126ac9df701fdde354e0e5d759f0177db5a28b
Signed-off-by: Pavankumar Kondeti <pkondeti@codeaurora.org>
sched-walt-debug module does not get compiled unless SCHED_WALT
config is turned on. However the config corresponding to the
debug module shows in menuconfig even without enabling SCHED_WALT.
Fix this by specifying the dependency in the Kconfig.
Change-Id: Ie466c0ca11bbcdec43af2624c3631ddff5c77024
Signed-off-by: Pavankumar Kondeti <pkondeti@codeaurora.org>
The walt_find_best_target() has a special case for the active task
migration attempt from tick path. It tries hard to find an idle
CPU by disabling most_spare_cap_cpu selection. However this is
all unnecessary if we can check for the idle CPU condition at
the caller site. This helps the overall execution time of wfbt().
Change-Id: I89de44824dca8729b8c62898f9e75dc8e0e1c544
Signed-off-by: Pavankumar Kondeti <pkondeti@codeaurora.org>
walt_find_best_target() which selects the best CPU has a special
handling for running tasks. The idle CPUs are only considered as
migrating a running task to a busy CPU will result in wait time.
A running task is detected via task->state == TASK_RUNNING check,
which is true for tick path active migration tasks. It is also
true for newly created tasks. These tasks are getting placed on
idle CPUs of the higher capacity clusters though the lower capacity
CPUs are lightly loaded. Detect the running task via task->on_rq to
fix this issue.
Change-Id: I154cd6c6f934061526bce30fe596d9ad36c9e07f
Signed-off-by: Pavankumar Kondeti <pkondeti@codeaurora.org>
core_ctl is part of WALT. There is no separate config for
core_ctl. So it makes sense to declare all the WALT related
public interfaces in a single header file.
Change-Id: I0ae5585c0037295eee4268705f0e553d14c60fd9
Signed-off-by: Pavankumar Kondeti <pkondeti@codeaurora.org>
Signed-off-by: Sai Harshini Nimmala <snimmala@codeaurora.org>
wake_up_idle related functions are meant for other modules
usage. Make them public by moving to appropriate header.
Change-Id: I888b511e656d6c160c98b5da44bc873573a9beed
Signed-off-by: Pavankumar Kondeti <pkondeti@codeaurora.org>
The walt_task_struct structure is used by WALT module. WALT
type cast the task_struct's android vendor data to
walt_task_struct and use it for the scheduler needs. However
there can be other users who wants to cache information per task.
Currently walt_task_struct is defined in WALT's private header and
can't be included by other modules. Move the walt_task_struct and
dependent structures to appropriate header file so that is becomes
public.
This movement also allows wake_up_idle related funcitons to define
in the public header and keep the inline functionality instead of
exporting them for the other modules usage.
Change-Id: If8c94c7d6cd709bbc56aa08eb964d135a492a530
Signed-off-by: Pavankumar Kondeti <pkondeti@codeaurora.org>
set_task_boost() API is meant for other modules, so export
this function. Also move the function declaration to
appropriate header.
Change-Id: I7213b137a0763140c631bea30653321674fef272
Signed-off-by: Pavankumar Kondeti <pkondeti@codeaurora.org>
Re-introduce the earlier sched_load_boost feature as
waltgov's boost feature. A tunable called "boost" is
exposed per cpufreq policy which can be used to boost
(either positive or negative) the utilization, thus
the selected frequency.
Change-Id: I3a41a985d449e5c753b5afcbc375749e67036089
Signed-off-by: Pavankumar Kondeti <pkondeti@codeaurora.org>
The sched_load_boost feature will be introduced in a different
form in the subsequent patch.
Change-Id: Ib528ccd0fcb78f85685399264cb58212752ca26b
Signed-off-by: Pavankumar Kondeti <pkondeti@codeaurora.org>
cpu_frequency trace events are coming twice when a frequency
switch happens. This is due to commit 08d8c65e84 ("cpufreq:
Move traces and update to policy->cur to cpufreq core") in the
upstream which takes care of cpu_frequency trace event and
updates to policy->cur when cpufreq_driver_fast_switch() is
called.
So remove the redundant code from waltgov cpufreq governor to
avoid duplicate cpu_frequency trave events.
Change-Id: I490e766abcc09ed2a580e5f09930a54d76367999
Signed-off-by: Pavankumar Kondeti <pkondeti@codeaurora.org>
Schedutil flags are no longer used in WALT as we moved to
waltgov cpufreq governor. So remove them.
Change-Id: Iedc68215ed0e994b8e051b282a575ac699ebf7e0
Signed-off-by: Pavankumar Kondeti <pkondeti@codeaurora.org>
The core_ctl has to know how many CPUs are active in each cluster.
If the current need is less than the number of active CPUs, it tries
to pause the CPUs provided tunables are in agreement. On the other
hand, if the current need is more than the number of active CPUs,
it tries to resume the CPUs, provided those CPUs are paused by us
in the first place.
To determine the number of active CPUs, we should consider only
active CPUs. The others could be in pause or hotplugged out. Since
we only use cpu_active_mask for both, we don't need wrappers to
tell this to us. All we need is to maintain how many CPUs that
are going to be paused/resumed in the current iteration and check
against the need/limits.
Change-Id: I0a29da869b52eff35ff1dfa3094c51d6ece4e49f
Signed-off-by: Pavankumar Kondeti <pkondeti@codeaurora.org>
static_key_disable_cpuslocked() API is called to disable TTWU_QUEUE
feature. This API expects read side of cpu_hotplug_lock lock is taken.
Since we are not acquiring that lock, use static_key_disable() API which
internally takes care of the locking.
Change-Id: Ic8c7023235dd2e03bfe93d3f87a6a68e4f5b9b0d
Signed-off-by: Pavankumar Kondeti <pkondeti@codeaurora.org>
There are references to kernel command line parameters like sched_predl
and sched_ravg_window. These are not supported any more. So remove the
left over code.
Change-Id: I865f44fafee3a1d8f3d0d6f8f7e2341aff9ca964
Signed-off-by: Pavankumar Kondeti <pkondeti@codeaurora.org>
LPM governor can query the scheduler preference about the CPU's
c-state via sched_lpm_disallowed_time() API. It returns 0 when
shallowest c-state is preferred and associated timeout is set.
Change-Id: Ie8246aa9ec4638730d3442468aa9dc1969a90369
Signed-off-by: Pavankumar Kondeti <pkondeti@codeaurora.org>
We export necessary symbols as msm_perf driver needs to use core_ctl and
expand the core_ctl header file to add non WALT stubs.
Change-Id: I2fa65d6e5b00cfc88b97579df7074e8f82cb23ac
Signed-off-by: Shaleen Agrawal <shalagra@codeaurora.org>
Signed-off-by: Sai Harshini Nimmala <snimmala@codeaurora.org>
should_we_pause() always return true. Update the conditional
check at the caller and remove this function.
Change-Id: I085bc7737888fcafc74350a7d22176ae27bdece2
Signed-off-by: Pavankumar Kondeti <pkondeti@codeaurora.org>
The core_ctl thread is woken up only after evaluating the need.
so another evaluation in core_ctl thread context is not needed.
Change-Id: I28162b758c18cda4ff05f1b1a4b1a5553c895e7e
Signed-off-by: Pavankumar Kondeti <pkondeti@codeaurora.org>
We have 1 core_ctl thread for all clusters. Evaluate all clusters
and then wakeup the core_ctl if required.
Change-Id: I8131ae97280d8c478fd061e0aac06f1e56c5b6ea
Signed-off-by: Pavankumar Kondeti <pkondeti@codeaurora.org>
cpuhp_pause trace point is in place. so remove the sched_pause
trace point from WALT module.
Change-Id: Ife250254207321a9033142a34319f04cdf3a2dbe
Signed-off-by: Pavankumar Kondeti <pkondeti@codeaurora.org>
sysctl_task_read_pid sysctl needs to be set to the desired task's
pid before reading any of the per-task sysctl. So use the same
mutex for both of these sysctl handlers to prevent data tearing.
While at it, fix an incorrect goto in sched_task_handler()
function and restrict the range of sysctl_task_read_pid.
Change-Id: I544010f51a28ff6e5c4f117d68816de147ef3d0f
Signed-off-by: Pavankumar Kondeti <pkondeti@codeaurora.org>
This change is for general scheduler improvement.
Change-Id: I8f521dfd31ff1c8cc1f16805e4e0ac78228d9383
Signed-off-by: Pavankumar Kondeti <pkondeti@codeaurora.org>
WALT is using padding avaialble in task_struct, rq and task_group
structures. Add a build time test to make sure that we are not
using more than the avaialble padding.
Change-Id: Icd7e23b4536075bc53a975eeff838eedd58b29cc
Signed-off-by: Pavankumar Kondeti <pkondeti@codeaurora.org>
In WALT task placement algorithm, we select the root domain to be the
root domain corresponding to the current cpu, however, this could be
null if the cpu is inactive. This, in fact, resulted in undesired
behaviour when evaluating a single old threaded usecase, since desired
cpu was inactive, thread was being placed on a fallback cpu.
Change-Id: I68eef10dc84db15796a0d43af7215f4dbbdc059e
Signed-off-by: Shaleen Agrawal <shalagra@codeaurora.org>
The TODO comments in walt_lb.c are not applicable anymore.
So remove them.
Change-Id: I3cd780ad3f05e974944d6aa60f0a3da155c7b1be
Signed-off-by: Pavankumar Kondeti <pkondeti@codeaurora.org>
walt_update_cluster_topology requires the availability
of the cpufreq policies. These are not available
immediately after insmoding walt, and this causes a
crash through SCHED_BUG_ON.
Register for the cpufreq notifier for POLICY creation,
and init the second half of walt init after the
policy creation for all cpus.
Change-Id: If1af62a4991f3226bd0b661f5a5e419ceea3817a
Signed-off-by: Stephen Dickey <dickey@codeaurora.org>
This change is for general scheduler improvement.
Change-Id: Id19f38a2b9a67b130876cb8df1e88c6a56aa6aad
Signed-off-by: Pavankumar Kondeti <pkondeti@codeaurora.org>
