core_ctl is launching a kthread which uses a spinlock, prior
to initializing the spinlock. This creates a race, and a lockup.
Change-Id: I170a77addc6e4c2cc90b6683b50723008760f2f4
Signed-off-by: Stephen Dickey <dickey@codeaurora.org>
When sched_ravg_window_nr_ticks tunable is read and write concurrently,
the read can return the previous value though the write is already
happened. Fix this by reading the sysctl_sched_ravg_window_nr_ticks
value under mutex.
Change-Id: Ie1146c47ac1a184fc8bf4a2824bf6db28fd7156d
Signed-off-by: Pavankumar Kondeti <pkondeti@codeaurora.org>
sched_dynamic_ravg_window_enable tunable is not currently used. If 0
is written to this tunable, the dynamic window size feature is disabled
in the kernel which results in bad performance. So remove this unused
tunable.
Change-Id: I64282146e526f98d049be25502ef638f538b0496
Signed-off-by: Pavankumar Kondeti <pkondeti@codeaurora.org>
As part of reducing the arguments to sched_task_util trace point,
remove is_rtg argument as well. It can be retrieved from task_struct.
The recent update to this trace point made walt_get_rtg_status()
an extern function. Move it to a header file and restore the inline
functionality.
Change-Id: If447600fd9835e3b78724eb7a94b87c79090fe7f
Signed-off-by: Pavankumar Kondeti <pkondeti@codeaurora.org>
Reduce redundant parameters passed to sched_task_util tracepoint
by invoking relevant functions that return these values during
runtime.
Change-Id: Ie146a5a7a2ac6e4879115f49d4ccc6f7dcadcb52
Signed-off-by: Sai Harshini Nimmala <snimmala@codeaurora.org>
preemptirq_long_init() return value is incorrectly being checked which
is causing few of the trace hooks to be not even registered. Update the
check to make sure return value is checked properly.
Change-Id: If68a23d585c9e460ec1e0f4902a4ed9c9cae4c68
Signed-off-by: Satya Durga Srinivasu Prabhala <satyap@codeaurora.org>
dequeue_task() hook invocation is being changed from post dequeue to
pre dequeue. Consider the task being dequeued is out of the scheduling
class managed list.
Change-Id: Ib95ec7f8a873587493487a3c31c3f53ea78b0d95
Signed-off-by: Pavankumar Kondeti <pkondeti@codeaurora.org>
The cpufreq constraints must be reflected in the capacity of
a CPU. Add a hook for cpufreq limits to implement this.
Change-Id: I779cb16bd66ea99c16e55faea7383e034d22440a
Signed-off-by: Pavankumar Kondeti <pkondeti@codeaurora.org>
With refcounting, a cpu will get stuck in a paused state
if client software doesn't account properly for paused CPUs.
To properly track which CPUs it has made a request to be paused,
the driver needs to record the success or failure of an operation
and only record a change in CPU state, in the case of success.
Change-Id: I7bab17026d67ac1733c77083eae97af51c30dd37
Signed-off-by: Stephen Dickey <dickey@codeaurora.org>
Signed-off-by: Sai Harshini Nimmala <snimmala@codeaurora.org>
During test cases where multiple entities (thermal, core_ctl, hotplug)
attempting to pause or offline CPUs, error conditions caused core_ctl
to get stuck, unable to pause or resume CPUs.
Improve the core_ctl implementation such that an error to pause or
resume is carefully tracked, such that core_ctl will retry those
operations at a later time. Simplify paused_by_us to eliminate
redundant information.
Change-Id: Id156e6644da5264f5552daea7d3092916c44c434
Signed-off-by: Stephen Dickey <dickey@codeaurora.org>
Client code can miss the actual paused state of a cpu, if ref-counting
is performed when the pause/resume operation is incomplete, or if
an additional pause/resume operation executes prior to updating ref-counts.
This becomes a problem with the online work handler, which can run
at the same time as any walt_pause/walt_resume call.
Protect the calls to pause/resume CPUs, such that the actual state of the
CPUS is known and stored, prior to beginning a new operation. Ensure
that the work function has accurate information before performing its
resume operation.
Change-Id: Iac279ac731f7c97b289547d36cce7de6e2b91c3a
Signed-off-by: Stephen Dickey <dickey@codeaurora.org>
With refcounting, the walt wrapper for pause/resume can keep
track of whether a CPU should be paused or not, where this
information used to be distributed to the clients using the api.
Simplify the design, such that a CPU is automatically paused
as it is coming online, affecting the active/inactive state of
the CPU, but not adjusting the ref-counts.
Going forward, client software using walt_pause/walt_resume
will not be required to keep track of the online state of a
cpu.
Change-Id: I83e3cc8743942c3a482eb588546b6ea9bb7bf2e3
Signed-off-by: Stephen Dickey <dickey@codeaurora.org>
This change is for general scheduler improvement.
Change-Id: Ib1ec2f655193a71d401d7e51f7890537d4f75aef
Signed-off-by: Shaleen Agrawal <shalagra@codeaurora.org>
Remove the TODO comment about frequency in-variance in walt governor
as the code is completed. Also remove a FIXME comment from WALT IRQ
work as the code looks correct there.
Change-Id: Ib5447009ab009d7cf2991c0f22d04a766de59f3a
Signed-off-by: Pavankumar Kondeti <pkondeti@codeaurora.org>
There is a revised member to track the wake_q length upstream.
Change-Id: Ie1b9619be3616d4749f5fa2da8584f5fd4b45a58
Signed-off-by: Shaleen Agrawal <shalagra@codeaurora.org>
This change is for general scheduler improvement.
Change-Id: If7c9fd9d76d54da94acec15018c82e0d66d4162c
Signed-off-by: Shaleen Agrawal <shalagra@codeaurora.org>
Make task_boost_type declarations public, as it is being used by other
modules, by moving to appropriate header.
Change-Id: I24baca67b9fdba883d7393247ca774ce9a664039
Signed-off-by: Shaleen Agrawal <shalagra@codeaurora.org>
Currently the code tries to pull RT tasks from balance_rt()
i.e when previous task is a RT task and it is de-activated.
To cover more cases, attempt this from newly idle load balance
which gets called before a CPU goes to idle.
Change-Id: Id3c9a68c19d2b5709f9953ac2383c14f5134b7de
Signed-off-by: Pavankumar Kondeti <pkondeti@codeaurora.org>
Add more debugging information to walt_newidle_balance trace point
to better understand newidle balance behavior.
Change-Id: I946d1d89644cc4fffb026e00686eedf805b06c01
Signed-off-by: Pavankumar Kondeti <pkondeti@codeaurora.org>
SCHED_BUG_ON is called from update_task_rq_cpu_cycles() when
current time is less than the previous recorded time for the
task. SCHED_BUG_ON isn't providing the precise details of
the time delta.
Improve the debugging such that the previous time, current, time
pid, and CPU are all printed before the scheduler dump.
Change-Id: I2156310be7db792e5579f8f56a0b653ed1ace225
Signed-off-by: Stephen Dickey <dickey@codeaurora.org>
It is possible that a cpu paused with thermal might be
unpaused by core-ctl, as well as the reverse. This means
that a cpu that has been paused for thermal reasons
can be unpaused when there is need.
Ensure that cpus paused by more than one entity are sticky,
such that only the last unpause request will actually
perform the operation.
Change-Id: Ieb2f836af7348d48a43ae1b22e11d97046ef9dea
Signed-off-by: Stephen Dickey <dickey@codeaurora.org>
Signed-off-by: Sai Harshini Nimmala <snimmala@codeaurora.org>
The first CPU that supports AArch32 in EL0 on asymmetric AArch32
systems can't be paused. Track such CPUs in core_ctl to not to
request pause_cpus().
Change-Id: I4a9132884c3e322e9a6bd7d1f06a82bd9113eec8
Signed-off-by: Pavankumar Kondeti <pkondeti@codeaurora.org>
When a CPU is not in idle state, the idle exit latency should be 0.
However walt_get_idle_exit_latency() returns UINT_MAX. Fix this.
Without this, the idle CPU that is serving an interrupt will not
win the battle against a CPU that is in a deep c-state.
Change-Id: Ic9ca36add5cd97cc7174c8e25b269ae40aadd451
Signed-off-by: Pavankumar Kondeti <pkondeti@codeaurora.org>
The cum_window_demand tracking is not supported in WALT. Remove the
left over code and use the PELT signal based util_avg as replacement
in RT task placement. The CFS task placement is already using the
same.
Change-Id: I078353ed33729b7bf4eab324c7ec861982a3bc57
Signed-off-by: Pavankumar Kondeti <pkondeti@codeaurora.org>
The tick path load balance is meant for fair tasks. Add a check
for the same before considering the task for migration. The
fair_policy() check alone does not work as it returns true
for idle task and also for RT boosted tasks. Add a wrapper
based on priority check.
Change-Id: I05cc996cb15051e4f6bb1a73005234e4d5188d19
Signed-off-by: Pavankumar Kondeti <pkondeti@codeaurora.org>
WALT core module register for android_rvh_update_misfit_status
restricted vendor hook. Remove the duplicate registration from
WALT CFS module. The later registered function has a compilation
issue when FAIR_GROUP_SCHED is enabled.
Change-Id: I2d89efa92ca704941f12263ee4994dc389202d5f
Signed-off-by: Pavankumar Kondeti <pkondeti@codeaurora.org>
The balance_rt vendor hook is implemented to pull a runnable RT
task from the other CPU.
Change-Id: I288fc9a8858cb29ae9c6993b398a349b4a8c7475
Signed-off-by: Pavankumar Kondeti <pkondeti@codeaurora.org>
Signed-off-by: Sai Harshini Nimmala <snimmala@codeaurora.org>
FIFO/RT tasks are supposed to run for short amount of time. So
when a CPU is busy with 1 RT and 1 CFS task, the other idle CPU
won't offer help thinking the CFS task will run immediately.
However, results indicate that realaxing this policy and pulling
the CFS task helps in reducing the runnables. This policy is
limited to the same capacity cluster load balance as we want
strict prefer idle behavior for it.
Change-Id: I4df6e00233503e096b1bb87fd17a784ea53fd9be
Signed-off-by: Pavankumar Kondeti <pkondeti@codeaurora.org>
The utilization return by cpu_util() is bounded by the original
capacity of the CPU. Load balancer uses the utilization to
break the ties. Use unbounded cpu utilization so that the most
loaded CPU gets help.
Change-Id: Id16cea205e6eb07a42a1e86fda533cd825df6908
Signed-off-by: Pavankumar Kondeti <pkondeti@codeaurora.org>
Active migration of running tasks is allowed only onto idle CPUs.
Add a check for it in walt_lb_find_busiest_lower_cap_cpu().
Change-Id: I195aadc8b5822d4f231edd2f57d9df5ef1a6e350
Signed-off-by: Pavankumar Kondeti <pkondeti@codeaurora.org>
Currently we are using task_fits_capacity() for misfit task
accounting. This does not consider boost scenarios. We also
end up accounting 100% tasks running on the highest capacity
CPU as misfit. A task on the highest capacity CPU should never
be treated as misfit.
rq->misfit_task_load should use task_util() not task_load(). The
later is task demand scaled to sched_ravg_window in nano seconds.
Change-Id: I1f923e33530d2d04d7bfca9dd42920f799794864
Signed-off-by: Pavankumar Kondeti <pkondeti@codeaurora.org>
walt_lb_find_busiest_lower_cap_cpu() is meant to be called when
the destination CPU has more capacity than the source CPU group.
However, the condition for calling this function is inverted.
Change-Id: I03dc62f42795479d21f676f8e27459955229dc82
Signed-off-by: Pavankumar Kondeti <pkondeti@codeaurora.org>
Implement select_task_rq_rt trace hook to be able to find suitable
CPU for the RT tasks from wake-up path.
Change-Id: Id1e8a4529d7391bf39a2a069ae37c146fb8a55c9
Signed-off-by: Satya Durga Srinivasu Prabhala <satyap@codeaurora.org>
Signed-off-by: Sai Harshini Nimmala <snimmala@codeaurora.org>
Currently there is no trace support for indicating
whether an application is 32 or 64 bit compatible.
Since SOCs going forward will have a mix of support
for 32 bit vs 64 bit applications, this is needed
information.
Update the trace_sched_enq_deq output such that the
task's compatibility (32 or 64) is printed.
Change-Id: I780b6b290a77a8405dd268b45a1e7affd035036b
Signed-off-by: Stephen Dickey <dickey@codeaurora.org>
The core kernel side has race conditions which lead to memory faults.
Disable this feature temporarily.
Change-Id: I0013753d571f1e7265ce385a1130f2671c574ab0
Signed-off-by: Shaleen Agrawal <shalagra@codeaurora.org>
The clean up including removing unused variables and functions,
converting global variables to static.
Change-Id: Ib5126ac9df701fdde354e0e5d759f0177db5a28b
Signed-off-by: Pavankumar Kondeti <pkondeti@codeaurora.org>
sched-walt-debug module does not get compiled unless SCHED_WALT
config is turned on. However the config corresponding to the
debug module shows in menuconfig even without enabling SCHED_WALT.
Fix this by specifying the dependency in the Kconfig.
Change-Id: Ie466c0ca11bbcdec43af2624c3631ddff5c77024
Signed-off-by: Pavankumar Kondeti <pkondeti@codeaurora.org>
The walt_find_best_target() has a special case for the active task
migration attempt from tick path. It tries hard to find an idle
CPU by disabling most_spare_cap_cpu selection. However this is
all unnecessary if we can check for the idle CPU condition at
the caller site. This helps the overall execution time of wfbt().
Change-Id: I89de44824dca8729b8c62898f9e75dc8e0e1c544
Signed-off-by: Pavankumar Kondeti <pkondeti@codeaurora.org>
walt_find_best_target() which selects the best CPU has a special
handling for running tasks. The idle CPUs are only considered as
migrating a running task to a busy CPU will result in wait time.
A running task is detected via task->state == TASK_RUNNING check,
which is true for tick path active migration tasks. It is also
true for newly created tasks. These tasks are getting placed on
idle CPUs of the higher capacity clusters though the lower capacity
CPUs are lightly loaded. Detect the running task via task->on_rq to
fix this issue.
Change-Id: I154cd6c6f934061526bce30fe596d9ad36c9e07f
Signed-off-by: Pavankumar Kondeti <pkondeti@codeaurora.org>
core_ctl is part of WALT. There is no separate config for
core_ctl. So it makes sense to declare all the WALT related
public interfaces in a single header file.
Change-Id: I0ae5585c0037295eee4268705f0e553d14c60fd9
Signed-off-by: Pavankumar Kondeti <pkondeti@codeaurora.org>
Signed-off-by: Sai Harshini Nimmala <snimmala@codeaurora.org>
wake_up_idle related functions are meant for other modules
usage. Make them public by moving to appropriate header.
Change-Id: I888b511e656d6c160c98b5da44bc873573a9beed
Signed-off-by: Pavankumar Kondeti <pkondeti@codeaurora.org>
The walt_task_struct structure is used by WALT module. WALT
type cast the task_struct's android vendor data to
walt_task_struct and use it for the scheduler needs. However
there can be other users who wants to cache information per task.
Currently walt_task_struct is defined in WALT's private header and
can't be included by other modules. Move the walt_task_struct and
dependent structures to appropriate header file so that is becomes
public.
This movement also allows wake_up_idle related funcitons to define
in the public header and keep the inline functionality instead of
exporting them for the other modules usage.
Change-Id: If8c94c7d6cd709bbc56aa08eb964d135a492a530
Signed-off-by: Pavankumar Kondeti <pkondeti@codeaurora.org>
set_task_boost() API is meant for other modules, so export
this function. Also move the function declaration to
appropriate header.
Change-Id: I7213b137a0763140c631bea30653321674fef272
Signed-off-by: Pavankumar Kondeti <pkondeti@codeaurora.org>
Re-introduce the earlier sched_load_boost feature as
waltgov's boost feature. A tunable called "boost" is
exposed per cpufreq policy which can be used to boost
(either positive or negative) the utilization, thus
the selected frequency.
Change-Id: I3a41a985d449e5c753b5afcbc375749e67036089
Signed-off-by: Pavankumar Kondeti <pkondeti@codeaurora.org>
The sched_load_boost feature will be introduced in a different
form in the subsequent patch.
Change-Id: Ib528ccd0fcb78f85685399264cb58212752ca26b
Signed-off-by: Pavankumar Kondeti <pkondeti@codeaurora.org>
cpu_frequency trace events are coming twice when a frequency
switch happens. This is due to commit 08d8c65e84 ("cpufreq:
Move traces and update to policy->cur to cpufreq core") in the
upstream which takes care of cpu_frequency trace event and
updates to policy->cur when cpufreq_driver_fast_switch() is
called.
So remove the redundant code from waltgov cpufreq governor to
avoid duplicate cpu_frequency trave events.
Change-Id: I490e766abcc09ed2a580e5f09930a54d76367999
Signed-off-by: Pavankumar Kondeti <pkondeti@codeaurora.org>
Schedutil flags are no longer used in WALT as we moved to
waltgov cpufreq governor. So remove them.
Change-Id: Iedc68215ed0e994b8e051b282a575ac699ebf7e0
Signed-off-by: Pavankumar Kondeti <pkondeti@codeaurora.org>
The core_ctl has to know how many CPUs are active in each cluster.
If the current need is less than the number of active CPUs, it tries
to pause the CPUs provided tunables are in agreement. On the other
hand, if the current need is more than the number of active CPUs,
it tries to resume the CPUs, provided those CPUs are paused by us
in the first place.
To determine the number of active CPUs, we should consider only
active CPUs. The others could be in pause or hotplugged out. Since
we only use cpu_active_mask for both, we don't need wrappers to
tell this to us. All we need is to maintain how many CPUs that
are going to be paused/resumed in the current iteration and check
against the need/limits.
Change-Id: I0a29da869b52eff35ff1dfa3094c51d6ece4e49f
Signed-off-by: Pavankumar Kondeti <pkondeti@codeaurora.org>
