Merge remote-tracking branch 'quic/tmp-5204e04' into msm-pineapple
* quic/tmp-5204e04:
sched: improve the scheduler
sched: walt: handle support for single and two cluster system
sched/walt: check for -1 for first_idle cpu
sched/walt: Remove avg_cap check in packing CPU feature
sched/walt: Add walt_disabled checks
sched/walt: Fix excess detach_task in walt_lb_pull_tasks
sched/walt: Check reserved CPU during walt_newidle_balance
treewide: Fixup copyright attribution
Revert "sched/walt: print all history samples"
Revert "sched: Use bitshift over division"
sched: walt: handle CTS failure for sched_set/getaffinity
sched: walt: add nested spinlock api
sched/walt: Ensure driving cpu is used for region3
sched: walt: Drop double rq locks for migrations
sched/walt: handle misaligned task window start
sched/walt: reject queue_wakelist() is cpu is same as waker cpu
sched/walt: Improve the scheduler
sched/walt/halt: track halt reasons
sched/walt: properly account irq entry/exit
sched/walt: Prevent within cluster load balance of misfits
sched/walt: Cleanup CPUFREQ flags and reasons
sched/walt: Add thermal pressure to cpu_util TP
sched/walt: Add global_ws to walt dump
sched: walt: allow silver to help prime during newidle balance
sched/walt cfs: rt: find a packing cpu
sched/walt: add functionality for choosing a packing cpu
sched/walt/avg: cache window based statistics
sched/walt: Kick newidle balance after unhalting
Revert "sched: walt: Remove finding historical runtimes in pred"
sched/walt: window rollover missing fix
sched/walt/halt: prevent selection of non-32 bit cpu
sched/walt: Ensure one 32bit capable cpu is never halted
sched/walt: print global ws in UTRA traces
sched/walt: remove set_window_start
sched/walt: prevent main walt_irq_work execution for no notif
sched/walt: add wrq->latest_clock to the walt dump
sched/walt: use rq->clock_task from scheduler tick hook
sched/walt: Improve the scheduler
sched/walt: use num_mvp_tasks for check if rq have mvp task
sched/walt: don't requeue mvp task if only have one mvp task on rq
sched/walt: add num_mvp_tasks for get the number of mvp task in walt rq
sched/walt: get rcu lock before use task's cgroups
sched/walt: check current task's mvp_list instead of mvp prio
sched/walt: skip long running RT task detection for stopper tasks
sched: Re-fix lock_cpus mask setting during IC migrations
sched: Improve the Scheduler
sched: Use bitshift over division
sched/walt: remove duplicate definitions of rt_task_arrival_time
sched/walt: check for incorrect cpu assignment
sched/walt: fix deadlock due to recursive rq locking
sched/walt: fix incorrect migrations for RT task
sched/walt: streamline affinity mask
sched: Revert "sched/walt: Account for changing task affinity"
sched/walt: Fix lock_cpus mask setting during IC migrations
sched/walt/halt: ra-arrange the code
sched/walt: Add yield case to LRRT task detection
sched/walt/core_control: disable need_all_cpus()
sched/walt: Add parameter for CPUFREQ reasons
sched: Align window start with tick boundary
sched/walt: Account for changing task affinity
sched/walt/rt: limit long running RT task detection to FIFO
sched/walt: print all history samples
sched/walt: stop accounting stopper threads as long running RT
sched/walt: Improve comments
sched/walt: Optimize update_task_ravg() calls
sched/walt/core_ctl: do core control immediately
sched/walt/halt: use kthread to launch cpu drain operations
sched: walt: fix stale window_start for walt load
sched: walt: properly account for migrating mvp task
sched/walt: Fix in long running RT task detection
sched/walt: Disable long running RT task detection
sched/walt: Remove unused sysctl node
sched/walt: cleanup copyright issues
sched/walt: walt hooks skipped during rt throttling
sched/walt: kick unhalted cpu for load-balancing
sched/walt: Fix setting RTG Boost Freq
sched: walt: add trace in update_cpu_capacity
sched: Fix negative capacity issue under thermal conditions
sched/walt/halt: prevent migration of migration_disabled tasks
sched/walt: cleanup hooks for halt
sched: walt: Use pred_demand_scaled for all predictive ops
sched: walt: Introduce bucket bitmask
sched: walt: Remove finding historical runtimes in pred
sched: walt: Use rq_clock instead of sched_ktime_get
sched/walt: avoid halted cpus in get_nohz_timer_target
sched: walt: use the is_cpu_allowed tracehook to restrict cpus
Revert "sched/walt_cfs: temporarily disable MVP feature"
sched/walt_cfs: set_next_entity hierarchy
sched/walt: do not perform unnecessary migrations
sched/walt: Deprecate RT task throttling
sched/walt: Handle long running RT tasks
sched/walt: rename to sched_get_cpu_util_pct
sched/walt: guarantee result from energy efficient cpu
sched: walt: take runnable into account for placement descision
sched/walt: Fix in walt_lb_tick
sched/walt_cfs: temporarily disable MVP feature
sched: walt: update the copyright
sched/walt: catch any set_task_cpu to unaffined cpus
sched/walt: allow checking of last halt time against threshold
sched/walt: create conditions for walt-bug and console output
sched/walt: Compare cluster IDs instead of capacities
sched: walt: fix rounding off error
sched/walt: update select_task_rq_rt to filter halted cpus
sched/walt: Deal with mvp_list at the very initial stage
sched: Simplify sliding window logic
sched/walt: remove unused can_halt cpu mask
sched/walt: cleanup cpu_active and cpu_online usage
sched/walt: catch tasks running on disallowed cpus
sched/walt/halt: cleaup issues post-review
sched/walt: Increase the number of history samples
sched: Remove references to min/max capacity cpus
sched/walt: make cmt aware of boosted/colocated task
sched/walt: move trace_sched_compute_energy tracepoint
sched/walt: kick Gold if farthest cluster needs help
sched/walt: avoid halted cpus in walt_find_energy_efficient_cpu
sched/walt: avoid halted cpus for rt in find_lowest_rq
sched/walt: limit the cpu selected by rto_next_cpu
sched/walt: limited the dest_cpu with set_cpus_allowed_ptr_locked
sched/walt: do not allow migration to halted CPU
sched/walt/rt: check halted flag for rt task wakeups
sched/walt: avoid halted cpus for prev cpu fastpath
sched/walt: avoid halted cpus in newidle balance
sched/walt/core_ctl: minimum changes for halt
sched/walt/halt: update tracepoints for halt/start
sched/walt/pause: remove pause functionality from build
sched/walt/halt: walt halt functionality
sched/walt: walt irq work reduce locking
sched/walt: remove task from mvp list if task in yield
sched/walt: Fix frequent frequency updates
sched/walt: Remove unsused function
sched/walt: Restore obj file in Makefile
sched/walt: Improve the scheduler
sched: Improve the scheduler
sched: walt: Improve the scheduler
sched: walt: Fix stale walt CPU reservation flag
sched: walt: Fix bad casting in trace
sched: Improve the Scheduler
walt_lb: Read push_task under lock
sched/walt: Remove duplicate get_task_struct
sched/walt: improve preemptirq template based tracepoints
sched/walt/walt_rt: Improve the scheduler
include/linux/cpu: Remove get_online_cpus and put_online_cpus wrapper
sched/walt: Remove vendor hook for resume_cpus
Revert "sched:walt: register tracehooks for race with pause"
sched/walt: Improve the scheduler
sched/walt: Clear scale_frequency_data during init
pause: port the users of pause
sched/walt: complain if num clusters is high
sched/walt: Change rq lock variable as per upstream
sched/walt: Remove redefinition of 'task_of' function
sched_walt: Change from MAX_USER_RT_PRIO to MAX_RT_PRIO
sched/walt: Change task state variable
sched/walt: Improve the scheduler
sched/walt: Improve the scheduler
sched/walt: Remove newline in traces
sched: walt: Create pipeline "low latency" tasks
sched/walt: fix target load crossover
sched: walt: Add target load tunables
sched/walt: Fix caching freq earlier than down rate limit check
sched/walt: improve task_util trace and variable names
sched: walt: Don't crash unless crash mode enabled
sched: Improve the scheduler
sched/walt: Improve the scheduler
sched: walt: Imrpove the scheduler
sched: Improve the Scheduler
sched/walt: Improve the scheduler
sched/walt: Improve the scheduler
sched/walt: Improve the scheduler
sched: walt: Improve the scheduler
sched: walt: Improve the Scheduler
sched/walt: Improve the scheduler
sched: walt: Do dequeue task accounting after dequeue
sched: walt: Use CRA for sum util
sched/walt: eval_need code and trace cleanup
sched/walt: Improve the scheduler
sched/walt: update the value of active_cpus
sched: walt: Increase nr_threshold to 40 percent
sched: walt: add sched_switch_ctrs_cfg event
sched: walt: Account rt scale to cpu_capacity
sched/walt: Improve the scheduler
sched/walt: Improve the scheduler
sched/walt: Improve the scheduler
sched/walt: Improve the scheduler
sched/walt: Check if necessary for sched_boost modify
sched/walt: Improve the scheduler
sched/walt: Improve the scheduler
sched/walt: Improve the scheduler
sched/walt/walt_cfs: Improve the scheduler
sched/walt: reduce severity of walt pause warning
sched/walt: reduce severity of core_ctl warning
sched/walt: convert existing walt debug to use macro
sched/walt: sysctl node to allow capture of walt issues
sched/walt: when idle task is scheduled cra must be zero
sched/walt: check next task state
Sched/walt/walt_cfs: Improve the scheduler
sched: walt: Register restricted trace hook for libunity fixup
sched/walt: test to ensure dequeue cpu same as enqueue cpu
sched/walt/walt_rt: Improve the scheduler
sched/walt: choose best cpu based on spare cap
sched/walt: skip finding max when energy evaluating
sched/walt: skip enery evaluation for need_idle when idle CPU is available
sched/walt/core_ctl: use printk_deferred() in case of errors
sched/walt: allow reference to walt debug capabilities
sched: walt: Use unsigned long type for counter values
sched/walt_rt: Honor sync wakeups
sched: walt: use scale_cpu for energy evals
sched/walt: correct return value for walt_pause/resume_cpus()
sched/walt: Improve the scheduler
sched: walt: Fix the logic for detecting enq/deq
sched: Use IS_ENABLED macro to check if config is enabled
sched: suppress silver region frequency boost
sched: walt: Add AMU based instructions and cycle counts
sched:walt: register restricted tracehooks for race with pause
sched: walt: Ensure only valid cpu selected
sched: walt: Add cgroup_attach trace point
Revert "sched:walt: register tracehooks for race with pause"
sched/walt: prevent lockup when system is suspending
sched:walt: register tracehooks for race with pause
sched/walt: Fix scheduling while atomic
sched: Enable walt debug module in first stage
sched: walt: Fix max_util calculation
sched/walt: print per cluster energy
sched/walt: consolidate multiple rcu unlock statements
sched: walt: Setup complex sibling hints at init
sched/walt: Improve the scheduler
sched/walt: use just 3% energy criteria
sched/walt: Improve the scheduler
sched/walt: initiate initialization after topology
sched/walt: Improve the scheduler
sched/walt: handle overlap between silver and gold
sched/walt: use prs for energy evaluation
sched/walt: skip energy for uninvolved cluster
sched/walt: create walt friendly energy apis
sched/walt: Track best target cpu for each cluster
sched: walt: Move perf counter trace to dynamic trace point
sched: walt: support dynamic trace event registration
sched: Disable LPM if CONSERVATIVE_BOOST is enabled
sched: walt: Fix waltgov_next_freq format specifiers
sched/walt: improve debug of strange rq situations in walt
sched/walt: add sched_busy_hyst_time trace event
sched: Add util_busy_hist feature
sched: Add util busy hist sysctl tunables
sched/walt: improve logging in the error paths
sched/walt: Do the rq clock update before MVP tasks update
sched/walt: Fix locking issue in MVP tasks update
sched/walt: Improve the scheduler
sched: walt: Adjust nomenclature
sched: walt: expand waltgov_util_update trace point
sched/fair: Add sched_sync_hint_enable node
sched/walt: fix lock ordering issue between RTG lock and task's RQ lock
sched/walt: Improve the scheduler
sched/walt: fixup_busy_time: Correct typo in lockdep_assert_held
sched/walt: Improve the scheduler
sched/walt: Fix newline issues
sched/walt: Improve the scheduler
walt: honor affinity in placement path
sched/walt: Add MVP tasks feature
sched/walt: cleanup references to pr_emerg
sched: Improve the scheduler
sched/walt: Add small task filtering to uclamp
sched/walt: Cache the affinity of a user space task
sched/walt: enhanced debug for double enqueue/dequeue
sched/walt: debug for fixup_cumulative_runnable_avg
sched/walt: Improve the scheduler
sched/walt: improved debug for tracking accounting cases
sched/walt: Apply binder vendor hook performance tune
sched/walt: Improve the scheduler
sched/walt: Initialize task groups from vendor hook
sched: walt: Cleanup fbt_env boosted member
sched: walt: Improve FEEC energy variable nomenclature
sched: walt: Shift boost status of current task
sched: walt: Move spare_cap calculation
sched: walt: remove extra function call
sched/walt: add rt_time to dump_throttled_rt_tasks
sched/walt: null ptr dereference fix in walt_lb_active_migration
sched/walt: core_ctl init out of order
sched/walt: Read sysctl_sched_ravg_window_nr_ticks under mutex
sched/walt: Remove unused sched_dynamic_ravg_window_enable tunable
sched/walt: sched_task_util trace point updates
sched/walt: Modify sched_task_util tracepoint
sched/walt: update check during debug module init
sched/walt: Adjust code as per new dequeue_task() hook invocation
sched/walt: Update CPU capacity based on cpufreq constraints
thermal/pause: thermal driver not tracking pause state
walt/core_ctl: handle error conditions
walt/pause: switch over to mutex and protect call to pause/resume
sched/pause: handle cpu online processing for pause
sched: walt: Improve the scheduler
sched/walt: Remove incorrect TODO/FIXME comments
sched/walt: Track wake_q length using newer parameter
sched: walt: Improve the Scheduler
sched: walt: Move task_boost_type declarations
sched/walt: Improve the chance of RT task pull
sched/walt: Improve walt_newidle_balance trace point
sched/walt: improve debug for time_delta < 0
sched/pause: support refcounting on pause cpus operation
sched/walt: Track CPUs that can't be paused
sched/walt: Fix !idle case in walt_get_idle_exit_latency
sched/walt: Remove left over references to cum_window_demand
sched/walt: Add fair task check in tick path load balance
sched/walt: Remove duplicate restricted vendor hook registration
sched/walt: Implement balance_rt vendor hook
binder: update low_latency selection for binder transactions
sched/walt: Use nr_running > 1 condition for same cluster lb
sched/walt: Use unbounded cpu util in load balancer
sched/walt: Don't migrate running task to a busy CPU
sched/walt: Fix misfit accounting issues
sched/walt: Fix walt_lb_find_busiest_lower_cap_cpu() condition
sched/walt: implement select_task_rq_rt trace hook
sched/trace: support compat in enq/deq trace
sched/walt: Temporarily disable sibling_count_hint
sched/walt: Clean up the code
sched/walt: Make debug module depend on SCHED_WALT
sched/walt: Remove the active task special case from wfbt()
sched/walt: Fix waking new tasks on higher capacity CPUs
sched/walt: Move public declarations of core_ctl.h to walt.h
sched/walt: remove unused functions and variables
sched/walt: Make wake_up_idle related functions public
sched/walt: Make walt_task_struct struct public
sched/walt: Export set_task_boost() API
sched/walt: Introduce waltgov boost feature
sched/walt: Remove sched_load_boost related code
sched/walt: Update cpufreq_driver_fast_switch() invocation
sched/walt: Remove references to schedutil flags
sched/walt: Simplify core_ctl active cpu count management
sched/walt: Fix lockdep warning
sched/walt: Remove left over code related to kernel command line param
sched/walt: Implement sched_lpm_disallowed_time() API
sched: walt: Cleanup core_ctl
sched/walt: Remove should_we_pause() function
sched/walt: Remove evaluation from the core_ctl thread context
sched/wakeup: Optimize core_ctl thread wakeup
sched/walt: Remove unused sched_pause trace point
sched/walt: Add window rollover trace event
sched/walt: Synchronize per-task sysctl and sysctl_task_read_pid
sched/walt: Improve the scheduler
sched/walt: Add a build time test for vendor data size check
sched: walt: Fixup root domain selection
sched/walt: Remove obsolete TODO comments
sched/walt: wait for cpufreq policies
sched/walt: Improve the scheduler
sched/walt: Improve the scheduler
sched/walt: Improve the scheduler
sched: Improve the scheduler
sched/walt: Restrict new idle load balance based on avg_idle
sched/walt: Introduce WALT as a module
Change-Id: Iad463c5cdd9384cc09f68b47fdadbc5e2a8143ac
Signed-off-by: Sai Harshini Nimmala <quic_snimmala@quicinc.com>
This change is for general scheduler improvement.
Change-Id: I98af51b7efd49aeec5a7b405625a1db12c654cd5
Signed-off-by: Shaleen Agrawal <quic_shalagra@quicinc.com>
Update walt_newidle load balance to support single and two
cluster topology.
Change-Id: Ic2bbbe7a273267c1a7aa340f9908b5e5325d21ba
Signed-off-by: Ashay Jaiswal <quic_ashayj@quicinc.com>
There is a typo in the current code it is checking a value
of 1 for unavailability of idle kick cpu. It should check for
-1 instead.
Fix it.
Change-Id: I8796b4c6315d157834ecc8dc2263bcfb7f50e582
Signed-off-by: Abhijeet Dharmapurikar <quic_adharmap@quicinc.com>
The current scheduler code uses avg_cap that is calculated by waltgov.
This is stored in a private structure wg_policy in waltgov.
The problem with this is wg_policy could be freed when the governor
changes or all the CPUs in the policy get offlined. This leads to a
use-after-free access of wg_policy->avg_cap. Besides, WALT introduced
checks between avg_cap and cur_freq to identify if scheduler guided
frequency is overridden by CPU frequency min/max limits. However, avg_cap
doesn't reflect the scheduler guided frequency and instead represents
the frequency from min/max limits. IOW, cur_freq will always be
equal to avg_cap when CPU frequency limits are active, rendering the
check useless and thus allowing packing.
Remove the code that retrieves and checks avg_cap.
Change-Id: I9b32c6798fb209d63d688ed89dc17674a7859475
Signed-off-by: Sai Harshini Nimmala <quic_snimmala@quicinc.com>
Entry into WALT code via tracehooks should be guarded based on the
completion of WALT initialization.
Add check for the same in functions where it is missing.
Change-Id: I2a04003977832087be681108c15bd6bbb24fc7e8
Signed-off-by: Sai Harshini Nimmala <quic_snimmala@quicinc.com>
There's three loops to find a task to pull.
In the first iteration, we look for a task that is not running,
which fits a strict criteria.
In the second iteration, we look for a task that is not running,
which fits a lax criteria.
If we have still not found a valid task to pull, we intended to pull the
running task servicing an active load migration.
However, there is some leftover code, wherein a task that is not running
will end up being detached and pulled. This was not intended, and leads
to incorrect behavior. Essentially, this is negating the effects of the
first two iterations.
Change-Id: Ia225d89043c76ec6e661771a39d058498346920d
Signed-off-by: Shaleen Agrawal <quic_shalagra@quicinc.com>
Currently, it is observed that multiple calls to active_load_balance can
take place.
For example, say two Big threads are running on Golds, and
become misfits at the same time. Then, Prime which is currently say
halted, will end up pulling both threads. This is because first, as
Prime is unhalted, the CPU is kicked to enable it to pull tasks.
However, while Prime is kicked, it goes back to idle briefly, causing
the upstream newidle_balance path to come into play.
As a consequence, both threads are queued on Prime, and eventually, the
one sitting in the runqueue is put back on a Gold. This behavior is
unoptimal and unnecessary.
Ensure that walt_newidle_balance only takes when the CPU is not marked
as reserved.
Change-Id: I00feb8df3c4433333a356f2e855fb7a3ee848789
Signed-off-by: Shaleen Agrawal <quic_shalagra@quicinc.com>
During CTS testing, the user-space code can get an affinity
that includes a halted cpu, and immediately use that to set
the affinity. Currently sched_setaffinity will cause an error
if a halted cpu is selected.
Port the change that was in use prior to the introduction of
pause, correcting the CTS test case.
Change-Id: I9bfc819e4a6904494c199931e76f2702b7990758
Signed-off-by: Stephen Dickey <quic_dickey@quicinc.com>
Use nested spin lock api to avoid deadlock warning
Change-Id: Ia2aed8f025bb209876ff87ba5a9e32b2b80718d8
Signed-off-by: Ashay Jaiswal <quic_ashayj@quicinc.com>
Signed-off-by: Sai Harshini Nimmala <quic_snimmala@quicinc.com>
Currently, wg_cpu is used, however, this is incorrect, as the driving
CPU should be used when evaluating region3 conditions.
Change-Id: I4f86ec6021b718e35d84f589dc9be68a53ab9523
Signed-off-by: Shaleen Agrawal <quic_shalagra@quicinc.com>
Currently, WALT holds rq lock for both the cpus when handling task
migration. Upstream code however (PELT) holds rq locks consecutively for
migrations. This results in cumbersome tricks in WALT to hold them at
the same time - rq locks need to be acquired in order of cpu numbers.
Besides that, holding two rq locks for every migration results in
contentions in busy situations.
Split the migration accounting in two steps - subtracting time from src
cpu and adding time to the dest cpu. Subtract when we know the task is
moving i.e. the migration hook and add on the dest cpu during subsequent
enqueue.
Upstream code guarantees the rq lock for src cpu is held during
migration hook for queued or running task. For a waking task grab the
src cpu lock. No need to grab rq lock for dest cpu as enqueue hook is
always run with that lock held.
The basic idea of walt migration accounting, non group tasks move their
contributions only for inter cluster migration while group tasks do that
for every migration, stays.
Change-Id: I5d319e26bcbc088448954fd83bbcd1981ee7fb5c
Signed-off-by: Shaleen Agrawal <quic_shalagra@quicinc.com>
Signed-off-by: Sai Harshini Nimmala <quic_snimmala@quicinc.com>
Introduce awareness for task window start and identify if task
window_start doesn't match that of the CPUs just before accounting.
Change-Id: I58d806e2a75bc647dc28b1f95d00352d29c5c447
Signed-off-by: Shaleen Agrawal <quic_shalagra@quicinc.com>
The upstream code has a WARN when wakelist is being enforced on the same
cpu where the waker runs. Although WALT tries its best to skip the
current cpu during many_wakeups situation, it may have to put the task
on it during highload/offline or 32 bit situations.
Simply disable queue_wakelist() if we wakeup the task on the waker cpu.
Change-Id: Ie13f1156bf0389570dbe0fbac70e271db2b9b211
Signed-off-by: Abhijeet Dharmapurikar <quic_adharmap@quicinc.com>
Signed-off-by: Shaleen Agrawal <quic_shalagra@quicinc.com>
This change is for general scheduler improvement.
Change-Id: I2b95dc15c0dbe88966c90acfa4dca6aebf1a3f95
Signed-off-by: Shaleen Agrawal <quic_shalagra@quicinc.com>
Currently, there are no protections under conditions where CPUs may be
halted or unhalted twice, which could lead to undesired behavior. Track
reasons for refcounts per CPU to ensure proper accounting.
Change-Id: Ic85a13f3536e3287fc76a02004d2b4601ada7a1a
Signed-off-by: Shaleen Agrawal <quic_shalagra@quicinc.com>
Due to restructuring of upstream logic, it is necessary to properly
deduce start and end times of irqs to maintain correct accounting.
Change-Id: I8f0ff9b0f9137283315e5b67cdf2cfb1dede033f
Signed-off-by: Abhijeet Dharmapurikar <quic_adharmap@quicinc.com>
Signed-off-by: Shaleen Agrawal <quic_shalagra@quicinc.com>
A misfit task can be pulled to a CPU within the same cluster in the
newly-idle balance path.
Since that just increases migration count and does not show any
significant benefits, prevent pulling of a task if it is going to be a
misfit on the destination CPU. Only pull if there are many such tasks and
the source CPU needs help from a newly idle CPU.
Change-Id: Ief55e3916d5902e473c0eaa101e8aefa01c04398
Signed-off-by: Sai Harshini Nimmala <quic_snimmala@quicinc.com>
Currently, the flags and reasons are defined based off left shifts,
which requires manual translation when printed in traces. Change those
definitions to be directly in hex format, maintaining congruence.
Additionally, in the event that a boost_util is 0, ensure that a reason
is not inappropriately assigned.
Change-Id: Ia8e9c1628373f3355a3fcd9707a5950a9757e42b
Signed-off-by: Shaleen Agrawal <quic_shalagra@quicinc.com>
In the event that thermal throttling is under way, and user has not
enabled update_cpu_capacity trace point, add a backup way of quickly
determining scheduler under thermal constraints by displaying the
thermal pressure in the sched_cpu_util trace point.
Change-Id: I36b8f23e37b4e9b42770e1535feabb9d513fb730
Signed-off-by: Shaleen Agrawal <quic_shalagra@quicinc.com>
This change is for general scheduler improvement.
Change-Id: I88cf8678b918aed530f3af122edea2bca5b60503
Signed-off-by: Shaleen Agrawal <quic_shalagra@quicinc.com>
Present newidle balance design doesn't allow farthest cluster to help
during newidle balance (silver's newidle balance cannot help prime and
prime cannot help silver).
For Example:
If silver is entering idle and it find's prime core busy then newidle
balance will try to find and kick an idle gold core but it will not
participate in pulling task from prime to silver.
Update the logic to allow silver cluster to help and pull task during
newidle balance if none of the cores in the nearest cluster (Gold here)
are idle.
Change-Id: I0908058d55f344467f559b50209a6bf8bb18b0ef
Signed-off-by: Ashay Jaiswal <quic_ashayj@quicinc.com>
Under low-util conditions, small tasks can be packed onto
a single dedicated cpu per cluster, the first cpu in the
cluster. This can improve performance by ensuring fewer
exits from wfi or deep-idle, and reduce power by keeping
more of the other cpus in wfi or deep-idle, for longer.
Add the functionality to perform choosing of the packing cpu
and incorporate this into the select_rq path for fair and rt
in walt hooks.
Change-Id: I02675ceb6dd2e5fb40f4b14308159cc1fdde245e
Signed-off-by: Stephen Dickey <quic_dickey@quicinc.com>
Very small tasks can end up spread across all cpus
in a cluster, and cause excessive wfi and lpm exit
latencies. This can be reduced significantly by combining
these tasks on to a single cpu.
Create the functionality to be used in cfs/rt that can
make the decision as to whether a packing cpu should
be chosen. Add supporting sysctl nodes to tune the
thresholds for this.
Change-Id: I68723b8dc461efffeba3069bfd5d41724f071bd2
Signed-off-by: Stephen Dickey <quic_dickey@quicinc.com>
Previously it was possible to update core control with new
statistics and those statistics would be unavailable to
any other decisions that need to be made.
Update sched_avg to cache the statistics locally and simply
pass a reference to that information back to core control.
Provide a cluster util function to provide window based load.
Create a sysctl node and threshold to use to compare against
cluster util.
Change-Id: I8cc40603e49e29e7f8a343f6ff667ce9abe09cb5
Signed-off-by: Stephen Dickey <quic_dickey@quicinc.com>
When a cpu is unhalted currently, it will attempt to perform
a balance through the normal ipi kick interface. However, this
can fail because a balance might have happened in the current
4mS tick, while the cpu was still halted. The next kick as a
result of unhalting, fails to execute (min 1 tick between kicks).
Instead, use walt_newidle_balance() to ensure that this cpu can
take a task. Perform an smp async function call to the target
cpu, invoke newidle balance, and allow the cpu to pull tasks
immediately after unhalting.
Change-Id: Ide1bb51f80dc8c4eba079588ffb6a7ed341e8628
Signed-off-by: Stephen Dickey <quic_dickey@quicinc.com>
Currently, the code has a flow where a window rollover is not issued
during migration. This could lead to a situation where no future window
rollovers are handled.
For example in a 4 + 3 + 1 topology, A B C indicate the window boundaries
while X indicates the rq->window_start of each cpu and "GLB" indicates
the timestamp of the global window tracker (walt_irq_work_lastq_ws)
1.
A B C
CPU0----X-----------------------------------------------------------------
CPU1----X-----------------------------------------------------------------
CPU2----X-----------------------------------------------------------------
CPU3----X-----------------------------------------------------------------
CPU4----X-----------------------------------------------------------------
CPU5----X-----------------------------------------------------------------
CPU6----X-----------------------------------------------------------------
CPU7----X-----------------------------------------------------------------
GLB X
Now CPU1 gets an event(e) right before C, in which case it advances his
rq1->window_start to B, also advances GLB to B and issues a rollover irq
work, which will advance window_starts for all other cpus.
2.
A B C
CPU0----X-----------------------------------------------------------------
CPU1----------------------------X---------------------e-------------------
CPU2----X-----------------------------------------------------------------
CPU3----X-----------------------------------------------------------------
CPU4----X-----------------------------------------------------------------
CPU5----X-----------------------------------------------------------------
CPU6----X-----------------------------------------------------------------
CPU7----X-----------------------------------------------------------------
GLB X
Since the event was very close to C, the irq_work(i) could run right after
C, which advances all the window_starts to C, including that of cpu1.
3.
A B C
CPU0----------------------------------------------------X-----------------
CPU1--------------------------------------------------e-X-i---------------
CPU2----------------------------------------------------X-----------------
CPU3----------------------------------------------------X-----------------
CPU4----------------------------------------------------X-----------------
CPU5----------------------------------------------------X-----------------
CPU6----------------------------------------------------X-----------------
CPU7----------------------------------------------------X-----------------
GLB X
This advancement of window_start on cpu1 causes it to advance G (and
reissue another irq_work, which is not relevant to this discussion) like
below. CPU1 will advance the global because it matches its window_start
(atomic_cmpxchg)
4.
A B C
CPU0----------------------------------------------------X-----------------
CPU1--------------------------------------------------e-X-i---------------
CPU2----------------------------------------------------X-----------------
CPU3----------------------------------------------------X-----------------
CPU4----------------------------------------------------X-----------------
CPU5----------------------------------------------------X-----------------
CPU6----------------------------------------------------X-----------------
CPU7----------------------------------------------------X-----------------
GLB X
However we could run in to a situation where the irq work (i) is held up
because cpu0's rq lock isn't available (BTW a window rollover holds all the
rqlocks) for some other reason. And while the irq work is held there,
the rq locks of cpus1-7 are available, an inter cluster migration between
cpu1 and cpu7 could happen causing cpu1,7 to advance their window_start.
Redrawing the 3rd picture
3a.
A B C
CPU0----X-----------------------------------------------------------------
CPU1--------------------------------------------------e-X---m-----i-------
CPU2----X-----------------------------------------------------------------
CPU3----X-----------------------------------------------------------------
CPU4----X-----------------------------------------------------------------
CPU5----X-----------------------------------------------------------------
CPU6----X-----------------------------------------------------------------
CPU7----------------------------------------------------X---m-------------
GLB X
Eventually when the irq work gets around to hold the locks on all the cpus,
it will move the rest of the cpu's window_start to C. However the Global is
left at B and cpu1 that should have advanced the Global, won't do it
because of an unchecked advancement by the migration with cpu7. After this
point the global is old and will never match any cpu's window_start and
WALT will never do a window_rollover
4a.
A B C
CPU0----------------------------------------------------X-----------------
CPU1--------------------------------------------------e-X---m-----i-------
CPU2----------------------------------------------------X-----------------
CPU3----------------------------------------------------X-----------------
CPU4----------------------------------------------------X-----------------
CPU5----------------------------------------------------X-----------------
CPU6----------------------------------------------------X-----------------
CPU7----------------------------------------------------X---m-------------
GLB X
Fix this by issuing migration walt_irq_work when an intercluster
migration ends up rolling over the window in the dest rq.
Change-Id: Ib3985a1cdf72342305398a9c648159195ca99658
Signed-off-by: Shaleen Agrawal <quic_shalagra@quicinc.com>
When __migrate_task() is given a new cpu to run a 32bit
task, it may be migrating to a halted cpu. is_cpu_allowed
tracehook currently rejects halted cpus, forcing
__migrate_task() to leave the task on its current cpu.
The current cpu may not be 32 bit capable.
If the halted state of the task is changed just prior
to the migrate operation, the migration is rejected and
the prev_cpu is chosen. If the prev_cpu is not 32 bit
capable, then the task will be migrated to an improper
cpu. This can only happen while a 32bit task is being
execv'd.
Detect the situation where a halted cpu is about to be
rejected. If the task is in execve, and it's a 32 bit
task, allow it to run on a halted cpu.
Change-Id: I0a0ca79efeab08aed8136f92cc86dcc6b4179aab
Signed-off-by: Stephen Dickey <quic_dickey@quicinc.com>
It is possible that all 32bit cpus are halted. In this case
a 32 bit task can be woken, and the scheduler will not find
a cpu for it to run on that is 32 bit capable.
Prevent the least-significant 32bit capable cpu from ever
being halted.
Change-Id: I23f616a5988d7a7eaa6b0d7dd7ef82dbe6f5ec9b
Signed-off-by: Stephen Dickey <quic_dickey@quicinc.com>
Add a parameter in UTRA and UTRA_mini traces to help identify the global
window_start.
Change-Id: I1eb7432f47fe2f622ba1efc78047551b97f08952
Signed-off-by: Shaleen Agrawal <quic_shalagra@quicinc.com>
set_window_start is a deprecated function that is never used. Remove it.
Change-Id: I636a2a37e55c4d812606ea1f52c4ce80b70d44d2
Signed-off-by: Shaleen Agrawal <quic_shalagra@quicinc.com>
If there are no notifications pending for performing migration
work, there is no migration work to be done because the work
must have already been handled.
Optimize the walt_irq_work execution for migrations. If no
cpus got locked, then no notifications were handled, and
the irq work function should exit.
Change-Id: I57730503ebb8f440a288666c79c4cb4f31ed1023
Signed-off-by: Stephen Dickey <quic_dickey@quicinc.com>
Tracking latest_clock from walt run queue structure.
Change-Id: I50322b4efc91edbeaf61b72dd031e2a7e96edf28
Signed-off-by: Stephen Dickey <quic_dickey@quicinc.com>
Since we are called from the main tick, rq clock task must
have been updated very recently. Use it directly, instead of
update_rq_clock_task() to avoid warnings.
Change-Id: I8ea644cb28844b9ded6e45703aa72e72c21486f5
Signed-off-by: Stephen Dickey <quic_dickey@quicinc.com>
This change is for general scheduler improvements.
Change-Id: I611eb9aa071e4fac1fad962fb98139f9aff79dcd
Signed-off-by: Satya Durga Srinivasu Prabhala <quic_satyap@quicinc.com>
Signed-off-by: Sai Harshini Nimmala <quic_snimmala@quicinc.com>
There is mvp task running on rq if num_mvp_task not equal 0,
so use num_mvp_tasks check if rq have mvp task.
Change-Id: Ia653b8d1dbbaa642ccc06296338617252bf1225b
Signed-off-by: Tengfei Fan <quic_tengfan@quicinc.com>
Don't requeue mvp task if only have one mvp task on rq, because
only have one mvp task on rq, current mvp task will be select for
running if this mvp task runtime still less than limit.
Change-Id: Idaf343e7471665c01fe711e86a3d7747e501ed76
Signed-off-by: Tengfei Fan <quic_tengfan@quicinc.com>
Add num_mvp_tasks for get the number of mvp task in walt rq.
Change-Id: I4367723cd61f7ee5fd63c2afcfefd6eb3258c118
Signed-off-by: Tengfei Fan <quic_tengfan@quicinc.com>
uclamp_latency_sensitive() function will use struct
task_struct's member of cgroups, but rcu lock should
be got before use it.
Change-Id: I1988ed7fe836f9f1ba99d59c5d46f26f3418b51e
Signed-off-by: Tengfei Fan <quic_tengfan@quicinc.com>
Current task's mvp prio get from walt_get_mvp_task_prio will still be
there even if the task is not in the mvp list. For example task removed
from mvp_list due to runtime slice limition.
Change-Id: Iae5d6e420213111acd0b0e71095e0a6caffbe0aa
Signed-off-by: Tengfei Fan <quic_tengfan@quicinc.com>
As stopper task's scheduling policy is set to FIFO and task may run
long time depends on the number of tasks to be migrated etc., current
long running RT task detection feature crashes system due to task's
arrivial time stamp is taken when stopper task scheduled in during
CPU hot-plug out operation and used it later when CPU gets hot-plugged
in back.
Fix this false positive issue by skipping update of arrivial time stamp
for stopper tasks.
Fixes: b429bee50fe2c8f ("sched/walt/rt: limit long running RT task detection to FIFO")
Change-Id: I714494fee49b6e8e5d50885bd47b38ad0d535579
Signed-off-by: Satya Durga Srinivasu Prabhala <quic_satyap@quicinc.com>
In an earlier fix, we intended to clear lock CPUs with a
given cluster if and only all CPUs belonging to the
cluster did not have a notification pending.
However, the fix was incomplete, as we were continuing to do the
opposite, resulting in undesired behavior.
Fixes: 95d58bdfe8d ("sched/walt: Fix lock_cpus mask setting during IC migrations")
Change-Id: Ia3c0d886a2a27b8f4cf8d6f60673aeb7213d5fb9
Signed-off-by: Shaleen Agrawal <quic_shalagra@quicinc.com>
This change is for general scheduler improvement.
Change-Id: Iebc41134b83b1c83d02f433d92ec2b44ed972511
Signed-off-by: Shaleen Agrawal <quic_shalagra@quicinc.com>
Now that our window history size is a power of 2, we can take
advantage of this to replace division operations in a critical code
path by using bit-shifts instead.
Change-Id: Idc0ed37a87814783fcab2f4d5204b79fb9ce9996
Signed-off-by: Shaleen Agrawal <quic_shalagra@quicinc.com>
rt_task_arrival_time is being defined in a header as well as in
a source file which is incorrect, fix it by changing definition
to declaration in header.
While at it, clean-up related code and make sure long running RT
detection happens for FIFO sched class only.
Change-Id: Ia9f227ad3af0059d354eb0e4df3ab719c1348ac5
Signed-off-by: Satya Durga Srinivasu Prabhala <quic_satyap@quicinc.com>
When a task is in execve, it can be assigned to any cpu regardless
of its status as a 32bit task. However, once execve is done, the
task needs to be assigned to a 32 bit cpu.
Capture the case where a 32 bit task, not in execve, is assigned
to an incorrect cpu.
Change-Id: I7880521716f6ed16d81aad13e42c0ad4374bab15
Signed-off-by: Stephen Dickey <quic_dickey@quicinc.com>
