Remove member no longer used by the scheduler core.
Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@igalia.com>
Cc: Lucas Stach <l.stach@pengutronix.de>
Cc: Russell King <linux+etnaviv@armlinux.org.uk>
Cc: Christian Gmeiner <christian.gmeiner@gmail.com>
Cc: etnaviv@lists.freedesktop.org
Reviewed-by: Christian Gmeiner <cgmeiner@igalia.com>
Signed-off-by: Philipp Stanner <phasta@kernel.org>
Link: https://patch.msgid.link/20260417103744.76020-20-tvrtko.ursulin@igalia.com
Remove member no longer used by the scheduler core.
Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@igalia.com>
Cc: Rob Herring (Arm) <robh@kernel.org>
Cc: Tomeu Vizoso <tomeu@tomeuvizoso.net>
Cc: Oded Gabbay <ogabbay@kernel.org>
Acked-by: Rob Herring (Arm) <robh@kernel.org>
Signed-off-by: Philipp Stanner <phasta@kernel.org>
Link: https://patch.msgid.link/20260417103744.76020-18-tvrtko.ursulin@igalia.com
Remove member no longer used by the scheduler core.
Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@igalia.com>
Cc: Min Ma <mamin506@gmail.com>
Cc: Lizhi Hou <lizhi.hou@amd.com>
Cc: Oded Gabbay <ogabbay@kernel.org>
Reviewed-by: Lizhi Hou <lizhi.hou@amd.com>
Signed-off-by: Philipp Stanner <phasta@kernel.org>
Link: https://patch.msgid.link/20260417103744.76020-16-tvrtko.ursulin@igalia.com
Now that the run queue to scheduler relationship is always 1:1 we can
embed it (the run queue) directly in the scheduler struct and save on
some allocation error handling code and such.
Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@igalia.com>
Cc: Christian König <christian.koenig@amd.com>
Cc: Danilo Krummrich <dakr@kernel.org>
Cc: Matthew Brost <matthew.brost@intel.com>
Cc: Philipp Stanner <phasta@kernel.org>
Acked-by: Danilo Krummrich <dakr@kernel.org>
Tested-by: Vitaly Prosyak <vitaly.prosyak@amd.com>
Signed-off-by: Philipp Stanner <phasta@kernel.org>
Link: https://patch.msgid.link/20260417103744.76020-15-tvrtko.ursulin@igalia.com
Since the new FAIR policy is in general better than FIFO and almost as
good as round-robin in interactive use cases, plus the latter has not been
the default policy in a long time, we can afford to remove both and leave
just FAIR.
By doing so we can simplify the scheduler code by making the scheduler to
run queue relationship always 1:1 and remove some code.
Also, now that the FIFO policy is gone the tree of entities is not a FIFO
tree any more so rename it to just the tree.
Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@igalia.com>
Cc: Christian König <christian.koenig@amd.com>
Cc: Danilo Krummrich <dakr@kernel.org>
Cc: Matthew Brost <matthew.brost@intel.com>
Cc: Philipp Stanner <phasta@kernel.org>
Acked-by: Danilo Krummrich <dakr@kernel.org>
Tested-by: Vitaly Prosyak <vitaly.prosyak@amd.com>
Signed-off-by: Philipp Stanner <phasta@kernel.org>
Link: https://patch.msgid.link/20260417103744.76020-14-tvrtko.ursulin@igalia.com
FAIR policy works better than FIFO for all known use cases and either
matches or gets close to RR. Lets make it a default to improve the user
experience especially with interactive workloads competing with heavy
clients.
Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@igalia.com>
Cc: Christian König <christian.koenig@amd.com>
Cc: Danilo Krummrich <dakr@kernel.org>
Cc: Matthew Brost <matthew.brost@intel.com>
Cc: Philipp Stanner <phasta@kernel.org>
Cc: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Acked-by: Danilo Krummrich <dakr@kernel.org>
Tested-by: Vitaly Prosyak <vitaly.prosyak@amd.com>
Signed-off-by: Philipp Stanner <phasta@kernel.org>
Link: https://patch.msgid.link/20260417103744.76020-13-tvrtko.ursulin@igalia.com
GPUs do not always implement preemption and DRM scheduler definitely
does not support it at the front end scheduling level. This means
execution quanta can be quite long and is controlled by userspace,
consequence of which is picking the "wrong" entity to run can have a
larger negative effect than it would have with a virtual runtime based CPU
scheduler.
Another important consideration is that rendering clients often have
shallow submission queues, meaning they will be entering and exiting the
scheduler's runnable queue often.
Relevant scenario here is what happens when an entity re-joins the
runnable queue with other entities already present. One cornerstone of the
virtual runtime algorithm is to let it re-join at the head and rely on the
virtual runtime accounting and timeslicing to sort it out.
However, as explained above, this may not work perfectly in the GPU world.
Entity could always get to overtake the existing entities, or not,
depending on the submission order and rbtree equal key insertion
behaviour.
Allow interactive jobs to overtake entities already queued up for the
limited case when interactive entity is re-joining the queue after
being idle.
This gives more opportunity for the compositors to have their rendering
executed before the GPU hogs even if they have been configured with the
same scheduling priority.
To classify a client as interactive we look at its average job duration
versus the average for the whole scheduler. We can track this easily by
plugging into the existing job runtime tracking and applying the
exponential moving average window on the past submissions. Then, all other
things being equal, we let the more interactive jobs go first.
Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@igalia.com>
Cc: Christian König <christian.koenig@amd.com>
Cc: Danilo Krummrich <dakr@kernel.org>
Cc: Matthew Brost <matthew.brost@intel.com>
Cc: Philipp Stanner <phasta@kernel.org>
Cc: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Acked-by: Danilo Krummrich <dakr@kernel.org>
Tested-by: Vitaly Prosyak <vitaly.prosyak@amd.com>
Signed-off-by: Philipp Stanner <phasta@kernel.org>
Link: https://patch.msgid.link/20260417103744.76020-12-tvrtko.ursulin@igalia.com
The FAIR scheduling policy is built upon the same concepts as the well
known CFS CPU scheduler - entity run queue is sorted by the virtual GPU
time consumed by entities in a way that the entity with least vruntime
runs first.
It is able to avoid total priority starvation, which is one of the
problems with FIFO, and it also does not need for per priority run queues.
As it scales the actual GPU runtime by an exponential factor as the
priority decreases, the virtual runtime for low priority entities grows
faster than for normal priority, pushing them further down the runqueue
order for the same real GPU time spent.
Apart from this fundamental fairness, fair policy is especially strong in
oversubscription workloads where it is able to give more GPU time to short
and bursty workloads when they are running in parallel with GPU heavy
clients submitting deep job queues.
Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@igalia.com>
Cc: Christian König <christian.koenig@amd.com>
Cc: Danilo Krummrich <dakr@kernel.org>
Cc: Matthew Brost <matthew.brost@intel.com>
Cc: Philipp Stanner <phasta@kernel.org>
Cc: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Acked-by: Danilo Krummrich <dakr@kernel.org>
Tested-by: Vitaly Prosyak <vitaly.prosyak@amd.com>
Signed-off-by: Philipp Stanner <phasta@kernel.org>
Link: https://patch.msgid.link/20260417103744.76020-11-tvrtko.ursulin@igalia.com
There is no need to keep entities with no jobs in the tree so lets remove
it once the last job is consumed. This keeps the tree smaller which is
nicer and more efficient as entities are removed and re-added on every
popped job.
Apart from that, the upcoming fair scheduling algorithm will rely on the
tree only containing runnable entities.
Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@igalia.com>
Cc: Christian König <christian.koenig@amd.com>
Cc: Danilo Krummrich <dakr@kernel.org>
Cc: Matthew Brost <matthew.brost@intel.com>
Cc: Philipp Stanner <phasta@kernel.org>
Acked-by: Danilo Krummrich <dakr@kernel.org>
Tested-by: Vitaly Prosyak <vitaly.prosyak@amd.com>
Signed-off-by: Philipp Stanner <phasta@kernel.org>
Link: https://patch.msgid.link/20260417103744.76020-10-tvrtko.ursulin@igalia.com
To implement fair scheduling we need a view into the GPU time consumed by
entities. Problem we have is that jobs and entities objects have decoupled
lifetimes, where at the point we have a view into accurate GPU time, we
cannot link back to the entity any longer.
Solve this by adding a light weight entity stats object which is reference
counted by both entity and the job and hence can safely be used from
either side.
With that, the only other thing we need is to add a helper for adding the
job's GPU time into the respective entity stats object, and call it once
the accurate GPU time has been calculated.
The most convenient place to do that is the free job worker for several
reasons. Doing the accounting from the job completion callback would mean
a few locks would need to become irq safe and we would also need to worry
about out of order completions (via dma_fence_is_signaled calls which we
cannot control). In-order completions are critical for GPU time accuracy
which is currently adjusted per fence in the free worker and requires
looking at the next job in the scheduler pending list. We would also need
to add a new lock to protect the scheduler average stats update.
In contrast to those complications, having the accounting done from the
free worker is serialized by definition and all the above complications
are avoided. Downside is there is potential for a time lag between job
completions and GPU time being accounted against the entity. Since that is
partly alleviated by batch processing the completed job queue, and the
scheduling algorithm does not attempt to be completely fair, which would
even be rather impossible to achieve in the GPU world with the current
DRM scheduler design and hardware with no or poor preemption support,
this downside is not considered critical. Plus, in practice the scheduler
is also affected by worker scheduling delays from other angles too. Not
least being able to promptly feed the GPU with new work.
We therefore choose the simple option and can later consider improving
upon it if the need arises.
Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@igalia.com>
Cc: Christian König <christian.koenig@amd.com>
Cc: Danilo Krummrich <dakr@kernel.org>
Cc: Matthew Brost <matthew.brost@intel.com>
Cc: Philipp Stanner <phasta@kernel.org>
Acked-by: Danilo Krummrich <dakr@kernel.org>
Tested-by: Vitaly Prosyak <vitaly.prosyak@amd.com>
Signed-off-by: Philipp Stanner <phasta@kernel.org>
Link: https://patch.msgid.link/20260417103744.76020-9-tvrtko.ursulin@igalia.com
To implement fair scheduling we will need as accurate as possible view
into per entity GPU time utilisation. Because sched fence execution time
are only adjusted for accuracy in the free worker we need to process
completed jobs as soon as possible so the metric is most up to date when
view from the submission side of things.
Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@igalia.com>
Cc: Christian König <christian.koenig@amd.com>
Cc: Danilo Krummrich <dakr@kernel.org>
Cc: Matthew Brost <matthew.brost@intel.com>
Cc: Philipp Stanner <phasta@kernel.org>
Reviewed-by: Matthew Brost <matthew.brost@intel.com>
Acked-by: Danilo Krummrich <dakr@kernel.org>
Tested-by: Vitaly Prosyak <vitaly.prosyak@amd.com>
Signed-off-by: Philipp Stanner <phasta@kernel.org>
Link: https://patch.msgid.link/20260417103744.76020-8-tvrtko.ursulin@igalia.com
Round-robin being the non-default policy and unclear how much it is used,
we can notice that it can be implemented using the FIFO data structures if
we only invent a fake submit timestamp which is monotonically increasing
inside drm_sched_rq instances.
So instead of remembering which was the last entity the scheduler worker
picked we can simply bump the picked one to the bottom of the tree, which
ensures round-robin behaviour between all active queued jobs.
If the picked job was the last from a given entity, we remember the
assigned fake timestamp and use it to re-insert the job once it re-joins
the queue. This ensures the job neither overtakes all already queued jobs,
neither it goes last. Instead it keeps the position after the currently
queued jobs and before the ones which haven't yet been queued at the point
the entity left the queue.
Advantage is that we can consolidate to a single code path and remove a
bunch of code. Downside is round-robin mode now needs to lock on the job
pop path but that should not have a measurable performance impact.
Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@igalia.com>
Cc: Christian König <christian.koenig@amd.com>
Cc: Danilo Krummrich <dakr@kernel.org>
Cc: Matthew Brost <matthew.brost@intel.com>
Cc: Philipp Stanner <phasta@kernel.org>
Acked-by: Danilo Krummrich <dakr@kernel.org>
Tested-by: Vitaly Prosyak <vitaly.prosyak@amd.com>
Signed-off-by: Philipp Stanner <phasta@kernel.org>
Link: https://patch.msgid.link/20260417103744.76020-7-tvrtko.ursulin@igalia.com
This time round we explore the rate of submitted job queue processing
with multiple identical parallel clients.
Example test output:
3 clients:
t cycle: min avg max : ...
+ 0ms 0 0 0 : 0 0 0
+ 102ms 2 2 2 : 2 2 2
+ 208ms 5 6 6 : 6 5 5
+ 310ms 8 9 9 : 9 9 8
...
+ 2616ms 82 83 83 : 83 83 82
+ 2717ms 83 83 83 : 83 83 83
avg_max_min_delta(x100)=60
Every 100ms for the duration of the test it logs how many jobs each
client had completed, prefixed by minimum, average and maximum numbers.
When finished overall average delta between max and min is output as a
rough indicator to scheduling fairness.
Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@igalia.com>
Cc: Christian König <christian.koenig@amd.com>
Cc: Danilo Krummrich <dakr@kernel.org>
Cc: Matthew Brost <matthew.brost@intel.com>
Cc: Philipp Stanner <phasta@kernel.org>
Cc: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Acked-by: Christian König <christian.koenig@amd.com>
Acked-by: Danilo Krummrich <dakr@kernel.org>
Tested-by: Vitaly Prosyak <vitaly.prosyak@amd.com>
Signed-off-by: Philipp Stanner <phasta@kernel.org>
Link: https://patch.msgid.link/20260417103744.76020-6-tvrtko.ursulin@igalia.com
To make evaluating different scheduling policies easier (no need for
external benchmarks) and perfectly repeatable, lets add some synthetic
workloads built upon mock scheduler unit test infrastructure.
Focus is on two parallel clients (two threads) submitting different job
patterns and logging their progress and some overall metrics. This is
repeated for both scheduler credit limit 1 and 2.
Example test output:
Normal and low:
pct1 cps1 qd1; pct2 cps2 qd2
+ 0ms: 0 0 0; 0 0 0
+ 104ms: 100 1240 112; 100 1240 125
+ 209ms: 100 0 99; 100 0 125
+ 313ms: 100 0 86; 100 0 125
+ 419ms: 100 0 73; 100 0 125
+ 524ms: 100 0 60; 100 0 125
+ 628ms: 100 0 47; 100 0 125
+ 731ms: 100 0 34; 100 0 125
+ 836ms: 100 0 21; 100 0 125
+ 939ms: 100 0 8; 100 0 125
+ 1043ms: ; 100 0 120
+ 1147ms: ; 100 0 107
+ 1252ms: ; 100 0 94
+ 1355ms: ; 100 0 81
+ 1459ms: ; 100 0 68
+ 1563ms: ; 100 0 55
+ 1667ms: ; 100 0 42
+ 1771ms: ; 100 0 29
+ 1875ms: ; 100 0 16
+ 1979ms: ; 100 0 3
0: prio=normal sync=0 elapsed_ms=1015ms (ideal_ms=1000ms) cycle_time(min,avg,max)=134,222,978 us latency_time(min,avg,max)=134,222,978
us
1: prio=low sync=0 elapsed_ms=2009ms (ideal_ms=1000ms) cycle_time(min,avg,max)=134,215,806 us latency_time(min,avg,max)=134,215,806 us
There we have two clients represented in the two respective columns, with
their progress logged roughly every 100 milliseconds. The metrics are:
- pct - Percentage progress of the job submit part
- cps - Cycles per second
- qd - Queue depth - number of submitted unfinished jobs
The cycles per second metric is inherent to the fact that workload
patterns are a data driven cycling sequence of:
- Submit 1..N jobs
- Wait for Nth job to finish (optional)
- Sleep (optional)
- Repeat from start
In this particular example we have a normal priority and a low priority
client both spamming the scheduler with 8ms jobs with no sync and no
sleeping. Hence they build very deep queues and we can see how the low
priority client is completely starved until the normal finishes.
Note that the PCT and CPS metrics are irrelevant for "unsync" clients
since they manage to complete all of their cycles instantaneously.
A different example would be:
Heavy and interactive:
pct1 cps1 qd1; pct2 cps2 qd2
+ 0ms: 0 0 0; 0 0 0
+ 106ms: 5 40 3; 5 40 0
+ 209ms: 9 40 0; 9 40 0
+ 314ms: 14 50 3; 14 50 0
+ 417ms: 18 40 0; 18 40 0
+ 522ms: 23 50 3; 23 50 0
+ 625ms: 27 40 0; 27 40 1
+ 729ms: 32 50 0; 32 50 0
+ 833ms: 36 40 1; 36 40 0
+ 937ms: 40 40 0; 40 40 0
+ 1041ms: 45 50 0; 45 50 0
+ 1146ms: 49 40 1; 49 40 1
+ 1249ms: 54 50 0; 54 50 0
+ 1353ms: 58 40 1; 58 40 0
+ 1457ms: 62 40 0; 62 40 1
+ 1561ms: 67 50 0; 67 50 0
+ 1665ms: 71 40 1; 71 40 0
+ 1772ms: 76 50 0; 76 50 0
+ 1877ms: 80 40 1; 80 40 0
+ 1981ms: 84 40 0; 84 40 0
+ 2085ms: 89 50 0; 89 50 0
+ 2189ms: 93 40 1; 93 40 0
+ 2293ms: 97 40 0; 97 40 1
In this case client one is submitting 3x 2.5ms jobs, waiting for the 3rd
and then sleeping for 2.5ms (in effect causing 75% GPU load, minus the
overheads). Second client is submitting 1ms jobs, waiting for each to
finish and sleeping for 9ms (effective 10% GPU load). Here we can see
the PCT and CPS reflecting real progress.
Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@igalia.com>
Cc: Christian König <christian.koenig@amd.com>
Cc: Danilo Krummrich <dakr@kernel.org>
Cc: Matthew Brost <matthew.brost@intel.com>
Cc: Philipp Stanner <phasta@kernel.org>
Cc: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Acked-by: Christian König <christian.koenig@amd.com>
Acked-by: Danilo Krummrich <dakr@kernel.org>
Tested-by: Vitaly Prosyak <vitaly.prosyak@amd.com>
Signed-off-by: Philipp Stanner <phasta@kernel.org>
Link: https://patch.msgid.link/20260417103744.76020-5-tvrtko.ursulin@igalia.com
Lets move all the code dealing with struct drm_sched_rq into a separate
compilation unit. Advantage being sched_main.c is left with a clearer set
of responsibilities.
Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@igalia.com>
Cc: Christian König <christian.koenig@amd.com>
Cc: Danilo Krummrich <dakr@kernel.org>
Cc: Matthew Brost <matthew.brost@intel.com>
Cc: Philipp Stanner <phasta@kernel.org>
Reviewed-by: Matthew Brost <matthew.brost@intel.com> # v1
Acked-by: Danilo Krummrich <dakr@kernel.org>
Tested-by: Vitaly Prosyak <vitaly.prosyak@amd.com>
Signed-off-by: Philipp Stanner <phasta@kernel.org>
Link: https://patch.msgid.link/20260417103744.76020-4-tvrtko.ursulin@igalia.com
Move the code dealing with entities entering and exiting run queues to
helpers to logically separate it from jobs entering and exiting entities.
Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@igalia.com>
Cc: Christian König <christian.koenig@amd.com>
Cc: Danilo Krummrich <dakr@kernel.org>
Cc: Matthew Brost <matthew.brost@intel.com>
Cc: Philipp Stanner <phasta@kernel.org>
Acked-by: Danilo Krummrich <dakr@kernel.org>
Tested-by: Vitaly Prosyak <vitaly.prosyak@amd.com>
Signed-off-by: Philipp Stanner <phasta@kernel.org>
Link: https://patch.msgid.link/20260417103744.76020-3-tvrtko.ursulin@igalia.com
Since we have removed the case where amdgpu was initializing entitites
with either no schedulers on the list, or with a single NULL scheduler,
and there appears no other drivers which rely on this, we can simplify the
scheduler by explicitly rejecting that early.
Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@igalia.com>
Cc: Christian König <christian.koenig@amd.com>
Cc: Danilo Krummrich <dakr@kernel.org>
Cc: Matthew Brost <matthew.brost@intel.com>
Cc: Philipp Stanner <phasta@kernel.org>
Reviewed-by: Christian König <christian.koenig@amd.com>
Acked-by: Philipp Stanner <phasta@kernel.org>
Tested-by: Vitaly Prosyak <vitaly.prosyak@amd.com>
Signed-off-by: Philipp Stanner <phasta@kernel.org>
Link: https://patch.msgid.link/20260417103744.76020-2-tvrtko.ursulin@igalia.com
nouveau_abi16_chan_fini() does invoke drm_sched_entity_fini() twice:
Once directly, and a second time through nouveau_sched_destroy().
That's likely undesired behavior and might be a bug since
drm_sched_entity_fini() decrements reference counts.
Fix the issue by using the appropriate function,
drm_sched_entity_kill(), to kill all remaining jobs within the entity.
Cc: stable@kernel.vger.org
Fixes: 9a0c32d698 ("drm/nouveau: don't fini scheduler if not initialized")
Suggested-by: Danilo Krummrich <dakr@kernel.org>
Signed-off-by: Philipp Stanner <phasta@kernel.org>
Link: https://patch.msgid.link/20260415144956.272506-3-phasta@kernel.org
Signed-off-by: Danilo Krummrich <dakr@kernel.org>
Some drivers do not care on teardown whether the last jobs pending in an
entity are actually executed before teardown completed. For such
scenarios, drm_sched_entity_flush() is not the ideal function since it's
intended to wait for jobs to complete.
Make drm_sched_entity_kill() public for that use-case and update the
documentation.
Suggested-by: Danilo Krummrich <dakr@kernel.org>
Signed-off-by: Philipp Stanner <phasta@kernel.org>
Link: https://patch.msgid.link/20260415144956.272506-2-phasta@kernel.org
Signed-off-by: Danilo Krummrich <dakr@kernel.org>
Static analysis issue:
In all other uses of the function drm_object_property_get_default_value,
the return value of the function is checked before the output is saved to
the relevant object parameter. Though likely unnecessary given the
execution path involved, keep the behavior consistent across uses and only
set colorop_state->curve_1d_type in __drm_colorop_state_reset if
drm_object_property_get_default_value succeeds.
Signed-off-by: Jonathan Cavitt <jonathan.cavitt@intel.com>
Reviewed-by: Chaitanya Kumar Borah <chaitanya.kumar.borah@intel.com>
Link: https://patch.msgid.link/20260202214709.8037-2-jonathan.cavitt@intel.com
Statis analysis issue:
drm_gpuvm_prepare_range issues an exec_object_prepare call to all
drm_gem_objects mapped between addr and addr + range. However, it is
possible (albeit very unlikely) that the objects found through
drm_gpuvm_for_each_va_range (as connected to va->gem) are NULL, as seen
in other functions such as drm_gpuva_link and drm_gpuva_unlink_defer.
Do not prepare NULL objects.
Fixes: 50c1a36f59 ("drm/gpuvm: track/lock/validate external/evicted objects")
Signed-off-by: Jonathan Cavitt <jonathan.cavitt@intel.com>
Cc: Matthew Brost <matthew.brost@intel.com>
Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Reviewed-by: Krzysztof Karas <krzysztof.karas@intel.com>
Link: https://patch.msgid.link/20260130191953.61718-2-jonathan.cavitt@intel.com
In amdxdna_iommu_alloc(), if iommu_map() fails after successfully
allocating both iova and cpu_addr, the code jumps to free_iova
which only frees the iova, leaking the allocated pages.
Fixes: ece3e89809 ("accel/amdxdna: Allow forcing IOVA-based DMA via module parameter")
Signed-off-by: Felix Gu <ustc.gu@gmail.com>
Reviewed-by: Lizhi Hou <lizhi.hou@amd.com>
Signed-off-by: Lizhi Hou <lizhi.hou@amd.com>
Link: https://patch.msgid.link/20260416-amdxdna-v1-1-30c13008365c@gmail.com
Add support for configuring the hardware scheduler time quantum to
improve fairness across concurrent contexts.
The scheduler enforces a fixed time slice per context, preventing
long-running workloads from monopolizing the device and allowing
other contexts to make forward progress.
The default time quantum is 30ms and can be configured via the
time_quantum_ms module parameter.
Signed-off-by: Max Zhen <max.zhen@amd.com>
Reviewed-by: Mario Limonciello (AMD) <superm1@kernel.org>
Signed-off-by: Lizhi Hou <lizhi.hou@amd.com>
Link: https://patch.msgid.link/20260415171139.904947-1-lizhi.hou@amd.com
Modernize the open coded test framework by using kunit.
Add a num_online_cpus() check to test_race_signal_callback() as the
default kunit.py runs the VM with a single CPU and this test depends on
two truly parallel kthreads. Skip it instead of hanging.
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Christian König <christian.koenig@amd.com>
Link: https://lore.kernel.org/r/2-v1-0a349a394eff+14110-dmabuf_kunit_jgg@nvidia.com
Modernize the open coded test framework by using kunit.
The kunit tool can be used to build a kernel and run it in a VM with:
$ tools/testing/kunit/kunit.py run --build_dir build_kunit_x86_64 --arch x86_64 --kunitconfig ./drivers/dma-buf/.kunitconfig
Along with the other ways to run kunits.
To make the kunit tool work like this the DMABUF_KUNIT_TEST kconfig must
select DMA_SHARED_BUFFER to get it turned on without building a driver
using it.
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Christian König <christian.koenig@amd.com>
Link: https://lore.kernel.org/r/1-v1-0a349a394eff+14110-dmabuf_kunit_jgg@nvidia.com
of_drm_find_bridge() is deprecated. Move to its replacement
of_drm_find_and_get_bridge() which gets a bridge reference, and ensure it
is put when done.
We need to handle the two cases: when a panel_bridge is added and when it
isn't. So:
* in the 'else' case a panel_bridge is not added and bridge is found: use
of_drm_find_and_get_bridge() to get a reference to the found bridge
* in the 'then' case a panel_bridge is found using a devm function which
already takes a refcount and will put it on removal, but we need to take
another so the following code in this function always get exactly one
reference that it needs to put
In order to put the reference, add the needed drm_bridge_put() calls in the
existing cleanup function.
Reviewed-by: Linus Walleij <linusw@kernel.org>
Reviewed-by: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
Reviewed-by: Louis Chauvet <louis.chauvet@bootlin.com>
Link: https://patch.msgid.link/20260409-drm-bridge-alloc-getput-drm_of_find_bridge-4-v5-2-d7381c07788a@bootlin.com
Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com>
of_drm_find_bridge() is deprecated. Move to its replacement
of_drm_find_and_get_bridge() which gets a bridge reference, and ensure it
is put when done.
This is made somewhat simpler by the fact that 'bridge' is a local
variable.
However we need to handle both branches of the main if().
In the 'else' case, just switch to of_drm_find_and_get_bridge() to ensure
the bridge is not freed while in use in the function tail
(drm_bridge_attach() mainly).
In the 'then' case, devm_drm_panel_bridge_add_typed() already increments
the refcount using devres which ties the bridge allocation lifetime to the
device lifetime, so we would not need to do anything. However to have the
same behaviour in both branches take an additional reference here, so that
the bridge needs to be put whichever branch is taken without more
complicated logic.
Reviewed-by: Biju Das <biju.das.jz@bp.renesas.com>
Tested-by: Biju Das <biju.das.jz@bp.renesas.com>
Reviewed-by: Linus Walleij <linusw@kernel.org>
Reviewed-by: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
Reviewed-by: Louis Chauvet <louis.chauvet@bootlin.com>
Link: https://patch.msgid.link/20260409-drm-bridge-alloc-getput-drm_of_find_bridge-4-v5-1-d7381c07788a@bootlin.com
Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com>
Take the encoder chain mutex while iterating over the encoder chain in
drm_atomic_bridge_chain_pre_enable() and
drm_atomic_bridge_chain_post_disable() to ensure the lists won't change
while being inspected.
These functions have nested list_for_each_*() loops, which makes them
complicated. list_for_each_entry_from() loops could be replaced by
drm_for_each_bridge_in_chain_from(), but it would not work in a nested way
in its current implementation. Besides, there is no "_reverse" variant of
drm_for_each_bridge_in_chain_from().
Keep code simple and readable by explicitly locking around the outer
loop. Thankfully there are no break or return points inside the loops, so
the change is trivial and readable.
Reviewed-by: Maxime Ripard <mripard@kernel.org>
Reviewed-by: Louis Chauvet <louis.chauvet@bootlin.com>
Link: https://patch.msgid.link/20260324-drm-bridge-alloc-encoder-chain-mutex-v5-7-8bf786c5c7e6@bootlin.com
Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com>
These loops in drm_bridge.c iterate over the encoder chain using
list_for_each_entry_reverse(), which does not prevent changes to the bridge
chain while iterating over it.
Take the encoder chain mutex while iterating to avoid chain changes while
iterating.
All the "simple" loops are converted. drm_atomic_bridge_chain_pre_enable()
and drm_atomic_bridge_chain_post_disable() are handled by a separate
commit.
Reviewed-by: Louis Chauvet <louis.chauvet@bootlin.com>
Link: https://patch.msgid.link/20260324-drm-bridge-alloc-encoder-chain-mutex-v5-6-8bf786c5c7e6@bootlin.com
Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com>
These loops in drm_bridge.c iterate over the encoder chain using
list_for_each_entry_from(), which does not prevent changes to the bridge
chain while iterating over it.
Convert most of those loops to instead use
drm_for_each_bridge_in_chain_from(), which locks the chain.
This also simplifies code.
All the "simple" loops are converted here. The only ones not touched are
those in drm_atomic_bridge_chain_pre_enable() and
drm_atomic_bridge_chain_post_disable(), because they have nested loops
which are not well handled by drm_for_each_bridge_in_chain_from(). Those
two functions are handled by a separate commit.
Reviewed-by: Louis Chauvet <louis.chauvet@bootlin.com>
Link: https://patch.msgid.link/20260324-drm-bridge-alloc-encoder-chain-mutex-v5-5-8bf786c5c7e6@bootlin.com
Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com>
drm_for_each_bridge_in_chain_scoped() and
drm_for_each_bridge_in_chain_from() currently get/put the bridge at each
iteration. But they don't protect the encoder chain, so it could change
(bridges added/removed) while some code is iterating over the list
itself. Such code can then derail on incorrect pointers.
To make iterations safe, augment these for_each macros to lock the encoder
chain mutex at the beginning and unlock it at the end of the loop (be it at
the end of the list, or earlier due to a 'break' or 'return' statement).
This change requires more operations when starting and ending the loop. To
avoid making the macros even more complex, move these operations to helper
functions. Also remname some of the existing helper functions for
consistency.
Reviewed-by: Louis Chauvet <louis.chauvet@bootlin.com>
Link: https://patch.msgid.link/20260324-drm-bridge-alloc-encoder-chain-mutex-v5-4-8bf786c5c7e6@bootlin.com
Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com>
drm_bridge_attach() modifies the encoder bridge chain, so take a mutex
around such operations to allow users of the chain to protect themselves
from chain modifications while iterating.
Reviewed-by: Maxime Ripard <mripard@kernel.org>
Reviewed-by: Louis Chauvet <louis.chauvet@bootlin.com>
Link: https://patch.msgid.link/20260324-drm-bridge-alloc-encoder-chain-mutex-v5-3-8bf786c5c7e6@bootlin.com
Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com>
drm_encoder_cleanup() modifies the encoder chain by removing bridges via
drm_bridge_detach(). Protect this whole operation by taking the mutex, so
that:
* any users iterating over the chain will not access it during the change
* other code willing to modify the list (drm_bridge_attach()) will wait
until drm_encoder_cleanup() is done
Note that the _safe macro in use here is providing a different and
orthogonal kind of protection than the mutex:
1. list_for_each_entry_safe() allows removing the current entry from the
list it is iterating on, synchronously; the non-safe version would be
unable to find the next entry after the current entry has been removed
2. the mutex being added allows to ensure that the list is not used
asynchronously by other code while it is being modified; this prevents
such other concurrent code to derail because it is iterating over an
element while it is removed
The _safe macro, which works by taking the "next" pointer in addition to
the "current" one, does not even try to provide the protection at item 2
above. This is visible e.g. when the "next" element is removed by other
concurrent code. This is what would happen without the added mutex:
1. start loop: list_for_each_entry_safe(pos, n, ...) sets:
pos = list_first_entry() = (bridge 1)
n = list_next_entry(pos) = (bridge 2)
2. enter the loop 1st time, do something with *pos (bridge 1)
3. in the meanwhile bridge 2 is hot-unplugged
-> another thread removes bridge 2
-> drm_bridge_detach()
-> list_del() sets (bridge 2)->next = LIST_POISON1
4. loop iteration 1 finishes, list_for_each_entry_safe() sets:
pos = n (previously set to bridge 2)
n = (bridge 2)->next = LIST_POISON1
5. enter the loop 2nd time, do something with *pos (bridge 2)
6. loop iteration 2 finishes, list_for_each_entry_safe() sets:
pos = n = LIST_POISON1 ==> bug!
However, simply adding mutex_[un]lock(&encoder->bridge_chain_mutex)
before/after the list_for_each_entry_safe() seems a simple and good
solution, but it is introducing a possible ABBA deadlock (found by
PROVE_LOCKING). The two code paths involved are:
* drm_encoder_cleanup():
- takes the bridge_chain_mutex (A)
- calls drm_bridge_detach -> drm_atomic_private_obj_fini ->
DRM_MODESET_LOCK_ALL_BEGIN() which takes all locks in the
acquisition context (B)
* drm_mode_getconnector() (and other code paths):
- calls drm_helper_probe_single_connector_modes() which:
- takes a drm_modeset_lock in the acquisition context (B)
- calls __drm_helper_update_and_validate ->
drm_bridge_chain_mode_valid -> drm_for_each_bridge_in_chain_from()
which takes the bridge_chain_mutex (A)
To avoid this potential ABBA deadlock, move all list items to a temporary
list while holding the bridge_chain_mutex, then detach all elements from
the temporary list without the mutex.
Reviewed-by: Louis Chauvet <louis.chauvet@bootlin.com>
Link: https://patch.msgid.link/20260324-drm-bridge-alloc-encoder-chain-mutex-v5-2-8bf786c5c7e6@bootlin.com
Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com>
The per-encoder bridge chain is currently assumed to be static once it is
fully initialized. Work is in progress to add hot-pluggable bridges,
breaking that assumption.
With bridge removal, the encoder chain can change without notice, removing
tail bridges. This can be problematic while iterating over the chain.
Add a mutex to be taken whenever looping or changing the encoder chain.
Reviewed-by: Maxime Ripard <mripard@kernel.org>
Reviewed-by: Louis Chauvet <louis.chauvet@bootlin.com>
Link: https://patch.msgid.link/20260324-drm-bridge-alloc-encoder-chain-mutex-v5-1-8bf786c5c7e6@bootlin.com
Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com>
The RZ/G2L hardware manual (Rev. 1.50, May 2025), Section 34.4.2.1,
requires deasserting the CMN_RSTB signal after setting the Link registers.
Move the reset_control_deassert() call from rzg2l_mipi_dsi_dphy_init() to
rzg2l_mipi_dsi_startup(), placing it after the Link register writes. This
reset signal is optional for RZ/V2H SoCs, so add a NULL check. Drop the
unused ret variable from rzg2l_mipi_dsi_dphy_init().
The CMN_RSTB signal is not required for reading PHY registers in the
probe. Move reset_control_assert() from rzg2l_mipi_dsi_dphy_exit() to
rzg2l_mipi_dsi_stop(), placing it before the dphy_exit() call. Since this
reset signal is optional for RZ/V2H, the call is a no-op on that SoC.
Tested-by: Tommaso Merciai <tommaso.merciai.xr@bp.renesas.com>
Reviewed-by: Tommaso Merciai <tommaso.merciai.xr@bp.renesas.com>
Link: https://patch.msgid.link/20260330104450.128512-4-biju.das.jz@bp.renesas.com
Signed-off-by: Biju Das <biju.das.jz@bp.renesas.com>
The RZ/G2L hardware manual (Rev. 1.50, May 2025), Section 34.4.2.1,
requires waiting at least 1 msec after deasserting the CMN_RSTB signal
before the DSI-Tx module is ready. Increase the delay from 1 usec to
1 msec by replacing udelay(1) with fsleep(1000) for RZ/G2L SoCs.
Fixes: 7a043f978e ("drm: rcar-du: Add RZ/G2L DSI driver")
Cc: stable@vger.kernel.org
Reviewed-by: Tommaso Merciai <tommaso.merciai.xr@bp.renesas.com>
Tested-by: Tommaso Merciai <tommaso.merciai.xr@bp.renesas.com>
Link: https://patch.msgid.link/20260330104450.128512-3-biju.das.jz@bp.renesas.com
Signed-off-by: Biju Das <biju.das.jz@bp.renesas.com>
The RZ/G2L hardware manual (Rev. 1.50, May 2025), Section 34.4.2.1,
requires display timings to be set after the HS clock is started. Move
rzg2l_mipi_dsi_set_display_timing() from
rzg2l_mipi_dsi_atomic_pre_enable() to rzg2l_mipi_dsi_atomic_enable(),
placing it after rzg2l_mipi_dsi_start_hs_clock(). Drop the unused ret
variable from rzg2l_mipi_dsi_atomic_pre_enable().
Fixes: 5ce16c169a ("drm: renesas: rz-du: Add atomic_pre_enable")
Fixes: 7a043f978e ("drm: rcar-du: Add RZ/G2L DSI driver")
Cc: stable@vger.kernel.org
Reviewed-by: Tommaso Merciai <tommaso.merciai.xr@bp.renesas.com>
Tested-by: Tommaso Merciai <tommaso.merciai.xr@bp.renesas.com>
Link: https://patch.msgid.link/20260330104450.128512-2-biju.das.jz@bp.renesas.com
Signed-off-by: Biju Das <biju.das.jz@bp.renesas.com>
Adjust struct fatal_error_info to match the expected driver/firmware
interface layout.
The structure is used to retrieve debug information from firmware when
a command becomes stuck on the device. The reserved field currently
uses 128 u32 entries, which makes the structure larger than intended
and causes the layout to no longer match the firmware definition.
Reduce the reserved array size from 128 to 127 entries so the
structure matches the expected interface format.
Fixes: 25854131c0 ("accel/amdxdna: Support retrieving hardware context debug information")
Signed-off-by: Max Zhen <max.zhen@amd.com>
Reviewed-by: Mario Limonciello (AMD) <superm1@kernel.org>
Signed-off-by: Lizhi Hou <lizhi.hou@amd.com>
Link: https://patch.msgid.link/20260414165625.788853-1-lizhi.hou@amd.com
Trace_dma_fence_signaled, trace_dma_fence_wait_end and
trace_dma_fence_destroy can all currently dereference a null fence->ops
pointer after it has been reset on fence signalling.
Lets use the safe string getters for most tracepoints to avoid this class
of a problem, while for the signal tracepoint we move it to before ops are
cleared to avoid losing the driver and timeline name information. Apart
from moving it we also need to add a new tracepoint class to bypass the
safe name getters since the signaled bit is already set.
For dma_fence_init we also need to use the new tracepoint class since the
rcu read lock is not held there, and we can do the same for the enable
signaling since there we are certain the fence cannot be signaled while
we are holding the lock and have even validated the fence->ops.
Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@igalia.com>
Fixes: 541c8f2468 ("dma-buf: detach fence ops on signal v3")
Cc: Christian König <christian.koenig@amd.com>
Cc: Philipp Stanner <phasta@kernel.org>
Cc: Boris Brezillon <boris.brezillon@collabora.com>
Cc: linux-media@vger.kernel.org
Cc: linaro-mm-sig@lists.linaro.org
Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Tvrtko Ursulin <tursulin@ursulin.net>
Link: https://lore.kernel.org/r/20260415083207.40513-2-tvrtko.ursulin@igalia.com
Mailbox message IDs are allocated cyclically. When destroying a mailbox
channel, pending messages are canceled starting from message ID 0. This
results in an incorrect cancellation order when the ID of the last posted
message wraps around and is smaller than the ID of the first posted
message.
Fix this by canceling pending messages starting from the next available
message ID, ensuring the correct ordering across wraparound.
Fixes: a37d78470b ("accel/amdxdna: Replace idr api with xarray")
Reviewed-by: Mario Limonciello (AMD) <superm1@kernel.org>
Signed-off-by: Lizhi Hou <lizhi.hou@amd.com>
Link: https://patch.msgid.link/20260413181843.670796-1-lizhi.hou@amd.com
iommu_map_sgtable() returns negative error codes on failure, but the
result is stored in an unsigned variable. This prevents proper error
detection.
Change the variable type to ssize_t so negative error values can be
handled correctly.
Fixes: ece3e89809 ("accel/amdxdna: Allow forcing IOVA-based DMA via module parameter")
Reported-by: Dan Carpenter <error27@gmail.com>
Closes: https://lore.kernel.org/all/adk7kOUBwIyYnX1M@stanley.mountain/
Reviewed-by: Mario Limonciello (AMD) <superm1@kernel.org>
Signed-off-by: Wendy Liang <wendy.liang@amd.com>
Signed-off-by: Lizhi Hou <lizhi.hou@amd.com>
Link: https://patch.msgid.link/20260413180238.668441-1-lizhi.hou@amd.com
Convert this driver to DRM_BRIDGE_ATTACH_NO_CONNECTOR and to the
drm_bridge_connector framework which is the current DRM bridge best
practice.
Based on the in-tree dts[i] files this introduces no regression. Based on
the kernel doc of drm_bridge_connector.c:
* To make use of this helper, all bridges in the chain shall report bridge
* operation flags (&drm_bridge->ops) and bridge output type
* (&drm_bridge->type), as well as the DRM_BRIDGE_ATTACH_NO_CONNECTOR attach
* flag (none of the bridges shall create a DRM connector directly).
and each of the 3 LCDIF blocks in the i.MX8MP, all of them comply with the
above requirement:
* For the LCDIF3, the pipeline is:
LCDIF3 -> fsl,imx8mp-hdmi-pvi -> fsl,imx8mp-hdmi-tx -> HDMI connector
And the involved bridges are:
* fsl,imx8mp-hdmi-pvi has ops = 0 (it doesn't set it) because it
implements none the optional features mentioned by those flags, and it
honors the DRM_BRIDGE_ATTACH_NO_CONNECTOR by propagating it
* fsl,imx8mp-hdmi-tx is implemented based on dw-hdmi, which sets ops as
appropriate and also propagates the DRM_BRIDGE_ATTACH_NO_CONNECTOR flag
* display-connector (enabled via the DT overlay if needed) sets ops and
makes DRM_BRIDGE_ATTACH_NO_CONNECTOR mandatory
* The LCDIF2 involves the panel-bridge, display-connector and lvds-decoder
(even though only the pane-bridge is currently supported), and all these
three also set ops as needed and propagate
DRM_BRIDGE_ATTACH_NO_CONNECTOR or make it mandatory.
* The LCDIF1 is used with the adv7511, tc358767 and the panel bridge
drivers which also comply with the requirements.
Tested-by: Martyn Welch <martyn.welch@collabora.com>
Tested-by: Alexander Stein <alexander.stein@ew.tq-group.com> # TQMa8MPxL/MBa8MPxL
Reviewed-by: Liu Ying <victor.liu@nxp.com>
Link: https://patch.msgid.link/20260407-drm-lcdif-dbanc-v4-11-247a16e61ef9@bootlin.com
Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com>
The imx8mp-hdmi-tx one of many drivers based on dw-hdmi. dw-hdmi in turn
can operate in two different modes, depending on the platform data as set
by the driver:
A. hdmi->plat_data->output_port = 0:
the HDMI output (port@1) in device tree is not used [0]
B. hdmi->plat_data->output_port = 1:
the HDMI output (port@1) is parsed to find the next bridge
The imx8mp-hdmi-tx driver falls in case A. Switching to case B, which is
the current best practice, requires that the HDMI connector is always
described in the live device tree, and a previous commit ensured this is
always the case by adding an overlay in case the device tree does not
comply.
So now we can simply switch to the new mode and support
DRM_BRIDGE_ATTACH_NO_CONNECTOR.
Reviewed-by: Liu Ying <victor.liu@nxp.com>
Link: https://patch.msgid.link/20260407-drm-lcdif-dbanc-v4-10-247a16e61ef9@bootlin.com
Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com>
Describing the HDMI connector in device tree is recommended. While the
overlay insertion is a workaround to avoid breaking existing devices, every
dts should be improved by adding a connector description.
Add a warning to make users aware as far as possible.
As a warning line cannot hold all the relevant info, add a detailed comment
in the code so it easy to find when the warning is seen.
Reviewed-by: Liu Ying <victor.liu@nxp.com>
Link: https://patch.msgid.link/20260407-drm-lcdif-dbanc-v4-9-247a16e61ef9@bootlin.com
Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com>
