The descriptor FIFO requires that all words of a descriptor are written
in order, with the control word written last to flush it into the DMA
engine. Using memcpy() with __force to __iomem is not the correct API
and does not guarantee appropriate MMIO access on all architectures.
Replace the descriptor body copy with memcpy_toio(), using
offsetof(struct msgdma_extended_desc, control) to exclude the control
word. This matches the previous sizeof(desc->hw_desc) - sizeof(u32)
length only when control is the last struct member; add a static_assert
to enforce that layout so a future field after control cannot silently
break FIFO ordering.
Keep writing the control word separately with write barriers, so it
remains the final word pushed into the FIFO.
Signed-off-by: Adrian Ng Ho Yin <adrianhoyin.ng@altera.com>
Signed-off-by: Tze Yee Ng <tze.yee.ng@altera.com>
Link: https://patch.msgid.link/f6f3b4a2e2eb0eb1a51976de3f5d1ef5bab9bd76.1779697226.git.tze.yee.ng@altera.com
Signed-off-by: Vinod Koul <vkoul@kernel.org>
Add documentation for the dma-channel-mask property in the fsl-edma
binding. This property uses an inverted bit definition: bit value 0
indicates the channel is available, while bit value 1 indicates
unavailable.
That was already used widely for i.MX8, i.MX9. Correcting the definition
will break backward compatibility. This reversal only impacts the eDMA
dts node and driver, and doesn't impact DMA consumer. Therefore,
keep the inverted definition.
Also add a note at the top of the binding to highlight this inverted
definition to prevent confusion.
Signed-off-by: Joy Zou <joy.zou@nxp.com>
Reviewed-by: Frank Li <Frank.Li@nxp.com>
Acked-by: Rob Herring (Arm) <robh@kernel.org>
Link: https://patch.msgid.link/20260211-b4-imx95-v2x-v4-1-10852754b267@nxp.com
Signed-off-by: Vinod Koul <vkoul@kernel.org>
Currently, the Tegra GPC DMA hardware requires the transfer length to
be a multiple of the max burst size configured for the channel. When a
client requests a transfer where the length is not evenly divisible by
the configured max burst size, the DMA hangs with partial burst at
the end.
Fix this by reducing the burst size to the largest power-of-2 value
that evenly divides the transfer length. For example, a 40-byte
transfer with a 16-byte max burst will now use an 8-byte burst
(40 / 8 = 5 complete bursts) instead of causing a hang.
This issue was observed with the PL011 UART driver where TX DMA
transfers of arbitrary lengths were stuck.
Fixes: ee17028009 ("dmaengine: tegra: Add tegra gpcdma driver")
Cc: stable@vger.kernel.org
Signed-off-by: Kartik Rajput <kkartik@nvidia.com>
Reviewed-by: Frank Li <Frank.Li@nxp.com>
Reviewed-by: Jon Hunter <jonathanh@nvidia.com>
Link: https://patch.msgid.link/20260422064134.1323610-1-kkartik@nvidia.com
Signed-off-by: Vinod Koul <vkoul@kernel.org>
The Intel IOPx3xx platform was completely removed in commit b91a69d162
("ARM: iop32x: remove the platform"), and it'd be safe to remove an unused
and leftover platform data specific header file dma-iop32x.h also.
Signed-off-by: Vladimir Zapolskiy <vz@mleia.com>
Reviewed-by: Frank Li <Frank.Li@nxp.com>
Link: https://patch.msgid.link/20260114051508.3908807-1-vz@mleia.com
[vkoul: fixed subsystem tag]
Signed-off-by: Vinod Koul <vkoul@kernel.org>
For architectures like Microblaze or arm64 (where this IP is used),
DMA_DIRECT_REMAP is set which means that dma_alloc_coherent() might
remap (and hence vmalloc()) some memory. This became visible in a design
where dma_direct_use_pool() is not possible.
With the above, when calling dma_free_coherent(), vunmap() would be
called from softirq context and thus leading to a BUG().
To fix it, use a dma pool that is allocated in
.device_alloc_chan_resources() and allocate blocks from it. The key
point is that now dma_pool_free() is used in axi_dmac_free_desc() to
free the blocks and that just frees the blocks from the pool in the
sense they can be used again. In other words, no actual call to
dma_free_coherent() happens. That only happens when destroying the pool
in axi_dmac_free_chan_resources() which does not happen in any interrupt
context.
Fixes: 3f8fd25936 ("dmaengine: axi-dmac: Allocate hardware descriptors")
Signed-off-by: Nuno Sá <nuno.sa@analog.com>
Reviewed-by: Frank Li <Frank.Li@nxp.com>
Link: https://patch.msgid.link/20260424-dma-dmac-handle-vunmap-v4-4-90f43412fdc0@analog.com
Signed-off-by: Vinod Koul <vkoul@kernel.org>
There's no reason to keep struct clk in struct axi_dmac. Hence, use a
local clk variable in .probe() and drop it from struct axi_dmac.
Signed-off-by: Nuno Sá <nuno.sa@analog.com>
Reviewed-by: Frank Li <Frank.Li@nxp.com>
Link: https://patch.msgid.link/20260424-dma-dmac-handle-vunmap-v4-3-90f43412fdc0@analog.com
Signed-off-by: Vinod Koul <vkoul@kernel.org>
Use axi_dmac_free_desc() to free fully the descriptor at fail path when
call axi_dmac_alloc_desc() in axi_dmac_prep_peripheral_dma_vec().
Fixes: 74609e5686 ("dmaengine: dma-axi-dmac: Implement device_prep_peripheral_dma_vec")
Signed-off-by: Nuno Sá <nuno.sa@analog.com>
Reviewed-by: Frank Li <Frank.Li@nxp.com>
Link: https://patch.msgid.link/20260424-dma-dmac-handle-vunmap-v4-2-90f43412fdc0@analog.com
Signed-off-by: Vinod Koul <vkoul@kernel.org>
In dma_release_channel(), check chan->device->privatecnt after call
dma_chan_put(). However, dma_chan_put() call dma_device_put() which could
release the last reference of the device if the DMA provider is already
gone and hence free it.
Fixes it by moving dma_chan_put() after the check.
Fixes: 0f571515c3 ("dmaengine: Add privatecnt to revert DMA_PRIVATE property")
Signed-off-by: Nuno Sá <nuno.sa@analog.com>
Reviewed-by: Frank Li <Frank.Li@nxp.com>
Link: https://patch.msgid.link/20260424-dma-dmac-handle-vunmap-v4-1-90f43412fdc0@analog.com
Signed-off-by: Vinod Koul <vkoul@kernel.org>
The DONE_INT_MASK and ABORT_INT_MASK registers are shared by all DMA
channels, and modifying them requires a read-modify-write sequence.
Because this operation is not atomic, concurrent calls to
dw_edma_v0_core_start() can introduce race conditions if two channels
update these registers simultaneously.
Add a spinlock to serialize access to these registers and prevent race
conditions.
Fixes: 7e4b8a4fbe ("dmaengine: Add Synopsys eDMA IP version 0 support")
Cc: stable@vger.kernel.org
Signed-off-by: Frank Li <Frank.Li@nxp.com>
[den: update dw_edma.lock comment]
Link: https://lore.kernel.org/dmaengine/20260109-edma_ll-v2-1-5c0b27b2c664@nxp.com/
Signed-off-by: Koichiro Den <den@valinux.co.jp>
Link: https://patch.msgid.link/20260521142153.2957432-5-den@valinux.co.jp
Signed-off-by: Vinod Koul <vkoul@kernel.org>
dw_edma_pcie_probe() treats the PCI device ID driver_data as the
template for the controller layout and copies it unconditionally. A
device bound dynamically via sysfs can match the driver without that
data, which leads to a NULL pointer dereference.
Reject such matches before enabling the device.
Fixes: 41aaff2a2a ("dmaengine: Add Synopsys eDMA IP PCIe glue-logic")
Cc: stable@vger.kernel.org
Signed-off-by: Koichiro Den <den@valinux.co.jp>
Reviewed-by: Frank Li <Frank.Li@nxp.com>
Link: https://patch.msgid.link/20260521142153.2957432-3-den@valinux.co.jp
Signed-off-by: Vinod Koul <vkoul@kernel.org>
Some peripherals on RZ/G3E SoCs (SSIU, SPDIF, SCU/SRC, DVC, PFC) require
explicit ACK signal routing through the ICU for level-based DMA handshaking.
Rather than extending the DT binding with an optional second #dma-cells
(which would require all DMA consumers to supply two cells even when ACK
routing is not needed), derive the ACK signal number directly from the
MID/RID request number using the linear mapping defined in RZ/G3E hardware
manual Table 4.6-28:
PFC external DMA pins (DREQ0..DREQ4):
req_no 0x000-0x004 -> ACK No. 84-88
SSIU BUSIFs (ssip00..ssip93):
req_no 0x161-0x198 -> ACK No. 28-83
SPDIF (CH0..CH2) + SCU SRC (sr0..sr9) + DVC (cmd0..cmd1):
req_no 0x199-0x1b4 -> ACK No. 0-27
ACK routing is programmed when a channel is prepared for transfer and
cleared when the channel is released or the transfer times out, following
the same pattern as MID/RID request routing.
Signed-off-by: John Madieu <john.madieu.xa@bp.renesas.com>
Link: https://patch.msgid.link/20260525110750.4020112-3-john.madieu.xa@bp.renesas.com
[fixes subsystem name tag]
Signed-off-by: Vinod Koul <vkoul@kernel.org>
Some peripherals on RZ/G3E SoCs (SSIU, SPDIF, SCU/SRC, DVC) require
explicit ACK signal routing through the ICU via the ICU_DMACKSELk
registers for level-based DMA handshaking.
Add rzv2h_icu_register_dma_ack() to configure ICU_DMACKSELk, routing
a DMAC channel's ACK signal to the specified peripheral.
Signed-off-by: John Madieu <john.madieu.xa@bp.renesas.com>
Acked-by: Thomas Gleixner <tglx@kernel.org>
Link: https://patch.msgid.link/20260525110750.4020112-2-john.madieu.xa@bp.renesas.com
Signed-off-by: Vinod Koul <vkoul@kernel.org>
Function dw_edma_add_irq_mask() sets the mask of the
interrupts alloted to read / write channels in a variable.
The mask set for read / write channels is niether used nor
this function is called else where, making it redundant.
The redundant function can be removed safely as it is
not affecting anything.
Signed-off-by: Devendra K Verma <devendra.verma@amd.com>
Reviewed-by: Frank Li <Frank.Li@nxp.com>
Link: https://patch.msgid.link/20260526053111.3244488-1-devverma@amd.com
Signed-off-by: Vinod Koul <vkoul@kernel.org>
The dma-nbpf driver only probes devices from device tree and fails to
probe devices relying on the traditional platform device probe path. So
the platform_device_id array is unused apart from providing misleading
module meta data.
Drop it.
Signed-off-by: Uwe Kleine-König (The Capable Hub) <u.kleine-koenig@baylibre.com>
Link: https://patch.msgid.link/5f7380828873e2375e319ef091178d11a277a0ac.1779965563.git.u.kleine-koenig@baylibre.com
Signed-off-by: Vinod Koul <vkoul@kernel.org>
Since commit 2e7f55ce43 ("dmaengine: cirrus: Convert to DT for Cirrus
EP93xx") the driver cannot probe devices using the traditional platform
device way any more. Thus the driver's .id_table serves no purpose any
more and can be dropped.
Signed-off-by: Uwe Kleine-König (The Capable Hub) <u.kleine-koenig@baylibre.com>
Link: https://patch.msgid.link/c3830cb95b0bb939f9cc9543dfa3047e41532c47.1779976024.git.ukleinek@kernel.org
Signed-off-by: Vinod Koul <vkoul@kernel.org>
Combining the struct definition with its variable initializer confuses the
kernel-doc parser because __MUTEX_INITIALIZER() expands to contain braces,
breaking brace counting and causing:
Warning: drivers/dma/dmatest.c:152 struct member '' not described in 'dmatest_info'
Split into separate struct definition and variable declaration, which is
the standard kernel pattern.
Assisted-by: Opencode:Big-pickle
Signed-off-by: Rosen Penev <rosenp@gmail.com>
Link: https://patch.msgid.link/20260530200322.7584-1-rosenp@gmail.com
Signed-off-by: Vinod Koul <vkoul@kernel.org>
The GPI DMA controller is only responsible for QUP peripherals, and
cannot work as a general-purpose DMA accelerator.
Set DMA_PRIVATE capability for it.
This fixes error messages about GPI being shown when an async-tx
consumer is loaded.
Fixes: 5d0c3533a1 ("dmaengine: qcom: Add GPI dma driver")
Signed-off-by: Icenowy Zheng <zhengxingda@iscas.ac.cn>
Reviewed-by: Dmitry Baryshkov <dmitry.baryshkov@oss.qualcomm.com>
Link: https://patch.msgid.link/20260602070344.3707256-1-zhengxingda@iscas.ac.cn
Signed-off-by: Vinod Koul <vkoul@kernel.org>
Convert the separately-offset phy_chans pointer to a C99 flexible array
member at the end of struct d40_base, and switch the allocation to
struct_size(). The log_chans and memcpy_chans slots continue to live
in the same allocation immediately after phy_chans, indexed via
base->log_chans. This removes the hand-rolled pointer fixup that
recomputed phy_chans from base + ALIGN(sizeof(struct d40_base), 4).
The ALIGN(sizeof(struct d40_base), 4) requirement is met implicitly by the
C compiler when using a flexible array member. With struct d40_chan
phy_chans[] as the last member, the C standard guarantees
sizeof(struct d40_base) includes trailing padding to satisfy the alignment
of the flexible array element type (struct d40_chan). Since struct d40_chan
contains members like spinlock_t, pointers, and struct dma_chan — all with
alignment ≥ 4 — the compiler ensures sizeof(struct d40_base) is already a
multiple of _Alignof(struct d40_chan) >= 4. The struct_size() macro then
computes sizeof(struct d40_base) + sizeof(struct d40_chan) * num_phy_chans,
so phy_chans[0] lands at a properly aligned offset without needing the manual
ALIGN.
Assisted-by: Claude:Opus-4.7
Signed-off-by: Rosen Penev <rosenp@gmail.com>
Reviewed-by: Linus Walleij <linusw@kernel.org>
Link: https://patch.msgid.link/20260531020843.594892-1-rosenp@gmail.com
Signed-off-by: Vinod Koul <vkoul@kernel.org>
Add dev_err/dev_err_probe logging across failure paths to improve
debuggability of DMA errors during runtime and probe.
Use return dev_err_probe() pattern where no cleanup is required in the
probe function. On error paths that need explicit unwind, store the
dev_err_probe() return value in ret before jumping to the cleanup label.
Also convert existing dev_err calls in probe to dev_err_probe for
consistency, and use dev_err in non-probe functions.
Keep explicit runtime PM and DMA registration unwind instead of managed or
scoped cleanup. The scoped runtime PM guard releases the usage count with
pm_runtime_put(), while this probe error path needs pm_runtime_put_sync()
before pm_runtime_disable(). The OF DMA registration failure path also
needs to unregister the DMA engine before dropping the runtime PM reference.
Signed-off-by: Sheetal <sheetal@nvidia.com>
Reviewed-by: Jon Hunter <jonathanh@nvidia.com>
Link: https://patch.msgid.link/20260517163045.363444-1-sheetal@nvidia.com
Signed-off-by: Vinod Koul <vkoul@kernel.org>
Add Device ID for AMD (Xilinx) CPM6 DMA IP. This IP enables
64 Read and 64 Write Channels.
Adding the relevant dw_edma_pcie_data to use 8 Read and 8 Write
channels for initial commit.
Signed-off-by: Devendra K Verma <devendra.verma@amd.com>
Reviewed-by: Frank Li <Frank.Li@nxp.com>
Link: https://patch.msgid.link/20260605112829.679697-1-devendra.verma@amd.com
Signed-off-by: Vinod Koul <vkoul@kernel.org>
On an RZ/G2L-based system, it has been observed that when the DMA channels
for all enabled IPs are active (TX and RX for one serial IP, TX and RX for
one audio IP, and TX and RX for one SPI IP), shortly after all of them are
started, the system can become irrecoverably blocked. In one debug session
the system did not block, and the DMA HW registers were inspected. It was
found that the DER (Descriptor Error) bit in the CHSTAT register for one of
the SPI DMA channels was set.
According to the RZ/G2L HW Manual, Rev. 1.30, chapter 14.4.7 Channel
Status Register n/nS (CHSTAT_n/nS), description of the DER bit, the DER
bit is set when the LV (Link Valid) value loaded with a descriptor in link
mode is 0. This means that the DMA engine has loaded an invalid
descriptor (as defined in Table 14.14, Header Area, of the same manual).
The same chapter states that when a descriptor error occurs, the transfer
is stopped, but no DMA error interrupt is generated.
Set the LE bit on the last descriptor of a transfer. This informs the DMA
engine that this is the final descriptor for the transfer.
Tested-by: John Madieu <john.madieu.xa@bp.renesas.com>
Signed-off-by: Claudiu Beznea <claudiu.beznea.uj@bp.renesas.com>
Tested-by: Tommaso Merciai <tommaso.merciai.xr@bp.renesas.com>
Link: https://patch.msgid.link/20260526084710.3491480-19-claudiu.beznea@kernel.org
Signed-off-by: Vinod Koul <vkoul@kernel.org>
On Renesas RZ/G2L and RZ/G3S SoCs (where this was tested), captured audio
files occasionally contained random spikes when viewed with a tool such
as Audacity. These spikes were also audible as popping noises.
Using cyclic DMA resolves this issue. The driver was reworked to use the
existing support provided by the generic PCM dmaengine APIs. In addition
to eliminating the random spikes, the following issues were addressed:
- blank periods at the beginning of recorded files, which occurred
intermittently, are no longer present
- no overruns or underruns were observed when continuously recording
short audio files (e.g. 5 seconds long) in a loop
- concurrency issues in the SSI driver when enqueuing DMA requests were
eliminated; previously, DMA requests could be prepared and submitted
both from the DMA completion callback and the interrupt handler, which
led to crashes after several hours of testing
- the SSI driver logic is simplified
- the number of generated interrupts is reduced by approximately 250%
In the SSI platform driver probe function, the following changes were
made:
- the driver-specific DMA configuration was removed in favor of the
generic PCM dmaengine APIs. As a result, explicit cleanup goto labels
are no longer required and the driver remove callback was dropped,
since resource management is now handled via devres helpers
- special handling was added for IP variants operating in half-duplex
mode, where the DMA channel name in the device tree is "rt"; this DMA
channel name is taken into account and passed to the generic PCM
dmaengine configuration data
All code previously responsible for preparing and completing DMA
transfers was removed, as this functionality is now handled entirely by
the generic PCM dmaengine APIs.
Since DMA channels must be paused and resumed during recovery paths
(overruns and underruns reported by the hardware), the DMA channel
references are stored in rz_ssi_hw_params().
The logic in rz_ssi_is_dma_enabled() was updated to reflect that the
driver no longer manages DMA transfers directly.
To avoid software reported underruns (e.g. when running aplay during
consecutive suspend/resume cycles, or when the CPU is nearly 100%
loaded), rz_ssi_pcm_hardware.buffer_bytes_max was increased to 192K.
At the same time, rz_ssi_pcm_hardware.period_bytes_max was set to 48K
to reduce interrupt overhead.
Finally, rz_ssi_stream_is_play() was removed, as it had only a single
remaining user after this rework, and its logic was inlined at the call
site.
Acked-by: Mark Brown <broonie@kernel.org>
Tested-by: John Madieu <john.madieu.xa@bp.renesas.com>
Signed-off-by: Claudiu Beznea <claudiu.beznea.uj@bp.renesas.com>
Link: https://patch.msgid.link/20260526084710.3491480-18-claudiu.beznea@kernel.org
Signed-off-by: Vinod Koul <vkoul@kernel.org>
Add pause support as a preparatory step to switch to PCM dmaengine APIs.
Acked-by: Mark Brown <broonie@kernel.org>
Tested-by: John Madieu <john.madieu.xa@bp.renesas.com>
Signed-off-by: Claudiu Beznea <claudiu.beznea.uj@bp.renesas.com>
Link: https://patch.msgid.link/20260526084710.3491480-17-claudiu.beznea@kernel.org
Signed-off-by: Vinod Koul <vkoul@kernel.org>
The Renesas RZ/G3S SoC supports a power saving mode in which power to most
of the SoC components is turned off, including the DMA IP. Add suspend to
RAM support to save and restore the DMA IP registers.
Cyclic DMA channels require special handling. Since they can be paused and
resumed during system suspend/resume, the driver restores additional
registers for these channels during the system resume phase. If a channel
was not explicitly paused during suspend, the driver ensures that it is
paused and resumed as part of the system suspend/resume flow.
Tested-by: John Madieu <john.madieu.xa@bp.renesas.com>
Signed-off-by: Claudiu Beznea <claudiu.beznea.uj@bp.renesas.com>
Tested-by: Tommaso Merciai <tommaso.merciai.xr@bp.renesas.com>
Link: https://patch.msgid.link/20260526084710.3491480-16-claudiu.beznea@kernel.org
Signed-off-by: Vinod Koul <vkoul@kernel.org>
Protect the driver exposed APIs with runtime PM suspend/resume calls
before accessing HW registers. As the current driver leaves runtime PM
enabled in probe, the purpose of the changes in this patch is to avoid
accessing HW registers after a failed system suspend leaves the runtime
PM state of the device improperly reinitialized.
In that case, the driver remains bound to the device, the APIs are still
exposed, and any access to HW registers without runtime resuming the
device may lead to synchronous aborts.
To avoid leaking resources in case of runtime PM failures, save the error
code returned by PM_RUNTIME_ACQUIRE_ERR() in rz_dmac_terminate_all() and
return it only at the end of the function to allow the cleanup code to
run. A similar approach is used in rz_dmac_free_chan_resources().
Because some exposed APIs (e.g. ->device_terminate_all()) may be called
from atomic context according to the documentation, mark the DMA device as
pm_runtime_irq_safe().
This patch prepares the driver for suspend-to-RAM support.
Tested-by: John Madieu <john.madieu.xa@bp.renesas.com>
Signed-off-by: Claudiu Beznea <claudiu.beznea.uj@bp.renesas.com>
Tested-by: Tommaso Merciai <tommaso.merciai.xr@bp.renesas.com>
Link: https://patch.msgid.link/20260526084710.3491480-15-claudiu.beznea@kernel.org
Signed-off-by: Vinod Koul <vkoul@kernel.org>
Adjust rz_dmac_chan_get_residue() to return error codes on failure and
provide the residue to callers through the residue parameter. This
prepares the code for the addition of runtime PM support.
Signed-off-by: Claudiu Beznea <claudiu.beznea.uj@bp.renesas.com>
Tested-by: Tommaso Merciai <tommaso.merciai.xr@bp.renesas.com>
Link: https://patch.msgid.link/20260526084710.3491480-14-claudiu.beznea@kernel.org
Signed-off-by: Vinod Koul <vkoul@kernel.org>
Add cyclic DMA support to the RZ DMAC driver. A per-channel status bit is
introduced to mark cyclic channels and is set during the DMA prepare
callback. The IRQ handler checks this status bit and calls
vchan_cyclic_callback() accordingly.
Tested-by: John Madieu <john.madieu.xa@bp.renesas.com>
Signed-off-by: Claudiu Beznea <claudiu.beznea.uj@bp.renesas.com>
Tested-by: Tommaso Merciai <tommaso.merciai.xr@bp.renesas.com>
Link: https://patch.msgid.link/20260526084710.3491480-13-claudiu.beznea@kernel.org
Signed-off-by: Vinod Koul <vkoul@kernel.org>
The CHCTRL_SETEN bit is explicitly set in rz_dmac_enable_hw(). Updating
struct rz_dmac_chan::chctrl with this bit in
rz_dmac_prepare_desc_for_memcpy() and rz_dmac_prepare_descs_for_slave_sg()
is unnecessary in the current code base. Moreover, it conflicts with the
configuration sequence that will be used for cyclic DMA channels during
suspend to RAM. Cyclic DMA support will be introduced in subsequent
commits.
This is a preparatory commit for cyclic DMA suspend to RAM support.
Reviewed-by: Frank Li <Frank.Li@nxp.com>
Tested-by: John Madieu <john.madieu.xa@bp.renesas.com>
Signed-off-by: Claudiu Beznea <claudiu.beznea.uj@bp.renesas.com>
Tested-by: Tommaso Merciai <tommaso.merciai.xr@bp.renesas.com>
Link: https://patch.msgid.link/20260526084710.3491480-12-claudiu.beznea@kernel.org
Signed-off-by: Vinod Koul <vkoul@kernel.org>
Subsequent patches will add suspend/resume and cyclic DMA support to the
rz-dmac driver. This support needs to work on SoCs where power to most
components (including DMA) is turned off during system suspend. For this,
some channels (for example cyclic ones) may need to be paused and resumed
manually by the DMA driver during system suspend/resume.
Refactor the pause/resume support so the same code can be reused in the
system suspend/resume path.
Tested-by: John Madieu <john.madieu.xa@bp.renesas.com>
Signed-off-by: Claudiu Beznea <claudiu.beznea.uj@bp.renesas.com>
Tested-by: Tommaso Merciai <tommaso.merciai.xr@bp.renesas.com>
Link: https://patch.msgid.link/20260526084710.3491480-11-claudiu.beznea@kernel.org
Signed-off-by: Vinod Koul <vkoul@kernel.org>
The driver used a mix of virt-dma APIs and driver specific logic to
process descriptors. It maintained three internal queues: ld_free,
ld_queue, and ld_active as follows:
- ld_free: stores the descriptors pre-allocated at probe time
- ld_queue: stores descriptors after they are taken from ld_free and
prepared. At the same time, vchan_tx_prep() queues them to
vc->desc_allocated. The vc->desc_allocated list is then checked in
rz_dmac_issue_pending() and rz_dmac_irq_handler_thread() before
starting a new transfer via rz_dmac_xfer_desc(). In turn,
rz_dmac_xfer_desc() grabs the next descriptor from vc->desc_issued and
submits it for transfer
- ld_active: stores the descriptors currently being transferred
The interrupt handler moved a completed descriptor to ld_free before
invoking its completion callback. Once returned to ld_free, the
descriptor can be reused to prepare a new transfer. In theory, this
means the descriptor could be re-prepared before its completion
callback is called.
Commit fully back the driver by the virt-dma APIs. With this, only ld_free
need to be kept to track how many free descriptors are available. This
is now done as follows:
- the prepare stage removes the first descriptor from the ld_free and
prepares it
- the completion calls for it vc->desc_free() (rz_dmac_virt_desc_free())
which re-adds the descriptor at the end of ld_free
With this, the critical areas in prepare callbacks were minimized to only
getting the descriptor from the ld_free list.
Introduce struct rz_dmac_chan::desc to keep track of the currently
transferred descriptor. It is cleared in rz_dmac_terminate_all(),
referenced from rz_dmac_issue_pending() to determine whether a new transfer
can be started, and from rz_dmac_irq_handler_thread() once a descriptor has
completed. Finally, the rz_dmac_device_synchronize() was updated with
vchan_synchronize() call to ensure the terminated descriptor is freed and
the tasklet is killed.
With this, residue computation is also simplified, as it can now be
handled entirely through the virt-dma APIs.
The spin_lock/unlock operations from rz_dmac_irq_handler_thread() were
replaced by guard as the final code after rework is simpler this way.
As subsequent commits will set the Link End bit on the last descriptor
of a transfer, rz_dmac_enable_hw() is also adjusted as part of the full
conversion to virt-dma APIs. It no longer checks the channel enable
status itself; instead, its callers verify whether the channel is
enabled and whether the previous transfer has completed before starting
a new one.
Tested-by: John Madieu <john.madieu.xa@bp.renesas.com>
Signed-off-by: Claudiu Beznea <claudiu.beznea.uj@bp.renesas.com>
Tested-by: Tommaso Merciai <tommaso.merciai.xr@bp.renesas.com>
Link: https://patch.msgid.link/20260526084710.3491480-10-claudiu.beznea@kernel.org
Signed-off-by: Vinod Koul <vkoul@kernel.org>
Add the rz_dmac_chan_is_paused() helper to check if the channel is paused.
This helper will be reused in subsequent patches.
Reviewed-by: Frank Li <Frank.Li@nxp.com>
Tested-by: John Madieu <john.madieu.xa@bp.renesas.com>
Signed-off-by: Claudiu Beznea <claudiu.beznea.uj@bp.renesas.com>
Tested-by: Tommaso Merciai <tommaso.merciai.xr@bp.renesas.com>
Link: https://patch.msgid.link/20260526084710.3491480-9-claudiu.beznea@kernel.org
Signed-off-by: Vinod Koul <vkoul@kernel.org>
Add the rz_dmac_chan_is_enabled() helper to check if a channel is
enabled. This helper will be reused in subsequent patches.
Reviewed-by: Frank Li <Frank.Li@nxp.com>
Tested-by: John Madieu <john.madieu.xa@bp.renesas.com>
Signed-off-by: Claudiu Beznea <claudiu.beznea.uj@bp.renesas.com>
Tested-by: Tommaso Merciai <tommaso.merciai.xr@bp.renesas.com>
Link: https://patch.msgid.link/20260526084710.3491480-8-claudiu.beznea@kernel.org
Signed-off-by: Vinod Koul <vkoul@kernel.org>
Save the start LM descriptor to avoid starting from the beginning of the
channel's LM descriptor list in rz_dmac_calculate_residue_bytes_in_vd().
This avoids unnecessary iterations.
Tested-by: John Madieu <john.madieu.xa@bp.renesas.com>
Signed-off-by: Claudiu Beznea <claudiu.beznea.uj@bp.renesas.com>
Tested-by: Tommaso Merciai <tommaso.merciai.xr@bp.renesas.com>
Link: https://patch.msgid.link/20260526084710.3491480-7-claudiu.beznea@kernel.org
Signed-off-by: Vinod Koul <vkoul@kernel.org>
Add a the rz_dmac_lmdesc_addr() helper function to compute the lmdesc
address, to make the code easier to understand. The helper will be used in
subsequent patches.
Reviewed-by: Frank Li <Frank.Li@nxp.com>
Tested-by: John Madieu <john.madieu.xa@bp.renesas.com>
Signed-off-by: Claudiu Beznea <claudiu.beznea.uj@bp.renesas.com>
Tested-by: Tommaso Merciai <tommaso.merciai.xr@bp.renesas.com>
Link: https://patch.msgid.link/20260526084710.3491480-6-claudiu.beznea@kernel.org
Signed-off-by: Vinod Koul <vkoul@kernel.org>
Use rz_dmac_disable_hw() instead of open coding it. This unifies the
code and prepares it for the addition of suspend to RAM and cyclic DMA.
The rz_dmac_disable_hw() from rz_dmac_chan_probe() was moved after
vchan_init() as it initializes the channel->vc.chan.device used in
rz_dmac_disable_hw().
Reviewed-by: Frank Li <Frank.Li@nxp.com>
Tested-by: John Madieu <john.madieu.xa@bp.renesas.com>
Signed-off-by: Claudiu Beznea <claudiu.beznea.uj@bp.renesas.com>
Tested-by: Tommaso Merciai <tommaso.merciai.xr@bp.renesas.com>
Link: https://patch.msgid.link/20260526084710.3491480-5-claudiu.beznea@kernel.org
Signed-off-by: Vinod Koul <vkoul@kernel.org>
Use list_first_entry_or_null() instead of open-coding it with a
list_empty() check and list_first_entry(). This simplifies the code.
Reviewed-by: Frank Li <Frank.Li@nxp.com>
Tested-by: John Madieu <john.madieu.xa@bp.renesas.com>
Signed-off-by: Claudiu Beznea <claudiu.beznea.uj@bp.renesas.com>
Tested-by: Tommaso Merciai <tommaso.merciai.xr@bp.renesas.com>
Link: https://patch.msgid.link/20260526084710.3491480-4-claudiu.beznea@kernel.org
Signed-off-by: Vinod Koul <vkoul@kernel.org>
list_first_entry() does not return NULL when the list is empty,
making the existing NULL check invalid. Use list_first_entry_or_null()
instead.
Fixes: 21323b118c ("dmaengine: sh: rz-dmac: Add device_tx_status() callback")
Cc: stable@vger.kernel.org
Tested-by: John Madieu <john.madieu.xa@bp.renesas.com>
Signed-off-by: Claudiu Beznea <claudiu.beznea.uj@bp.renesas.com>
Tested-by: Tommaso Merciai <tommaso.merciai.xr@bp.renesas.com>
Link: https://patch.msgid.link/20260526084710.3491480-3-claudiu.beznea@kernel.org
Signed-off-by: Vinod Koul <vkoul@kernel.org>
Once the interrupt is requested, the interrupt handler may run immediately.
Since the IRQ handler can access channel->ch_base, which is initialized
only after requesting the IRQ, this may lead to invalid memory access.
Likewise, the IRQ thread may access uninitialized data (the ld_free,
ld_queue, and ld_active lists), which may also lead to issues.
Request the interrupts only after everything is set up. To keep the error
path simpler, use dmam_alloc_coherent() instead of dma_alloc_coherent().
Fixes: 5000d37042 ("dmaengine: sh: Add DMAC driver for RZ/G2L SoC")
Cc: stable@vger.kernel.org
Reviewed-by: Frank Li <Frank.Li@nxp.com>
Tested-by: John Madieu <john.madieu.xa@bp.renesas.com>
Signed-off-by: Claudiu Beznea <claudiu.beznea.uj@bp.renesas.com>
Tested-by: Tommaso Merciai <tommaso.merciai.xr@bp.renesas.com>
Link: https://patch.msgid.link/20260526084710.3491480-2-claudiu.beznea@kernel.org
Signed-off-by: Vinod Koul <vkoul@kernel.org>
Add compatible and chip data to support GPCDMA in Tegra264, which has
differences in register layout and address bits compared to previous
versions.
Signed-off-by: Akhil R <akhilrajeev@nvidia.com>
Reviewed-by: Frank Li <Frank.Li@nxp.com>
Acked-by: Thierry Reding <treding@nvidia.com>
Reviewed-by: Jon Hunter <jonathanh@nvidia.com>
Link: https://patch.msgid.link/20260331102303.33181-10-akhilrajeev@nvidia.com
Signed-off-by: Vinod Koul <vkoul@kernel.org>
Use 'iommu-map', when provided, to get the stream ID to be programmed
for each channel. Iterate over the channels registered and configure
each channel device separately using of_dma_configure_id() to allow
it to use a separate IOMMU domain for the transfer. However, do this
in a second loop since the first loop populates the DMA device channels
list and async_device_register() registers the channels. Both are
prerequisites for using the channel device in the next loop.
Channels will continue to use the same global stream ID if the
'iommu-map' property is not present in the device tree.
Signed-off-by: Akhil R <akhilrajeev@nvidia.com>
Reviewed-by: Frank Li <Frank.Li@nxp.com>
Reviewed-by: Jon Hunter <jonathanh@nvidia.com>
Link: https://patch.msgid.link/20260331102303.33181-9-akhilrajeev@nvidia.com
Signed-off-by: Vinod Koul <vkoul@kernel.org>
Switch to managed registration in probe. This simplifies the error
paths in the probe and also removes the requirement of the driver
remove function.
Signed-off-by: Akhil R <akhilrajeev@nvidia.com>
Suggested-by: Frank Li <frank.li@nxp.com>
Acked-by: Thierry Reding <treding@nvidia.com>
Reviewed-by: Jon Hunter <jonathanh@nvidia.com>
Link: https://patch.msgid.link/20260331102303.33181-8-akhilrajeev@nvidia.com
Signed-off-by: Vinod Koul <vkoul@kernel.org>
Tegra264 supports address width of 41 bits. Unlike older SoCs which use
a common high_addr register for upper address bits, Tegra264 has separate
src_high and dst_high registers to accommodate this wider address space.
Add an addr_bits property to the device data structure to specify the
number of address bits supported on each device and use that to program
the appropriate registers.
Update the sg_req struct to remove the high_addr field and use
dma_addr_t for src and dst to store the complete addresses. Extract
the high address bits only when programming the registers.
Signed-off-by: Akhil R <akhilrajeev@nvidia.com>
Reviewed-by: Frank Li <Frank.Li@nxp.com>
Reviewed-by: Jon Hunter <jonathanh@nvidia.com>
Link: https://patch.msgid.link/20260331102303.33181-7-akhilrajeev@nvidia.com
Signed-off-by: Vinod Koul <vkoul@kernel.org>
Repurpose the struct tegra_dma_channel_regs to define offsets for all the
channel registers. Previously, the struct only held the register values
for each transfer and was wrapped within tegra_dma_sg_req. Move the
values directly into tegra_dma_sg_req and use channel_regs for
storing the register offsets. Update all register reads/writes to use
the struct channel_regs. This prepares for the register offset change
in Tegra264.
Signed-off-by: Akhil R <akhilrajeev@nvidia.com>
Reviewed-by: Frank Li <Frank.Li@nxp.com>
Acked-by: Thierry Reding <treding@nvidia.com>
Reviewed-by: Jon Hunter <jonathanh@nvidia.com>
Link: https://patch.msgid.link/20260331102303.33181-6-akhilrajeev@nvidia.com
Signed-off-by: Vinod Koul <vkoul@kernel.org>
On Tegra264, reset is not available for the driver to control as
this is handled by the boot firmware. Hence make the reset control
optional and update the error message to reflect the correct error.
Signed-off-by: Akhil R <akhilrajeev@nvidia.com>
Reviewed-by: Frank Li <Frank.Li@nxp.com>
Acked-by: Thierry Reding <treding@nvidia.com>
Reviewed-by: Jon Hunter <jonathanh@nvidia.com>
Link: https://patch.msgid.link/20260331102303.33181-5-akhilrajeev@nvidia.com
Signed-off-by: Vinod Koul <vkoul@kernel.org>
Add iommu-map property to specify separate stream IDs for each DMA
channel. This enables each channel to be in its own IOMMU domain,
keeping memory isolated from other devices sharing the same DMA
controller.
Define the constraints such that if the channel and stream IDs are
contiguous, a single entry can map all the channels, but if the
channels or stream IDs are non-contiguous support multiple entries.
Signed-off-by: Akhil R <akhilrajeev@nvidia.com>
Acked-by: Rob Herring (Arm) <robh@kernel.org>
Acked-by: Thierry Reding <treding@nvidia.com>
Reviewed-by: Jon Hunter <jonathanh@nvidia.com>
Link: https://patch.msgid.link/20260331102303.33181-4-akhilrajeev@nvidia.com
Signed-off-by: Vinod Koul <vkoul@kernel.org>
On Tegra264, GPCDMA reset control is not exposed to Linux and is handled
by the boot firmware.
Although reset was not exposed in Tegra234 as well, the firmware supported
a dummy reset which just returns success on reset without doing an actual
reset. This is also not supported in Tegra264 BPMP. Therefore mark 'reset'
and 'reset-names' properties as required only for devices prior to
Tegra264.
This also necessitates that the Tegra264 compatible be standalone and
cannot have the fallback compatible of Tegra186. Since there is no
functional impact, we keep reset as required for Tegra234 to avoid
breaking the ABI.
Fixes: bb8c97571d ("dt-bindings: dma: Add Tegra264 compatible string")
Signed-off-by: Akhil R <akhilrajeev@nvidia.com>
Acked-by: Rob Herring (Arm) <robh@kernel.org>
Acked-by: Thierry Reding <treding@nvidia.com>
Reviewed-by: Jon Hunter <jonathanh@nvidia.com>
Link: https://patch.msgid.link/20260331102303.33181-2-akhilrajeev@nvidia.com
Signed-off-by: Vinod Koul <vkoul@kernel.org>
There are multi spba-busses for i.MX8M* platforms, if only search for
the first spba-bus in DT, the found spba-bus may not the real bus of
audio devices, which cause issue for sdma p2p case, as the sdma p2p
script presently does not deal with the transactions involving two devices
connected to the AIPS bus.
Search the SDMA parent node first, which should be the AIPS bus, then
search the child node whose compatible string is spba-bus under that AIPS
bus for the above multi spba-busses case.
Fixes: 8391ecf465 ("dmaengine: imx-sdma: Add device to device support")
Signed-off-by: Shengjiu Wang <shengjiu.wang@nxp.com>
Reviewed-by: Frank Li <Frank.Li@nxp.com>
Link: https://patch.msgid.link/20260407032755.2758049-1-shengjiu.wang@nxp.com
Signed-off-by: Vinod Koul <vkoul@kernel.org>
By convention MODULE_DEVICE_TABLE() immediately follows the ID table it
exports, because this is easier to read and verify. It also makes more
sense since #ifdef for ACPI or OF could hide both of them.
Most of the drivers already have this correctly placed, so adjust
the missing ones. No functional impact.
Reviewed-by: Radhey Shyam Pandey <radhey.shyam.pandey@amd.com>
Reviewed-by: Frank Li <Frank.Li@nxp.com>
Signed-off-by: Krzysztof Kozlowski <krzysztof.kozlowski@oss.qualcomm.com>
Link: https://patch.msgid.link/20260514060525.9253-2-krzysztof.kozlowski@oss.qualcomm.com
Signed-off-by: Vinod Koul <vkoul@kernel.org>
Document the GPI DMA engine found on the Eliza SoC.
It is fully compatible with the GPI DMA engine found on SM6350,
thus using qcom,sm6350-gpi-dma as fallback compatible.
Acked-by: Krzysztof Kozlowski <krzysztof.kozlowski@oss.qualcomm.com>
Signed-off-by: Abel Vesa <abel.vesa@oss.qualcomm.com>
Reviewed-by: Pankaj Patil <pankaj.patil@oss.qualcomm.com>
Link: https://patch.msgid.link/20260515-eliza-gpi-dma-v2-1-1255b43d5ca9@oss.qualcomm.com
Signed-off-by: Vinod Koul <vkoul@kernel.org>
SpacemiT K3 reuses most of the PDMA IP design found on K1, with one
difference being the extended DRCMR base address. Add "spacemit,k3-pdma"
compatible string and define a new mmp_pdma_ops for K3 PDMA.
Signed-off-by: Guodong Xu <guodong@riscstar.com>
Signed-off-by: Troy Mitchell <troy.mitchell@linux.spacemit.com>
Link: https://patch.msgid.link/20260518-k3-pdma-v6-3-67fdf319a8f8@linux.spacemit.com
Signed-off-by: Vinod Koul <vkoul@kernel.org>
