Bunch of core and driver fixes:
- Couple of fixes in core around dma_chan_put() for kref underflow, user
after free bug and waiting for rcu readers for dma devices
- mmp sg length and wrong extended DRCMR base for SpacemiT K3
- hardware buffer descriptor chain fix for xilinx dma
- sun6i fixes for status behaviour and dma position registers
- runtime pm reference leak fix for sprd driver
-----BEGIN PGP SIGNATURE-----
iQIzBAABCgAdFiEE+vs47OPLdNbVcHzyfBQHDyUjg0cFAmqwEtUACgkQfBQHDyUj
g0dCWQ//ShhhOd2vQQttV+BSHB0xTdT/wENO4mhXcz50chHFh1diBFASMnhejOxG
dv4ZkePAxV7PLpNqRQrCz2gx89wn5Uxf71PmnLbleuJ5lOuSUMGbFL0MhLzrlfqr
2Di122aa5Va4Tp3zjNllQ4ihKMfCF02FLCXoZqelVIR/NQFFMeJhEjv/0P2foFvd
nrf5jzGJuCCbhKiV47H4a8hSb1bG68ct/mV0ZfOT5Koe4B16qPTe7Z9kW5FP0do5
++BeHLbEheA/btWUSzvnLMtGHhhUywlAab6cKEkYAcTsuxtapMMRtJmAq9bYyX1O
qS6hIC9qWMS4xc0+BUsIhwU8cXY6IINy8lPJmwc+/p4V+MJ8QMP+MrfWpWADFy0H
rTvkIdzU+Lc4fqIr97UbW+pi8Naf2NCiV2NJSYF8JT6ufeG1PymtwbeUlbhkThtZ
ISVF2/CIieyHr/eDfzNuGrYmdkSfJZgl6Rd6pCHdYQ5vgK6DImtaHfuBBm1OYXVG
NwA4vkTk6hKu2Sx2JShrSJC2js5q5gdz+9jQuvb9zZZS4I1qOARWjoe/E0wowlir
EB5bivcNX8VR49x2aYmT7Mg1CJovzs6hC1Aw9aqFIhp+1hQRku+iFOSki8Pwzl23
EcpkC7liL8S6WCo73Zh7cmT0aBvw3Y6Gpvfozh+Rr7kLZy6cfmw=
=9a97
-----END PGP SIGNATURE-----
Merge tag 'dmaengine-fix-7.3' of git://git.kernel.org/pub/scm/linux/kernel/git/vkoul/dmaengine
Pull dmaengine fixes from Vinod Koul:
- A couple of fixes in core around dma_chan_put() for kref underflow,
use-after-free and waiting for rcu readers for dma devices
- mmp sg length and wrong extended DRCMR base for SpacemiT K3
- hardware buffer descriptor chain fix for xilinx dma
- sun6i fixes for status behaviour and dma position registers
- runtime pm reference leak fix for sprd driver
* tag 'dmaengine-fix-7.3' of git://git.kernel.org/pub/scm/linux/kernel/git/vkoul/dmaengine:
dmaengine: mmp_pdma: fix wrong sg length in mmp_pdma_prep_slave_sg()
dmaengine: xilinx_dma: Fix hardware buffer descriptor chain after cyclic DMA
dmaengine: pxa: fix double counting of the hw descriptors
dmaengine: sun6i: fix undefined behaviour in sun6i_dma_tx_status
dmaengine: sun6i: fix non-atomic read of DMA position registers
dmaengine: xilinx_dma: Fix hardware buffer descriptor reuse order
dmaengine: wait for RCU readers before releasing dma_device
dmaengine: fix use-after-free in dma_chan_put() and dma_release_channel()
dmaengine: Fix device kref underflow in dma_chan_put()
dmaengine: add dma_device_get() helper
dmaengine: sprd: Fix runtime PM reference leak in probe
dmaengine: ti: k3-udma-glue: fix NULL dereference in k3_udma_glue_release_rx_chn()
dmaengine: mmp_pdma: fix wrong extended DRCMR base for SpacemiT K3
In mmp_pdma_prep_slave_sg(), for_each_sg() iterates the scatterlist
putting each entry into 'sg', but the entry length is read from 'sgl'
(the list head) instead of 'sg' (the current entry):
for_each_sg(sgl, sg, sg_len, i) {
addr = sg_dma_address(sg);
avail = sg_dma_len(sgl); /* should be 'sg' */
Consequently 'avail' is always the length of the first entry. For
multi-sg lists this causes out-of-bounds reads when a later entry is
shorter than the first, and silent data loss when it is longer.
Single-sg or uniformly-sized lists happen to mask the issue.
Fixes: c8acd6aa6b ("dmaengine: mmp-pdma support")
Signed-off-by: Baineng Shou <shoubaineng@gmail.com>
Reviewed-by: Frank Li <Frank.Li@nxp.com>
Link: https://patch.msgid.link/20260910021652.1296640-1-shoubaineng@gmail.com
Signed-off-by: Vinod Koul <vkoul@kernel.org>
Using the DMA in cyclic mode modifies the hardware buffer descriptor
chain in xilinx_dma_prep_dma_cyclic so that the last descriptor used by
the cyclic transfer points back to the first descriptor, but it never
restores the original descriptor ring. This breaks using non-cyclic mode
after cyclic mode with an error like:
xilinx-vdma 86000000.dma: Channel 00000000354d5c8d has errors 100, cdr 6de40000 tdr 6de40400
The only way to get out of this error state is to rebuild the hardware
buffer descriptor ring by releasing and re-acquiring the channel.
Fix using non-cyclic mode after cyclic mode by always restoring the
original buffer descriptor ring in the same manner as it is set up by
xilinx_dma_alloc_chan_resources().
Fixes: 23059408b6 ("dmaengine: xilinx_dma: Fix race condition in the driver for multiple descriptor scenario")
Signed-off-by: Alex Bereza <alex@bereza.email>
Reviewed-by: Frank Li <Frank.Li@nxp.com>
Reviewed-by: Suraj Gupta <suraj.gupta2@amd.com>
Link: https://patch.msgid.link/20260817-fix-hw-buf-desc-after-cyclic-mode-v1-1-1fe47e701d6c@bereza.email
Link: https://patch.msgid.link/20260818-fix-hw-buf-desc-after-cyclic-mode-v2-1-530ff44c6a81@bereza.email
Signed-off-by: Vinod Koul <vkoul@kernel.org>
pxad_alloc_desc() was converted from
kzalloc(struct_size(sw_desc, hw_desc, nb_hw_desc), GFP_NOWAIT)
to kzalloc_flex(), which sets the __counted_by() counter sw_desc->nb_desc
itself - but only where the compiler has __builtin_counted_by_ref(), so
from gcc 15.1 or clang 22.1 on. The loop below it still increments
nb_desc, which makes it come out doubled there and correct elsewhere.
nb_desc is what pxad_free_desc() iterates over and what
set_updater_desc() indexes from, so set it explicitly and drop the
increment. The error path has to lower it to the number of descriptors
allocated so far, otherwise pxad_free_desc() would free entries that were
never allocated.
Fixes: 69050f8d6d ("treewide: Replace kmalloc with kmalloc_obj for non-scalar types")
Assisted-by: Claude:claude-opus-5
Signed-off-by: Sascha Hauer <s.hauer@pengutronix.de>
Reviewed-by: Frank Li <Frank.Li@nxp.com>
Link: https://lore.kernel.org/r/20260817-dmaengine-pxa-v1-1-850c215c1196@pengutronix.de
Link: https://patch.msgid.link/20260817-dmaengine-pxa-v2-1-f42ab0569a48@pengutronix.de
Signed-off-by: Vinod Koul <vkoul@kernel.org>
sun6i_dma_tx_status() calls vchan_find_desc() to look up the virtual
descriptor for a given cookie, before checking whether the pointer
vd is NULL:
vd = vchan_find_desc(&vchan->vc, cookie);
txd = to_sun6i_desc(&vd->tx); /* vd may be NULL here */
if (vd) {
for (lli = txd->v_lli; ...)
vchan_find_desc() returns NULL when the descriptor has already been
completed or is in-flight on a physical channel and no longer present
in the virtual channel's descriptor list. When vd is NULL,
to_sun6i_desc() is called unconditionally on &vd->tx before the NULL
check, which is undefined behaviour. Move the call inside the if (vd)
guard to ensure it is only reached with a valid pointer.
vd = vchan_find_desc(&vchan->vc, cookie);
if (vd) {
struct sun6i_desc *txd = to_sun6i_desc(&vd->tx);
for (lli = txd->v_lli; ...)
Fixes: 5558593087 ("dmaengine: sun6i: Add driver for the Allwinner A31 DMA controller")
Cc: stable@vger.kernel.org
Assisted-by: Claude:claude-sonnet-4-6
Signed-off-by: Christian Lugnberg <christian.lugnberg@soundtrack.io>
Reviewed-by: Frank Li <Frank.Li@nxp.com>
Link: https://patch.msgid.link/20260817135723.12807-3-christian.lugnberg@soundtrack.io
Signed-off-by: Vinod Koul <vkoul@kernel.org>
sun6i_get_chan_size() reads DMA_CHAN_LLI_ADDR and DMA_CHAN_CUR_CNT in two
separate readl() calls with no synchronisation between them:
pos = readl(pchan->base + DMA_CHAN_LLI_ADDR);
bytes = readl(pchan->base + DMA_CHAN_CUR_CNT);
DMA_CHAN_LLI_ADDR holds the physical address of the *next* descriptor the
engine will load once the current one completes. DMA_CHAN_CUR_CNT holds the
remaining byte count for the *current* descriptor. If the DMA engine
advances to the next LLI entry between the two reads, pos becomes stale: it
still points to what was the next descriptor at the time of the first read,
but that descriptor is now the current one and CUR_CNT reflects its initial
(full) byte count. The subsequent virtual-chain walk starts one entry too
early and accumulates an extra full period's worth of bytes into the
residue estimate.
Fix this by re-reading DMA_CHAN_LLI_ADDR after DMA_CHAN_CUR_CNT and
retrying if the value changed. This double-read pattern guarantees that
both registers were sampled during the same descriptor interval. The cost
is at most one extra readl() pair per call in the racy case, which occurs
only at descriptor boundaries (~every 2 ms) and is negligible.
Fixes: a90e173f3f ("dmaengine: sun6i: Add cyclic capability")
Cc: stable@vger.kernel.org
Assisted-by: Claude:claude-sonnet-4-6
Signed-off-by: Christian Lugnberg <christian.lugnberg@soundtrack.io>
Reviewed-by: Frank Li <Frank.Li@nxp.com>
Link: https://patch.msgid.link/20260817135723.12807-2-christian.lugnberg@soundtrack.io
Signed-off-by: Vinod Koul <vkoul@kernel.org>
xilinx_dma_alloc_chan_resources() builds a static ring of hardware
buffer descriptors once and the driver uses this ring throughout the
lifetime of a channel. This requires the allocation order of hardware
buffer descriptors from chan->free_seg_list to stay in sync with the
hardware buffer descriptor ring built at channel allocation time by
returning oldest descriptors to chan->free_seg_list first.
When chan->pending_list is not empty e.g. during
xilinx_dma_terminate_all() the chan->free_seg_list and the order of the
static hardware buffer descriptor ring get out of sync. Descriptors age
in this order: pending -> active -> done. So freeing pending_list first
returns the newest buffer descriptors to the chan->free_seg_list first
and thus breaks the order required by the static hardware buffer
descriptor ring. Then when the channel is reused, after a wrap around of
the free_seg_list the DMA will find a hardware buffer descriptor with a
length field that is still zeroed and stop with something like this:
xilinx-vdma 86000000.dma: Channel 000000003a21d7b8 has errors 10, cdr 6de4c000 tdr 6de4c000
After this no more descriptors are completed and a consumer potentially
blocks and waits forever. The only way to get out of this error state is
to rebuild the static hardware buffer descriptor ring and the
free_seg_list by releasing and re-acquiring the channel.
Fix the order in which hardware buffer descriptors are returned to
free_seg_list to ensure the mentioned requirement holds.
Fixes: 23059408b6 ("dmaengine: xilinx_dma: Fix race condition in the driver for multiple descriptor scenario")
Signed-off-by: Alex Bereza <alex@bereza.email>
Reviewed-by: Frank Li <Frank.Li@nxp.com>
Reviewed-by: Suraj Gupta <suraj.gupta2@amd.com>
Link: https://patch.msgid.link/20260817-fix-hw-buf-desc-reuse-v1-1-d79827a844c7@bereza.email
Signed-off-by: Vinod Koul <vkoul@kernel.org>
dma_issue_pending_all() walks the dma_device_list with
list_for_each_entry_rcu() under rcu_read_lock(). dma_device_release()
unlinks the device with list_del_rcu() and then calls
device->device_release() (which in many drivers, such as plx_dma.c,
directly calls kfree()).
Because there is no grace period between unlinking the device and
freeing it, concurrent RCU readers in dma_issue_pending_all() can
access the device after it has been freed.
The lockless walk originally relied on clients holding a dmaengine
reference to pin the provider module, and therefore the device, for as
long as they might traverse the list. Commit 8ad342a863 ("dmaengine:
Add reference counting to dma_device struct") decoupled the dma_device
lifetime from the module reference, so the device can now be released
while a reader is still walking the list.
Add synchronize_rcu() before the device is freed, so RCU readers are
guaranteed to have finished. Keep it unconditional: providers that do
not implement device_release() free the device themselves once
dma_async_device_unregister() returns. This call will delay for a grace
period with dma_list_mutex held, which is safe and only teardown path is
delayed.
Fixes: 2ba05622b8 ("dmaengine: provide a common 'issue_pending_all' implementation")
Suggested-by: Sashiko <sashiko-bot@kernel.org>
Link: https://sashiko.dev/#/patchset/20260526-dmaengine-kref-fix-v2-0-3df60afac01d@amd.com
Reviewed-by: Frank Li <Frank.Li@nxp.com>
Reviewed-by: Logan Gunthorpe <logang@deltatee.com>
Signed-off-by: Shivank Garg <shivankg@amd.com>
Link: https://patch.msgid.link/20260822-dmaengine-kref-fix-v5-4-d4a4ee47d927@amd.com
Signed-off-by: Vinod Koul <vkoul@kernel.org>
When dma_device_put() drops the last reference on chan->device->ref,
dma_device_release() runs and may free the dma_device along with its
channels.
dma_chan_put() then still reads chan->device->owner via
dma_chan_to_owner() for the trailing module_put(). KASAN catches it:
slab-use-after-free in dma_chan_put+0x3e6/0x4c0
Read of size 8 by task insmod/6319
Freed by task 6319:
kfree+0x225/0x470
dma_chan_put+0x395/0x4c0
dmaengine_put+0xf8/0x160
Cache the module owner in dma_chan_put() before the put so the trailing
module_put() does not need chan->device.
Fixes: 8ad342a863 ("dmaengine: Add reference counting to dma_device struct")
Suggested-by: Sashiko <sashiko-bot@kernel.org>
Link: https://sashiko.dev/#/patchset/20260518-dmaengine-kref-fix-v1-1-4d6125048fb7@amd.com
Reviewed-by: Frank Li <Frank.Li@nxp.com>
Reviewed-by: Logan Gunthorpe <logang@deltatee.com>
Signed-off-by: Shivank Garg <shivankg@amd.com>
Link: https://patch.msgid.link/20260822-dmaengine-kref-fix-v5-3-d4a4ee47d927@amd.com
Signed-off-by: Vinod Koul <vkoul@kernel.org>
dma_chan_get() takes chan->device->ref only on the slow path:
/* no kref on fast path */
if (chan->client_count) {
__module_get(owner);
chan->client_count++;
return 0;
}
if (!try_module_get(owner))
return -ENODEV;
if (!dma_device_get(chan->device)) { // calls kref_get_unless_zero()
dma_chan_put() drops the ref unconditionally, so every fast-path
get/put pair drops one extra device reference.
The bug fires when two conditions hold together: a non-private
provider has a persistent client holding chan->client_count > 0
and another client cycles dmaengine_get()/dmaengine_put().
When the kref hits zero, the subsequent dma_find_channel() returns
NULL even though the provider module is still loaded.
Fix this by dropping device->ref only on the last put, matching the
single slow-path get.
Fixes: 8ad342a863 ("dmaengine: Add reference counting to dma_device struct")
Reviewed-by: Frank Li <Frank.Li@nxp.com>
Reviewed-by: Logan Gunthorpe <logang@deltatee.com>
Signed-off-by: Shivank Garg <shivankg@amd.com>
Link: https://patch.msgid.link/20260822-dmaengine-kref-fix-v5-2-d4a4ee47d927@amd.com
Signed-off-by: Vinod Koul <vkoul@kernel.org>
Add dma_device_get() helper to match dma_device_put() to make code
symmetric. It wraps open-coded kref_get_unless_zero() and asserts that
dma_list_mutex is held, matching its put counterpart.
No functional change intended.
Suggested-by: Frank Li <Frank.Li@nxp.com>
Reviewed-by: Logan Gunthorpe <logang@deltatee.com>
Signed-off-by: Shivank Garg <shivankg@amd.com>
Link: https://patch.msgid.link/20260822-dmaengine-kref-fix-v5-1-d4a4ee47d927@amd.com
Signed-off-by: Vinod Koul <vkoul@kernel.org>
This is another run of the Coccinelle script for converting kmalloc()
family of allocations to kmalloc_obj() via the existing rules in
scripts/coccinelle/api/kmalloc_objs.cocci
This catches both the set of kmalloc() uses added since the first
kmalloc_obj() conversions in v7.0 and adds a large group missed in the
first pass due to Coccinelle not interacting well with the cleanup.h
scoped_...() family of macros[1]. I worked around this with spatch's
"--macro-file" argument to a file with all the scoped_...() macros mapped
to Coccinelle's YACFE_ITERATOR[2] as that was the closest viable control
flow indicator I could find.
Build tested allmodconfig on x86, arm64, arm, loongarch, mips, powerpc,
riscv, and s390 with no new warnings.
Link: https://lore.kernel.org/lkml/202609021314.8A9C0B8@keescook/ [1]
Link: https://github.com/coccinelle/coccinelle/blob/master/standard.h [2]
Signed-off-by: Kees Cook <kees+treewide@kernel.org>
pm_runtime_get_sync() increments a device's usage counter even when it
fails. sprd_dma_probe() currently jumps directly to controller clock
cleanup on that error, bypassing both pm_runtime_put_noidle() and
pm_runtime_disable(). This can happen if the preceding unchecked
pm_runtime_set_active() fails and the following runtime-resume attempt
also returns an error.
Enter the existing runtime-PM unwind path instead. This drops the
reference without idling the partially initialized device, disables
runtime PM, and then releases the controller clocks. The success path
and propagated error code are unchanged.
This issue was found by a static analysis checker and confirmed by manual
source review.
Fixes: 9b3b8171f7 ("dmaengine: sprd: Add Spreadtrum DMA driver")
Signed-off-by: Ruoyu Wang <ruoyuw560@gmail.com>
Reviewed-by: Frank Li <Frank.Li@nxp.com>
Reviewed-by: Baolin Wang <baolin.wang@linux.alibaba.com>
Link: https://patch.msgid.link/20260813153149.3953497-1-ruoyuw560@gmail.com
Signed-off-by: Vinod Koul <vkoul@kernel.org>
If devm_kcalloc() for rx_chn->flows fails in a channel request function,
the error path calls k3_udma_glue_release_rx_chn(), which dereferences
the NULL rx_chn->flows pointer in k3_udma_glue_release_rx_flow().
Skip the flow release loop in k3_udma_glue_release_rx_chn() when
rx_chn->flows is not allocated.
Found by Linux Verification Center (linuxtesting.org) with SVACE.
Fixes: d702419134 ("dmaengine: ti: k3-udma: Add glue layer for non DMAengine users")
Cc: stable@vger.kernel.org
Reported-by: Pavel Zhigulin <Pavel.Zhigulin@kaspersky.com>
Signed-off-by: Alexander Chesnokov <Alexander.Chesnokov@kaspersky.com>
Reviewed-by: Frank Li <Frank.Li@nxp.com>
Link: https://patch.msgid.link/20260812053426.3521589-1-Alexander.Chesnokov@kaspersky.com
Signed-off-by: Vinod Koul <vkoul@kernel.org>
Core:
- New API to combine configuration and preparation and users
New Support:
- Mediatek MT8189 SoC uart dma support
Updates:
- Designware dma driver flatten desc structures and simplify code,
interrupt-path groundwork changes, first part of PCI EP DMA support
- Updates to zynqmp_dma with runtime PM and device removal improvments
- Xilinx dma optimizations for AXIDMA and MCDMA channel management
-----BEGIN PGP SIGNATURE-----
iQIyBAABCgAdFiEE+vs47OPLdNbVcHzyfBQHDyUjg0cFAmqMgZEACgkQfBQHDyUj
g0e82w/4oLwj2QUjUO3QEO2l4E/RVS2/7UyCYKoJDAh66aM8Z3QjQ+lUvy+4wHHE
H8AtTpRzGEPpqqU+BckHEySf8mNhlA4ULhX17aPc+NKqYNlW8p2L63kPFYggI1ja
Z6/zVFrJDLwWTI0bSxAyJ6OoaObSXY0P6qpMcXKcqoHc981+EHMGrt/iVJM4Et/c
Q6nFMxIMgxYPI1VCMSrJqOBA4ZvymyWBisJPZf6SK0iH3PnaBnGWWWilpT8eUJxm
/MbVf2aUMetdK1+QVx1nNQUdb1eyAw50zFgbeDFDTXQ85Mfbvwh8XxxuT4wXJS5b
8MFwIEUq6iRgyjZYlykmFYBXw1A3SZWuWMUuMic4l13lw7dxk92wR31j99pIRaV3
g2iykz+gzQNQ5RNPuo/9j0s26FAmMNxbFW1RPAMnoVc33vZakV4ueaYtIy7s7mMJ
eDrE6YNDW24DWfWrSxkQGCW1qFGBUY20dXZ0sBmhUxpTLB0ubPZrWLvKKZe/qTUJ
wmrGNtTBLAYJMsGBm5VMybXwrQXa2Hvqa6KKFR2TdjpdcJwyWVHzRPpR+sJ7GfxZ
Lz59ODW3x1MnukRQS/T+RXGT42Tzk+qQQ+xK/yCrXq0VkxE3010n7SglTEEygGYP
Zk2eos4fARpXOUNqAxoKTVRyvQBxrIrHuJuQjAys2RhLHARuCQ==
=lUcr
-----END PGP SIGNATURE-----
Merge tag 'dmaengine-7.3-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/vkoul/dmaengine
Pull dmaengine updates from Vinod Koul:
"Core:
- New API to combine configuration and preparation and users
New hardware support:
- Mediatek MT8189 SoC uart dma support
Updates:
- Designware dma driver flatten desc structures and simplify code,
interrupt-path groundwork changes, first part of PCI EP DMA support
- Updates to zynqmp_dma with runtime PM and device removal
improvments
- Xilinx dma optimizations for AXIDMA and MCDMA channel management"
* tag 'dmaengine-7.3-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/vkoul/dmaengine: (73 commits)
dmaengine: dw-edma: Mark emulated IRQ as level-triggered
dmaengine: idxd: assign all engines to group 0 in IAA defaults
dmaengine: qcom_hidma: remove conditional return with no effect
dmaengine: qcom-bam-dma: fix autosuspend cleanup during removal
dmaengine: fsl-edma: tracing: no ptr dereference during log output
dmaengine: dw-edma: Program endpoint function numbers
dmaengine: dw-edma-pcie: Add chip flags to match data
dmaengine: dw-edma-pcie: Handle optional data blocks
dmaengine: dw-edma-pcie: Factor out descriptor block address lookup
dmaengine: dw-edma-pcie: Add register offset match flag
dmaengine: dw-edma-pcie: Add platform ops to match data
dmaengine: dw-edma-pcie: Rename vsec_data to dma_data
dmaengine: dw-edma-pcie: Add capability match data
dmaengine: dw-edma-pcie: Track non-LL mode in DMA data
dmaengine: dw-edma: Add partial channel ownership mode
dmaengine: dw-edma: Initialize IRQ data before requesting IRQs
dmaengine: dw-edma: Add core quiesce operations
dmaengine: dw-edma: Add per-channel interrupt routing control
dmaengine: dw-edma: Factor out HDMA interrupt setup helper
dmaengine: dw-edma: Defer channel IRQ handling to workqueue
...
-----BEGIN PGP SIGNATURE-----
iQJIBAABCgAyFiEEgMe7l+5h9hnxdsnuWYigwDrT+vwFAmqIy5UUHGJoZWxnYWFz
QGdvb2dsZS5jb20ACgkQWYigwDrT+vw79g//dJCdLjB7Hu/pYOkkZ1VGpQC4x8eu
RR2if7TxHOceOkzIzd9OB68NU3KLuBthyMRdcfZ5VjayiRKZHbY1NnQX8qzoXsPM
F10QbJ/JOhZF0bvtr5nkTS2659AkHfMgiEPAN1hN6M0gHzFEB0vNoYb9lgMRkdXm
3jxMX8tq1x8QlbpiTx5nmfHtbK40u8BI/zcpsBW6P8LHmRMhlEgLViPDNVNfup9p
OK4Ra/jeExIkODwVI5ngBJgtetcXs5jFPgAkbk+efjU32VSLUwIETj2l1JsfDh79
taj7XKEYReghFIvIUmm0vkNZU3CRvkMdoZQnep/HypxpPc5cJAFcHRpAb70QRgA5
CbHeFKfFr4D1fOXyUq0atbpb4O3wZdCRgaULDgRmP7TmLIY+8VaUv4wjYDi9vVW9
QVxxFmaydF4lRLsKGrUX5755QVMTAN7Hyqs984R/zi66WjF1MLPqXkaqOKZxrjj8
8oJ9HCLX+B1hHnIENN/D0kcc1YSsruCke2RF0FFDpc3N48Y0LOGyof5S7GUx2Z1+
MohLF/Y8TSsdmzo9a25OZ8Oyq+kyZ1spDYbpBmN/Pqu3GZBNBan7EsbY5pg0LVG+
NtQv7+oX7Q4VGmbDvxfGCwauA3ekaSfyIW0KJtk5y1xLJO2pAQn4QOBqUZtYtyJf
wu10ek23s7w9wbI=
=cNUI
-----END PGP SIGNATURE-----
Merge tag 'pci-v7.3-changes' of git://git.kernel.org/pub/scm/linux/kernel/git/pci/pci
Pull PCI updates from Bjorn Helgaas:
"Resource management:
- Add hotplug reservation only once (not at each level of the
hierarchy) so bridge windows don't grow more than necessary (Ilpo
Järvinen)
Driver binding:
- Rework device matching so device ID lifetime only needs to cover
the probe path since dynamic IDs can be removed at any time (Gary
Guo)
Error handling:
- Update mappings of AER errors to agent & layer and log them for
each individual error when multiple errors detected (Lukas Wunner)
- Log Error Source only once, not twice in separate messages (Lukas
Wunner)
- Emit TLP Log only for unmasked errors (Lukas Wunner)
- Support Advisory Non-Fatal Errors (Lukas Wunner)
- Allow DPC on all Downstream Ports, not just Root Ports, when OS
controls AER (Darshit Shah)
ASPM:
- Program the same ASPM Control values for every function of
multi-function devices, as recommended by the PCIe spec (Krishna
Chaitanya Chundru)
- Avoid L0s for Realtek RTS525A, where it causes an AER interrupt
storm (Max Lee)
- Avoid ASPM L0s, L1, and L1 PM Substates based on 'aspm-no-l0s',
'aspm-no-l1' [1], and 'aspm-no-l1ss' DT properties (Krishna
Chaitanya Chundru)
Power management:
- Allow D3 for native hotplug-capable Root Ports on non-x86 platforms
(we avoid D3 for these ports on x86 because some old platforms
didn't validate it) (Manivannan Sadhasivam)
- Allow portdrv to claim Ports even if they don't support services
(AER, PME, DPC, hotplug, etc) so it can do power management (Brian
Norris)
Power control:
- Add support for PCIe WAKE# interrupt when described via DT (Krishna
Chaitanya Chundru)
- For the TC9563 PCIe switch:
- Take a reference on the I2C adapter to avoid uninterruptible
hang when unloading an I2C module while in-use (Johan Hovold)
- Update DT binding and driver to restrict Tx Amplitude, DFE and
N_FTS to USP, DSP1 and DSP2 (Manivannan Sadhasivam)
- Power off only external-facing ports (DSP1, DSP2), leaving USP
and DSP3 (aka VDSP) powered up (Manivannan Sadhasivam)
- Move integrated MAC Endpoint out of the list of internal ports
and configure it separately (Manivannan Sadhasivam)
Virtualization:
- Add ACS quirk for Pericom PI7C9X2G608 switches (Tim Harvey)
- Fix a long-standing bug in the Intel PCH Root Port MPC ACS quirk
that didn't update the intended INTEL_MPC_REG_IRBNCE bit because it
used a 16-bit config write when a 32-bit write was intended
(Mohamad Raizudeen)
Procfs:
- Avoid spurious runtime PM wakeup on config space accesses that are
outside config space and fail before reaching PCI (Krzysztof
Wilczyński)
- Warn on user-space writes to kernel-exclusive config space regions,
as we already do for sysfs (Krzysztof Wilczyński)
- Check credentials of opener, not reader, for config space reads, as
we already do for sysfs (Krzysztof Wilczyński)
Sysfs:
- In pci_write_legacy_io(), avoid out-of-bounds reads from the user
buffer and fix incorrect ioport write data (1-byte writes on
little-endian powerpc, 2- and 4-byte writes on big-endian powerpc)
(Krzysztof Wilczyński)
- In pci_read_legacy_io(), fix incorrect ioport read data for 2- and
4-byte reads on big-endian powerpc (Krzysztof Wilczyński)
- Fix I/O port accessor argument order in Alpha pci_legacy_write()
(Krzysztof Wilczyński)
- Avoid spurious runtime PM wakeup on config space accesses that are
outside config space and fail before reaching PCI (Krzysztof
Wilczyński)
- Return -EINVAL, not -ENODEV, for mmap of I/O BAR that fails because
the arch doesn't support it, as we do for procfs (Krzysztof
Wilczyński)
- Check for LOCKDOWN_PCI_ACCESS for legacy_io and legacy_mem, as we
do for other config space accessors (Krzysztof Wilczyński)
Peer-to-peer DMA:
- Add Nvidia Vera Rubin to list of platforms that support P2PDMA
(Leon Romanovsky)
Endpoint framework:
- Check doorbell SUCCESS bit in pci_endpoint_test to avoid treating
some failures as successes (Niklas Cassel)
- Fail doorbell test when the trigger IRQ is missed (Niklas Cassel)
New native PCIe controller drivers:
- Add DT binding and driver for NVIDIA Tegra264 (Thierry Reding)
Native PCIe controllers:
- Use common wait time definitions for PCIe link monitoring instead
of defining driver-private duplicates (Thierry Reding)
Generic host bridge driver:
- Fix NULL pointer dereference that caused enumeration failures on
32-bit CAM systems (Steffen Persvold)
Amlogic Meson PCIe controller driver:
- Correct the PERST# GPIO state so it remains asserted until power
and REFCLK become stable to fix enumeration failure (Ronald
Claveau)
ASPEED PCIe controller driver:
- Switch to irq_domain_create_linear() so we can obsolete
irq_domain_add_linear() (Jiri Slaby)
Cadence PCIe controller driver:
- Add MODULE_DEVICE_TABLE to generate module aliases for OF-based
module autoloading (Pengpeng Hou)
- Add debugfs 'ltssm_status' file for LGA- and HPA-based Cadence
controllers (Hans Zhang)
- Support up to x4 (not x2) lanes for J200 (Takuma Fujiwara)
- Fix host/endpoint dependencies for cadence-plat driver to fix link
error when cadence-plat is built-in but the host or endpoint driver
is modular (Aksh Garg)
Freescale i.MX6 PCIe controller driver:
- Add imx6 intr/aer/pme interrupt lines for i.MX95 (Richard Zhu)
- Remove PERST# checking from pci_host_common_parse_port() so callers
can decide whether to fall back to legacy DT binding with PERST# in
the host bridge (Sherry Sun)
- Fix build issues when PCI_PWRCTRL_GENERIC or PCI_HOST_COMMON is a
module (Arnd Bergmann)
- Create pwrctrl devices only once by doing it from imx_pcie_probe()
instead of imx_pcie_host_init(), which is used during both probe
and resume (Sherry Sun)
- Use 'dw_pcie_rp->skip_pwrctrl_off' to avoid powering off devices
during suspend to preserve wakeup capability (Sherry Sun)
- Add runtime PM support for i.MX95 to allow dynamic power management
when the link is idle (Richard Zhu)
Intel VMD host bridge driver:
- Support device ID 0x28C1 and assume that BIOS has already
enumerated the hierarchy below VMD and stored bus range info for OS
to use (Nirmal Patel)
- Add support for VMCONFIG BUS_RESTRICT_CFG=3, which makes it
possible to enumerate downstream devices on Intel Arrow Lake-HX
systems and probably others (Ali Alaei)
- Pay attention to _OSC negotiation for VMD hierarchy only when
running on bare metal, not when running in a VM (Nirmal Patel)
- Add Nova Lake (NVL) and Dunlow (DNL) Device IDs (Szymon Durawa)
MediaTek PCIe controller driver:
- Add support for PCIe controller in EcoNet EN7528 and EN751221 SoCs
(Caleb James DeLisle)
MediaTek PCIe Gen3 controller driver:
- Add mediatek-gen3 'memory-region' for restricted DMA buffer
(Chen-Yu Tsai)
NVIDIA Tegra264 PCIe controller driver:
- Distinguish Tegra264 C0 PCIe controller for internal GPU from C1-C5
controllers so the unit address matches the first 'reg' entry
(Thierry Reding)
- Add Tegra264 Root Port stanzas to prepare for generic WAKE#
handling (Thierry Reding)
Qualcomm PCIe controller driver:
- Add IPQ9650 compatible with global interrupt (Kathiravan
Thirumoorthy)
- Add IPQ5210 compatible with IPQ9574 fallback (Varadarajan
Narayanan)
- Add DT binding and driver support for Hawi SoC (Matthew Leung)
- Skip PERST# GPIOs provided by downstream PCIe devices, which should
be handled by drivers of those devices (Manivannan Sadhasivam)
- Stop advertising Attention Button Present (no Qcom SoCs support
Attention Buttons) so pciehp can use Presence Detect Changed events
(Qiang Yu)
Renesas R-Car PCIe controller driver:
- Add rcar-gen4-pci-host optional 'msi-parent' for GIT ITS (Marek
Vasut)
- When MSI is enabled but iMSI-RX is not used, configure AXIINTC to
allow GIT ITS to handle MSI (Marek Vasut)
- Refactor GIC600 implementation to make it easier to add platforms
that only support 32-bit addressing (Marek Vasut)
- Add Renesas R-Car Gen4 S4/V4H/V4M to the list of GIC600
integrations that only support 32-bit addressing (Marek Vasut)
Renesas RZ/G3S PCIe controller driver:
- Add DT binding and driver support for RZ/V2H(P) SoC, which contains
two PCIe controllers, configured either as a single x4 link or two
independent x2 link controllers (Lad Prabhakar)
SpacemiT K1 PCIe controller driver:
- Add missing MODULE_DEVICE_TABLE() to generate module alias info for
OF-based module autoloading (Pengpeng Hou)
StarFive PCIe controller driver:
- Fix resource leaks on error paths in host_init() (Ali Tariq)
- Fix runtime PM handling and teardown ordering to avoid register
access while power or clocks are disabled (Ali Tariq)
- Check for runtime PM resume failure to avoid register access while
power or clocks are disabled (Ali Tariq)
Synopsys DesignWare PCIe controller driver:
- Add LECARC PMU IDs to the DWC RAS/DES VSEC list so it can take
advantage of the existing debugfs support for silicon debug, error
injection, and event counters (Brett Zhou)
- Factor pcie_valid_speed() and pci_bus_speed2lnkctl2() out of bwctrl
so they can be shared by the DWC core (Hans Zhang)
- Flush MSI writes from endpoint before unmapping the iATU, as we
already do for MSI-X writes (Niklas Cassel)
- Unmap MSI iATU window before mapping MSI-X window, to avoid a
subsequent MSI write using a disabled aperture and losing the
interrupt (Niklas Cassel)
- Change endpoint .pre_init() and .init() callbacks to return errors
and handle them (Marek Vasut)
UltraRISC PCIe controller driver:
- Add 'core', 'dbi', and 'aux' clocks to DT binding and manage them
in the driver (Jia Wang)
- Use module_platform_driver() since this may be built as a module,
though not removable because IRQs can't be safely disposed (Jia
Wang)
MicroSemi Switchtec management driver:
- Add Microchip PCI1008 device ID and include it in NTB DMA alias
quirk (Logan Gunthorpe)
Miscellaneous:
- Document how to write PCI Host Controller drivers (Manivannan
Sadhasivam)
- Fix typos in documentation (D'Orus Tsitera)
- Use %pe format specifier to print error pointers so we get symbolic
errname when available (Krzysztof Wilczyński)"
* tag 'pci-v7.3-changes' of git://git.kernel.org/pub/scm/linux/kernel/git/pci/pci: (124 commits)
PCI: vmd: Add Nova Lake (NVL) and Dunlow (DNL) Device IDs
PCI: tegra264: Add Tegra264 support
dt-bindings: PCI: tegra264: Switch to PCIe Root Port bindings
dt-bindings: PCI: tegra264: Strictly distinguish C0 from C1-C5
PCI/AER: Support Advisory Non-Fatal Errors
PCI: Fix 32-bit config write in Intel PCH Root Port MPC ACS quirk
PCI: dwc: Handle return value from endpoint .pre_init callback
PCI: dwc: Handle return value from endpoint .init callback
PCI: dwc: Add PCI ID for LECARC PCIe PMU
PCI/ASPM: Mask ASPM states based on Devicetree properties
PCI/ASPM: Disable/restore ASPM on every function for multi-function devices
Documentation: PCI: Document how to write PCI Host Controller drivers
PCI/ASPM: Use pcie_capability_clear_and_set_word() for ASPM disable/restore
PCI: Add support for PCIe WAKE# interrupt
PCI: Allow D3 for native hotplug-capable Root Ports on non-x86 platforms
dt-bindings: PCI: Correct white-space style
PCI/ASPM: Avoid L0s for Realtek RTS525A
PCI: ultrarisc: Use module_platform_driver()
PCI: ultrarisc: Get and enable DP1000 PCIe controller clocks
dt-bindings: PCI: ultrarisc: Add required DP1000 PCIe clocks
...
The interrupt-emulation virtual IRQ uses handle_level_irq(), but the IRQ
descriptor has not been marked with IRQ_LEVEL.
The interrupt emulation is somewhat unusual: the eDMA interrupt handlers
dispatch the virtual IRQ for every edma_int[] interrupt because software
cannot reliably tell an interrupt-emulation event from one caused solely
by DONE/ABORT status. If an interrupt arrives before the doorbell
handler is registered for the virtual IRQ, the IRQ core marks it
pending. When the IRQ is later started, check_irq_resend() treats it as
non-level and replays the pending interrupt, causing the newly
registered handler to run for a stale event.
Mark the virtual IRQ with IRQ_LEVEL so the stale pending state is
cleared without being replayed. This was observed in pci_endpoint_test
as two doorbell handler calls when the DMA-variant test ran before
DOORBELL_TEST.
Fixes: d9d5e1bdd1 ("dmaengine: dw-edma: Add virtual IRQ for interrupt-emulation doorbells")
Cc: stable@vger.kernel.org
Reported-by: Niklas Cassel <cassel@kernel.org>
Closes: https://lore.kernel.org/r/ampndLtU32ODmncX@ryzen
Tested-by: Niklas Cassel <cassel@kernel.org>
Signed-off-by: Koichiro Den <den@valinux.co.jp>
Reviewed-by: Frank Li <Frank.Li@nxp.com>
Reviewed-by: Manivannan Sadhasivam <mani@kernel.org>
Link: https://patch.msgid.link/20260730160701.3550710-1-den@valinux.co.jp
Signed-off-by: Vinod Koul <vkoul@kernel.org>
The IAA device defaults only assigned engine 0 to group 0, leaving
engines 1 through max_engines-1 unassigned (group_id = -1). This means
that by default only a single engine processed descriptors, limiting
throughput to one engine's capacity.
Assign all available engines to group 0 so that the full hardware
parallelism is used out of the box without requiring manual
accel-config setup.
Signed-off-by: Giovanni Cabiddu <giovanni.cabiddu@intel.com>
Signed-off-by: Vinicius Costa Gomes <vinicius.gomes@intel.com>
Reviewed-by: Dave Jiang <dave.jiang@intel.com>
Link: https://patch.msgid.link/20260805-iaa-crypto-fixes-zswap-v2-1-55c02694f499@intel.com
Signed-off-by: Vinod Koul <vkoul@kernel.org>
Both branches of the check return the same value, so the check has
no effect. Remove it and return the value directly.
This is the result of running the Coccinelle script from
scripts/coccinelle/misc/cond_return_no_effect.cocci.
Signed-off-by: Sang-Heon Jeon <ekffu200098@gmail.com>
Link: https://patch.msgid.link/20260723184538.3888637-20-ekffu200098@gmail.com
Signed-off-by: Vinod Koul <vkoul@kernel.org>
bam_dma_probe() calls pm_runtime_use_autosuspend(), but
bam_dma_remove() does not call the matching
pm_runtime_dont_use_autosuspend() when removing the device.
If the autosuspend delay is set to a negative value while autosuspend
is enabled, the runtime PM core increments usage_count to prevent
runtime suspend. Without calling pm_runtime_dont_use_autosuspend()
during teardown, this reference is not dropped and usage_count remains
unbalanced.
Add the missing pm_runtime_dont_use_autosuspend() call before forcing
the device into runtime suspend during removal.
This issue was found by manual code inspection.
Fixes: 7d2545599f ("dmaengine: qcom-bam-dma: Add pm_runtime support")
Signed-off-by: Guangshuo Li <lgs201920130244@gmail.com>
Reviewed-by: Bartosz Golaszewski <bartosz.golaszewski@oss.qualcomm.com>
Link: https://patch.msgid.link/20260808110642.2770355-1-lgs201920130244@gmail.com
Signed-off-by: Vinod Koul <vkoul@kernel.org>
The fsl edma events store a pointer to a struct fsl_edma_engine in the
ringbuffer and dereference it when a log entry is printed. At this time,
the pointer may no longer be valid.
Event injection can be used to trigger a crash:
$ cd /sys/kernel/tracing
$ echo 'value = 0' > events/fsl_edma/edma_writeb/inject
$ cat trace
The log output needs only edma->membase. Add a membase field at the end
of the event and use the new field for log output. Keep the existing
fields for backward compatibility.
Fixes: 11102d0c34 ("dmaengine: fsl-edma: add trace event support")
Cc: stable@vger.kernel.org
Reviewed-by: Steven Rostedt <rostedt@goodmis.org>
Signed-off-by: Martin Kaiser <martin@kaiser.cx>
Reviewed-by: Frank Li <Frank.Li@nxp.com>
Link: https://patch.msgid.link/20260718130024.341243-1-martin@kaiser.cx
Signed-off-by: Vinod Koul <vkoul@kernel.org>
The eDMA/HDMA transfers the driver issues carry a requester function
number in their TLPs, but nothing ever programs it: eDMA v0 leaves the
FUNC_NUM field of the channel control word zero and HDMA leaves the
per-channel func_num register at its reset value, so every transfer is
attributed to function 0. That is invisible in single-function setups,
but once the DMA block serves a non-zero endpoint function, its
requests must carry that function's number for the host to attribute
and translate them correctly.
Record the function number in the chip data (PCI_FUNC() of the probing
device for dw-edma-pcie) and program it per channel.
Endpoint-local chip instances keep func_no at 0, so transfers issued by
the endpoint-side driver remain PF0-attributed. Delegated channels are
programmed by the host-side dw-edma-pcie instance when it takes over the
channel, using that instance's PCI_FUNC().
Reviewed-by: Frank Li <Frank.Li@nxp.com>
Signed-off-by: Koichiro Den <den@valinux.co.jp>
Link: https://patch.msgid.link/20260721062815.4117887-15-den@valinux.co.jp
Signed-off-by: Vinod Koul <vkoul@kernel.org>
Allow PCI ID match data to pass dw_edma_chip flags into dw_edma_probe().
This keeps per-device policy in the match data instead of open-coding it
in probe().
Reviewed-by: Frank Li <Frank.Li@nxp.com>
Signed-off-by: Koichiro Den <den@valinux.co.jp>
Link: https://patch.msgid.link/20260721062815.4117887-14-den@valinux.co.jp
Signed-off-by: Vinod Koul <vkoul@kernel.org>
Skip data block BAR mapping and debug output when a channel has no data
block size. This lets future providers describe channels that only need
descriptor memory exposed.
No functional change intended for existing Synopsys EDDA and
AMD (Xilinx) MDB/CPM6 devices. Their static channel descriptions still
provide data block sizes where data block windows are used. A zero-sized
data block now means "not present" for future metadata providers.
Reviewed-by: Frank Li <Frank.Li@nxp.com>
Signed-off-by: Koichiro Den <den@valinux.co.jp>
Link: https://patch.msgid.link/20260721062815.4117887-13-den@valinux.co.jp
Signed-off-by: Vinod Koul <vkoul@kernel.org>
Add an optional physical address override to struct dw_edma_block and
use a helper to compute descriptor block addresses.
No functional change intended. Existing Synopsys EDDA and AMD (Xilinx)
MDB/CPM6 block descriptors leave the override unset, so the helper still
returns the same values as before.
Reviewed-by: Frank Li <Frank.Li@nxp.com>
Signed-off-by: Koichiro Den <den@valinux.co.jp>
Link: https://patch.msgid.link/20260721062815.4117887-12-den@valinux.co.jp
Signed-off-by: Vinod Koul <vkoul@kernel.org>
Add a match-data flag for devices whose DMA register block starts at an
offset inside the mapped BAR. Existing Synopsys EDDA and AMD (Xilinx)
MDB/CPM6 matches keep using the BAR mapping base directly.
The offsets handled here come from static, trusted match data. Later
metadata-derived offsets are validated when the endpoint DMA metadata
path parses them at that trust boundary.
No functional change intended.
Reviewed-by: Frank Li <Frank.Li@nxp.com>
Signed-off-by: Koichiro Den <den@valinux.co.jp>
Link: https://patch.msgid.link/20260721062815.4117887-11-den@valinux.co.jp
Signed-off-by: Vinod Koul <vkoul@kernel.org>
Move the platform ops pointer into match data. Existing EDDA/MDB/CPM6
matches keep using dw_edma_pcie_plat_ops.
No functional changes intended.
Suggested-by: Frank Li <Frank.Li@nxp.com>
Reviewed-by: Frank Li <Frank.Li@nxp.com>
Signed-off-by: Koichiro Den <den@valinux.co.jp>
Link: https://patch.msgid.link/20260721062815.4117887-10-den@valinux.co.jp
Signed-off-by: Vinod Koul <vkoul@kernel.org>
dw_edma_pcie_probe() now obtains DMA layout data through device-specific
capability callbacks, not only from PCIe Vendor-Specific Extended
Capabilities. Rename the local data copy from vsec_data to dma_data
before adding endpoint DMA BAR metadata discovery, which does not rely
on VSEC.
No functional change intended.
Reviewed-by: Frank Li <Frank.Li@nxp.com>
Signed-off-by: Koichiro Den <den@valinux.co.jp>
Link: https://patch.msgid.link/20260721062815.4117887-9-den@valinux.co.jp
Signed-off-by: Vinod Koul <vkoul@kernel.org>
Move device-specific capability parsing behind per-device match data.
The existing probe path mixes two decisions: which static template a PCI
ID uses, and which device-specific capability parser adjusts that
template. Split those decisions so device-specific discovery can be
added through match data instead of adding more vendor checks to
dw_edma_pcie_probe().
No functional change is intended for the existing Synopsys EDDA and
AMD (Xilinx) MDB/CPM6 matches. They still copy the same static template
data and run the same capability parsing logic before BAR mapping. The
AMD (Xilinx) MDB/CPM6 entries also keep using endpoint memory physical
addresses for descriptor windows through a new match-data flag.
Suggested-by: Frank Li <Frank.Li@nxp.com>
Reviewed-by: Frank Li <Frank.Li@nxp.com>
Signed-off-by: Koichiro Den <den@valinux.co.jp>
Link: https://patch.msgid.link/20260721062815.4117887-8-den@valinux.co.jp
Signed-off-by: Vinod Koul <vkoul@kernel.org>
The dw-edma-pcie driver copies static template data into a mutable
dw_edma_pcie_data instance before applying capability-derived updates.
Keep the derived non-LL mode in that copy as well, instead of only
tracking it in a local variable in dw_edma_pcie_probe().
This prepares for keeping capability parsing behind match data without a
separate non-LL output parameter.
No functional change intended.
Suggested-by: Frank Li <Frank.Li@nxp.com>
Reviewed-by: Frank Li <Frank.Li@nxp.com>
Signed-off-by: Koichiro Den <den@valinux.co.jp>
Link: https://patch.msgid.link/20260721062815.4117887-7-den@valinux.co.jp
Signed-off-by: Vinod Koul <vkoul@kernel.org>
A DesignWare eDMA instance may represent only a subset of channels that
is also initialized by another OS instance, such as an endpoint-side OS.
Add a partial ownership flag for instances that must preserve
controller-wide state owned by that peer.
In partial ownership mode, dw-edma skips the initial core reset and uses
the limited quiesce path in probe() and remove() instead of the full
core-off path. The flag also makes the driver validate the ownership
granularity required by each register layout before registering
channels.
Probe fails if the limited quiesce cannot stop its resources; remove
reports the error after completing the remaining driver teardown.
Partial instances also skip interrupt-emulation doorbell allocation: the
emulated doorbell is a controller-level resource, and a partial owner
must not claim it on behalf of the whole block.
For EDMA_MF_EDMA_UNROLL and EDMA_MF_HDMA_COMPAT, the driver programs
per-direction registers, such as DMA_{WRITE,READ}_INT_MASK_OFF and
DMA_{WRITE,READ}_INT_CLEAR_OFF. These register layouts have at most
EDMA_MAX_{WR,RD}_CH channels per direction, so the capped hardware
channel count still represents the whole direction. A partial instance
can therefore expose write or read channels only if it owns every
channel in that direction; otherwise two OS instances could update the
same direction-wide registers without a shared locking protocol.
In contrast, HDMA native uses per-channel registers, so it can be owned
at channel granularity.
Signed-off-by: Koichiro Den <den@valinux.co.jp>
Link: https://patch.msgid.link/20260721062815.4117887-6-den@valinux.co.jp
Signed-off-by: Vinod Koul <vkoul@kernel.org>
dw_edma_irq_request() passes struct dw_edma_irq to request_irq() before
dw_edma_channel_setup() fills the back pointer. A shared interrupt can
therefore enter the handler with dw_irq->dw still NULL, leading to a
NULL pointer dereference.
Set the back pointer before installing each handler.
Fixes: e63d79d1ff ("dmaengine: Add Synopsys eDMA IP core driver")
Cc: stable@vger.kernel.org
Reviewed-by: Frank Li <Frank.Li@nxp.com>
Signed-off-by: Koichiro Den <den@valinux.co.jp>
Link: https://patch.msgid.link/20260721062815.4117887-5-den@valinux.co.jp
Signed-off-by: Vinod Koul <vkoul@kernel.org>
Add core operations that quiesce only the resources represented by a
dw-edma instance, separate from the existing full controller off path.
For unrolled eDMA, quiesce masks the direction, disables ENGINE_EN,
waits for it to read back clear, then clears pending interrupt status.
If ENGINE_EN does not clear, return the timeout to the caller.
HDMA compatibility mode does not implement ENGINE_EN, so quiesce clears
CHi_PWR_EN for each represented channel instead. Both register maps
share interrupt control per direction, so quiescing one channel
quiesces the whole direction. Callers must own that direction and stop
the peer from programming it first.
HDMA native has per-channel registers and can quiesce the represented
channel directly.
No caller is added yet, so this is a no-functional-change preparation
for delegated channel reclaim and partial-owned remove paths.
Signed-off-by: Koichiro Den <den@valinux.co.jp>
Link: https://patch.msgid.link/20260721062815.4117887-4-den@valinux.co.jp
Signed-off-by: Vinod Koul <vkoul@kernel.org>
DesignWare eDMA can signal completion locally through edma_int[] and
remotely through IMWr/MSI. When channels are delegated to a remote
frontend, the local endpoint side and the remote host side must not both
service the same DONE/ABORT status.
Add channel interrupt routing state and initialize it from the controller
instance configuration. Update the eDMA and HDMA native paths so
linked-list interrupt generation, HDMA non-linked-list interrupt enables,
and DONE/ABORT masking follow the selected mode. For HDMA native
non-linked-list channels, keep the local stop/abort enables set so status
is latched. In remote mode, also enable remote signaling and mask the local
interrupt pins.
Keep the existing dw-edma-pcie host-side instances in remote interrupt
routing mode so their IMWr/MSI completion model remains unchanged after
local routing becomes the zero value.
Note:
- The routing mode describes where a channel should report completion.
It does not by itself say whether this dw-edma instance owns the
interrupt status. A local instance must ignore remote-only channels,
and a remote instance must ignore local-only channels, even if such
interrupts are unexpectedly delivered. Otherwise the non-owner side
could steal the interrupt from the owner by clearing shared DONE/ABORT
status.
Cc: Devendra K Verma <devendra.verma@amd.com>
Suggested-by: Frank Li <Frank.Li@nxp.com>
Signed-off-by: Koichiro Den <den@valinux.co.jp>
Link: https://patch.msgid.link/20260721062815.4117887-3-den@valinux.co.jp
Signed-off-by: Vinod Koul <vkoul@kernel.org>
The HDMA linked-list and non-linked-list start paths both program the
stop/abort interrupt setup register using the same local/remote enable
policy. Only the interrupt-mask handling differs by transfer mode.
Factor the common setup into dw_hdma_v0_core_int_setup() before adding
per-channel interrupt routing support. No functional change intended.
Suggested-by: Frank Li <Frank.Li@nxp.com>
Signed-off-by: Koichiro Den <den@valinux.co.jp>
Link: https://patch.msgid.link/20260721062815.4117887-2-den@valinux.co.jp
Signed-off-by: Vinod Koul <vkoul@kernel.org>
On some SoCs (e.g. R-Car S4) the endpoint-side eDMA raises a single
fixed SPI that is hardwired to CPU0 and covers every read and write
channel. Handling channel events directly in that hard IRQ context
serializes the completion processing of all channels on one CPU:
descriptor recycling and refill, client callbacks (the vchan tasklet
runs on the scheduling CPU) and the doorbell writes all funnel through
CPU0, while the handler additionally spins on each channel's vc.lock.
Especially under heavy multichannel load, this contention becomes a
performance bottleneck.
Keep the hard IRQ handler minimal: clear the status, dispatch channel
events, and defer per-channel processing to work items. A work item per
channel preserves ordering while allowing different channels to run in
parallel on any CPU.
Reviewed-by: Frank Li <Frank.Li@nxp.com>
Signed-off-by: Koichiro Den <den@valinux.co.jp>
Link: https://patch.msgid.link/20260717180639.2643243-10-den@valinux.co.jp
Signed-off-by: Vinod Koul <vkoul@kernel.org>
The v0 interrupt handler reads the interrupt status register twice per
invocation, once through the DONE accessor and once through the ABORT
accessor, although both fields live in the same 32-bit register. On
remote setups (dw-edma-pcie) each read is a non-posted round trip across
the PCIe link costing on the order of a microsecond, and with one
completion interrupt per element the duplicate adds up. As an example,
profiling the R-Car S4 remote path put the handler at ~7us per
invocation, dominated by such reads.
Read the register once and derive the DONE and ABORT views from the
snapshot. No abort is lost to this because the pass only clears status
bits it observed, so an abort raised after the snapshot keeps its status
and its own interrupt delivery brings it to the next pass. An abort on
an observed channel cannot race the clear either. Software can restart
the halted channel only after abort() runs, and abort() is called after
dw_edma_v0_core_clear_abort_int().
Reviewed-by: Frank Li <Frank.Li@nxp.com>
Signed-off-by: Koichiro Den <den@valinux.co.jp>
Link: https://patch.msgid.link/20260717180639.2643243-9-den@valinux.co.jp
Signed-off-by: Vinod Koul <vkoul@kernel.org>
dw_edma_pcie enables the PCI device with pcim_enable_device(), so IRQ
vectors allocated by pci_alloc_irq_vectors() are released by
pcim_msi_release() on device release. The driver should not call
pci_free_irq_vectors() manually.
Drop the redundant remove-time cleanup and rely on the managed PCI
device lifetime instead, as documented by commit 03e4905402 ("PCI/MSI:
Clarify pci_free_irq_vectors() usage for managed devices").
Reviewed-by: Frank Li <Frank.Li@nxp.com>
Signed-off-by: Koichiro Den <den@valinux.co.jp>
Link: https://patch.msgid.link/20260717180639.2643243-8-den@valinux.co.jp
Signed-off-by: Vinod Koul <vkoul@kernel.org>
terminate_all() can finish immediately when the channel is unconfigured,
paused, idle, or already stopped in hardware. A pending PAUSE request can
survive these paths and block issue_pending() even after termination.
Clear the request whenever termination leaves the channel idle. A running
channel keeps its STOP request until the interrupt handler consumes it.
Fixes: e63d79d1ff ("dmaengine: Add Synopsys eDMA IP core driver")
Reviewed-by: Frank Li <Frank.Li@nxp.com>
Signed-off-by: Koichiro Den <den@valinux.co.jp>
Link: https://patch.msgid.link/20260717180639.2643243-7-den@valinux.co.jp
Signed-off-by: Vinod Koul <vkoul@kernel.org>
pause() and resume() read and update channel state without holding vc.lock,
while the interrupt handlers update the same state under it. Take the same
lock around those state checks so that request, status, and configured stay
consistent.
For example, pause() can observe EDMA_ST_BUSY right before the interrupt
handler completes the final descriptor and moves the channel to
EDMA_ST_IDLE, and then record EDMA_REQ_PAUSE on an already idle channel. No
further interrupt will acknowledge the request, and since issue_pending()
requires EDMA_REQ_NONE, the channel is wedged for good: terminate_all()
leaves the stale request behind, so even reconfiguring the channel does not
recover it.
issue_pending() already runs under vc.lock, but it tests configured before
taking it. Move that test under the lock as well, so configured, request,
and status are evaluated as one channel-state snapshot.
Fixes: e63d79d1ff ("dmaengine: Add Synopsys eDMA IP core driver")
Reviewed-by: Frank Li <Frank.Li@nxp.com>
Signed-off-by: Koichiro Den <den@valinux.co.jp>
Link: https://patch.msgid.link/20260717180639.2643243-6-den@valinux.co.jp
Signed-off-by: Vinod Koul <vkoul@kernel.org>
If PAUSE is requested while the final burst of a descriptor is in
flight, the DONE interrupt takes the PAUSE path without checking whether
the descriptor has been depleted. The depleted descriptor remains on the
issued list and the channel enters EDMA_ST_PAUSE.
On resume, dw_edma_start_transfer() can select that depleted descriptor
again even though no burst remains, leaving the channel in an invalid
busy state.
Check for descriptor completion before acknowledging PAUSE. If there is
no work to start on resume, leave the channel idle. Also ignore DONE
interrupts while the channel is paused so a stale or repeated interrupt
cannot change its state or start queued work.
Fixes: e63d79d1ff ("dmaengine: Add Synopsys eDMA IP core driver")
Cc: stable@vger.kernel.org
Reviewed-by: Frank Li <Frank.Li@nxp.com>
Signed-off-by: Koichiro Den <den@valinux.co.jp>
Link: https://patch.msgid.link/20260717180639.2643243-5-den@valinux.co.jp
Signed-off-by: Vinod Koul <vkoul@kernel.org>
dw_edma_abort_interrupt() drops vc.lock before changing request and
status. issue_pending() can acquire the lock in that small window,
observe the old busy state, and skip starting queued descriptors. Then
the abort handler overwrites the channel status as idle, leaving the new
descriptors stranded for good.
Keep descriptor completion and the state transition in the same critical
section.
Fixes: e63d79d1ff ("dmaengine: Add Synopsys eDMA IP core driver")
Reviewed-by: Frank Li <Frank.Li@nxp.com>
Signed-off-by: Koichiro Den <den@valinux.co.jp>
Link: https://patch.msgid.link/20260717180639.2643243-4-den@valinux.co.jp
Signed-off-by: Vinod Koul <vkoul@kernel.org>
The DMA Engine client documentation says in the "Terminate APIs" section
of Documentation/driver-api/dmaengine/client.rst:
"No callback functions will be called for any incomplete transfers."
dw-edma instead calls vchan_cookie_complete() when a deferred STOP reaches
the interrupt handler. This schedules a callback for the active descriptor
and leaves other issued or submitted descriptors queued. A late callback
after dmaengine_terminate_sync() can dereference client state that has
already been freed, while leftover descriptors may later restart into
reused buffers or leak.
Move all issued and submitted descriptors to the terminated list whenever
termination completes. For a pending STOP, do this from both the DONE and
ABORT paths. Complete their cookies in order without scheduling callbacks.
A STOP can remain pending until the running transfer raises an
interrupt. Make device_synchronize() wait for such a pending STOP to
complete before releasing terminated descriptors. Reuse it from
free_chan_resources(), then release the remaining virt-dma resources.
Sleep instead of busy-polling while waiting, and warn if the existing
timeout expires.
Fixes: e63d79d1ff ("dmaengine: Add Synopsys eDMA IP core driver")
Reviewed-by: Frank Li <Frank.Li@nxp.com>
Signed-off-by: Koichiro Den <den@valinux.co.jp>
Link: https://patch.msgid.link/20260717180639.2643243-3-den@valinux.co.jp
Signed-off-by: Vinod Koul <vkoul@kernel.org>
GET_CH_32() takes the direction before the channel ID, but
dw_hdma_v0_core_ch_status() passed them in the opposite order. This can
make the status callback read another HDMA channel status register.
Use the same argument order as the other HDMA register accesses.
Fixes: e74c39573d ("dmaengine: dw-edma: Add support for native HDMA")
Cc: stable@vger.kernel.org
Reviewed-by: Frank Li <Frank.Li@nxp.com>
Signed-off-by: Koichiro Den <den@valinux.co.jp>
Link: https://patch.msgid.link/20260717180639.2643243-2-den@valinux.co.jp
Signed-off-by: Vinod Koul <vkoul@kernel.org>
'struct dma_descriptor_metadata_ops' in not modified in these drivers.
Constifying these structures moves some data to a read-only section, so
increases overall security, especially when the structure holds some
function pointers.
On a x86_64, with allmodconfig, as an example:
Before:
======
text data bss dec hex filename
120635 21584 64 142283 22bcb drivers/dma/xilinx/xilinx_dma.o
After:
=====
text data bss dec hex filename
120699 21520 64 142283 22bcb drivers/dma/xilinx/xilinx_dma.o
Signed-off-by: Christophe JAILLET <christophe.jaillet@wanadoo.fr>
Reviewed-by: Frank Li <Frank.Li@nxp.com>
Reviewed-by: Sai Sree Kartheek Adivi <s-adivi@ti.com>
Reviewed-by: Manivannan Sadhasivam <mani@kernel.org>
Reviewed-by: Radhey Shyam Pandey <radhey.shyam.pandey@amd.com>
Link: https://patch.msgid.link/b0a22171f3ed68e156a2fa84383e99c23ec6b2ff.1784037977.git.christophe.jaillet@wanadoo.fr
Signed-off-by: Vinod Koul <vkoul@kernel.org>
dma_request_chan() assumes both @dev and @name are valid, but neither is
checked before use.
dev is dereferenced immediately via dev_fwnode(), which accesses
dev->of_node or dev->fwnode without checking for NULL. Likewise, if name is
NULL and the OF/ACPI lookup does not succeed, the legacy filter-map path
eventually passes it to strcmp(), resulting in a NULL pointer dereference.
These are caller bugs rather than normal lookup failures, so add a
WARN_ON() at function entry to catch invalid arguments early during
development instead of crashing later.
No functional change for valid callers.
Signed-off-by: bui duc phuc <phucduc.bui@gmail.com>
Reviewed-by: Frank Li <Frank.Li@nxp.com>
Link: https://patch.msgid.link/20260716052758.23465-1-phucduc.bui@gmail.com
Signed-off-by: Vinod Koul <vkoul@kernel.org>
Replace the open-coded platform_get_resource() plus devm_ioremap()
sequence with devm_platform_ioremap_resource(), which fetches the
resource, requests the region and maps it in one call. Switch the error
check to IS_ERR()/PTR_ERR() and drop the now-unused struct resource
pointer.
The raideng node has a single reg region (0x320000, 0x10000); the
job-queue/ring children are separate OF devices probed independently, so
the region reservation added by devm_ioremap_resource() is exclusive and
does not introduce overlap failures.
Assisted-by: opencode:hy3-free
Signed-off-by: Rosen Penev <rosenp@gmail.com>
Reviewed-by: Frank Li <Frank.Li@nxp.com>
Link: https://patch.msgid.link/20260716202949.677290-5-rosenp@gmail.com
Signed-off-by: Vinod Koul <vkoul@kernel.org>
