This is another run of the Coccinelle script for converting kmalloc()
family of allocations to kmalloc_obj() via the existing rules in
scripts/coccinelle/api/kmalloc_objs.cocci
This catches both the set of kmalloc() uses added since the first
kmalloc_obj() conversions in v7.0 and adds a large group missed in the
first pass due to Coccinelle not interacting well with the cleanup.h
scoped_...() family of macros[1]. I worked around this with spatch's
"--macro-file" argument to a file with all the scoped_...() macros mapped
to Coccinelle's YACFE_ITERATOR[2] as that was the closest viable control
flow indicator I could find.
Build tested allmodconfig on x86, arm64, arm, loongarch, mips, powerpc,
riscv, and s390 with no new warnings.
Link: https://lore.kernel.org/lkml/202609021314.8A9C0B8@keescook/ [1]
Link: https://github.com/coccinelle/coccinelle/blob/master/standard.h [2]
Signed-off-by: Kees Cook <kees+treewide@kernel.org>
- ARM SMMUv2:
- Device-tree binding updates for Qualcomm Eliza, Maili, Shikra and
IPQ9650 SoCs.
- Add support for Qualcomm SM8450.
- Numerous fixes for lifetime and ordering issues found by Sashiko in
the Qualcomm driver.
- ARM SMMUv3:
- Fix interrupt type in device-tree binding example for NVIDIA CMDQV.
- Numerous fixes for issues identified by Sashiko in the NVIDIA CMDQV
driver.
- Work around TLB erratum T264-SMMU-3 on Tegra264 by repeating the
invalidation sequence.
- Add support for HAFT (hardware access flag in table entries) when
using SVA.
- Probe for 52-bit addressing with a page size smaller than 64k ('DS')
but don't do anything with it for now.
- Minor driver improvements (remove sort_nonatomic(), use
readl_relaxed_poll_timeout_atomic(), fix IOPF teardown ordering).
- Intel VT-d:
- Consolidation of complex enablement logic into a clean, priority-based
state machine.
- Support for the DMA_REMAP_OPT_OUT flag from the VT-d v5.2
specification.
- An update to cache_tag_flush_devtlb_psi() to use full-range constants
instead of modifying shared variables for CACHE_TAG_NESTING_DEVTLB.
- A fix for the UCTP context-table slot when copying root entries.
- Fixes for several pre-existing issues reported by Sashiko.
- General code cleanup and refinement.
- AMD IOMMU:
- Add SNP page-mode-0 support, enabling passthrough, v2 DMA page
tables and host SVA on supporting systems.
- Fix invalid PPR handling, COMPLETE_PPR responses and guest-mode
reporting.
- Improve Southbridge IOAPIC validation and remove the dependency on
hard-coded device IDs.
- Fix PCI-device lifetime, debugfs and diagnostic issues.
- IOMMU core and IOMMUFD:
- Restore serialization of the shared MSI-page list.
- Fix SVA-handle publication and several IOMMUFD reference and error
path leaks.
- Return the expected zero result for invalid generic page-table
translations.
- Allocate per-CPU IOVA magazines lazily to reduce memory use on
large systems.
- PCI ATS:
- Make VF support checks account for the associated PF and validate
that VF and PF Smallest Translation Unit settings agree.
- Platform drivers:
- Fix Qualcomm runtime-PM, probe unwind, fault reporting and page
table initialization races.
- Rework Rockchip state handling and fix clock, probe and stale-fault
handling.
- Fix smaller issues in the MSM and MediaTek drivers.
- Device-tree bindings:
- Add new Qualcomm SMMU compatibles, convert the OMAP IOMMU binding
to YAML, and fix the Tegra264 CMDQV interrupt example.
- Various smaller cleanups, documentation fixes and a Rust IOMMU
safety/readability improvement.
-----BEGIN PGP SIGNATURE-----
iQIzBAABCgAdFiEEr9jSbILcajRFYWYyK/BELZcBGuMFAmqDJowACgkQK/BELZcB
GuOPNw//aE9PbA4JWkYou6W2kVQnCowxvX87yE0ZDA6WAr9dOGJh15ZgIqdBMZEo
L28ZtaaH8BXM/DmMuQwVMinuvQ4jq5PcFbgu+erfrN/1PpT9fSE8eLPKX63b3fng
Qgv2gf2lAW6aTmMr8N8XKKlZUOLKfFPjOKzeqz77gTu2Z/FspEvM14QgoXmHum68
xtaEtrV11Do69CgGUCtXMunjpYQ3PHwQafjcvlsYaczWZ/NjTU/snjAWaqgpVBHh
1A8tdwbR5CEsLEHbFLeeQMuZJlEsTedv9lE95QZSlHsoqEkrrgkC930w+CaK+a3C
rd5chPEBwltz6TmzIoAF8PubzANiUNCZXnFL7U3GTkOLqwfr3l0T8lO0NWDhVqU/
L3jSGbFc/QOqVH4nygdh28rh7DHfhkpa1NNHtbFgNdl2W5UDseCnok/2MAvn8+wV
gflGhtsLLxbNis9FMS+w7QTOgHkJDq2eWCJp+uaDtZtY2IsVEy9Qi5dxS1VQ9i9T
jzVz6gsDdP7uAJBXByZcns7acjC6q2X2iLFhcU+mAk2H68rFZj411WEz/eRw8xvi
RGA8nY9DCVuiIo2NR6dE0r1DhTzI+eJCS/F3xeKyjxDNM6P4EMi53PvKAaIL/wXt
JF7FsfFlmlSgGeDCoin6CRHBvb41drpLwczs6+qURmL1X9iNi9s=
=V5FT
-----END PGP SIGNATURE-----
Merge tag 'iommu-updates-v7.3' of git://git.kernel.org/pub/scm/linux/kernel/git/iommu/linux
Pull iommu updates from Joerg Roedel:
"ARM SMMUv2:
- Device-tree binding updates for Qualcomm Eliza, Maili, Shikra and
IPQ9650 SoCs
- Add support for Qualcomm SM8450
- Numerous fixes for lifetime and ordering issues found by Sashiko in
the Qualcomm driver
ARM SMMUv3:
- Fix interrupt type in device-tree binding example for NVIDIA CMDQV
- Numerous fixes for issues identified by Sashiko in the NVIDIA CMDQV
driver
- Work around TLB erratum T264-SMMU-3 on Tegra264 by repeating the
invalidation sequence
- Add support for HAFT (hardware access flag in table entries) when
using SVA
- Probe for 52-bit addressing with a page size smaller than 64k
('DS') but don't do anything with it for now
- Minor driver improvements (remove sort_nonatomic(), use
readl_relaxed_poll_timeout_atomic(), fix IOPF teardown ordering)
Intel VT-d:
- Consolidation of complex enablement logic into a clean,
priority-based state machine
- Support for the DMA_REMAP_OPT_OUT flag from the VT-d v5.2
specification
- An update to cache_tag_flush_devtlb_psi() to use full-range
constants instead of modifying shared variables for
CACHE_TAG_NESTING_DEVTLB
- A fix for the UCTP context-table slot when copying root entries
- Fixes for several pre-existing issues reported by Sashiko
- General code cleanup and refinement
AMD IOMMU:
- Add SNP page-mode-0 support, enabling passthrough, v2 DMA page
tables and host SVA on supporting systems
- Fix invalid PPR handling, COMPLETE_PPR responses and guest-mode
reporting
- Improve Southbridge IOAPIC validation and remove the dependency on
hard-coded device IDs
- Fix PCI-device lifetime, debugfs and diagnostic issues
IOMMU core and IOMMUFD:
- Restore serialization of the shared MSI-page list
- Fix SVA-handle publication and several IOMMUFD reference and error
path leaks
- Return the expected zero result for invalid generic page-table
translations
- Allocate per-CPU IOVA magazines lazily to reduce memory use on
large systems
PCI ATS:
- Make VF support checks account for the associated PF and validate
that VF and PF Smallest Translation Unit settings agree
Platform drivers:
- Fix Qualcomm runtime-PM, probe unwind, fault reporting and page
table initialization races
- Rework Rockchip state handling and fix clock, probe and stale-fault
handling
- Fix smaller issues in the MSM and MediaTek drivers
Device-tree bindings:
- Add new Qualcomm SMMU compatibles, convert the OMAP IOMMU binding
to YAML, and fix the Tegra264 CMDQV interrupt example
Various smaller cleanups, documentation fixes and a Rust IOMMU
safety/readability improvement"
* tag 'iommu-updates-v7.3' of git://git.kernel.org/pub/scm/linux/kernel/git/iommu/linux: (93 commits)
iommu/amd: Add SNP page mode 0 support
iommu/amd: Fix GN bit setting in COMPLETE_PPR_REQUEST command
iommu/amd: Rate limit INVALID_PPR_REQUEST error logging
iommu/amd: Fix missing CMD_COMPLETE_PPR response for invalid PPR requests
iommu/amd: Introduce PPR_TAG_LAST_PAGE() macro
iommu/amd: Fix incorrect device ID in invalid PASID error message
iommu/vt-d: Flush context cache with correct SID when tearing down aliases
iommu/vt-d: Tear down scalable-mode context on probe failure
iommu/vt-d: Fix iopf_refcount leak on RID domain replacement
iommu/vt-d: Clear Present bit before tearing down copied context entry
iommu/vt-d: Fix copied_tables bitmap leak on error in copy_translation_tables
iommu/vt-d: Cache max domain ID to avoid redundant calculation
iommu/vt-d: Support the new DMA_REMAP_OPT_OUT flag bit
iommu/vt-d: Remove dmar_disabled
iommu/vt-d: Remove the 'force_on' variable
iommu/vt-d: Call dmar_can_force_on() for tboot opt-in
iommu/vt-d: Use dmar_can_force_on() for platform opt-in
iommu/vt-d: Consolidate dmar policy management and force_on logic
iommu/vt-d: Remove dead code when CONFIG_INTEL_IOMMU is not set
iommu/vt-d: Force requesting ACS when tboot is enabled
...
ACPI:
- Combine reads of AMU counters into a single FFH feedback counter op.
Confidential computing:
- Fix smp_processor_id() in preemptible context when retrieving an
attestation token inside a realm.
- Convert pKVM over to a "CC platform".
- Clean-up our SWIOTLB configuration in preparation for reworking the
handling of encrypted/decryped DMA buffers in the dma-mapping tree.
CPU errata handling:
- Work around broken device memory ordering on NVIDIA Olympus cores.
- Fix broken 'nospectre_bhb' command-line option.
- Select the idle loop backend instruction on the command-line.
CPU features:
- Replace our BBML2-noabort feature with the new architectural BBML3
feature.
- Disable in-kernel BTI for recent versions of Clang due to issues with
livepatch that are still being investigated.
- Clean-up documentation describing which ID register fields are exposed
to userspace.
Interrupts:
- Preliminary work towards supporting FEAT_NMI, which cleans up our
IRQ entry code and fixes some latent issues with pseudo-NMI.
- Support for an SDEI backend to trigger an NMI backtrace.
Memory management:
- Treat all devices as coherent when CLIDR_EL1.LoC == 0.
- Fix no-map handling of sub-page-sized regions.
- Second attempt at unmapping the linear aliases of the kernel data and
bss sections.
- Fix EFI runtime calls when software-PAN is enabled.
Miscellaneous:
- Add Mark Rutland as a reviewer!
- Tidy-up our futex cmpxchg logic when using the new LSUI instructions.
- Drop the requirement on DYNAMIC_FTRACE_WITH_CALL_OPS when selecting
HAVE_DYNAMIC_FTRACE_WITH_DIRECT_CALLS.
- Fix a false-positive KCSCAN splat in the delay loop.
- Use a portable typedef for 128-bit scalar types in our UAPI headers.
- Non-critical fixes for Sashiko reports all over.
MPAM:
- Hook MPAM memory bandwidth counters into resctrl's counter assignment
interface.
- Fix a quirk in the MPAM bandwidth counting on Nvidia T241 so that it
also applies to 63 bit counters.
Perf:
- Workarounds for hardware issues in the CMN-S3 PMU (Graviton 5) and CPU
PMU (NVIDIA Olympus again!)
- Add support for the DDR PMU on Marvell CN20K SoCs.
- Add support for Picoheart implementations of the DCW PCIe PMU.
- Add support for Channel/Rank/Bank filtering in the CXL PMU driver.
- Add support for 64-bit counters in the CSPMU device.
- Add support for revision 2 of the CMN S3 PMU.
Ptrace:
- Fix a decade-old bug in our handling of seccomp and tracing on syscall
entry.
- Fix regset handling for inactive SVE and SSVE registers.
Selftests
- Add some tests for the decade-old bug that we just tried to fix in our
syscall entry path.
- Fix SVE test crash on SME-only CPUs.
-----BEGIN PGP SIGNATURE-----
iQFEBAABCgAuFiEEPxTL6PPUbjXGY88ct6xw3ITBYzQFAmp+9HkQHHdpbGxAa2Vy
bmVsLm9yZwAKCRC3rHDchMFjNEjXB/4y8KueLwgMOcpjzJom6ujmE2Ab7IOI+I5v
rYQkGDdHLcQd0WW9gtFXoxlq/EUcWuVCGAdOF3PxkUmVhHJC45ZNSmBGS9/LbU3f
AqeFskSE46D2gOdMXSscwGAQ+OiQ6WlSSgqy2dU1ZfTXy7ioxydWVUtvmqMmB85O
ctN9Ocr3zAZuptuyLJurCJNnyCuzd55SIuy+WKU6NIpWseiIqiLEb/6HFbWcw6iV
vVZjD/Je5my8Gfjuu12z7kJ6o5jwGNpI8U7Z0b68ESUG3b5fh9E2zVLIq0fvn83B
rjPqg/Wj2TVGag/+yZgafLwQ/7111hhkdf1bPBitClx4GxOoZEjG
=Iypw
-----END PGP SIGNATURE-----
Merge tag 'arm64-upstream' of git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux
Pull arm64 updates from Will Deacon:
"There's a reasonable amount of stuff here, including a bunch of
updates to the perf PMU drivers and some MPAM updates to expose the
memory bandwidth counters via resctrl.
On the architecture side, some highlights include support for BBML3
and steps towards support for an architectural NMI solution, all
wrapped up in a web of fixes for latent issues identified by Sashiko.
ACPI:
- Combine reads of AMU counters into a single FFH feedback counter op
Confidential computing:
- Fix smp_processor_id() in preemptible context when retrieving an
attestation token inside a realm
- Convert pKVM over to a "CC platform"
- Clean-up our SWIOTLB configuration in preparation for reworking the
handling of encrypted/decryped DMA buffers in the dma-mapping tree
CPU errata handling:
- Work around broken device memory ordering on NVIDIA Olympus cores
- Fix broken 'nospectre_bhb' command-line option
- Select the idle loop backend instruction on the command-line
CPU features:
- Replace our BBML2-noabort feature with the new architectural BBML3
feature
- Disable in-kernel BTI for recent versions of Clang due to issues
with livepatch that are still being investigated
- Clean-up documentation describing which ID register fields are
exposed to userspace
Interrupts:
- Preliminary work towards supporting FEAT_NMI, which cleans up our
IRQ entry code and fixes some latent issues with pseudo-NMI
- Support for an SDEI backend to trigger an NMI backtrace
Memory management:
- Treat all devices as coherent when CLIDR_EL1.LoC == 0
- Fix no-map handling of sub-page-sized regions
- Second attempt at unmapping the linear aliases of the kernel data
and bss sections
- Fix EFI runtime calls when software-PAN is enabled
Miscellaneous:
- Add Mark Rutland as a reviewer!
- Tidy-up our futex cmpxchg logic when using the new LSUI
instructions
- Drop the requirement on DYNAMIC_FTRACE_WITH_CALL_OPS when
selecting HAVE_DYNAMIC_FTRACE_WITH_DIRECT_CALLS
- Fix a false-positive KCSCAN splat in the delay loop
- Use a portable typedef for 128-bit scalar types in our UAPI headers
- Non-critical fixes for Sashiko reports all over
MPAM:
- Hook MPAM memory bandwidth counters into resctrl's counter
assignment interface
- Fix a quirk in the MPAM bandwidth counting on Nvidia T241 so that
it also applies to 63 bit counters
Perf:
- Workarounds for hardware issues in the CMN-S3 PMU (Graviton 5) and
CPU PMU (NVIDIA Olympus again!)
- Add support for the DDR PMU on Marvell CN20K SoCs
- Add support for Picoheart implementations of the DCW PCIe PMU
- Add support for Channel/Rank/Bank filtering in the CXL PMU driver
- Add support for 64-bit counters in the CSPMU device
- Add support for revision 2 of the CMN S3 PMU
Ptrace:
- Fix a decade-old bug in our handling of seccomp and tracing on
syscall entry
- Fix regset handling for inactive SVE and SSVE registers
Selftests
- Add some tests for the decade-old bug that we just tried to fix in
our syscall entry path
- Fix SVE test crash on SME-only CPUs"
* tag 'arm64-upstream' of git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux: (95 commits)
arm64/efi: Avoid voluntary preemption with efi_mm installed
arm64: bti: Disable in-kernel BTI with recent versions of Clang
arm64: entry: Avoid unnecessary local_irq_disable() on kernel exit
irqchip/gic-v3: make the unmasking of pseudo-NMIs explicit when handling IRQs
arm64: Disable KCSAN instrumentation in delay.o
arm_mpam: Disable driver unbind to avoid UAF
arm_mpam: Fix a NULL pointer dereference on unbinding after an error interrupt
perf: arm_pmuv3: Zero initialize hw_id branch stack field
arm64: mm: Unmap kernel data/bss entirely from the linear map
iommu/arm-smmu-v3-sva: Use system_supports_bbml3() to detect CPU feature
perf/arm-cmn: Support CMN S3 r2
perf/arm-cmn: Plumb in new filter types
perf/arm-cmn: Refactor event filter data
perf/arm-cmn: Refactor event filter programming
perf/arm-cmn: Rename filter variables for clarity
arm64: mm: fix accidental linear mapping of no-map reserved memory
tools: Ensure tools copy of linux/filter.h exports the UAPI
kselftest/arm64: Fix abi test compilation errors
arch: arm64: add early_param idle=<wfi|yield|nop>
arm64: entry: mask DAIF before returning from C EL1 handlers
...
Commit 94104e3cfa ("arm64: cpufeature: Rename BBML2_NOABORT as BBML3")
renamed the cpu_supports_bbml2_noabort() helper to cpu_supports_bbml3(),
as the Linux-defined "noabort" semantics have now been incorporated into
the architecture under the BBML3 feature.
Update the caller in the SMMUv3 SVA driver to use the new function.
There is a slightly oddity in that the SMMUv3 architecture already
defined BBML2 in such a way that aborts were prohibited, so we use the
BBML3 feature on the CPU to enable BBML2 in the SMMU.
Fixes: 94104e3cfa ("arm64: cpufeature: Rename BBML2_NOABORT as BBML3")
Signed-off-by: Will Deacon <will@kernel.org>
The number of master->num_streams per master device is typically very
small in practice. Sorting this array takes a very small amount of
time, so there is no practical risk of triggering a soft lockup that
would necessitate calling cond_resched() during the sort.
Replace sort_nonatomic() with the standard sort(). Since this is the
only remaining in-tree caller of sort_nonatomic(), this change paves
the way to eventually remove the unused sort_nonatomic() API from the
core library.
Signed-off-by: Kuan-Wei Chiu <visitorckw@gmail.com>
Signed-off-by: Will Deacon <will@kernel.org>
An SMMU implementing SMMU_IDR5.DS extends the range invalidation commands:
the SCALE field grows a 6th bit, raising its maximum value from 31 to 39,
and TTL == 0b01 becomes a valid level hint for a 16KB translation granule.
Add a new ARM_SMMU_FEAT_DS feature detecting the DS bit, and widen the
CMDQ_TLBI_0_SCALE field to its architectural 6 bits. Mask the scale value
explicitly in arm_smmu_cmdq_batch_add_range(), so the range invalidation
path emits the same commands as before, keeping the pre-existing 5-bit
truncation of a scale above 31.
Also list DS as a valid IDR5 field in the iommu_hw_info_arm_smmuv3 kdoc:
iommufd has always reported the raw IDR5 register, so a VMM may conclude
from that bit alone that it can expose DS to its guest.
Suggested-by: Jason Gunthorpe <jgg@nvidia.com>
Reviewed-by: Jason Gunthorpe <jgg@nvidia.com>
Reviewed-by: Pranjal Shrivastava <praan@google.com>
Assisted-by: Claude:claude-fable-5
Signed-off-by: Nicolin Chen <nicolinc@nvidia.com>
Signed-off-by: Will Deacon <will@kernel.org>
The arm_smmu_write_reg_sync() helper is currently implemented using
readl_relaxed_poll_timeout() (that relies on usleep_range() internally)
which becomes a critical issue when used in the gerror irq handler.
If the SMMU hits a gerror and enters Service Failure Mode
(GERROR_SFM_ERR), the gerror handler calls arm_smmu_device_disable() in
hard-irq context. This becomes a problem as arm_smmu_device_disable()
inevitably calls arm_smmu_write_reg_sync() which might attempt to sleep
inside a hard-irq context.
Fix this by converting the arm_smmu_write_reg_sync to use the
readl_relaxed_poll_timeout_atomic() polling helper.
(Discovered while running Sashiko locally on another patch series).
Reported-by: Sashiko <sashiko-bot@kernel.org>
Fixes: 48ec83bcbc ("iommu/arm-smmu: Add initial driver support for ARM SMMUv3 devices")
Signed-off-by: Pranjal Shrivastava <praan@google.com>
Signed-off-by: Will Deacon <will@kernel.org>
Since table access flags cannot be software-managed, if process
pagetables are using HAFT then SVA must require the SMMU to support and
enable it too, otherwise page aging is liable to get out of whack.
For unbinding, we can't disable HAFT atomically with HA as might be
desired, but luckily we can get away with just not disabling HA either.
Cc: stable@vger.kernel.org
Fixes: 62df5870eb ("arm64: Enable ARCH_HAS_NONLEAF_PMD_YOUNG")
Signed-off-by: Robin Murphy <robin.murphy@arm.com>
Reviewed-by: Jason Gunthorpe <jgg@nvidia.com>
Signed-off-by: Will Deacon <will@kernel.org>
tegra241_cmdqv_isr() logs the error-map registers on every error interrupt.
A malfunctioning device, or a guest deliberately faulting its own VCMDQs,
can raise these interrupts rapidly, and the unconditional dev_warn() then
floods the kernel log.
Rate-limit the message with dev_warn_ratelimited(), and pass the error-map
registers straight to it so their four MMIO reads run only when the limiter
prints, instead of building the string on every interrupt.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Nicolin Chen <nicolinc@nvidia.com>
Signed-off-by: Will Deacon <will@kernel.org>
tegra241_cmdqv_init_structures() allocates VINTF0 with kzalloc_obj(), inits
it, and preallocates its logical VCMDQs. Two of its error paths leak.
When tegra241_cmdqv_init_vintf() fails it returns before VINTF0 reaches the
cmdqv->vintfs[] array, so the devres unwind on probe failure cannot reach
it; free it directly there.
A later VCMDQ preallocation failure instead leaves VINTF0 published, and so
this time the unwind does reach tegra241_cmdqv_remove_vintf(), which then
frees it from vintf->hyp_own. But tegra241_vintf_hw_init() sets that flag
only afterward, from a HW read-back, so the still-uninited VINTF0 reads as
guest-owned and leaks, with mutex_destroy() and ida_destroy() run on fields
it never set up.
Decide ownership from vintf->idx instead, the index assigned when its id is
allocated: idx 0 is the kernel-owned VINTF0, while idx >= 1 marks a guest
VINTF. So the in-kernel free decision in tegra241_cmdqv_remove_vintf() and
tegra241_vintf_free_lvcmdq() now keys on idx too, and hyp_own stays a pure
HW-readback state.
Fixes: 918eb5c856 ("iommu/arm-smmu-v3: Add in-kernel support for NVIDIA Tegra241 (Grace) CMDQV")
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Nicolin Chen <nicolinc@nvidia.com>
Signed-off-by: Will Deacon <will@kernel.org>
tegra241_vintf_init_vsid() maps a guest vSID to a single physical Stream ID
taken from master->streams[0], and only warns when the device does not have
exactly one stream. A device with several streams gets only its first one
mapped, so a guest vSID invalidation cannot reach the others' ATC and IOTLB
entries; a device with none makes master->streams a ZERO_SIZE_PTR, read out
of bounds.
Reject the mapping with -EOPNOTSUPP if master->num_streams is not one.
Fixes: 4dc0d12474 ("iommu/tegra241-cmdqv: Add user-space use support")
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Nicolin Chen <nicolinc@nvidia.com>
Signed-off-by: Will Deacon <will@kernel.org>
tegra241_vintf_init_vsid() programs the guest-provided vSID into SID_MATCH,
whose VIRT_SID field spans bits [20:1] with bit 0 as the match-enable flag.
The HW therefore matches only a 20-bit Stream ID.
The bound check rejects only virt_sid > UINT_MAX, which admits a value far
wider than the field. The write "virt_sid << 1 | 0x1" then drops every bit
above 20: a virt_sid of 0x80000000 lands as SID_MATCH = 0x1, a valid match
on vSID 0, so the entry aliases the wrong Stream ID. Because vdev->virt_id
is guest-controlled, a VMM can trigger it.
Validate virt_sid against the field width with FIELD_MAX(), and program the
register with FIELD_PREP() so the value and the field stay consistent.
Fixes: 4dc0d12474 ("iommu/tegra241-cmdqv: Add user-space use support")
Cc: stable@vger.kernel.org
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Nicolin Chen <nicolinc@nvidia.com>
Signed-off-by: Will Deacon <will@kernel.org>
tegra241_cmdqv_remove() tears each VINTF down first, then calls free_irq().
Tearing a VINTF down frees vintf0 and clears cmdqv->vintfs[0]. An error in
that window makes tegra241_cmdqv_isr() read the stale slot and hand it to
tegra241_vintf0_handle_error(), which dereferences a NULL or freed pointer.
Free the IRQ before tearing the VINTFs down. free_irq() waits for in-flight
handlers to finish and blocks new ones, so no ISR can observe a VINTF as it
is torn down.
Note: a user-owned VINTF (viommu) could outlive this teardown, which unmaps
cmdqv->base and frees cmdqv->vintfs, so a later viommu close then touches
freed memory. This is neither introduced nor fixed here: a physical IOMMU
is not a pluggable device, so iommufd by design holds no reference on the
one behind a viommu, and this teardown is not expected while that viommu is
still alive.
Fixes: 918eb5c856 ("iommu/arm-smmu-v3: Add in-kernel support for NVIDIA Tegra241 (Grace) CMDQV")
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Nicolin Chen <nicolinc@nvidia.com>
Signed-off-by: Will Deacon <will@kernel.org>
__tegra241_cmdqv_probe() uses devm_krealloc() to grow @smmu into the larger
tegra241_cmdqv, which frees the original @smmu once it relocates. A failure
after that returned NULL, and the caller then dereferenced the freed @smmu
on its fallback path.
Return an int and take @smmu by reference instead, then update *smmu to the
reallocated pointer after devm_krealloc() succeeds, so the caller and its
fallback path both use the live @smmu rather than the freed original.
Fixes: 918eb5c856 ("iommu/arm-smmu-v3: Add in-kernel support for NVIDIA Tegra241 (Grace) CMDQV")
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Nicolin Chen <nicolinc@nvidia.com>
Signed-off-by: Will Deacon <will@kernel.org>
__tegra241_cmdqv_probe() requests the error IRQ before it has allocated the
cmdqv->vintfs array and set cmdqv->num_vintfs. A CMDQV left enabled with a
latched error across a kexec fires the IRQ as soon as it is requested, and
tegra241_cmdqv_isr() then walks the uninitialized cmdqv->vintfs array.
Request the IRQ only after cmdqv->vintfs is allocated and zeroed, so that
a latched interrupt firing early runs the ISR against a valid array of NULL
slots that it safely skips.
Fixes: 918eb5c856 ("iommu/arm-smmu-v3: Add in-kernel support for NVIDIA Tegra241 (Grace) CMDQV")
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Nicolin Chen <nicolinc@nvidia.com>
Signed-off-by: Will Deacon <will@kernel.org>
tegra241_vintf0_handle_error() reads both 64-bit LVCMDQ error-map registers
but used the register-local __ffs64() bit directly as the vintf->lvcmdqs[]
index. For the second register that selects the wrong queue instead of 64 *
i + bit, clearing the wrong queue's error status.
The index is unbounded too: a bit at or beyond num_lvcmdqs_per_vintf would
walk the read off vintf->lvcmdqs[].
tegra241_cmdqv_isr() has the same flaw one level up: a VINTF_ERR_MAP bit
at or beyond num_vintfs would walk the read off cmdqv->vintfs[].
Use 64 * i + bit for the index and clear the snapshot with the local bit.
In both handlers, WARN_ON_ONCE() and skip an out-of-bounds index. Only a
malfunctioning device sets such a bit, so the _ONCE form keeps a wedged map
from flooding the log.
Note that 64 * i + bit is not reachable with the current configuration as a
VINTF is pre-assigned with 2 lvcmdqs, this is not treated as bug fix but an
defensive hardening.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Nicolin Chen <nicolinc@nvidia.com>
Signed-off-by: Will Deacon <will@kernel.org>
A user VINTF is torn down by tegra241_cmdqv_deinit_vintf(), which runs from
the destroy callback and from the init-failure unwind in the alloc handler.
It clears the cmdqv->vintfs[] slot and lets the iommufd core free it, but
nothing serializes that against the error interrupt: tegra241_cmdqv_isr()
reads cmdqv->vintfs[idx] and dereferences the vintf. A concurrent error can
make the ISR read a slot mid-clear (a NULL deref) or use a vintf which is
about to be freed (a use-after-free).
deinit_vintf() also returns idx to the IDA before clearing the slot, so a
concurrent create that reuses idx can publish its new vintf into the slot,
only for this teardown to erase it again with the stale NULL store.
On the other end, tegra241_cmdqv_init_vintf() publishes a new vintf with a
plain store to the cmdqv->vintfs[] slot, and the ISR dereferences fields of
a published vintf such as vintf->base. A plain store gives no ordering on a
weakly-ordered CPU, and a stale VINTF_ERR_MAP bit on a reused idx can make
the ISR pick a vintf the moment it is published, before its fields are set
or tegra241_vintf_hw_init() runs.
The cmdqv->vintfs[0] slot stays NULL until tegra241_cmdqv_init_structures()
first creates VINTF0, so the slot 0 read needs the same NULL check.
Publish every slot with an smp_store_release(), and read each slot in the
ISR with an smp_load_acquire() under a NULL check, so the ISR always sees
a fully built vintf or NULL. Also make deinit_vintf() clear the slot, and
synchronize_irq() prior to returning idx to the IDA, so no vintf is freed
under a running handler and no reused idx is clobbered.
Fixes: 4dc0d12474 ("iommu/tegra241-cmdqv: Add user-space use support")
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Nicolin Chen <nicolinc@nvidia.com>
Signed-off-by: Will Deacon <will@kernel.org>
tegra241_vintf_init_lvcmdq() stores the freshly allocated vcmdq pointer to
the vintf->lvcmdqs[] array, before tegra241_vcmdq_alloc_smmu_cmdq() builds
the vcmdq->cmdq. The error ISR dereferences that cmdq, so a latched LVCMDQ
error (e.g. one inherited across a kexec) firing in this window would make
tegra241_vintf0_handle_error() pass the still-zeroed arm_smmu_cmdq down to
__arm_smmu_cmdq_skip_err(), dereferencing NULL queue register pointers.
Drop the store from tegra241_vintf_init_lvcmdq() and publish the vcmdq at
the end of the allocation instead, with an smp_store_release() that pairs
with an smp_load_acquire() in the ISR, which can see a fully built LVCMDQ
or NULL.
The user-owned LVCMDQ allocation moves accordingly, publishing the vcmdq
once tegra241_vcmdq_hw_init_user() succeeds, using a plain store since a
user VINTF's lvcmdqs[] has no lockless reader -- the error ISR only walks
the VINTF0 array.
Fixes: 918eb5c856 ("iommu/arm-smmu-v3: Add in-kernel support for NVIDIA Tegra241 (Grace) CMDQV")
Assisted-by: Claude:claude-fable-5
Signed-off-by: Nicolin Chen <nicolinc@nvidia.com>
Signed-off-by: Will Deacon <will@kernel.org>
Nvidia Tegra264 SMMU is affected by an erratum where a TLB entry can
survive an invalidation that races with concurrent traffic targeting
the same entry. The hardware-recommended software workaround is to
issue every CFGI/TLBI command (each followed by CMD_SYNC) twice, and
that infrastructure is already in place behind
arm_smmu_erratum_repeat_tlbi_cfgi_key.
Neither IDR nor IIDR flags this Tegra264-specific bug, so hardware
detection is not possible. Tegra264 is device-tree-only (no ACPI/IORT
support) and already has a dedicated "nvidia,tegra264-smmu" compatible,
so DT-probe is the only viable detection path.
Enable the workaround on instances matching the existing
"nvidia,tegra264-smmu" compatible by calling static_branch_enable() on
arm_smmu_erratum_repeat_tlbi_cfgi_key. Document the erratum in
Documentation/arch/arm64/silicon-errata.rst.
Signed-off-by: Ashish Mhetre <amhetre@nvidia.com>
Reviewed-by: Nicolin Chen <nicolinc@nvidia.com>
Reviewed-by: Jason Gunthorpe <jgg@nvidia.com>
Signed-off-by: Will Deacon <will@kernel.org>
A guest with access to VCMDQ generates its own invalidation commands
and must apply any invalidation errata before submitting them. If the
host also repeats those commands, each affected invalidation is issued
four times instead of twice.
Add IOMMU_HW_INFO_ARM_SMMUV3_ERRATA_REPEAT_TLBI_CFGI to report the
CFGI/TLBI-repeat erratum to user space. This allows the VMM to expose
the erratum to the guest or apply the workaround itself.
Use the raw __arm_smmu_cmdq_issue_cmdlist() helper for user-provided
invalidations so the host does not apply the workaround a second time.
Add arm_smmu_erratum_repeat_tlbi_cfgi() to query the static key when
populating the SMMUv3 hardware information.
Signed-off-by: Ashish Mhetre <amhetre@nvidia.com>
Reviewed-by: Nicolin Chen <nicolinc@nvidia.com>
Reviewed-by: Jason Gunthorpe <jgg@nvidia.com>
Signed-off-by: Will Deacon <will@kernel.org>
Tegra264 SMMU instances need every CFGI/TLBI command sequence issued
twice, with the second issue executing only after the first issue's
CMD_SYNC has completed:
TLBI/CFGI ... CMD_SYNC TLBI/CFGI ... CMD_SYNC
ATC_INV is not affected and must never be doubled.
Add arm_smmu_erratum_repeat_tlbi_cfgi_key and a file-local
arm_smmu_erratum_cmd_needs_repeating() helper that gates on the static
key first and then range-checks the opcode (CFGI_STE .. ATC_INV).
Rename the existing arm_smmu_cmdq_issue_cmdlist() to
__arm_smmu_cmdq_issue_cmdlist() and add a thin wrapper of the original
name that re-issues the same cmdlist a second time when the predicate
fires. Register the new condition with
arm_smmu_cmdq_batch_force_sync() too.
No callers enable the static key yet, so there is no functional change.
A subsequent change will enable the key on affected instances.
Suggested-by: Nicolin Chen <nicolinc@nvidia.com>
Reviewed-by: Nicolin Chen <nicolinc@nvidia.com>
Reviewed-by: Jason Gunthorpe <jgg@nvidia.com>
Signed-off-by: Ashish Mhetre <amhetre@nvidia.com>
Signed-off-by: Will Deacon <will@kernel.org>
arm_smmu_cmdq_batch_add_cmd_p() carries two distinct reasons for
flushing the current batch with a CMD_SYNC before appending the
new command:
- The batch's pre-assigned cmdq does not support the new command.
- The Arm erratum 2812531 workaround (ARM_SMMU_OPT_CMDQ_FORCE_SYNC)
forces a SYNC at one entry before the batch is full.
Lift those checks into a new arm_smmu_cmdq_batch_force_sync() helper
so that adding another force-sync condition becomes a one-line
addition. No functional change.
Signed-off-by: Nicolin Chen <nicolinc@nvidia.com>
Reviewed-by: Jason Gunthorpe <jgg@nvidia.com>
Signed-off-by: Ashish Mhetre <amhetre@nvidia.com>
Signed-off-by: Will Deacon <will@kernel.org>
arm_smmu_impl_remove() is registered as a devres action in
arm_smmu_impl_probe(), before arm_smmu_init_queues() allocates
smmu->cmdq.q.base. On a devres unwind, whether a failed probe or an
unbind, the queue is freed first and arm_smmu_impl_remove() then runs
tegra241_cmdqv_remove_vintf(), whose VINTF deinit issues a CMD_SYNC on
the freed memory.
Observed during testing with a QEMU hack that makes the VCMDQ fail to
enable, so the impl reset fails and probe aborts into the devres unwind:
platform NVDA200C:00: tegra241_cmdqv: VINTF0: VCMDQ0/LVCMDQ0: failed to enable, STATUS=0x00000000
platform NVDA200C:00: tegra241_cmdqv: VINTF0: VCMDQ0/LVCMDQ0: GERRORN=0x0, GERROR=0x4, CONS=0x0
platform NVDA200C:00: tegra241_cmdqv: VINTF0: VCMDQ0/LVCMDQ0: uncleared error detected, resetting
arm-smmu-v3 arm-smmu-v3.0.auto: failed to reset impl
arm-smmu-v3 arm-smmu-v3.0.auto: probe with driver arm-smmu-v3 failed with error -110
Unable to handle kernel paging request at virtual address ffff8000891e0098
...
Internal error: Oops: 0000000096000047 [#1] SMP
...
Call trace:
arm_smmu_cmdq_issue_cmdlist+0x320/0x6fc (P)
tegra241_vcmdq_hw_deinit+0x98/0x168
tegra241_vintf_hw_deinit+0x5c/0x1b0
tegra241_cmdqv_remove_vintf+0x34/0xec
tegra241_cmdqv_remove+0x40/0x9c
arm_smmu_impl_remove+0x20/0x30
devm_action_release+0x14/0x20
devres_release_all+0xa8/0x110
device_unbind_cleanup+0x18/0x84
really_probe+0x1f0/0x29c
Drop the VINTF deinit from tegra241_cmdqv_remove_vintf() so the unwind no
longer touches the freed queue. Quiesce the VINTFs earlier instead. Add a
device_disable() impl op and run it from arm_smmu_disable_action() while
the CMDQ is still up. That handles a live unbind. A failed reset is already
handled because tegra241_vintf_hw_init() deinits the VINTF on its own error
path. tegra241_cmdqv_remove_vintf() is also used by the iommufd viommu
destroy path, so quiesce there too.
Fixes: 4dc0d12474 ("iommu/tegra241-cmdqv: Add user-space use support")
Cc: stable@vger.kernel.org
Reviewed-by: Nicolin Chen <nicolinc@nvidia.com>
Signed-off-by: Shameer Kolothum <skolothumtho@nvidia.com>
Reviewed-by: Jason Gunthorpe <jgg@nvidia.com>
Signed-off-by: Will Deacon <will@kernel.org>
arm_smmu_device_remove() manually frees the IOPF queue, destroys the
vmid_map and disables the device, while the IRQs and queues are devm
managed. devm unwinds only after remove() returns, so the cleanup runs
in the wrong order. The IOPF queue is freed before the event-queue IRQ
whose handler uses it.
Manage all of it with devm so the unwind order is correct. Free the IOPF
queue and vmid_map via devm actions, and disable the device from one
registered after arm_smmu_device_reset().
This is also a prerequisite for fixing a Tegra241 CMDQV CMD_SYNC
use-after-free in the subsequent patch.
Cc: stable@vger.kernel.org
Suggested-by: Jason Gunthorpe <jgg@ziepe.ca>
Reviewed-by: Nicolin Chen <nicolinc@nvidia.com>
Signed-off-by: Shameer Kolothum <skolothumtho@nvidia.com>
Reviewed-by: Jason Gunthorpe <jgg@nvidia.com>
Signed-off-by: Will Deacon <will@kernel.org>
arm_smmu_get_ste_update_safe() declares the eats_s1chk and eats_trans
locals as __le64, but initializes them from FIELD_PREP(), which returns a
host-endian value, and passes them through cpu_to_le64() at the use sites.
Sparse reports the following warnings:
>> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c:1122:38: sparse: sparse: cast from restricted __le64
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c:1124:33: sparse: sparse: cast from restricted __le64
Declare both locals as u64 so the type matches FIELD_PREP() and the
existing cpu_to_le64() at the use sites performs the host-to-little-endian
conversion. No functional change.
Fixes: 7cad800485 ("iommu/arm-smmu-v3: Mark EATS_TRANS safe when computing the update sequence")
Reported-by: kernel test robot <lkp@intel.com>
Closes: https://lore.kernel.org/all/202606151017.QU0evpH9-lkp@intel.com/
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Nicolin Chen <nicolinc@nvidia.com>
Reviewed-by: Jason Gunthorpe <jgg@nvidia.com>
Reviewed-by: Pranjal Shrivastava <praan@google.com>
Signed-off-by: Will Deacon <will@kernel.org>
Since commit 55b48e23f5 ("genirq/devres: Add error handling in
devm_request_*_irq()"), devm_request_irq() and devm_request_threaded_irq()
automatically log detailed error messages on failure. Remove
the now-redundant driver-specific dev_err() and dev_err_probe() calls.
Signed-off-by: Pan Chuang <panchuang@vivo.com>
Reviewed-by: Robin Murphy <robin.murphy@arm.com>
Signed-off-by: Will Deacon <will@kernel.org>
qcom_iommu_ctx_probe() reads and writes the CB_FSR register to clear any
stale IRQ left by the bootloader. This happens during
devm_of_platform_populate() which is called from the parent device's
probe before any pm_runtime_get(). The parent's clocks (iface, bus, tbu)
are therefore not guaranteed to be on, making the register access
unreliable on rebind or after a suspend cycle.
Use pm_runtime_resume_and_get() on the parent device to ensure clocks
are enabled before the register access, and release the reference
immediately after.
Reviewed-by: Konrad Dybcio <konrad.dybcio@oss.qualcomm.com>
Signed-off-by: Mukesh Ojha <mukesh.ojha@oss.qualcomm.com>
Signed-off-by: Will Deacon <will@kernel.org>
qcom_domain->pgtbl_ops was assigned after mutex_unlock(). Another thread
calling qcom_iommu_init_domain() would see qcom_domain->iommu already set
(domain fully initialized) and skip re-initialization under the mutex.
If it then called qcom_iommu_map() before the first thread set pgtbl_ops,
it would observe a NULL ops pointer and return -ENODEV for valid mappings.
Move the assignment to before mutex_unlock() so that once the mutex is
released the domain is fully visible to concurrent operations.
Reviewed-by: Konrad Dybcio <konrad.dybcio@oss.qualcomm.com>
Signed-off-by: Mukesh Ojha <mukesh.ojha@oss.qualcomm.com>
Signed-off-by: Will Deacon <will@kernel.org>
alloc_io_pgtable_ops() can succeed and then qcom_scm_restore_sec_cfg()
can fail for one of the context banks. The goto out_clear_iommu path
only cleared qcom_domain->iommu; the locally allocated pgtbl_ops was
never freed, leaking it permanently since qcom_domain->pgtbl_ops is only
assigned on the success path.
free_io_pgtable_ops() safely handles a NULL argument (covers the case
where alloc_io_pgtable_ops() itself failed), so add it unconditionally in
the out_clear_iommu handler.
Reviewed-by: Konrad Dybcio <konrad.dybcio@oss.qualcomm.com>
Signed-off-by: Mukesh Ojha <mukesh.ojha@oss.qualcomm.com>
Signed-off-by: Will Deacon <will@kernel.org>
The SMMU_INTR_SEL_NS register write in qcom_iommu_device_probe() uses
pm_runtime_get_sync() without checking the return value. If runtime
resume fails the subsequent writel_relaxed() would access hardware with
clocks potentially disabled.
Switch to pm_runtime_resume_and_get() which handles the usage-count
cleanup on failure, check the return value, and unwind the already
registered iommu device on error.
Reviewed-by: Konrad Dybcio <konrad.dybcio@oss.qualcomm.com>
Signed-off-by: Mukesh Ojha <mukesh.ojha@oss.qualcomm.com>
Signed-off-by: Will Deacon <will@kernel.org>
Switch from pm_runtime_enable() to devm_pm_runtime_enable() so that
the matching pm_runtime_disable() is handled automatically via devres,
both on probe failure and on device removal.
This removes the err_pm_disable error label from the probe function and
the explicit pm_runtime_disable() call from qcom_iommu_device_remove().
Signed-off-by: Mukesh Ojha <mukesh.ojha@oss.qualcomm.com>
Reviewed-by: Konrad Dybcio <konrad.dybcio@oss.qualcomm.com>
Signed-off-by: Will Deacon <will@kernel.org>
report_iommu_fault() returns 0 when a fault handler successfully handles
the fault, and -ENOSYS when no handler is installed. The condition
'!report_iommu_fault()' evaluates to true (printing "Unhandled context
fault") precisely when the fault *was* handled, and stays silent when no
handler is present — the opposite of what is intended.
Remove the '!' so the driver logs unhandled faults correctly.
Fixes: 049541e178 ("iommu: qcom: wire up fault handler")
Reviewed-by: Konrad Dybcio <konrad.dybcio@oss.qualcomm.com>
Signed-off-by: Mukesh Ojha <mukesh.ojha@oss.qualcomm.com>
Signed-off-by: Will Deacon <will@kernel.org>
In qcom_iommu_device_probe(), if iommu_device_register()
fails, the sysfs device created by iommu_device_sysfs_add()
is not released. Add a goto label to do the cleanup.
Fixes: 0ae349a0f3 ("iommu/qcom: Add qcom_iommu")
Signed-off-by: Haoxiang Li <haoxiang_li2024@163.com>
Reviewed-by: Konrad Dybcio <konrad.dybcio@oss.qualcomm.com>
Reviewed-by: Mukesh Ojha <mukesh.ojha@oss.qualcomm.com>
Signed-off-by: Will Deacon <will@kernel.org>
Add the compatible for the MDSS client on the Snapdragon 8 Gen 1 so it can
be properly configured by the IOMMU driver.
Otherwise, there is an unhandled context fault, because the framebuffer is
already configured in UEFI.
Reviewed-by: Konrad Dybcio <konrad.dybcio@oss.qualcomm.com>
Signed-off-by: Esteban Urrutia <esteuwu@proton.me>
Signed-off-by: Will Deacon <will@kernel.org>
arm_vsmmu_vsid_to_sid() maps a guest's vSID to a single physical Stream ID
taken from master->streams[0], assuming a device has exactly one stream. A
device with several streams gets only its first one mapped, so a guest vSID
invalidation cannot reach the others' ATC and IOTLB entries; a device with
none makes master->streams a ZERO_SIZE_PTR, read out of bounds.
Add an arm_vsmmu_vdevice_init() op to reject the vDEVICE with -EOPNOTSUPP
when master->num_streams is not one, rather than mapping it silently.
Fixes: d68beb276b ("iommu/arm-smmu-v3: Support IOMMU_HWPT_INVALIDATE using a VIOMMU object")
Link: https://patch.msgid.link/r/b15f2b73520f389f3f57881da2f040e7bdc18876.1783311134.git.nicolinc@nvidia.com
Reviewed-by: Kevin Tian <kevin.tian@intel.com>
Assisted-by: Claude:claude-opus-4-8
Reviewed-by: Pranjal Shrivastava <praan@google.com>
Signed-off-by: Nicolin Chen <nicolinc@nvidia.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
Update arm_smmu_enable_ats() to wrap the pci_enable_ats() call in a
WARN(). Since probe-time checks now preclude configuration errors
any failure during hardware enablement is considered a kernel bug.
Reviewed-by: Kevin Tian <kevin.tian@intel.com>
Reviewed-by: Nicolin Chen <nicolinc@nvidia.com>
Reviewed-by: Samiullah Khawaja <skhawaja@google.com>
Reviewed-by: Jason Gunthorpe <jgg@nvidia.com>
Signed-off-by: Pranjal Shrivastava <praan@google.com>
Signed-off-by: Joerg Roedel <joerg.roedel@amd.com>
Replace the #include of <linux/mod_devicetable.h> by the more specific
<linux/device-id/*.h> where applicable. For most cases the include
can be dropped completely, only a few drivers need one or two headers
added.
Acked-by: Danilo Krummrich <dakr@kernel.org>
Acked-by: Takashi Sakamoto <o-takashi@sakamocchi.jp>
Acked-by: Bjorn Helgaas <bhelgaas@google.com>
Link: https://patch.msgid.link/1a3f2007c5c5dcf555c09a4035ce3ae8ef1b6c49.1782808461.git.u.kleine-koenig@baylibre.com
Signed-off-by: Uwe Kleine-König (The Capable Hub) <u.kleine-koenig@baylibre.com>
When a device's default substream attaches to an identity domain, the SMMU
driver currently sets the device's STE between two modes:
Mode 1: Cfg=Translate, S1DSS=Bypass, EATS=1
Mode 2: Cfg=bypass (EATS is ignored by HW)
When there is an active PASID (non-default substream), mode 1 is used. And
when there is no PASID support or no active PASID, mode 2 is used.
The driver will also downgrade an STE from mode 1 to mode 2, when the last
active substream becomes inactive.
However, there are PCIe devices that demand ATS to be always on. For these
devices, their STEs have to use the mode 1 as HW ignores EATS with mode 2.
Change the driver accordingly:
- always use the mode 1
- never downgrade to mode 2
- allocate and retain a CD table (see note below)
Note that these devices might not support PASID, i.e. doing non-PASID ATS.
In such a case, the ssid_bits is set to 0. However, s1cdmax must be set to
a !0 value in order to keep the S1DSS field effective. Thus, when a master
requires ats_always_on, set its s1cdmax to at least 1, meaning that the CD
table will have a dummy entry (SSID=1) that will never be used.
Now for these devices, arm_smmu_cdtab_allocated() will always return true,
v.s. false prior to this change. When its default substream is attached to
an IDENTITY domain, its first CD is NULL in the table, which is a totally
valid case. Thus, add "!master->ats_always_on" to the condition.
Reviewed-by: Jonathan Cameron <jonathan.cameron@huawei.com>
Tested-by: Nirmoy Das <nirmoyd@nvidia.com>
Acked-by: Nirmoy Das <nirmoyd@nvidia.com>
Reviewed-by: Jason Gunthorpe <jgg@nvidia.com>
Reviewed-by: Kevin Tian <kevin.tian@intel.com>
Reviewed-by: Dave Jiang <dave.jiang@intel.com>
Signed-off-by: Nicolin Chen <nicolinc@nvidia.com>
Signed-off-by: Joerg Roedel <joerg.roedel@amd.com>
HTTU is introduced by utilizing the Dirty Bit Modifier (DBM) in the PTE.
When kernel maps a clean but writable page, it will set PTE_READONLY and
PTE_DBM (aka PTE_WRITE) at the same time. When a write occurs, an HTTU-
capable MMU will automatically clear the PTE_RDONLY bit without software
intervention.
On the other hand, SMMU has the same HTTU feature, yet it is not enabled
in the SVA CD. As a result, SMMU will not clear the PTE_RDONLY bit while
sharing the CPU page table, resulting in unnecessary stalls.
Thus, enable CTXDESC_CD_0_TCR_HA and CTXDESC_CD_0_TCR_HD in the SVA CD.
Suggested-by: Jason Gunthorpe <jgg@nvidia.com>
Signed-off-by: Nicolin Chen <nicolinc@nvidia.com>
Signed-off-by: Will Deacon <will@kernel.org>
TLBI is more complicated than all the other commands because the
invalidation loop builds a template command from the struct
arm_smmu_inv which is then expanded into many TLBI commands for the
invalidation.
Reviewed-by: Mostafa Saleh <smostafa@google.com>
Reviewed-by: Pranjal Shrivastava <praan@google.com>
Tested-by: Pranjal Shrivastava <praan@google.com>
Tested-by: Mostafa Saleh <smostafa@google.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
Reviewed-by: Nicolin Chen <nicolinc@nvidia.com>
Tested-by: Samiullah Khawaja <skhawaja@google.com>
Tested-by: Nicolin Chen <nicolinc@nvidia.com>
Signed-off-by: Will Deacon <will@kernel.org>
Change the flow so the caller controls the CS field and remove the
weird u64p_replace_bits() thing to override it.
Reviewed-by: Pranjal Shrivastava <praan@google.com>
Reviewed-by: Mostafa Saleh <smostafa@google.com>
Tested-by: Pranjal Shrivastava <praan@google.com>
Tested-by: Mostafa Saleh <smostafa@google.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
Reviewed-by: Nicolin Chen <nicolinc@nvidia.com>
Tested-by: Samiullah Khawaja <skhawaja@google.com>
Tested-by: Nicolin Chen <nicolinc@nvidia.com>
Signed-off-by: Will Deacon <will@kernel.org>
Add a new command make function and convert all the places using
ATC_INV.
Split out full invalidation to directly make the cmd instead of
overloading size=0 to mean full invalidation.
In section "3.9.1 ATS Interface" of F.b the specification says:
When the SMMU returns an ATS Translation Completion for a request that
had a PASID, the Global bit of the Translation Completion Data Entry
must be zero.
Even though it faithfully forwards the G bit through to the ATS
invalidation command there is no way to create G mappings so there is
never any need to send a G invalidation. Thus don't expose global in the
new helpers and leave CMDQ_ATC_0_GLOBAL unused.
Reviewed-by: Mostafa Saleh <smostafa@google.com>
Reviewed-by: Pranjal Shrivastava <praan@google.com>
Tested-by: Pranjal Shrivastava <praan@google.com>
Tested-by: Mostafa Saleh <smostafa@google.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
Reviewed-by: Nicolin Chen <nicolinc@nvidia.com>
Tested-by: Samiullah Khawaja <skhawaja@google.com>
Tested-by: Nicolin Chen <nicolinc@nvidia.com>
Signed-off-by: Will Deacon <will@kernel.org>
Add make functions to build commands for
CMDQ_OP_TLBI_EL2_ALL
CMDQ_OP_TLBI_NSNH_ALL
CMDQ_OP_CFGI_ALL
CMDQ_OP_PREFETCH_CFG
CMDQ_OP_CFGI_STE
CMDQ_OP_CFGI_CD
CMDQ_OP_RESUME
CMDQ_OP_PRI_RESP
Convert all of these call sites to use the make function instead of
going through arm_smmu_cmdq_build_cmd(). Use a #define so the general
pattern is always:
arm_smmu_cmdq_issue_cmd(smmu, arm_smmu_make_cmd_XX(..));
Add arm_smmu_cmdq_batch_add_cmd() which takes struct arm_smmu_cmd
directly to match the new flow.
Reviewed-by: Pranjal Shrivastava <praan@google.com>
Reviewed-by: Mostafa Saleh <smostafa@google.com>
Tested-by: Pranjal Shrivastava <praan@google.com>
Tested-by: Mostafa Saleh <smostafa@google.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
Reviewed-by: Nicolin Chen <nicolinc@nvidia.com>
Tested-by: Samiullah Khawaja <skhawaja@google.com>
Tested-by: Nicolin Chen <nicolinc@nvidia.com>
Signed-off-by: Will Deacon <will@kernel.org>
Convert the batch's type to also get the remaining helper functions to
use the new type and complete replacing naked u64s with the new struct.
The low-level queue_write()/queue_read()/queue_remove_raw() functions
remain u64-based since they are shared by event and PRI queues which
have different entry sizes.
Reviewed-by: Mostafa Saleh <smostafa@google.com>
Reviewed-by: Pranjal Shrivastava <praan@google.com>
Tested-by: Pranjal Shrivastava <praan@google.com>
Tested-by: Mostafa Saleh <smostafa@google.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
Reviewed-by: Nicolin Chen <nicolinc@nvidia.com>
Tested-by: Samiullah Khawaja <skhawaja@google.com>
Tested-by: Nicolin Chen <nicolinc@nvidia.com>
Signed-off-by: Will Deacon <will@kernel.org>
Continue removing struct arm_smmu_cmdq_ent in favour of the HW based
struct arm_smmu_cmd. Switch the lower level issue commands to work on
the native struct by lifting arm_smmu_cmdq_build_cmd() into all the
callers.
Following patches will revise each of the arm_smmu_cmdq_build_cmd()
call sites to replace it with the HW struct.
Reviewed-by: Mostafa Saleh <smostafa@google.com>
Reviewed-by: Pranjal Shrivastava <praan@google.com>
Tested-by: Pranjal Shrivastava <praan@google.com>
Tested-by: Mostafa Saleh <smostafa@google.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
Reviewed-by: Nicolin Chen <nicolinc@nvidia.com>
Tested-by: Samiullah Khawaja <skhawaja@google.com>
Tested-by: Nicolin Chen <nicolinc@nvidia.com>
Signed-off-by: Will Deacon <will@kernel.org>
Start removing the use of struct arm_smmu_cmdq_ent, starting at the
lower levels of the call chain. Change the functions that determine what
cmdq to issue the batch to into using struct arm_smmu_cmd directly.
Reviewed-by: Pranjal Shrivastava <praan@google.com>
Reviewed-by: Mostafa Saleh <smostafa@google.com>
Tested-by: Pranjal Shrivastava <praan@google.com>
Tested-by: Mostafa Saleh <smostafa@google.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
Reviewed-by: Nicolin Chen <nicolinc@nvidia.com>
Tested-by: Samiullah Khawaja <skhawaja@google.com>
Tested-by: Nicolin Chen <nicolinc@nvidia.com>
Signed-off-by: Will Deacon <will@kernel.org>
