- Add initial definitions and discovery for the Smcsrind, Sscsrind,
Smcntrpmf, Ssccfg, Smcdeleg, Zicclsm, Ziccamoa, Ziccif, Ziccrse,
Za64rs, and Ssqosid RISC-V ISA extensions
- Improve the RISC-V update_mmu_cache_range() implementation by using
flush-by-ASID, enabling performance improvements on
microarchitectures that support related optimizations; and by taking
advantage of the Svinval RISC-V ISA extension on microarchitectures
that support it
- Shrink CFI shadow stack allocation further (to 512MB) to save
virtual address space (and physical memory on systems with strict
overcommit policies)
- Add initial CPU context switch support for QoS tagging (Ssqosid)
- Change our vector misaligned access speed test code to be
synchronous, avoiding a bunch of bugs related to unnecessary
asynchronicity
- Enable ARCH_HAS_ACPI_TABLE_UPGRADE for RISC-V to help with ACPI
table debugging (similar to ARM64 and x86)
- Extract vDSO section offsets at build time, rather than run time, to
avoid the boot time overhead
- Use assembler directives to control the use of instructions from the
RISC-V ISA extensions Zacas and Zabha, rather than compiler -march
flags that could affect the whole kernel (similar to ARM64)
- Add a kselftest for kprobes support for the c.jal instruction on
RISC-V
- When UEFI runtime services are available, use them to restart and
power off
- Fix ARCH_HAS_ACPI_TABLE_UPGRADE for RISC-V by adding a missing
include file to the ACPI code
- Move contiguous DMA memory reservation later in the boot, to ensure
that the correct global and per-node pools are allocated when
CONFIG_CMA_SIZE_PERNUMA is enabled
- Disallow probes on breakpoint handlers to avoid recursing
indefinitely into do_trap_break()
- Patch the compat vDSO during runtime alternatives processing, not
only the standard vDSO
- Remove some leftover XIP support elements missed by commit
9b3a2be848 ("riscv: Remove support for XIP kernel")
- Fix broadcast timer switching in ACPI LPI power states by
implementing a RISC-V version of arch_get_idle_state_flags()
- Miscellaneous fixes, including: vDSO makefile simplification;
marking our default_power_off() as __noreturn; error path fixes in
the PMU SBI perf driver; RISC-V extension capitalization
consistency; the use of BIT() macros in one of our include files;
and some documentation fixes
-----BEGIN PGP SIGNATURE-----
iQIzBAABCgAdFiEElRDoIDdEz9/svf2Kx4+xDQu9KksFAmqGahMACgkQx4+xDQu9
Kks5lg//d6+oTEXAKYm5e4TQFi7yQFBVfRLskL+t2d1wAPYvDPz3dJEHxU2MSGpC
I7lYI57hVUMK+PZhZ7IVjNF6WMdxOioAT1FyY4yDaS41AyYWeWUW6IGJCYqtYRrm
Bf80JORXVrGI4UOw3SbDWr4IWkYKCo6CD/ZPq3vam9um7u0B8dzXxSAuMwH6hqNf
A/e65c4+Nr/RBzaBc6/s4lPHD2X+YtfgSlhuPlUyDJSVmomvhkLHrxdndRDZKMT6
BiyJSHlvK78w71beiE6yf+1AysHPffmLuYjCfWxHGQNlAGAUjoDin3EhKgfRZdQ3
VIMTl/LdhUJZMWFjjr8P8jciZNVsXaRk6IULQ6JIY1cKxioaQhF4PUt4ZQ0HOafO
/22Uts5fLMrgtxpnMO0CTdHKKpsQ+UZtQFEl2R9PiJSrxnlbw7zYt+I/e+gAPDgY
ve9VOvT74a1k2KkhfwLU0b8QwoVi4wWJdGoFvj9Tb42fzQvoXaWs3TZehdGK0bbi
btoNNnve4kRnDQTp+Jr/9iQr1L9CYq/hgn2ClipMUl5pj+oDHk8vDQ+jeboNRlfZ
FiYud4N4uh6CmJj0XlHLwNar547l9KwTqoAJK9bz4Ti9LxCnLziBiOjyjeXvEA41
fD9pdHX/kMmz1YjMNvhcd4uRAN48H40UpWvbermuVWpe8/pQEII=
=Wz+/
-----END PGP SIGNATURE-----
Merge tag 'riscv-for-linus-7.3-mw1' of git://git.kernel.org/pub/scm/linux/kernel/git/riscv/linux
Pull RISC-V updates from Paul Walmsley
- Add initial definitions and discovery for the Smcsrind, Sscsrind,
Smcntrpmf, Ssccfg, Smcdeleg, Zicclsm, Ziccamoa, Ziccif, Ziccrse,
Za64rs, and Ssqosid RISC-V ISA extensions
- Improve the RISC-V update_mmu_cache_range() implementation by using
flush-by-ASID, enabling performance improvements on
microarchitectures that support related optimizations; and by taking
advantage of the Svinval RISC-V ISA extension on microarchitectures
that support it
- Shrink CFI shadow stack allocation further (to 512MB) to save virtual
address space (and physical memory on systems with strict overcommit
policies)
- Add initial CPU context switch support for QoS tagging (Ssqosid)
- Change our vector misaligned access speed test code to be
synchronous, avoiding a bunch of bugs related to unnecessary
asynchronicity
- Enable ARCH_HAS_ACPI_TABLE_UPGRADE for RISC-V to help with ACPI table
debugging (similar to ARM64 and x86)
- Extract vDSO section offsets at build time, rather than run time, to
avoid the boot time overhead
- Use assembler directives to control the use of instructions from the
RISC-V ISA extensions Zacas and Zabha, rather than compiler -march
flags that could affect the whole kernel (similar to ARM64)
- Add a kselftest for kprobes support for the c.jal instruction on
RISC-V
- When UEFI runtime services are available, use them to restart and
power off
- Fix ARCH_HAS_ACPI_TABLE_UPGRADE for RISC-V by adding a missing
include file to the ACPI code
- Move contiguous DMA memory reservation later in the boot, to ensure
that the correct global and per-node pools are allocated when
CONFIG_CMA_SIZE_PERNUMA is enabled
- Disallow probes on breakpoint handlers to avoid recursing
indefinitely into do_trap_break()
- Patch the compat vDSO during runtime alternatives processing, not
only the standard vDSO
- Remove some leftover XIP support elements missed by commit
9b3a2be848 ("riscv: Remove support for XIP kernel")
- Fix broadcast timer switching in ACPI LPI power states by
implementing a RISC-V version of arch_get_idle_state_flags()
- Miscellaneous fixes, including: vDSO makefile simplification; marking
our default_power_off() as __noreturn; error path fixes in the PMU
SBI perf driver; RISC-V extension capitalization consistency; the use
of BIT() macros in one of our include files; and some documentation
fixes
* tag 'riscv-for-linus-7.3-mw1' of git://git.kernel.org/pub/scm/linux/kernel/git/riscv/linux: (39 commits)
RISC-V: hwprobe: Use BIT macro to avoid warnings
riscv: alternative: Also patch the compat vDSO
riscv: alternative: Use the statically extracted vDSO section offsets
riscv: vdso: Add symbols for the alternative section boundaries
riscv: Add Ziccamoa, Ziccif, Ziccrse, and Za64rs to cpufeature and hwprobe
riscv: Add Zicclsm to cpufeature and hwprobe
riscv: cpufeature: Clarify ISA spec version for canonical order
riscv: Standardize extension capitalization
riscv: kprobes: Prevent probes in breakpoint handlers
riscv: cmpxchg: Use .option arch for Zacas and Zabha
riscv: acpi: Handle LPI architectural context loss flags
riscv: further remove XIP
riscv: Add support for srmcfg CSR from Ssqosid extension
riscv: Detect the Ssqosid extension
dt-bindings: riscv: Add Ssqosid extension description
dt-bindings: riscv: add Counter delegation ISA extensions description
RISC-V: Add Ssccfg/Smcdeleg ISA extension definition and parsing
RISC-V: Add Ssccfg extension CSR definition
dt-bindings: riscv: add Smcntrpmf ISA extension description
RISC-V: Add Smcntrpmf extension parsing
...
- bitmap_find_next_zero_area_off() optimization (Sunyi);
- bitmap_find_next_zero_area_off(): return size when no zero area is
found (Yury);
- bitmap vs IDA vs Maple Tree performance test (Yury);
- get rid of cpumap_print_to_pagebuf() (Yury);
- use nr_node_ids in __nodemask_pr_numnodes() (Li RongQing);
- bitops: make the *_bit_le functions use unsigned long (Benjamin);
- bitmap scatter & gather test fix (Christophe);
- use __ASSEMBLER__ in bitmap header files (Thomas);
-----BEGIN PGP SIGNATURE-----
iQGzBAABCgAdFiEEi8GdvG6xMhdgpu/4sUSA/TofvsgFAmqDX1YACgkQsUSA/Tof
vsjcpwwAw7130xQPNFREr+ttdoUn3gN/aVUOl7K2Mi3kUOdlg8i9FEaQ/a07nE5L
ozXRAWgSM7VxW1I7nkj5McsGRiRX0SRoXmTbjDfPugMiXbD1UBPqCz2n0rYHYR4P
GfQ0RJK8LUN3Sq1eIu27HNirLG1HU1jCm3fsaTft2naaY6TnSPjklZdVE6Phz87E
eblgm+AO8//EoKsggoMCnGVCVWKE3N1RrWzsaT4ltErGyHvIzvWEbCCk2cI764Qj
orLLeefPf2xVDlbzywU7uUklNWYRe4miPxrbnT5TH2KmTipb3M62MkGjPa5dzYIq
ssOvSnKoQEV9od9cNv1hPFGs5b+RO+kOolcrWKTqWeHFgVTv7dZZL0QvuGHeOfOa
Fc5GPxE2SiXLPVuUEgMhlrg9DsXGf0d9u7JHTnJF+ZL/QhIE+4wOVnIYWUyBZiqx
zF7Ic62xmeJOKz9kfYg9JQYPwUblhhyYtiWOmzLIisf8FXi5M3L9eH6rO6CEWcrk
viEkTMbW
=q8Lf
-----END PGP SIGNATURE-----
Merge tag 'bitmap-for-7.3' of https://github.com/norov/linux
Pull bitmap updates from Yury Norov:
"The usual set of fixes, cleanups and performance improvements together
with a couple of new tests:
- bitmap_find_next_zero_area_off() optimization (Sunyi)
- bitmap_find_next_zero_area_off(): return size when no zero area is
found (Yury)
- bitmap vs IDA vs Maple Tree performance test (Yury)
- get rid of cpumap_print_to_pagebuf() (Yury)
- use nr_node_ids in __nodemask_pr_numnodes() (Li RongQing)
- bitops: make the *_bit_le functions use unsigned long (Benjamin)
- bitmap scatter & gather test fix (Christophe)
- use __ASSEMBLER__ in bitmap header files (Thomas)"
* tag 'bitmap-for-7.3' of https://github.com/norov/linux: (25 commits)
lib: test bitmap vs IDA vs Maple Tree performance for region allocations
bitmap: Return size when no zero area is found
media: s5p-mfc: Treat bitmap size as allocation failure
crypto: ccp: Treat bitmap size as allocation failure
powerpc/msi: Treat bitmap size as allocation failure
ARM: dma-mapping: Treat bitmap size as allocation failure
bitmap: drop bitmap_next_set_region()
nodemask: reduce bitmap width to nr_node_ids in __nodemask_pr_numnodes()
bitmap: Properly initialise destination bitmap for scatter & gather test
lib/bitmap-str: get rid of cpumap_print_to_pagebuf()
perf: Use sysfs_emit() for cpumask show callbacks
PCI/sysfs: Use sysfs_emit() for cpumask show callbacks
RDMA/hfi1: Use sysfs_emit() for cpumask show helper
hwtracing: hisi_ptt: Use sysfs_emit() for cpumask show
fpga: dfl-fme-perf: Use sysfs_emit() for cpumask show
devfreq: Use sysfs_emit() for cpumask show callbacks
cpu: Use sysfs_emit() for cpumask show callback
x86/events: Use sysfs_emit() for cpumask show callbacks
powerpc: Use sysfs_emit() for cpumask show callbacks
arm: Use sysfs_emit() for cpumask show callbacks
...
Sashiko pointed out various UAF and memory leak issues around
pmu_sbi_device_probe() error paths.
If the probe fails, here are list of cleanups needed.
a. Already registered pmu must be freed
b. per cpu IRQ must be released
c. pmu_ctr_list data structure must be freed
d. cpu hotplug state must be cleaned up only if added.
Fix the resource cleanup by reorganizing the code around probe failure.
Reported-by: Sashiko AI <sashiko-bot@kernel.org>
Reviewed-by: Charlie Jenkins <thecharlesjenkins@gmail.com>
Signed-off-by: Atish Patra <atishp@meta.com>
Link: https://patch.msgid.link/20260807-counter_delegation-v9-1-58658104e487@meta.com
Signed-off-by: Paul Walmsley <pjw@kernel.org>
PERF_SAMPLE_BRANCH_HW_INDEX is supported by BRBE so hw_id is passed to
userspace, but it's never set by the BRBE driver. Zero initialize it as
it should be according to the docs:
* For the architectures whose raw branch records are
* already stored in age order, the hw_idx should be 0.
It's probably too risky to remove PERF_SAMPLE_BRANCH_HW_INDEX from BRBE
now in case anyone is setting it and reading the value, but zero
initializing the whole struct also protects against the same issue with
new fields that are added in the future.
Fixes: 58074a0fce ("perf: arm_pmuv3: Add support for the Branch Record Buffer Extension (BRBE)")
Signed-off-by: James Clark <james.clark@linaro.org>
Reviewed-by: Anshuman Khandual <anshuman.khandual@arm.com>
Signed-off-by: Will Deacon <will@kernel.org>
If you were disappointed at how minimal the initial CMN S3 support
looked compared to previous versions, then oh boy is it time to put your
party hats on... The biggest batch of incompatible changes yet comes not
with a new CMN product, but a point release of an existing one. We've
got new filters, loads of changes to existing events, register fields
moved around for no good reason, and much, much more! On the upside, we
do at least gain a means of working around the isolation feature.
As such, for the sake of sanity in the driver it is easiest to split it
into a distict "model" for our internal abstractions despite it bearing
the same part number as r0/r1.
Signed-off-by: Robin Murphy <robin.murphy@arm.com>
Reviewed-by: Ilkka Koskinen <ilkka@os.amperecomputing.com>
Signed-off-by: Will Deacon <will@kernel.org>
Add the logic to handle events with the upcoming new filter controls.
Since for now we will have the sole invariant of all EVICT_STATE_SEL
events having HBT_LBT_SEL as a secondary filter, for the sake of
simplicity we can just special-case that, and save the complication
of a full multi-filter abstraction until unavoidably necessary.
Reviewed-by: Leo Yan <leo.yan@arm.com>
Signed-off-by: Robin Murphy <robin.murphy@arm.com>
Reviewed-by: Ilkka Koskinen <ilkka@os.amperecomputing.com>
Signed-off-by: Will Deacon <will@kernel.org>
The ABI hole I have dug myself into requires the driver to know which
event encodings are associated with which particular filter control.
Since we will soon have a notion of multiple filters per event, refactor
the event data to encapsulate filters in an explicit structure, which
can then more easily scale as an array in future.
Signed-off-by: Robin Murphy <robin.murphy@arm.com>
Reviewed-by: Ilkka Koskinen <ilkka@os.amperecomputing.com>
Signed-off-by: Will Deacon <will@kernel.org>
We're soon going to need to cope with events having multiple filters,
plus the filter fields themselves moving around, wherein any more inline
if/else logic will struggle to scale. Add a more general abstraction for
the node-specific filter controls, and rejig the pmu_event_sel filter
programming around it in a more extensible manner.
Reviewed-by: Ilkka Koskinen <ilkka@os.amperecomputing.com>
Signed-off-by: Robin Murphy <robin.murphy@arm.com>
Signed-off-by: Will Deacon <will@kernel.org>
CMN has already grown many more event-specific filters than the original
Occupancy ID, but since they are all independent of each other we've
just overloaded them onto the same name. Before we add yet more, and
they begin to overlap, rename all our "occupid" variables to "filter" so
that things can be a bit clearer and more consistent (but leaving the
format attribute itself, to avoid UAPI concerns).
Reviewed-by: Ilkka Koskinen <ilkka@os.amperecomputing.com>
Signed-off-by: Robin Murphy <robin.murphy@arm.com>
Signed-off-by: Will Deacon <will@kernel.org>
The 64-bit Programmers' model extension, now named FEAT_CSPMU_EXT64,
makes all the non-counter registers 64-bit as well, of which we most
need to care PMEVFILTR<n>, PMEVTYPER<n>, and PMCR since it changes
location. Our event config fields are ready for this internally, but
we need a few more tweaks to propagate 64-bit values properly from
end to end.
Since 64-bit events leave no room for our existing "cycles" encoding,
and we don't really expect to see implementations of it anyway, we
deliberately omit generic support for the architectural fixed cycle
counter in this case to save some bother. At worst, vendor modules
can still implement their own encoding if they really want to.
Signed-off-by: Robin Murphy <robin.murphy@arm.com>
Reviewed-by: Ilkka Koskinen <ilkka@os.amperecomputing.com>
Signed-off-by: Will Deacon <will@kernel.org>
Since commit 55b48e23f5 ("genirq/devres: Add error handling in
devm_request_*_irq()"), devm_request_irq() automatically logs
detailed error messages on failure. Remove the now-redundant
driver-specific dev_err() and dev_err_probe() calls.
Signed-off-by: Pan Chuang <panchuang@vivo.com>
Acked-by: Yushan Wang <wangyushan12@huawei.com>
Signed-off-by: Will Deacon <will@kernel.org>
Since commit 55b48e23f5 ("genirq/devres: Add error handling in
devm_request_*_irq()"), devm_request_irq() automatically logs
detailed error messages on failure. Remove the now-redundant
driver-specific dev_err() calls.
Signed-off-by: Pan Chuang <panchuang@vivo.com>
Reviewed-by: Ilkka Koskinen <ilkka@os.amperecomputing.com>
Signed-off-by: Will Deacon <will@kernel.org>
Since commit 55b48e23f5 ("genirq/devres: Add error handling in
devm_request_*_irq()"), devm_request_irq() automatically logs
detailed error messages on failure. Remove the now-redundant
driver-specific dev_err() and dev_err_probe() calls.
Signed-off-by: Pan Chuang <panchuang@vivo.com>
Reviewed-by: Shuai Xue <xueshuai@linux.alibaba.com>
Acked-by: Frank Li <Frank.Li@kernel.org>
Reviewed-by: Xu Yang <xu.yang_2@nxp.com>
Signed-off-by: Will Deacon <will@kernel.org>
The ACPI spec bizarrely indicates that the PMU interrupt can be
edge-triggered, which contradicts the very basics of the PMU
architecture (SW is required to clear the interrupt condition for
the level to drop).
Remove the code parsing this flag and always flag the interrupt
as level triggered, no matter what firmware says.
Signed-off-by: Marc Zyngier <maz@kernel.org>
Reviewed-by: Zenghui Yu <yuzenghui@huawei.com>
Signed-off-by: Will Deacon <will@kernel.org>
If we have 64-bit counters, we can reasonably assume we'll never have to
handle an overflow before the end of the universe (since we're a system
PMU with no sampling). Thus even if firmware does specify an IRQ, we can
still continue in the event of being unable to request it. This can help
systems where IRQs cannot be claimed exclusively, or are broken in other
ways.
Reviewed-by: Ilkka Koskinen <ilkka@os.amperecomputing.com>
Signed-off-by: Robin Murphy <robin.murphy@arm.com>
Signed-off-by: Will Deacon <will@kernel.org>
When waiting for a sub-module to register, we return a bare
-EPROBE_DEFER that ends up showing the end user:
platform arm-cs-arch-pmu.1: deferred probe pending (no reason)
wherein it's not necessarily clear that they might need to take some
action to ensure the appropriate module is available to load. Let's use
dev_err_probe() here so we can show exactly what we're waiting for.
Similarly, in the case where something's gone horribly wrong with an
already-registered module, we can use dev_WARN() to standardise the
device/driver attribution rather than just open-coding "arm_cspmu".
Reviewed-by: Ilkka Koskinen <ilkka@os.amperecomputing.com>
Signed-off-by: Robin Murphy <robin.murphy@arm.com>
Signed-off-by: Will Deacon <will@kernel.org>
On ACPI systems, it has not actually been possible for userspace to
reliably tell which PMU corresponds to which APMT entry for types
other than "ACPI device" - the evidence trail only leads from the
arbitrarily-numbered PMU device to its arbitrarily-numbered parent
platform device that has no distinguishing features either.
While we've now improved the platform device creation to associate the
actual APMT unique ID, we may as well also tweak the PMU devices to
substitute the arbitrary number with a different arbitrary number that
might be more directly meaningful based on the APMT definitions.
We don't have an equivalent for Devicetree, but in that case the
platform devices are at least identifiable via their sysfs-visible
of_node.
Signed-off-by: Robin Murphy <robin.murphy@arm.com>
Reviewed-by: Ilkka Koskinen <ilkka@os.amperecomputing.com>
[will: Remove unneeded semicolon reported by coccinelle]
Signed-off-by: Will Deacon <will@kernel.org>
Implement CRB filtering per CXL 4.0 8.2.7.2.2, and extend the
current filtering support beyond HDM. CRB filtering is only
permitted for the DDR Interface, Queue Occupancy, Queue Residency
and Retry event groups (CXL 4.0 Table 13-5), and only when counting
a single event (a single mask bit). Because these group IDs are
scoped by the CXL vendor ID, events from other vendors are also
rejected. For example, to count DDR activates on channel 2 only:
perf stat -a -e cxl_pmu_mem0.0/ddr_act,crb_filter_en=1,crb=0x02FFFFFF/
Placing the 32-bit CRB value at config2:32-63 leaves the
existing HDM value at config2:0-15 untouched and avoids needing
a new config3.
Signed-off-by: Harshal Thakkar <harshal.t@samsung.com>
[davidlohr: multiple fixes]
Signed-off-by: Davidlohr Bueso <dave@stgolabs.net>
Reviewed-by: Richard Cheng <icheng@nvidia.com>
Reviewed-by: Dave Jiang <dave.jiang@intel.com>
Signed-off-by: Will Deacon <will@kernel.org>
Add support for CXL 4.0 events that are exposed by the CPMU hardware
but not present in the driver. Such events are defined in Table 13-5
of the spec.
Signed-off-by: Harshal Thakkar <harshal.t@samsung.com>
[davidlohr: add missing throttle and queue occupancy events]
Signed-off-by: Davidlohr Bueso <dave@stgolabs.net>
Reviewed-by: Richard Cheng <icheng@nvidia.com>
Reviewed-by: Dave Jiang <dave.jiang@intel.com>
Signed-off-by: Will Deacon <will@kernel.org>
The HDM decoder filter configuration register is 32 bits wide, but the
driver programs it with a 64-bit writeq(). The filter value never
exceeds 32 bits, so the upper half of the write is always zero and
lands in the adjacent Filter ID 1 (Channel/Rank/Bank) configuration
register at offset+4.
Fixes: 5d7107c727 ("perf: CXL Performance Monitoring Unit driver")
Signed-off-by: Davidlohr Bueso <dave@stgolabs.net>
Reviewed-by: Richard Cheng <icheng@nvidia.com>
Reviewed-by: Dave Jiang <dave.jiang@intel.com>
Signed-off-by: Will Deacon <will@kernel.org>
The DWC PCIe Time-Based Analysis Data Register (the counter for time-based
events) is architected as 64-bit, but some hardware implementations do not
implement the full width. On these implementations the counter stops after
reaching its implemented width. This will limit the usage for short time
monitoring only. The counter will only cover ~15s for monitoring RX TLP
payloads on our platform.
Add an optional hrtimer that fires every 2 seconds. It'll take the role
as the counter overflow interrupt to read-update-reset the counter and
event counts to break the limits of the narrow counters. It'll only
apply on timer-based counter. The 2 seconds update period is the half
of the maximum counting period (4s) of the time-based counter under
period counting mode of the hardware.
Because fully-implemented 64-bit counters do not need this workaround,
enable this hrtimer on the platforms known to have narrowed counter.
Before this patch, when counting fio for 10m the counts is incorrect:
root@localhost:/tmp# perf stat -e dwc_rootport_20000/rx_pcie_tlp_data_payload/ -- fio --runtime=10m fio_job.config
[...]
Run status group 0 (all jobs):
READ: bw=5594MiB/s (5865MB/s), 5594MiB/s-5594MiB/s (5865MB/s-5865MB/s), io=3278GiB (3519GB), run=600010-600010msec
[...]
Performance counter stats for 'system wide':
137,438,953,456 dwc_rootport_20000/rx_pcie_tlp_data_payload/
After this patch the counts is as expected:
root@localhost:/tmp# perf stat -e dwc_rootport_20000/rx_pcie_tlp_data_payload/ -- fio --runtime=10m fio_job.config
[...]
Run status group 0 (all jobs):
READ: bw=5632MiB/s (5905MB/s), 5632MiB/s-5632MiB/s (5905MB/s-5905MB/s), io=3300GiB (3543GB), run=600013-600013msec
[...]
Performance counter stats for 'system wide':
3,543,850,268,576 dwc_rootport_20000/rx_pcie_tlp_data_payload/
Signed-off-by: Yufan Dou <douyufan@picoheart.com>
Signed-off-by: Yicong Yang <yang.yicong@picoheart.com>
Signed-off-by: Will Deacon <will@kernel.org>
cn10k_ddr_perf_remove() did not cancel the poll hrtimer before returning.
If the device was unbound while perf events were still active the timer
callback could run post-free.
To fix the issue by adding hrtimer_cancel() in remove().
Signed-off-by: Geetha sowjanya <gakula@marvell.com>
Signed-off-by: Will Deacon <will@kernel.org>
The CN20K DRAM Subsystem exposes eight programmable
performance counters and two fixed counters for DDR
read and write traffic. Software selects events for
the programmable counters from traffic at the DDR PHY
interface, the CHI interconnect, or inside the DDR controller.
Add CN20K register offsets, event maps, and sysfs attributes;
match the device via OF (marvell,cn20k-ddr-pmu) and ACPI (MRVL000B).
Represent the SoC variant in platform data with bit flags so
CN20K can reuse the CN10K PMU code path where appropriate.
Signed-off-by: Geetha sowjanya <gakula@marvell.com>
Signed-off-by: Will Deacon <will@kernel.org>
The current code checks whether the wakeup point is in the current
writable range by comparing it with handle->head + handle->size.
The perf AUX head is a monotonically increasing index, so that addition
can overflow when head is close to ULONG_MAX. In that case, a wakeup
point which is still inside the free space range can be missed.
Use unsigned subtraction to compare the distance from head to wakeup
against the handle->size. This can dismiss the issue when addition
overflow.
This is unlikely to happen in practice, but the change makes the
watermark check logically correct.
Fixes: d5d9696b03 ("drivers/perf: Add support for ARMv8.2 Statistical Profiling Extension")
Signed-off-by: Leo Yan <leo.yan@arm.com>
Signed-off-by: Will Deacon <will@kernel.org>
These callbacks are sysfs show paths. Use sysfs_emit() and
cpumask_pr_args() to emit the masks.
This prepares for removing cpumap_print_to_pagebuf().
Link: https://lore.kernel.org/all/akANJ-AT7nHpRMq-@yury/
Acked-by: Robin Murphy <robin.murphy@arm.com>
Signed-off-by: Yury Norov <ynorov@nvidia.com>
The PMCCNTR_EL0 in NVIDIA Olympus CPU may increment while
in WFI/WFE, which does not align with counting CPU_CYCLES
on a programmable counter. Add a MIDR range entry and refuse
PMCCNTR_EL0 for cycle events on affected parts so perf does
not mix the two behaviors.
Also keep PMCCNTR_EL0 unavailable to EL0 direct counter reads
on affected CPUs. When userspace counter access is enabled,
avoid setting PMUSERENR_EL0.CR for PMUs that must avoid
PMCCNTR_EL0, while still allowing direct reads from programmable
event counters. For 64-bit userspace CPU_CYCLES events on PMUs
without native long event counters, reject the event if the only
valid direct-read path would be PMCCNTR_EL0.
Signed-off-by: Besar Wicaksono <bwicaksono@nvidia.com>
Signed-off-by: Will Deacon <will@kernel.org>
Graviton5 uses a customised CMN-S3 implementation where certain
discovery registers report zeroed fields. Add the following workarounds:
- Introduce a dedicated ACPI HID to identify the Graviton5 CMN variant.
- Derive the DTC domain from the XP node ID, since the unit info
register reports it as zero.
- Set the DTC logical ID from the XP's logical ID, since the node info
register's logical ID field is also zeroed.
Signed-off-by: Aviv Bakal <avivb@amazon.com>
Reviewed-by: Robin Murphy <robin.murphy@arm.com>
Reviewed-by: Ilkka Koskinen <ilkka@os.amperecomputing.com>
Signed-off-by: Robin Murphy <robin.murphy@arm.com>
Signed-off-by: Will Deacon <will@kernel.org>
The amount of data we need to store all the per-DTM counter and
watchpoint allocations is already testing the limits of hw_perf_event,
and future CMNs are only likely to keep growing larger, so move these
arrays out to separate memory allocations. As part of that we can use
an explicit union for allocating cycle counters to dtc_cycles events,
which is arguably nicer anyway.
Reviewed-by: Ilkka Koskinen <ilkka@os.amperecomputing.com>
Signed-off-by: Robin Murphy <robin.murphy@arm.com>
Signed-off-by: Will Deacon <will@kernel.org>
Replace the #include of <linux/mod_devicetable.h> by the more specific
<linux/device-id/*.h> where applicable. For most cases the include
can be dropped completely, only a few drivers need one or two headers
added.
Acked-by: Danilo Krummrich <dakr@kernel.org>
Acked-by: Takashi Sakamoto <o-takashi@sakamocchi.jp>
Acked-by: Bjorn Helgaas <bhelgaas@google.com>
Link: https://patch.msgid.link/1a3f2007c5c5dcf555c09a4035ce3ae8ef1b6c49.1782808461.git.u.kleine-koenig@baylibre.com
Signed-off-by: Uwe Kleine-König (The Capable Hub) <u.kleine-koenig@baylibre.com>
The new DVM node events added in CMN-700 also apply to CMN S3; fix
the model encoding so that we can expose the aliases and handle
occupancy filtering on newer CMNs too.
Cc: stable@vger.kernel.org
Fixes: 0dc2f4963f ("perf/arm-cmn: Support CMN S3")
Signed-off-by: Robin Murphy <robin.murphy@arm.com>
Signed-off-by: Will Deacon <will@kernel.org>
Various names for Qualcomm as a company are used in user-visible config
options: QCOM, Qualcomm and Qualcomm Technologies. Switch to unified
"Qualcomm" so it will be easier for users to identify the options when
for example running menuconfig.
Signed-off-by: Krzysztof Kozlowski <krzysztof.kozlowski@oss.qualcomm.com>
Signed-off-by: Will Deacon <will@kernel.org>
Core features:
- Add support for FEAT_LSUI, allowing futex atomic operations without
toggling Privileged Access Never (PAN)
- Further refactor the arm64 exception handling code towards the
generic entry infrastructure
- Optimise __READ_ONCE() with CONFIG_LTO=y and allow alias analysis
through it
Memory management:
- Refactor the arm64 TLB invalidation API and implementation for better
control over barrier placement and level-hinted invalidation
- Enable batched TLB flushes during memory hot-unplug
- Fix rodata=full block mapping support for realm guests (when
BBML2_NOABORT is available)
Perf and PMU:
- Add support for a whole bunch of system PMUs featured in NVIDIA's
Tegra410 SoC (cspmu extensions for the fabric and PCIe, new drivers
for CPU/C2C memory latency PMUs)
- Clean up iomem resource handling in the Arm CMN driver
- Fix signedness handling of AA64DFR0.{PMUVer,PerfMon}
MPAM (Memory Partitioning And Monitoring):
- Add architecture context-switch and hiding of the feature from KVM
- Add interface to allow MPAM to be exposed to user-space using resctrl
- Add errata workaround for some existing platforms
- Add documentation for using MPAM and what shape of platforms can use
resctrl
Miscellaneous:
- Check DAIF (and PMR, where relevant) at task-switch time
- Skip TFSR_EL1 checks and barriers in synchronous MTE tag check mode
(only relevant to asynchronous or asymmetric tag check modes)
- Remove a duplicate allocation in the kexec code
- Remove redundant save/restore of SCS SP on entry to/from EL0
- Generate the KERNEL_HWCAP_ definitions from the arm64 hwcap
descriptions
- Add kselftest coverage for cmpbr_sigill()
- Update sysreg definitions
-----BEGIN PGP SIGNATURE-----
iQIzBAABCgAdFiEE5RElWfyWxS+3PLO2a9axLQDIXvEFAmnc8DEACgkQa9axLQDI
XvFauRAAhc1cIgoRpgtdZd7+3/g457teDPYA3L/CjJzI28aesIpV/ECrEw2GL4xs
HrQfijF4oyCDbBwh0sAascO/H7RoyOranlbuc+fVJ6Bj6gP9STzR4GmscsWkAMSJ
vA3Jd1DREdDBO2sjw+hGhht84nRlcfY1FyORJP+1JaFH4oWTWsRNeOZIiI3BhxR8
EtFP9E8r2Esxi/FmZb/47m7kYCEH+XsrzQvBQNLVCH899QX2Hn0kAY70ndq2ZiQl
n+zLAe7FBFwKzUVmlgWuhjrWMmK+1TthK/XQuOtxg13dHmX+vE/j+A+dOqRWSfHY
ktNcWaf6m4+TWKVeVTe4E1cnSuwTQTm4VQKd9zaeQxiZYyYJhCQjXuEZg3vDmDbq
F6D3MpTaJHRRWp0rEurxnSBlmQPCBE2IxEBdSrjd/WJ6T9e1oYwWiSJSS7bGCgGr
dd/XLsOY7Um5n4ooIFEZc1de6VO6/VTKjmxnBMgU+Sa1REbLpD438IX/6CjzG5qM
l5Ulke/c6/a/faeVCEpZpD8JuvNOzo9RISDPrNg1KKAL+OSU+9tgmVjIFPhDDB0w
zNTqT7YJIhxlJxnUGWDk8YNsTjT3OzyquY9UT1tBTBqC0k13J2i2ev30toUez7xj
2aV+9qMpunbLtwYhXNun1hBFiYrCxpX7I8ha0hXiXL0CywVOPTI=
=CnVn
-----END PGP SIGNATURE-----
Merge tag 'arm64-upstream' of git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux
Pull arm64 updates from Catalin Marinas:
"The biggest changes are MPAM enablement in drivers/resctrl and new PMU
support under drivers/perf.
On the core side, FEAT_LSUI lets futex atomic operations with EL0
permissions, avoiding PAN toggling.
The rest is mostly TLB invalidation refactoring, further generic entry
work, sysreg updates and a few fixes.
Core features:
- Add support for FEAT_LSUI, allowing futex atomic operations without
toggling Privileged Access Never (PAN)
- Further refactor the arm64 exception handling code towards the
generic entry infrastructure
- Optimise __READ_ONCE() with CONFIG_LTO=y and allow alias analysis
through it
Memory management:
- Refactor the arm64 TLB invalidation API and implementation for
better control over barrier placement and level-hinted invalidation
- Enable batched TLB flushes during memory hot-unplug
- Fix rodata=full block mapping support for realm guests (when
BBML2_NOABORT is available)
Perf and PMU:
- Add support for a whole bunch of system PMUs featured in NVIDIA's
Tegra410 SoC (cspmu extensions for the fabric and PCIe, new drivers
for CPU/C2C memory latency PMUs)
- Clean up iomem resource handling in the Arm CMN driver
- Fix signedness handling of AA64DFR0.{PMUVer,PerfMon}
MPAM (Memory Partitioning And Monitoring):
- Add architecture context-switch and hiding of the feature from KVM
- Add interface to allow MPAM to be exposed to user-space using
resctrl
- Add errata workaround for some existing platforms
- Add documentation for using MPAM and what shape of platforms can
use resctrl
Miscellaneous:
- Check DAIF (and PMR, where relevant) at task-switch time
- Skip TFSR_EL1 checks and barriers in synchronous MTE tag check mode
(only relevant to asynchronous or asymmetric tag check modes)
- Remove a duplicate allocation in the kexec code
- Remove redundant save/restore of SCS SP on entry to/from EL0
- Generate the KERNEL_HWCAP_ definitions from the arm64 hwcap
descriptions
- Add kselftest coverage for cmpbr_sigill()
- Update sysreg definitions"
* tag 'arm64-upstream' of git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux: (109 commits)
arm64: rsi: use linear-map alias for realm config buffer
arm64: Kconfig: fix duplicate word in CMDLINE help text
arm64: mte: Skip TFSR_EL1 checks and barriers in synchronous tag check mode
arm64/sysreg: Update ID_AA64SMFR0_EL1 description to DDI0601 2025-12
arm64/sysreg: Update ID_AA64ZFR0_EL1 description to DDI0601 2025-12
arm64/sysreg: Update ID_AA64FPFR0_EL1 description to DDI0601 2025-12
arm64/sysreg: Update ID_AA64ISAR2_EL1 description to DDI0601 2025-12
arm64/sysreg: Update ID_AA64ISAR0_EL1 description to DDI0601 2025-12
arm64/hwcap: Generate the KERNEL_HWCAP_ definitions for the hwcaps
arm64: kexec: Remove duplicate allocation for trans_pgd
ACPI: AGDI: fix missing newline in error message
arm64: Check DAIF (and PMR) at task-switch time
arm64: entry: Use split preemption logic
arm64: entry: Use irqentry_{enter_from,exit_to}_kernel_mode()
arm64: entry: Consistently prefix arm64-specific wrappers
arm64: entry: Don't preempt with SError or Debug masked
entry: Split preemption from irqentry_exit_to_kernel_mode()
entry: Split kernel mode logic from irqentry_{enter,exit}()
entry: Move irqentry_enter() prototype later
entry: Remove local_irq_{enable,disable}_exit_to_user()
...
Update arm_cspmu to use acpi_get_cpu_uid() instead of
get_acpi_id_for_cpu(), aligning with unified ACPI CPU UID interface.
No functional changes are introduced by this switch (valid inputs retain
original behavior).
Signed-off-by: Chengwen Feng <fengchengwen@huawei.com>
Reviewed-by: Jonathan Cameron <jonathan.cameron@huawei.com>
Link: https://patch.msgid.link/20260401081640.26875-7-fengchengwen@huawei.com
Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
When building for 32-bit ARM, there is a warning when using the %llx
specifier to print a resource_size_t variable:
drivers/perf/arm-cmn.c: In function 'arm_cmn_init_dtc':
drivers/perf/arm-cmn.c:2149:73: error: format '%llx' expects argument of type 'long long unsigned int', but argument 4 has type 'resource_size_t' {aka 'unsigned int'} [-Werror=format=]
2149 | "Failed to request DTC region 0x%llx\n", base);
| ~~~^ ~~~~
| | |
| | resource_size_t {aka unsigned int}
| long long unsigned int
| %x
Use the %pa specifier to handle the possible sizes of phys_addr_t
properly. This requires passing the variable by reference.
Fixes: 5394396ff5 ("perf/arm-cmn: Stop claiming entire iomem region")
Signed-off-by: Nathan Chancellor <nathan@kernel.org>
Reviewed-by: Robin murphy <robin.murphy@arm.com>
Signed-off-by: Will Deacon <will@kernel.org>
Check devm_ioremap() return value for NULL instead of ERR_PTR and return
-ENOMEM on failure. devm_ioremap() never returns ERR_PTR, using IS_ERR()
skips the error path and may cause a NULL pointer dereference.
Fixes: 5394396ff5 ("perf/arm-cmn: Stop claiming entire iomem region")
Signed-off-by: Chen Ni <nichen@iscas.ac.cn>
Signed-off-by: Will Deacon <will@kernel.org>
Adds NVIDIA C2C PMU support in Tegra410 SOC. This PMU is
used to measure memory latency between the SOC and device
memory, e.g GPU Memory (GMEM), CXL Memory, or memory on
remote Tegra410 SOC.
Reviewed-by: Ilkka Koskinen <ilkka@os.amperecomputing.com>
Signed-off-by: Besar Wicaksono <bwicaksono@nvidia.com>
Signed-off-by: Will Deacon <will@kernel.org>
Adds CPU Memory (CMEM) Latency PMU support in Tegra410 SOC.
The PMU is used to measure latency between the edge of the
Unified Coherence Fabric to the local system DRAM.
Reviewed-by: Ilkka Koskinen <ilkka@os.amperecomputing.com>
Signed-off-by: Besar Wicaksono <bwicaksono@nvidia.com>
Signed-off-by: Will Deacon <will@kernel.org>
Adds PCIE-TGT PMU support in Tegra410 SOC. This PMU is
instanced in each root complex in the SOC and it captures
traffic originating from any source towards PCIE BAR and CXL
HDM range. The traffic can be filtered based on the
destination root port or target address range.
Reviewed-by: Ilkka Koskinen <ilkka@os.amperecomputing.com>
Signed-off-by: Besar Wicaksono <bwicaksono@nvidia.com>
Signed-off-by: Will Deacon <will@kernel.org>
Adds PCIE PMU support in Tegra410 SOC. This PMU is instanced
in each root complex in the SOC and can capture traffic from
PCIE device to various memory types. This PMU can filter traffic
based on the originating root port or BDF and the target memory
types (CPU DRAM, GPU Memory, CXL Memory, or remote Memory).
Reviewed-by: Ilkka Koskinen <ilkka@os.amperecomputing.com>
Signed-off-by: Besar Wicaksono <bwicaksono@nvidia.com>
Signed-off-by: Will Deacon <will@kernel.org>
Add interface to get ACPI device associated with the
PMU. This ACPI device may contain additional properties
not covered by the standard properties.
Reviewed-by: Ilkka Koskinen <ilkka@os.amperecomputing.com>
Signed-off-by: Besar Wicaksono <bwicaksono@nvidia.com>
Signed-off-by: Will Deacon <will@kernel.org>
The Unified Coherence Fabric (UCF) contains last level cache
and cache coherent interconnect in Tegra410 SOC. The PMU in
this device can be used to capture events related to access
to the last level cache and memory from different sources.
Reviewed-by: Ilkka Koskinen <ilkka@os.amperecomputing.com>
Signed-off-by: Besar Wicaksono <bwicaksono@nvidia.com>
Signed-off-by: Will Deacon <will@kernel.org>
So far, the PMU has been the only thing of interest in the vast mass
of CMN registers, so we've gotten away with simply claiming the entire
iomem region. However, now that we can support other features like MPAM
controllers for the system caches, the PMU driver needs to stop being
selfish and learn to share. Similarly to arm-ni, requesting just the
DTC node(s) should suffice for staking our exclusive claim to the PMU
features, as requesting hundreds of tiny regions for all the individual
pmu_event_sel registers is definitely not worth the considerable bother.
As a consequence, we can also streamline the annoying CMN-600 special
cases even more. The ACPI binding has in fact always specified a strict
order for all resources, so we can reasonably drop the ancient pretence
of swapping base and cfg, which IIRC was more just a moment of doubt on
my part than anything else.
Cc: James Morse <james.morse@arm.com>
Signed-off-by: Robin Murphy <robin.murphy@arm.com>
Tested-by: Ben Horgan <ben.horgan@arm.com>
Reviewed-by: Ilkka Koskinen <ilkka@os.amperecomputing.com>
Signed-off-by: Will Deacon <will@kernel.org>
This converts some of the visually simpler cases that have been split
over multiple lines. I only did the ones that are easy to verify the
resulting diff by having just that final GFP_KERNEL argument on the next
line.
Somebody should probably do a proper coccinelle script for this, but for
me the trivial script actually resulted in an assertion failure in the
middle of the script. I probably had made it a bit _too_ trivial.
So after fighting that far a while I decided to just do some of the
syntactically simpler cases with variations of the previous 'sed'
scripts.
The more syntactically complex multi-line cases would mostly really want
whitespace cleanup anyway.
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
This was done entirely with mindless brute force, using
git grep -l '\<k[vmz]*alloc_objs*(.*, GFP_KERNEL)' |
xargs sed -i 's/\(alloc_objs*(.*\), GFP_KERNEL)/\1)/'
to convert the new alloc_obj() users that had a simple GFP_KERNEL
argument to just drop that argument.
Note that due to the extreme simplicity of the scripting, any slightly
more complex cases spread over multiple lines would not be triggered:
they definitely exist, but this covers the vast bulk of the cases, and
the resulting diff is also then easier to check automatically.
For the same reason the 'flex' versions will be done as a separate
conversion.
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
This is the result of running the Coccinelle script from
scripts/coccinelle/api/kmalloc_objs.cocci. The script is designed to
avoid scalar types (which need careful case-by-case checking), and
instead replace kmalloc-family calls that allocate struct or union
object instances:
Single allocations: kmalloc(sizeof(TYPE), ...)
are replaced with: kmalloc_obj(TYPE, ...)
Array allocations: kmalloc_array(COUNT, sizeof(TYPE), ...)
are replaced with: kmalloc_objs(TYPE, COUNT, ...)
Flex array allocations: kmalloc(struct_size(PTR, FAM, COUNT), ...)
are replaced with: kmalloc_flex(*PTR, FAM, COUNT, ...)
(where TYPE may also be *VAR)
The resulting allocations no longer return "void *", instead returning
"TYPE *".
Signed-off-by: Kees Cook <kees@kernel.org>
So far we've been fairly lax about accepting both unknown CMN models
(at least with a warning), and unknown revisions of those which we
do know, as although things do frequently change between releases,
typically enough remains the same to be somewhat useful for at least
some basic bringup checks. However, we also make assumptions of the
maximum supported sizes and numbers of things in various places, and
there's no guarantee that something new might not be bigger and lead
to nasty array overflows. Make sure we only try to run on things that
actually match our assumptions and so will not risk memory corruption.
We have at least always failed on completely unknown node types, so
update that error message for clarity and consistency too.
Cc: stable@vger.kernel.org
Fixes: 7819e05a0d ("perf/arm-cmn: Revamp model detection")
Reviewed-by: Ilkka Koskinen <ilkka@os.amperecomputing.com>
Signed-off-by: Robin Murphy <robin.murphy@arm.com>
Signed-off-by: Will Deacon <will@kernel.org>
When arm_spe_pmu_next_off() fails to calculate a valid limit, it returns
zero to indicate that tracing should not start. However, the caller
arm_spe_perf_aux_output_begin() does not propagate this failure by
updating hwc->state, cause the error to be silently ignored by upper
layers.
Because hwc->state remains zero after a failure, arm_spe_pmu_start()
continues to programs filter registers unnecessarily. The driver
still reports success to the perf core, so the core assumes the SPE
event was enabled and proceeds to enable other events. This breaks
event group semantics: SPE is already stopped while other events in the
same group are enabled.
Fix this by updating arm_spe_perf_aux_output_begin() to return a status
code indicating success (0) or failure (-EIO). Both the interrupt
handler and arm_spe_pmu_start() check the return value and call
arm_spe_pmu_stop() to set PERF_HES_STOPPED in hwc->state.
In the interrupt handler, the period (e.g., period_left) needs to be
updated, so PERF_EF_UPDATE is passed to arm_spe_pmu_stop(). When the
error occurs during event start, the trace unit is not yet enabled, so
a flag '0' is used to drain buffer and update state only.
Fixes: d5d9696b03 ("drivers/perf: Add support for ARMv8.2 Statistical Profiling Extension")
Signed-off-by: Leo Yan <leo.yan@arm.com>
Signed-off-by: Will Deacon <will@kernel.org>
Passing IRQF_ONESHOT ensures that the interrupt source is masked until
the secondary (threaded) handler is done. If only a primary handler is
used then the flag makes no sense because the interrupt can not fire
(again) while its handler is running.
The flag also disallows force-threading of the primary handler and the
irq-core will warn about this.
The intention here was probably not allowing forced-threading.
Replace IRQF_ONESHOT with IRQF_NO_THREAD.
Reviewed-by: Jonathan Cameron <jonathan.cameron@huawei.com>
Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Signed-off-by: Will Deacon <will@kernel.org>
Since we do not use the divide-by-64 option, there should be no
significant difference between the dedicated cycle counter and the
standard cycles event. Since using the latter on DSU-120 now has
the side-effect of allowing multiple cycles events to be scheduled
simultaneously (beneficial for multiple cycle-based metrics), there
seems little reason not to allow the same on older DSUs as well.
Signed-off-by: Robin Murphy <robin.murphy@arm.com>
Signed-off-by: Will Deacon <will@kernel.org>
DSU-120 has the same system register interface as previous DSUs, but
no longer offers a dedicated cycle counter. While this is not directly
discoverable via PMCR, the PMCCNTR register is still defined to exist
with RAZ/WI behaviour, allowing for a straightforward heuristic.
Signed-off-by: Robin Murphy <robin.murphy@arm.com>
Signed-off-by: Will Deacon <will@kernel.org>
