For each vCPU, pKVM's EL2 builds its own HCR_EL2 and takes only TWI, TWE
and VSE from the host's value on each entry. For a non-protected VM it
has fallen behind the host's: on a CPU with MTE the guest can read
GMID_EL1, on one with FEAT_EVT2 but no FGT it can execute a TLBI OS its
ID registers hide, an interrupt injected without a vGIC (VI, VF) never
arrives, and set/way emulation loses the TVM trap it relies on.
For a non-protected VM, also take the bits the host varies with the VM's
configuration or at runtime: VI, VF, TVM, TID2, TID4, TID5 and TTLBOS.
They only pend interrupts for, or add traps to, a VM the host controls.
The traps exit to the host, which handles them as it does without pKVM.
The bits that change what EL2 does on entry and exit (E2H, RW, API/APK)
or depend only on the CPU (TEA, TERR, FWB) stay EL2's. A protected VM
still takes only TWI, TWE and VSE.
EL2 now sets TID2 or TID4 only for a protected VM, and never sets ATA,
as the host rejects KVM_CAP_ARM_MTE in pKVM.
Fixes: b56680de9c ("KVM: arm64: Initialize trap register values in hyp in pKVM")
Signed-off-by: Fuad Tabba <fuad.tabba@linux.dev>
Link: https://patch.msgid.link/20260929090031.3829185-4-fuad.tabba@linux.dev
Signed-off-by: Oliver Upton <oupton@kernel.org>
pKVM keeps its own copy of each vCPU's HCR_EL2 at EL2, initialised with
RW set unconditionally. Nothing stops userspace from creating a
non-protected AArch32 VM, and with RW set, entering one of its vCPUs is
an illegal exception return: KVM_RUN fails with KVM_EXIT_FAIL_ENTRY.
Clear RW for a vCPU that is AArch32 at EL1, when the CPU has AArch32
EL1. On a CPU without it, RW stays set and the entry still fails, rather
than EL2 switching *32_EL2 registers that are UNDEFINED there. The host
already rejects such a vCPU, but EL2 takes the vCPU's features from the
host and doesn't rely on that.
Fixes: b56680de9c ("KVM: arm64: Initialize trap register values in hyp in pKVM")
Cc: stable@vger.kernel.org
Signed-off-by: Fuad Tabba <fuad.tabba@linux.dev>
Link: https://patch.msgid.link/20260929090031.3829185-3-fuad.tabba@linux.dev
Signed-off-by: Oliver Upton <oupton@kernel.org>
fix_host_ownership() transfers only what it walks, so a hyp mapping
outside the linear map is not manipulated by the walk.
Walk the quarter of the VA space holding the private range and the
vmemmap once the transfer is done, and fail init unless every valid
leaf is hyp-owned: in the vmemmap when the page is memory, and in the
host stage-2, where hyp text may instead be mapped without write
access. A leaf that is not memory has no vmemmap entry and is checked
against the host stage-2 alone. Hyp text is matched by physical
address, since the only executable mapping in the range is the
Spectre-v3a vectors, whose VA is a private allocation, and an
executable mapping of anything else must not be host-readable. The
vmemmap can be block-mapped, so the walker checks each page of a leaf.
Suggested-by: Will Deacon <will@kernel.org>
Signed-off-by: Fuad Tabba <fuad.tabba@linux.dev>
Reviewed-by: Vincent Donnefort <vdonnefort@google.com>
Tested-by: Vincent Donnefort <vdonnefort@google.com>
Reviewed-by: Marc Zyngier <maz@kernel.org>
Link: https://patch.msgid.link/20260908110713.1540304-5-fuad.tabba@linux.dev
Signed-off-by: Oliver Upton <oupton@kernel.org>
__io_map_base is the start of the private VA range only until the first
allocation from it, after which it is the allocation cursor. Keep it as
the start and move the cursor to __io_map_next, for the walk of the
range the next patch adds.
No functional change intended.
Signed-off-by: Fuad Tabba <fuad.tabba@linux.dev>
Reviewed-by: Vincent Donnefort <vdonnefort@google.com>
Tested-by: Vincent Donnefort <vdonnefort@google.com>
Reviewed-by: Marc Zyngier <maz@kernel.org>
Link: https://patch.msgid.link/20260908110713.1540304-4-fuad.tabba@linux.dev
Signed-off-by: Oliver Upton <oupton@kernel.org>
On a non-hVHE host, fix_host_ownership_walker()'s test for PAGE_HYP_EXEC
never matches: KVM_PGTABLE_PROT_UX is cleared at map time and only PX is
reported on read-back. Hyp text is therefore donated rather than left
read-only in the host stage-2, and the instruction dump in
nvhe_hyp_panic_handler() reads a page the host has no access to.
Match the text by physical address instead, in a helper a later patch
reuses. A test on the permissions would leave any other executable
mapping host-readable too.
Fixes: 80cbfd7174 ("KVM: arm64: Honor UX/PX attributes for EL2 S1 mappings")
Signed-off-by: Fuad Tabba <fuad.tabba@linux.dev>
Reviewed-by: Vincent Donnefort <vdonnefort@google.com>
Tested-by: Vincent Donnefort <vdonnefort@google.com>
Reviewed-by: Marc Zyngier <maz@kernel.org>
Link: https://patch.msgid.link/20260908110713.1540304-3-fuad.tabba@linux.dev
Signed-off-by: Oliver Upton <oupton@kernel.org>
fix_host_ownership() walks only the linear-map alias of each memblock
region, and the per-CPU hyp stack, mapped in the private VA range for
its guard page, has none.
Walk each stack's VA range with the same walker.
Fixes: 1a919b17ef ("KVM: arm64: Add guard pages for pKVM (protected nVHE) hypervisor stack")
Reported-by: Hiroyuki Katsura <hk590@cam.ac.uk>
Cc: stable@vger.kernel.org
Signed-off-by: Fuad Tabba <fuad.tabba@linux.dev>
Reviewed-by: Vincent Donnefort <vdonnefort@google.com>
Tested-by: Vincent Donnefort <vdonnefort@google.com>
Reviewed-by: Marc Zyngier <maz@kernel.org>
Link: https://patch.msgid.link/20260908110713.1540304-2-fuad.tabba@linux.dev
Signed-off-by: Oliver Upton <oupton@kernel.org>
pkvm_init_features_from_host() takes KVM_ARCH_FLAG_GUEST_HAS_SVE and
KVM_ARM_VCPU_SVE from the host separately, but pkvm_vcpu_init_sve()
tests the bit while vcpu_has_sve() reads the flag. A host that sets the
flag without the bit gets a vCPU with a NULL sve_state that the world
switch loads the guest's SVE state from.
Derive the flag from the bit, and drop the protected path's copy of the
host's flag, which is dead code since protected VMs are not allowed SVE.
Fixes: 41d6028e28 ("KVM: arm64: Convert the SVE guest vcpu flag to a vm flag")
Signed-off-by: Fuad Tabba <fuad.tabba@linux.dev>
Link: https://patch.msgid.link/20260825085948.1674721-5-fuad.tabba@linux.dev
Signed-off-by: Oliver Upton <oupton@kernel.org>
unpin_host_sve_state() gates on the VM's SVE feature bit, but what it
unpins is the state pkvm_vcpu_init_sve() pinned. A vCPU that completed
init has sve_state set exactly when that bit is set, so the two agree.
Gate on sve_state, which is what is being unpinned.
Signed-off-by: Fuad Tabba <fuad.tabba@linux.dev>
Link: https://patch.msgid.link/20260825085948.1674721-4-fuad.tabba@linux.dev
Signed-off-by: Oliver Upton <oupton@kernel.org>
pkvm_vcpu_init_sve() clears KVM_ARM_VCPU_SVE in kvm->arch.vcpu_features
when it fails, but vcpu_has_sve() tests KVM_ARCH_FLAG_GUEST_HAS_SVE,
which is left set. Later vCPUs on that VM then skip the SVE setup and
register with a NULL sve_state, which the guest's first FP access hands
to sve_load_state().
Return the error without touching vcpu_features.
Fixes: 5db1bef933 ("KVM: arm64: Track SVE state in the hypervisor vcpu structure")
Signed-off-by: Fuad Tabba <fuad.tabba@linux.dev>
Link: https://patch.msgid.link/20260825085948.1674721-3-fuad.tabba@linux.dev
Signed-off-by: Oliver Upton <oupton@kernel.org>
pkvm_vcpu_init_sve() clamps only the upper bound of the host-provided
sve_max_vl, so an invalid vector length reaches sve_state_size_from_vl()
and the WARN_ON() there, which is fatal at EL2. The existing
!sve_state_size test rejects such a length, but only after the macro has
run.
Check sve_vl_valid() before deriving the state size. A valid length
cannot yield a zero size, so the !sve_state_size test goes with it.
Fixes: 5db1bef933 ("KVM: arm64: Track SVE state in the hypervisor vcpu structure")
Reported-by: Stefan Teodorescu <fane@google.com>
Reviewed-by: Marc Zyngier <maz@kernel.org>
Signed-off-by: Fuad Tabba <fuad.tabba@linux.dev>
Link: https://patch.msgid.link/20260825085948.1674721-2-fuad.tabba@linux.dev
Signed-off-by: Oliver Upton <oupton@kernel.org>
- Add support for 'slot' based PMU events, paired with new UAPI that
compels the user to select a specific PMU implementation
- Lazy save/restore of vCPU state for pKVM, along with various fixes
and cleanups to the management of vCPU state between the untrusted
host and pKVM hypervisor
- Disable traps of EL1 registers for nested hypervisors when FEAT_NV2p1
is present, guaranteeing that EL2-specific register bits are stateful
in the EL1 counterpart
- Leverage FEAT_NV3 to avoid unnecessary ERET/TLBI traps when the scope
of those instructions remains 'in host' (i.e. L1 kernel/userspace)
- Pile of fixes for the management of the VNCR pseudo-TLB, such as
under-invalidations and races with concurrent TLBIs on other vCPUs
- Consolidate the non-protected and pKVM view of ICH_VTR_EL2 to a
runtime-patched constant, allowing the same data to be shared with
pKVM prior to dropping host privileges
- Considerable pile of LLM-assisted fixes around the shop but mostly in
the VGIC, our in-kernel generator of bugs (and sometimes interrupts)
-----BEGIN PGP SIGNATURE-----
iI0EABYKADUWIQSNXHjWXuzMZutrKNKivnWIJHzdFgUCaoZ/khccb2xpdmVyLnVw
dG9uQGxpbnV4LmRldgAKCRCivnWIJHzdFkK7AP9zvAcDf+DVte8+mZjCMDrvNB1n
UWsBnQkdZ1nDDYBJlwD8D1pvurK4ZV3UIl68IgpTqYKhMspCuOWDHrnBt1QyqQU=
=2iPX
-----END PGP SIGNATURE-----
Merge tag 'kvmarm-7.3' of https://git.kernel.org/pub/scm/linux/kernel/git/kvmarm/kvmarm into HEAD
KVM/arm64 changes for 7.3
- Add support for 'slot' based PMU events, paired with new UAPI that
compels the user to select a specific PMU implementation
- Lazy save/restore of vCPU state for pKVM, along with various fixes
and cleanups to the management of vCPU state between the untrusted
host and pKVM hypervisor
- Disable traps of EL1 registers for nested hypervisors when FEAT_NV2p1
is present, guaranteeing that EL2-specific register bits are stateful
in the EL1 counterpart
- Leverage FEAT_NV3 to avoid unnecessary ERET/TLBI traps when the scope
of those instructions remains 'in host' (i.e. L1 kernel/userspace)
- Pile of fixes for the management of the VNCR pseudo-TLB, such as
under-invalidations and races with concurrent TLBIs on other vCPUs
- Consolidate the non-protected and pKVM view of ICH_VTR_EL2 to a
runtime-patched constant, allowing the same data to be shared with
pKVM prior to dropping host privileges
- Considerable pile of LLM-assisted fixes around the shop but mostly in
the VGIC, our in-kernel generator of bugs (and sometimes interrupts)
* kvm-arm64/misc-7.3:
: Miscellaneous fixes for KVM/arm64, 7.3
:
: - Fixes for saving invalid table entries as part of saving the ITS
: tables (Fuad Tabba)
:
: - Don't reallocate the SPI array for re-attempted vgic_init(), avoiding
: a memory leak (Fuad Tabba)
:
: - Hold a reference on an LPI when saving the pending state (Qihang)
:
: - Don't WARN for out-of-range, guest-supplied INTID (Karl)
:
: - Avoid corrupting GPRs for 32-bit CP64 reads (Karl)
:
: - Reset 'in kernel' VGIC state when private IRQ allocation fails (Fuad)
:
: - Avoid kallsyms lookup in nVHE panic unless the host stage-2 is also
: disabled (Vincent)
:
: - Disregard Pending+Active state when computing maintenance IRQ for
: ICH_MISR_EL2.NP (Kajetan)
:
: - Various Sashiko-identified issues dealing with GICv5 (Sascha)
:
: - Fix CPU onlining in pKVM due to mismatched accesses when the MMU is
: disabled (Will)
KVM: arm64: Validate GICv5 timer PPIs before claiming ownership
KVM: arm64: vgic: Reject out-of-range GICv5 PPI IDs
KVM: arm64: vgic: Prevent speculative SPI array underflow
KVM: arm64: vgic: Free gic_kvm_info on initialization failure
KVM: arm64: Avoid mismatched accesses to 'struct kvm_nvhe_init_params'
KVM: arm64: vgic: Fix detection of MI on no pending LR
KVM: arm64: Drop %pB on nVHE panic when stage-2 is active
KVM: arm64: vgic: Reset in_kernel on private IRQ allocation failure
KVM: arm64: GICv2: Don't WARN on out-of-range GICV_DIR INTID
KVM: arm64: Preserve GPRs for AArch32 CP64 reads generating an UNDEF
KVM: arm64: vgic-v3: take an LPI reference in vgic_v3_save_pending_tables
KVM: arm64: vgic-its: Point saved ITEs at the next valid entry
KVM: arm64: vgic-its: Don't save collections the table cannot hold
KVM: arm64: vgic: Don't leak the SPI array when init is retried
KVM: arm64: vgic-its: Don't dereference a NULL collection on ITT save
Signed-off-by: Oliver Upton <oupton@kernel.org>
* kvm-arm64/vtr-patch:
: Inline patching of ICH_VTR_EL2 constant, courtesy of Marc Zyngier
:
: Unify readers of ICH_VTR_EL2 on an instruction-patched constant value,
: avoiding system register accesses known to trap under nested
: virtualization and sharing the implementation between pKVM and 'regular'
: KVM.
KVM: arm64: vgic-v3: Kill kvm_vgic_global_state.ich_vtr_el2
KVM: arm64: vgic-v3: Simplify initial GICv3 configuration sampling
KVM: arm64: Convert most ICH_VTR_EL2 accesses to inlined literal value
KVM: arm64: Add a helper providing an inlined literal value for ICH_VTR_EL2
KVM: arm64: Move GICv3 broken SEIS implementation detection to a CPU errrata
KVM: arm64: vgic-v3: Make vtr_to_* helpers use architectural field symbols
Signed-off-by: Oliver Upton <oupton@kernel.org>
* kvm-arm64/vncr-fixes:
: Fixes for handling L1 VNCR mappings, courtesy of Marc Zyngier
:
: - Recompute the VNCR software TLB when the MMU is toggled at stage-1
:
: - Improve TLB invalidation intersection to handle TLBIs affecting the
: end of the VA space
:
: - Fix race to invalidate the VNCR fixmap between TLBI emulation and
: vcpu_put()
:
: - Add missing sign extension for computing TLBI ranges
:
: - Make VNCR invalidation participate in the MMU notifier seqcount,
: preventing a concurrent VNCR TLB fill from consuming a stale
: translation
KVM: arm64: Correctly cap TLBI Range to the architural limit
KVM: arm64: Add VNCR TLB tracking again
KVM: arm64: Make VNCR invalidation participate in MMU invalidation retry
KVM: arm64: Sign-extend VA for range-based TLBI invalidation
KVM: arm64: Handle VNCR TLB invalidation race with vcpu_put() VNCR unmapping
KVM: arm64: Correctly handle end of VA space TLBI invalidation
KVM: arm64: Consider SCTLR_EL2.M when mapping the L1 VNCR page
KVM: arm64: Handle negative S1 walk levels in VNCR TLB size evaluation
KVM: arm64: Remove VM-wide VNCR mapping counter
Signed-off-by: Oliver Upton <oupton@kernel.org>
* kvm-arm64/pkvm-7.3: (26 commits)
: pKVM updates for 7.3
:
: - Avoid name collision on trace_clock() when CONFIG_NVHE_EL2_TRACING is
: disabled (Mostafa Saleh)
:
: - Clean up state tracking for whether the EL2 shadow VM has been
: created (Fuad Tabba)
:
: - Synchronize SCTLR_EL1 when injecting an exception to use current
: PAN/SSBS state (Fuad Tabba)
:
: - Avoid unnecessary cache maintenance when I/D-cache are known to be
: coherent in pKVM (Mostafa Saleh)
:
: - Lazy vCPU context save/restore for pKVM (Fuad Tabba)
:
: - Various fixes to the stage-2 MMU for pKVM (Fuad Tabba)
:
: - Allow counter offsetting of non-protected guests in protected mode
: (Mostafa Saleh)
:
: - Condition the 'broken CNTVOFF' mitigation on a VM actually having a
: nonzero offset, fixing boot failures of pVMs on affected hardware
: (Mostafa Saleh)
KVM: arm64: Fix hvhe and broken CNTVOFF_EL2
KVM: arm64: Fix timer offsets for non-protected VMs
KVM: arm64: Make timer_get_offset() work in all contexts
KVM: arm64: selftests: Add stage-2 block transition test
KVM: arm64: Don't advertise eager page splitting under pKVM
KVM: arm64: Don't WARN on pKVM stage-2 map failures
KVM: arm64: Skip pKVM stage-2 flush when FWB is enabled
KVM: arm64: Top up stage-2 memcache for dirty logging faults
KVM: arm64: Top up the memcache for pKVM permission faults
KVM: arm64: Skip cache maintenance for non-cacheable pKVM mappings
KVM: arm64: Implement lazy vCPU state sync for non-protected guests
KVM: arm64: Add primitives to flush/sync the VGIC state at EL2
KVM: arm64: Minimise EL2's exposure of host VGIC state during world switch
KVM: arm64: Add host and hypervisor vCPU lookup primitives
KVM: arm64: Move PSCI helper functions to a shared header
KVM: arm64: Factor out reusable vCPU reset helpers
KVM: arm64: Make vcpu_{read,write}_sys_reg available to HYP code
KVM: arm64: Extract MPIDR computation into a shared header
KVM: arm64: selftests: Add a userspace watchpoint test
KVM: arm64: Flush external_mdscr_el1 to the pKVM hyp vCPU
...
Signed-off-by: Oliver Upton <oupton@kernel.org>
When running with hVHE enabled, ___kvm_hyp_init() calls
__kvm_init_el2_state() on the CPU initialisation path during onlining
and resume from suspend.
In order to avoid clobbering the link register across this call, it is
stashed away in the 'tmp' member of 'struct kvm_nvhe_init_params',
however this save/restore operation is performed with the stage-1 MMU
disabled at EL2 and therefore gives rise to coherency problems because
the field is not aligned or padded to the CWG. For example, a cacheable
write to a physically-adjacent structure sharing the same cacheline
could lead to an eviction and subsequent write-back, overwriting the
saved LR while the incoming CPU is executing __kvm_init_el2_state().
Save the lr in far_el2 and remove the 'tmp' member from
'struct kvm_nvhe_init_params' altogether.
Cc: Oliver Upton <oupton@kernel.org>
Cc: Marc Zyngier <maz@kernel.org>
Fixes: afa9b48f32 ("KVM: arm64: Shave a few bytes from the EL2 idmap code")
Signed-off-by: Will Deacon <will@kernel.org>
Reviewed-by: Marc Zyngier <maz@kernel.org>
Link: https://patch.msgid.link/20260813131717.5885-1-will@kernel.org
Signed-off-by: Oliver Upton <oupton@kernel.org>
With pKVM, protected VMs always have offset of zero.
However, timer offsets for non-protected guests fail to take effect
for two reasons:
1) In __timer_enable_traps(), enabling of traps check for
is_protected_kvm_enabled() rather than vcpu_is_protected(vcpu)
2) The vcpu timer offsets were never initialised and kept as NULL.
This is problematic for cases when the timer is trapped in the
hypervisor as the with the case of broken CNTVOFF_EL2, which leads
to the hypervisor and host using different offsets and causing VM
hangs.
This can be confirmed by running the arch_timer selftest which fails:
./arch_timer -o 100000000
Random seed: 0x6b8b4567
Guest assert failed, vcpu 0; stage; 3; iter: 0
==== Test Assertion Failure ====
arm64/arch_timer.c:137: config_iter + 1 == irq_iter
pid=310 tid=312 errno=4 - Interrupted system call
Guest assert failed, vcpu 3; stage; 3; iter: 0
Guest assert failed, vcpu 1; stage; 3; iter: 0
==== Test Assertion Failure ====
arm64/arch_timer.c:137: config_iter + 1 == irq_iter
pid=310 tid=313 errno=4 - Interrupted system call
Guest assert failed, vcpu 2; stage; 3; iter: 0
==== Test Assertion Failure ====
arm64/arch_timer.c:137: config_iter + 1 == irq_iter
pid=310 tid=314 errno=4 - Interrupted system call
[...]
After the fix:
./arch_timer -o 100000000
Random seed: 0x6b8b4567
PASS(vCPU-1).
PASS(vCPU-3).
PASS(vCPU-0).
PASS(vCPU-2)
Reported-by: Sashiko <sashiko-bot@kernel.org>
Fixes: cb0c272ace ("KVM: arm64: Initialize the hypervisor's VM state at EL2")
Signed-off-by: Mostafa Saleh <smostafa@google.com>
Link: https://patch.msgid.link/20260808085824.732659-3-smostafa@google.com
Signed-off-by: Oliver Upton <oupton@kernel.org>
We currently have two implementations of get_timer offset(), one
in arm_arch_timer.h, and another one in switch.h.
These two only differ by a pair of kern_hyp_va(), which seems a
pretty weak reason to open-code it.
Turn this function into a macro to avoid the include dependency hell
on kern_hyp_va(), and make it work correctly in all contexts.
Signed-off-by: Marc Zyngier <maz@kernel.org>
Signed-off-by: Mostafa Saleh <smostafa@google.com>
Link: https://patch.msgid.link/20260808085824.732659-2-smostafa@google.com
Signed-off-by: Oliver Upton <oupton@kernel.org>
Having established that our VNCR TLB tracking was flawed and having
dropped it from KVM, it is time to replace it with something that
actually works in order to get some performance back, as the TLBI
overhead is otherwise pretty high.
The goal of such tracking is to hit the TLBI slow path if there are
any VNCR TLBs in the guest, irrespective of their mapping state.
For this purpose, we introduce an VM wide counter (vncr_tlb_count)
that tracks how many valid VNCR TLB are present. This means that
creating such TLB must increment the counter, and invalidation
decrement it, and both these operations must be done with the MMU lock
held for write.
On TLBI handling affecting EL2 S1, a non-zero counter forces the
handling to take the slow path to consider the VNCR TLBs. Note that
the bumping up is done "speculatively" in order to make sure that a
concurrent invalidation will be taken on the slow path.
Not exactly rocket science. Hopefully I got it right this time.
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://patch.msgid.link/20260806091026.620700-9-maz@kernel.org
Signed-off-by: Oliver Upton <oupton@kernel.org>
The global VNCR mapping counter is used to decide whether an L1
provided VNCR page is mapped in L0 on any CPU at the point of
dealing with a TLB invalidation. It is incremented when a mapping
is made in the fixmap, and decremented when unmapped.
As it turns out, this tracking has several flaws:
- we are trying to invalidate TLBs, and the mapping is only an
opportunistic consequence of the TLB. Checking this counter to
decide whether a TLB needs to be invalidated may result in missed
invalidations.
- an L1 vcpu invalidating its own TLB (a very likely case) will not
succeed in invalidating the VNCR pseudo TLB because that page is
not mapped in L0 at this stage.
Given that this tracking fails at delivering the minimum guarantees
that are required and is only a performance optimisation, remove it
completely.
Fixes: 4ffa72ad8f ("KVM: arm64: nv: Add S1 TLB invalidation primitive for VNCR_EL2")
Reviewed-by: Yuan Yao <yaoyuan@linux.alibaba.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Cc: stable@vger.kernel.org
Acked-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Link: https://patch.msgid.link/20260806091026.620700-2-maz@kernel.org
Signed-off-by: Oliver Upton <oupton@kernel.org>
When running on NV3 HW, always engage ERET trap elision when running
the L1 context, as there is no benefit in not doing so.
An L1 can itself engage trap elision by setting its own view of
HCRX_EL2.NVTGE==1, which will subsequently be honored.
Reviewed-by: Joey Gouly <joey.gouly@arm.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://patch.msgid.link/20260730071022.296811-25-maz@kernel.org
Signed-off-by: Oliver Upton <oupton@kernel.org>
Since NVHCR_EL2 represents the HCR_EL2 state of the EL1 guest, it
must be dealt with in some particular way:
- for a guest in hyp context (an L1 by definition), NVHCR_EL2 directly
reflects HCR_EL2 as read and written by the guest itself. It must
therefore be eagerly synced back with the emulation code which only
knows about HCR_EL2. This is unconditional if NV3 is available on
the host.
- For an L2 guest, NVHCR_EL2 is controlled by the L1 guest, and we
just context switch it like any other EL1 register. Yes, EL1, as
that's where this thing runs from the PoV of L1. This is conditioned
on the guest using NV3.
Reviewed-by: Joey Gouly <joey.gouly@arm.com>
Reviewed-by: Yuan Yao <yaoyuan@linux.alibaba.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://patch.msgid.link/20260730071022.296811-24-maz@kernel.org
Signed-off-by: Oliver Upton <oupton@kernel.org>
FEAT_NV3 makes a fundamental change to the architecture, by moving
guest-initiated HCR_EL2 accesses to the NVHCR_EL2 register. As the
names suggests, this is HCR_EL2 for a NV guest.
But where do NVHCR_EL2 accesses from a guest go? The are redirected
to the VNCR page, right where HCR_EL2 is stored in the NV2 case.
Does it hurt? Good. There's more coming.
The challenge here is to make KVM work seamlessly, without rewriting
everything. Which implies that things such as __vcpu_sys_reg(HCR_EL2)
must work, no matter the underlying NV implementation.
A simple way to deal with it is to move HCR_EL2's canonical storage
outside of VNCR for the vast majority of the KVM code, and only have
a copy at entry/exit times. Given that we don't really support NV3
yet, this is pretty simple.
In the process, advertise NVHCR_EL2 as the register that now holds
offset 0x78 in the VNCR page.
Reviewed-by: Joey Gouly <joey.gouly@arm.com>
Reviewed-by: Yuan Yao <yaoyuan@linux.alibaba.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://patch.msgid.link/20260730071022.296811-20-maz@kernel.org
Signed-off-by: Oliver Upton <oupton@kernel.org>
In order to facilitate further changes, move the NV handling of
early fixups in its own helper. This also makes the code slightly
simpler to parse.
Reviewed-by: Yuan Yao <yaoyuan@linux.alibaba.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://patch.msgid.link/20260730071022.296811-17-maz@kernel.org
Signed-off-by: Oliver Upton <oupton@kernel.org>
With NV2p1, it is no longer necessary to use the split approach
where bits of CNTHCTL_EL2 cannot be accessed via CNTKCTL_EL1,
and we can treat the CNTKCTL_EL1 accessor as if it was "normal".
Key the special casing on FEAT_NV2P1 not being implemented.
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://patch.msgid.link/20260730071022.296811-11-maz@kernel.org
Signed-off-by: Oliver Upton <oupton@kernel.org>
With FEAT_NV2P1, it is no longer necessary to trap CPTR_EL2 accesses
via CPACR_EL1, as CPACR_EL1.TCPAC is guaranteed to be stateful.
Prevent such trapping and context switch CPACTR_EL1 in NV contexts
when NV2P1 is present.
Reviewed-by: Joey Gouly <joey.gouly@arm.com>
Reviewed-by: Yuan Yao <yaoyuan@linux.alibaba.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://patch.msgid.link/20260730071022.296811-10-maz@kernel.org
Signed-off-by: Oliver Upton <oupton@kernel.org>
We currently avoid using the ERET fast path if the guest has HCR_EL2.NV
set. This is an odd check, as NV doesn't mean much if HCR_EL2.TGE==1.
Similarly, evaluating HFGITR_EL2.ERET makes little sense, as
this only applies to the nested context, while the ERET fast-path
is purely for the benefit of L1.
Replace these bizarre checks with is_nested_ctxt() which makes a lot
more sense: if we are running an L2, the ERET trap must go to L1.
Fixes: dd0717a998 ("KVM: arm64: nv: Fast-track 'InHost' exception returns")
Reviewed-by: Yuan Yao <yaoyuan@linux.alibaba.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://patch.msgid.link/20260730071022.296811-8-maz@kernel.org
Signed-off-by: Oliver Upton <oupton@kernel.org>
The way we merge the guest-provided HCRX_EL2 value with the host's
is bonkers. We try to make it look like the FGT registers by using
positive and negative polarities for traps, but most of these bits
are not strictly about trapping, as they actively change the way
some architectural state is managed.
It would be far better to deal with these bits like we do for
HCR_EL2, by enumerating the list of bits we don't allow the guest
to override. This is simplified by the fact that HCRX_EL2 only
affects EL1, and not EL2.
Re-jig the HCRX_EL2 handling with a macro that list the bits excluded
from the merge (TMEA, PTTWI, EnIDCP128).
Reviewed-by: Yuan Yao <yaoyuan@linux.alibaba.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Reviewed-by: Joey Gouly <joey.gouly@arm.com>
Link: https://patch.msgid.link/20260730071022.296811-4-maz@kernel.org
Signed-off-by: Oliver Upton <oupton@kernel.org>
pKVM copies a non-protected guest's register context between the host
and the hypervisor on every world switch, even when the host never
inspects it. Defer the copy: on entry, flush the host context into the
hyp vCPU only when the host marked it dirty (PKVM_HOST_STATE_DIRTY); on
exit, leave it in the hyp vCPU and copy it back only when the host needs
it, via a __pkvm_vcpu_sync_state hypercall or at vcpu put. A protected
guest's context is copied as before, since lazy sync only helps where
the host is trusted to see the guest's registers.
PC and PSTATE are the exception, copied back on every exit: the kvm_exit
tracepoint reports the guest's real exit PC, and
vcpu_mode_is_bad_32bit(), the SError-masking checks and the host
software-step state machine all read PSTATE before the next on-demand
sync. Without it an ARM_EXCEPTION_IRQ exit syncs nothing, so
flush_debug_state() pushes a stale PSTATE.SS back on the next entry and
single-stepping runs an extra instruction before the step is reported.
The host needs the full context when it is about to read it (trap
handling) or write it (the SError injection that writes ESR_EL1). Sync
both from handle_exit_early(), which runs non-preemptible so the loaded
hyp vCPU is stable without a preempt guard.
Reviewed-by: Vincent Donnefort <vdonnefort@google.com>
Signed-off-by: Fuad Tabba <fuad.tabba@linux.dev>
Link: https://patch.msgid.link/20260729131823.2021516-9-fuad.tabba@linux.dev
Signed-off-by: Oliver Upton <oupton@kernel.org>
pKVM performs its own world switch for protected VMs but has no
primitives to move the per-vCPU VGIC state between the host and
hypervisor vCPU contexts.
Add flush_hyp_vgic_state() and sync_hyp_vgic_state(). Flush copies
vgic_hcr, the in-use list registers and used_lrs from the host into the
hyp vCPU and pins vgic_sre to a fixed value; sync copies vgic_hcr,
vgic_vmcr and the in-use list registers back. The active priority
registers are handled separately by the save/restore-aprs path.
Bound used_lrs by hyp_gicv3_nr_lr, the cached implemented-LR count,
instead of reading ICH_VTR_EL2 on each entry. That clamps the
host-supplied value and avoids a per-entry sysreg read that is costly
under NV.
Reviewed-by: Vincent Donnefort <vdonnefort@google.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Co-developed-by: Fuad Tabba <fuad.tabba@linux.dev>
Signed-off-by: Fuad Tabba <fuad.tabba@linux.dev>
Link: https://patch.msgid.link/20260729131823.2021516-8-fuad.tabba@linux.dev
Signed-off-by: Oliver Upton <oupton@kernel.org>
The host passes a vgic_v3_cpu_if pointer to the __vgic_v3_save_aprs and
__vgic_v3_restore_vmcr_aprs hypercalls, which EL2 dereferences
wholesale. That exposes the host's full VGIC emulation state to the
hypervisor, against pKVM's isolation goals.
Recover the host vCPU from the supplied cpu_if via container_of() and
copy only vgic_vmcr and the active priority registers between EL2's
hyp-side state and the host vCPU, so EL2 no longer dereferences the
host's vgic_v3_cpu_if directly.
Reviewed-by: Vincent Donnefort <vdonnefort@google.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Co-developed-by: Fuad Tabba <fuad.tabba@linux.dev>
Signed-off-by: Fuad Tabba <fuad.tabba@linux.dev>
Link: https://patch.msgid.link/20260729131823.2021516-7-fuad.tabba@linux.dev
Signed-off-by: Oliver Upton <oupton@kernel.org>
The nVHE hypervisor repeatedly resolves a host vCPU into the EL2
address space and validates that the loaded hyp vCPU matches it, with
that logic open-coded in each handler.
Add __get_host_hyp_vcpus() and the get_host_hyp_vcpus() macro, which
translate the host vCPU into the hypervisor's address space and, when
pKVM is enabled, also return the loaded hyp vCPU if it matches. If pKVM
is enabled but the loaded hyp vCPU does not correspond to the requested
host vCPU, both the host and hyp vCPU are returned as NULL. Convert
handle___kvm_vcpu_run() to use it.
No functional change intended.
Reviewed-by: Vincent Donnefort <vdonnefort@google.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Co-developed-by: Fuad Tabba <fuad.tabba@linux.dev>
Signed-off-by: Fuad Tabba <fuad.tabba@linux.dev>
Link: https://patch.msgid.link/20260729131823.2021516-6-fuad.tabba@linux.dev
Signed-off-by: Oliver Upton <oupton@kernel.org>
The vcpu_{read,write}_sys_reg() accessors abstract where a system
register lives, but their VHE implementation cannot be linked into the
nVHE hypervisor, so exception.c open-codes has_vhe() wrappers instead.
Redirect the accessors to the raw context accessors for nVHE hyp builds,
where registers are always in memory, and drop the local wrappers. This
lets hyp code reuse helpers built on the canonical accessors.
No functional change intended.
Suggested-by: Marc Zyngier <maz@kernel.org>
Signed-off-by: Fuad Tabba <fuad.tabba@linux.dev>
Link: https://patch.msgid.link/20260729131823.2021516-3-fuad.tabba@linux.dev
Signed-off-by: Oliver Upton <oupton@kernel.org>
flush_debug_state() propagates the guest's debug_owner and the
owner-selected debug register state to the hyp vCPU, but not
external_mdscr_el1. While the host owns the debug registers, the world
switch loads MDSCR_EL1 from external_mdscr_el1 (ctxt_mdscr_el1()),
where the host's KDE/MDE/SS bits live. A non-protected guest under
KVM_GUESTDBG_USE_HW or single-step therefore runs with MDSCR_EL1.MDE/SS
clear in hardware, and its watchpoints, breakpoints and single-step
never fire.
Propagate external_mdscr_el1 to the hyp vCPU alongside the host-owned
debug state.
Fixes: 4ad3a0b87f ("KVM: arm64: Don't hijack guest context MDSCR_EL1")
Signed-off-by: Fuad Tabba <fuad.tabba@linux.dev>
Link: https://patch.msgid.link/20260726143643.303456-2-fuad.tabba@linux.dev
Signed-off-by: Oliver Upton <oupton@kernel.org>
KVM opportunistically enables FWB if supported by the system for guest
VMs, which allows it to elude cache maintenance for data as they are
forced to be cacheable from stage-2.
In that case, __clean_dcache_guest_page() will immediately return.
However in protected mode, before calling __clean_dcache_guest_page()
it loops over the range and fix_map/unmap it, issuing TLB
invalidations, dsb() and isb() unnecessarily.
This can be optimized by returning early if FWB is supported,
kvm_pgtable_stage2_map() already issues dsb() and tlb invalidation
functions issue dsb() for the unmap path.
Similarly for DIC, we can omit the call in
invalidate_icache_guest_page()
No isb() needed as ERET is context synchronization event.
Signed-off-by: Mostafa Saleh <smostafa@google.com>
Acked-by: Marc Zyngier <maz@kernel.org>
Reviewed-by: Fuad Tabba <fuad.tabba@linux.dev>
Link: https://patch.msgid.link/20260723162002.3848054-1-smostafa@google.com
Signed-off-by: Oliver Upton <oupton@kernel.org>
Add a missing hypervisor event call for hyp_enter on sysreg trapping,
causing an unbalanced hyp_enter/hyp_exit.
The enum hyp_enter_exit_reason is not ABI, so we can keep the ERET
reasons at the end for clarity.
Fixes: 696dfec22b ("KVM: arm64: Add hyp_enter/hyp_exit events to nVHE/pKVM hyp")
Signed-off-by: Vincent Donnefort <vdonnefort@google.com>
Reviewed-by: Fuad Tabba <tabba@google.com>
Tested-by: Fuad Tabba <tabba@google.com>
Link: https://patch.msgid.link/20260617095238.1530121-1-vdonnefort@google.com
Signed-off-by: Marc Zyngier <maz@kernel.org>
When pKVM injects a synchronous exception into a protected guest,
enter_exception64() reads SCTLR_EL1 to set the new PSTATE's PAN and
SSBS bits. pKVM refreshes VBAR_EL1 from the live value before injecting
but not SCTLR_EL1, which is untrapped for protected guests, so a guest
that updates it and then traps takes the exception with stale PAN/SSBS.
Sync SCTLR_EL1 alongside VBAR_EL1.
Fixes: 798eb59787 ("KVM: arm64: Sync protected guest VBAR_EL1 on injecting an undef exception")
Signed-off-by: Fuad Tabba <fuad.tabba@linux.dev>
Link: https://patch.msgid.link/20260721161624.3983041-1-fuad.tabba@linux.dev
Signed-off-by: Oliver Upton <oupton@kernel.org>
Now that we have our magic inline helper for ICH_VTR_EL2, we can
get rid of the hack that was reporting a combination of that
register and of the indication of the CPU interface supporting
GICv2 compatibility. We now only report the latter.
As a small benefit, GICv5 is not involved in this stuff anymore,
since it never has GICv2 compatibility..
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://patch.msgid.link/20260721170754.3150521-6-maz@kernel.org
Signed-off-by: Oliver Upton <oupton@kernel.org>
Now that we have a new helper that inlines the value of ICH_VTR_EL2
into the code, use this to replace most of the sysreg accesses to
that register.
This avoids a lot of traps when running KVM under NV.
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://patch.msgid.link/20260721170754.3150521-5-maz@kernel.org
Signed-off-by: Oliver Upton <oupton@kernel.org>
vgic-v3-sr.c still contains some hardcoded constants for some of
the ICH)VTR_EL2 fields. Bring them into the modern world by using
the named symbols from the sysreg file.
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://patch.msgid.link/20260721170754.3150521-2-maz@kernel.org
Signed-off-by: Oliver Upton <oupton@kernel.org>
- Move locking for kvm_io_bus_get_dev() into the caller, ensuring
race-free checks that the returned object is of the correct type
- Fix initialisation of the page-table walk level when relaxing
permissions
- Correctly update the XN attribute when relaxing permissions
- Fix the sign extension of loads from emulated MMIO regions
- Assorted collection of fixes for pKVM's FFA proxy, together with a
couple of FFA driver adjustments
-----BEGIN PGP SIGNATURE-----
iQIzBAABCgAdFiEEn9UcU+C1Yxj9lZw9I9DQutE9ekMFAmpOLOIACgkQI9DQutE9
ekPH5Q//VrbqIVfJekxRoI0B8zWpnG52q9ILdZBEhTL+xHEP+Z8WawzbpfHzv9LQ
tt7rUk8gmkGBvMXdY3fZaLkaQRDGvg51SKclxIaoNdmrav7Sscz/sar7P8QWUuIb
d0EQ3hqV+2hvNFqUVy0cTAFoog0U00KDpdBV136ePTYgT7/N0B8fiFStdHzVP80H
0WbeWnB0LB15mnkCkO0QY4QmEozfnjKYPrhv52fZFSmnc0BLY3muSR0y+lZ1FtN0
c2aUiYH/iyIkmmpEZyMYzzhwnMkPwwTIof1OgQ93//JUPkmEUtun5SNQ9YaQw/cd
/IKDo5xhEn/3jH6Zo2th8oKhH4q5mSOxBd8WH/hkwnTpZYbUZ5Q0647QHHEtWWEp
mEdQFroaORrPXtHfKSeltkr5JKWK9vweARwbTcLmNwbGfLqZGgJe8/FgOw8Q3lXB
wZcmNc5TqGe4QWiatH7qKTdqulWdQMrRE2XRAinfrfOfZMg69cnNizWAdNPVhPfw
B5yewkPrPwZNNXGksZOXfbFN7TnIM2M8j7uwXl5D3rGi/ebDp5ByuVKkP5ZNLnz3
D666cORhX1zdCfqbLx4mhPs+vcqmL8HHMASlaJg+hH1se/veLbkEZ9GIrjw8Y0iO
SMUJvwJ5Z2SZpvCKGsC85Wdm/pIBLLh5ISIW8wDIIw+7BoMrXFY=
=RmEk
-----END PGP SIGNATURE-----
Merge tag 'kvmarm-fixes-7.2-2' of git://git.kernel.org/pub/scm/linux/kernel/git/kvmarm/kvmarm into HEAD
KVM/arm64 fixes for 7.2, take #2
- Move locking for kvm_io_bus_get_dev() into the caller, ensuring
race-free checks that the returned object is of the correct type
- Fix initialisation of the page-table walk level when relaxing
permissions
- Correctly update the XN attribute when relaxing permissions
- Fix the sign extension of loads from emulated MMIO regions
- Assorted collection of fixes for pKVM's FFA proxy, together with a
couple of FFA driver adjustments
- Fix an accounting buglet when reclaiming pages from a protected
guest
- Fix a bunch of architectural compliance issues when injecting a
synthesised exception, most of which were missing the PSTATE.IL bit
indicating a 32bit-wide instruction
- Another set of fixes addressing issues with translation of VNCR_EL2,
including corner cases where the guest point that register at a RO
page...
- Don't warn when trapping accesses to ZCR_EL2 from an L2 guest, as
that's not unexpected at all
- Address a bunch of races with LPI migration vs LPIs being disabled
- Fix a total howler of a bug combining FEAT_MOPS and NV, resulting in
exception returning in the wrong place...
- Coerce Fuad Tabba into a reviewer role, and may his Inbox catch
fire!
-----BEGIN PGP SIGNATURE-----
iQIzBAABCgAdFiEEn9UcU+C1Yxj9lZw9I9DQutE9ekMFAmo+fUoACgkQI9DQutE9
ekOgBA/8D1XhSpPjYu7NDBOf6+62n02BRhihlFgA3KQkFmiVwUoeiAGXfDyZqQiM
ms1LfUmU+7Ry7zoeasHzVi5k/FEk+gMR911eEcciJF7C6BbC3uHUOgEg30DWQo9J
bb23PgeAcYnAqgjRw8vcXsvArK6YcY9Y6Qvp3SA1fM8Ls62P0+OdxCE/HM4TE1Ny
1t1vokNHaJCAkf0h6gAI12yEZGhK2bFZsVnkOYgkIuqKI5730/B2hbjqK+hcMPHL
TO7tVeRk8qQ0ZFJq3hpnOcRjO6uLA/f/6IHkYej/ADxaqH2uzOQeKY3awoS1UtNJ
okRKS3OQjFXW8nsarVRIzGDMp41EjydVB8po6HXWg1zhiD6PEH92W+rnVPq273xh
gKi0KOMchgBr+61zeZU/wdPZiU/wnMTxukcpqyCSR5WdoOxwMpGZvdCwHzGOyg7v
8QebZzkAxSB854aaHf5JjdTH9zI/HOk184fxtdgYbxGKMKaZ/wdTM7kos1bBHt8t
BP/yu7/jLBxEvrO9iLBHYzlk40i1oqyM4sZpM2Z2w8MBbOaHni1xm1URwFwTX5aM
FJ8BxObnnUcB8ARzRHafPcLwgyC0LubEVYsbddppijEhBu8f/wF/sMcIoHNppVCR
Vp9KK9+BH6OyYbG7TsKj2zy2S8KqgVonehoDrr66cPySo0l9KWY=
=mmLT
-----END PGP SIGNATURE-----
Merge tag 'kvmarm-fixes-7.2-1' of git://git.kernel.org/pub/scm/linux/kernel/git/kvmarm/kvmarm into HEAD
KVM/arm64 fixes for 7.2, take #1
- Fix an accounting buglet when reclaiming pages from a protected
guest
- Fix a bunch of architectural compliance issues when injecting a
synthesised exception, most of which were missing the PSTATE.IL bit
indicating a 32bit-wide instruction
- Another set of fixes addressing issues with translation of VNCR_EL2,
including corner cases where the guest point that register at a RO
page...
- Don't warn when trapping accesses to ZCR_EL2 from an L2 guest, as
that's not unexpected at all
- Address a bunch of races with LPI migration vs LPIs being disabled
- Fix a total howler of a bug combining FEAT_MOPS and NV, resulting in
exception returning in the wrong place...
- Coerce Fuad Tabba into a reviewer role, and may his Inbox catch
fire!
init_pkvm_hyp_vm() sets is_created on the EL2-private VM struct, but the
hypervisor never reads it: pkvm_hyp_vm_is_created() and every other
consumer operate on the host's struct kvm, a distinct allocation from
the EL2-private copy. The field is write-only at EL2.
Remove the store; host-side is_created tracking is unaffected.
Reviewed-by: Keir Fraser <keirf@google.com>
Signed-off-by: Fuad Tabba <fuad.tabba@linux.dev>
Link: https://patch.msgid.link/20260706103129.706974-2-fuad.tabba@linux.dev
Signed-off-by: Oliver Upton <oupton@kernel.org>
When the hypervisor tracing (CONFIG_NVHE_EL2_TRACING) is disabled, it
defines a static inline stub for trace_clock().
However, trace_clock() is already declared as an extern function in
linux/trace_clock.h which is pulled in EL2 compilation.
If the file <nvhe/clock.h> is included when CONFIG_NVHE_EL2_TRACING
is disabled (by including it manually in setup.c) it will cause:
In file included from arch/arm64/kvm/hyp/nvhe/setup.c:22:
./arch/arm64/kvm/hyp/include/nvhe/clock.h:14:19: error: static declaration of ‘trace_clock’ follows non-static declaration
14 | static inline u64 trace_clock(void) { return 0; }
| ^~~~~~~~~~~
on GCC and a linker error on LLVM (it seems to change the linkage to
global)
Although that is not a problem at the moment, as no other files
include <nvhe/clock.h>. That does not seem to be the intent of
this code and that will cause issues with more users as the SMMUv3
driver.
Signed-off-by: Mostafa Saleh <smostafa@google.com>
Reviewed-by: Fuad Tabba <fuad.tabba@linux.dev>
Tested-by: Fuad Tabba <fuad.tabba@linux.dev>
Reviewed-by: Vincent Donnefort <vdonnefort@google.com>
Link: https://patch.msgid.link/20260713141320.4065600-1-smostafa@google.com
Signed-off-by: Oliver Upton <oupton@kernel.org>
Assigning the invalidation level (an s8 value) with TLBI_TTL_UNKNOWN
(a 32bit signed value) is not ideal, to say the least. Instead of
this, only pass TLBI_TTL_UNKNOWN to __kvm_tlb_flush_vmid_ipa_nsh()
when we know for sure that we don't have a provided level.
Fixes: 100baf0184 ("KVM: arm64: Ensure level is always initialized when relaxing perms")
Reported-by: Mark Brown <broonie@kernel.org>
Reviewed-by: Oliver Upton <oupton@kernel.org>
Link: https://lore.kernel.org/r/akztC7H2IsEKaq4i@sirena.org.uk
Link: https://patch.msgid.link/20260707162935.1900874-1-maz@kernel.org
Signed-off-by: Marc Zyngier <maz@kernel.org>
Harden the check for the constituent memory region page alignment
to prevent over-sharing when the negotiated FFA_PAGE_SIZE size is
smaller than the system PAGE_SIZE.
At the moment we only check that the size of the range is page
aligned, and truncate the address to the page boundary which can
annotate more memory than needed as being used by the FF-A.
Fixes: 4360900017 ("KVM: arm64: Handle FFA_MEM_SHARE calls from the host")
Signed-off-by: Mostafa Saleh <smostafa@google.com>
Reviewed-by: Vincent Donnefort <vdonnefort@google.com>
Signed-off-by: Sebastian Ene <sebastianene@google.com>
Link: https://patch.msgid.link/20260702103848.1647249-6-sebastianene@google.com
Signed-off-by: Marc Zyngier <maz@kernel.org>
Prevent the pKVM hypervisor from making assumptions that the
endpoint memory access descriptor (EMAD) comes right after the
FF-A memory region header.
Prior to FF-A version 1.1 the header of the memory region
didn't contain an offset to the endpoint memory access descriptor.
The layout of a memory transaction looks like this from 1.1 onward:
Type | Field name | Offset
[ Header | ffa_mem_region | 0
EMAD 1 | ffa_mem_region_attributes) | ffa_mem_region.ep_mem_offset
]
Verify that the offset to the first endpoint memory access descriptor
is within the mailbox buffer bounds.
Also, fix one hardcoded sizeof(struct ffa_mem_region_attributes) that
should be replaced ffa_emad_size_get() for compatibility with FFA v1.0.
Fixes: 42fb33dde4 ("KVM: arm64: Use FF-A 1.1 with pKVM")
Signed-off-by: Mostafa Saleh <smostafa@google.com>
Signed-off-by: Sebastian Ene <sebastianene@google.com>
Reviewed-by: Vincent Donnefort <vdonnefort@google.com>
Link: https://patch.msgid.link/20260702103848.1647249-5-sebastianene@google.com
Signed-off-by: Marc Zyngier <maz@kernel.org>
