github-mirror/linux
1
0
Fork 0
mirror of https://github.com/torvalds/linux.git synced 2026-11-07 06:17:02 +01:00
Code Issues Actions Packages Projects Releases Wiki Activity Graph

Merge branch kvm-arm64/pkvm-state-7.4 into kvmarm-master/next

Browse Source 
* kvm-arm64/pkvm-state-7.4:
  : \
  : State protection for protected pKVM guests, finally starting to deliver
  : on its promises... Patches courtesy of Fuad Tabba.
  :
  : "Following the vCPU state-sync series, this series completes the
  : job for protected VMs: a protected guest's register state stays at
  : EL2, and the host sees only what handling each exit requires."
  : /
  KVM: arm64: Only emulate an SError's entry for vCPUs with NV
  KVM: arm64: Document the protected VM userspace API
  KVM: arm64: Advertise the capabilities that protected VMs support
  KVM: arm64: Reject host power-on of a vCPU that EL2 holds powered off
  KVM: arm64: Reject host access to protected VM private state
  KVM: arm64: Add per-EC entry/exit state marshalling for protected guests
  KVM: arm64: Inject an UNDEF at EL2 for unhandled protected guest exits
  KVM: arm64: Prevent host PC adjustments for protected vCPUs
  KVM: arm64: Restrict KVM_ARM_VCPU_INIT and PSCI version for protected VMs
  KVM: arm64: Handle PSCI calls for protected VMs at EL2
  KVM: arm64: Implement HVC handling for protected guests at EL2
  KVM: arm64: Add system register reset framework for protected VMs
  KVM: arm64: Add {flush,sync}_hyp_timer_state() primitives
  KVM: arm64: Skip fixed-feature state flush for protected vCPUs
  KVM: arm64: Introduce per-EC entry handlers for pKVM
  KVM: arm64: Disable steal time for protected VMs
  KVM: arm64: Pin the host vCPU before adjusting its PC under pKVM
  KVM: arm64: Validate the host vCPU's VM before reading it under pKVM
  KVM: arm64: Sync HCR_EL2.VSE back to the host vCPU under pKVM

Signed-off-by: Marc Zyngier <maz@kernel.org>
This commit is contained in:
Marc Zyngier 2026-10-11 10:54:06 +01:00
commit 0ff05136bb
21 changed files with 1506 additions and 91 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

27
Documentation/virt/kvm/api.rst
View File

@ -1323,6 +1323,9 @@ guests, across different userspace implementations. Nevertheless, userspace
can still emulate all Arm exceptions by manipulating individual registers
using the KVM_SET_ONE_REG API.
For a protected VM, setting ext_dabt_pending returns -EPERM; see
Documentation/virt/kvm/arm/pkvm.rst. Injecting an SError is unaffected.
See KVM_GET_VCPU_EVENTS for the data structure.
Calling this ioctl on a vCPU that hasn't been initialized will return
@ -1546,7 +1549,9 @@ Possible values are:
KVM_MP_STATE_RUNNABLE the vcpu is currently running
[x86,arm64,riscv,loongarch]
KVM_MP_STATE_UNINITIALIZED the vcpu is an application processor (AP)
which has not yet received an INIT signal [x86]
which has not yet received an INIT signal
[x86], or a protected VM's vcpu that the
guest holds powered off [arm64]
KVM_MP_STATE_INIT_RECEIVED the vcpu has received an INIT signal, and is
now ready for a SIPI [x86]
KVM_MP_STATE_HALTED the vcpu has executed a HLT instruction and
@ -1625,6 +1630,12 @@ For arm64/riscv:
The only states that are valid are KVM_MP_STATE_STOPPED and
KVM_MP_STATE_RUNNABLE which reflect if the vcpu should be paused or not.
On arm64, once a protected VM's vcpu has run, it reads as
KVM_MP_STATE_UNINITIALIZED while the guest holds it powered off, with
CPU_OFF or by not yet bringing it online with CPU_ON, and
KVM_MP_STATE_RUNNABLE and KVM_MP_STATE_SUSPENDED return -EPERM on it: only
the guest can power it on. See Documentation/virt/kvm/arm/pkvm.rst.
On LoongArch, only the KVM_MP_STATE_RUNNABLE state is used to reflect
whether the vcpu is runnable.
@ -2325,7 +2336,8 @@ Errors:
ENOENT no such register
EINVAL invalid register ID, or no such register or used with VMs in
protected virtualization mode on s390
EPERM (arm64) register access not allowed before vcpu finalization
EPERM (arm64) register access not allowed before vcpu
finalization, or after a protected VM's vcpu has run
EBUSY (riscv) changing register value not allowed after the vcpu
has run at least once
====== ============================================================
@ -2950,7 +2962,8 @@ Errors include:
ENOENT no such register
EINVAL invalid register ID, or no such register or used with VMs in
protected virtualization mode on s390
EPERM (arm64) register access not allowed before vcpu finalization
EPERM (arm64) register access not allowed before vcpu
finalization, or after a protected VM's vcpu has run
======== ============================================================
(These error codes are indicative only: do not rely on a specific error
@ -3485,6 +3498,7 @@ Errors:
====== =================================================================
EINVAL the target is unknown, or the combination of features is invalid.
ENOENT a features bit specified is unknown.
EPERM the vcpu belongs to a protected VM and has already run.
====== =================================================================
This tells KVM what type of CPU to present to the guest, and what
@ -3513,6 +3527,10 @@ after the vcpu has been run. This will reset the vcpu to its initial
state. All calls to this function after the initial call must use the same
target and same set of feature flags, otherwise EINVAL will be returned.
For a protected VM this ioctl returns EPERM once the vcpu has run, and the
features it accepts are restricted: KVM_ARM_VCPU_PSCI_0_2 is required. See
Documentation/virt/kvm/arm/pkvm.rst.
Possible features:
- KVM_ARM_VCPU_POWER_OFF: Starts the CPU in a power-off state.
@ -3785,6 +3803,9 @@ can be determined by querying the KVM_CAP_GUEST_DEBUG_HW_BPS and
KVM_CAP_GUEST_DEBUG_HW_WPS capabilities which return a positive number
indicating the number of supported registers.
On arm64, this ioctl returns -EPERM for a protected VM: debugging a
protected guest is not supported. See Documentation/virt/kvm/arm/pkvm.rst.
For ppc, the KVM_CAP_PPC_GUEST_DEBUG_SSTEP capability indicates whether
the single-step debug event (KVM_GUESTDBG_SINGLESTEP) is supported.

2
Documentation/virt/kvm/arm/fw-pseudo-registers.rst
View File

@ -33,6 +33,8 @@ The following registers are defined:
highest PSCI version implemented by KVM and compatible with v0.2)
- Allows any PSCI version implemented by KVM and compatible with
v0.2 to be set with SET_ONE_REG
- Returns -EINVAL for a version below v1.1 on a protected VM (see
Documentation/virt/kvm/arm/pkvm.rst)
- Affects the whole VM (even if the register view is per-vcpu)
* KVM_REG_ARM_SMCCC_ARCH_WORKAROUND_1:

162
Documentation/virt/kvm/arm/pkvm.rst
View File

@ -15,7 +15,9 @@ Overview
Booting a host kernel with '``kvm-arm.mode=protected``' enables
"Protected KVM" (pKVM). During boot, pKVM installs a stage-2 identity
map page-table for the host and uses it to isolate the hypervisor
running at EL2 from the rest of the host running at EL1/0.
running at EL2 from the rest of the host running at EL1/0. pKVM supports
GICv3 only: GICv2 is not supported in protected mode, and neither are
GICv5-based guests.
pKVM permits creation of protected virtual machines (pVMs) by passing
the ``KVM_VM_TYPE_ARM_PROTECTED`` machine type identifier to the
@ -31,7 +33,7 @@ See hypercalls.rst for more details.
Isolation mechanisms
====================
pKVM relies on a number of mechanisms to isolate PVMs from the host:
pKVM relies on a number of mechanisms to isolate pVMs from the host:
CPU memory isolation
--------------------
@ -50,8 +52,9 @@ stage-2 identity map as part of the donation procedure. This gives rise
to some user-visible differences when compared to non-protected VMs,
largely due to the lack of MMU notifiers:
* Memslots cannot be moved or deleted once the pVM has started running.
* Read-only memslots and dirty logging are not supported.
* Memslots cannot be moved or deleted once the pVM has started running
(``-EPERM``).
* Read-only memslots and dirty logging are not supported (``-EPERM``).
* With the exception of swap, file-backed pages cannot be mapped into a
pVM.
* Donated pages are accounted against ``RLIMIT_MLOCK`` and so the VMM
@ -72,7 +75,14 @@ largely due to the lack of MMU notifiers:
CPU state isolation
-------------------
Status: **Unimplemented.**
Status: CPU register state of protected vCPUs is managed entirely at EL2.
pKVM performs the complete context switch for protected vCPUs within the
hypervisor. The hypervisor sets a protected vCPU's initial state, not the
host, and only what each exit needs is synchronised back.
The user-visible consequences are described under `API behaviour for
protected VMs`_.
DMA isolation using an IOMMU
----------------------------
@ -89,13 +99,153 @@ The FF-A proxy ensures that the host cannot share pVM or hypervisor
memory with Trustzone as part of a "confused deputy" attack.
The PSCI proxy ensures that CPUs always have the stage-2 identity map
installed when they are executing in the host.
installed when they are executing in the host. This proxy is distinct
from the PSCI handling provided to protected guests, which is described
under `API behaviour for protected VMs`_.
Protected VM firmware (pvmfw)
-----------------------------
Status: **Unimplemented.**
API behaviour for protected VMs
===============================
Protected vCPU state is owned by EL2 (see `CPU state isolation`_). The VMM
configures a vCPU before its first ``KVM_RUN``; afterwards the state is
private to the guest and the ioctls that access it return ``-EPERM``. The
vCPU ioctls' errors follow one rule: ``-EPERM`` means the host asked for
state that the guest owns, ``-EINVAL`` means the request is not valid for a
protected VM, and a feature pKVM doesn't support for protected VMs is
rejected the way KVM rejects one that isn't implemented (``-ENXIO`` from
``KVM_ARM_VCPU_PVTIME_CTRL``, for example). The ioctls themselves are
described in Documentation/virt/kvm/api.rst.
Boot
----
A protected VM boots from a single primary vCPU. Before the first
``KVM_RUN``, the VMM prepares the boot state:
* Set ``KVM_MP_STATE_RUNNABLE`` on the primary vCPU and
``KVM_MP_STATE_STOPPED`` on every other vCPU. EL2 allows only one
RUNNABLE primary per protected VM. A second RUNNABLE vCPU fails at its
first ``KVM_RUN``.
* Set the primary vCPU's boot state with ``KVM_SET_ONE_REG``: the kernel
entry address in ``PC`` and the DTB pointer in ``x0``.
``PC`` and ``x0`` are the only registers EL2 takes from the host. Other
pre-run writes are accepted, but the guest starts from the state the
hypervisor resets it to.
Secondary vCPUs are started by the guest itself through PSCI ``CPU_ON``
(see `Power state`_), which supplies their entry point and context ID.
The VMM cannot choose where they boot.
vCPU state
----------
* ``KVM_GET_ONE_REG`` and ``KVM_SET_ONE_REG`` return ``-EPERM`` once the
vCPU has run. Before that, they access the host-side copy from which
EL2 builds the guest's boot state (see `Boot`_).
* ``KVM_ARM_VCPU_INIT`` accepts only the vCPU features that a protected
guest supports and returns ``-EINVAL`` otherwise.
``KVM_ARM_VCPU_PSCI_0_2`` is required, as EL2 implements PSCI 1.1 for
the guest (see `Power state`_). ``KVM_ARM_VCPU_EL1_32BIT`` is not
supported: protected guests run in AArch64 only and see no AArch32
support in ``ID_AA64PFR0_EL1``. Once the vCPU has run,
``KVM_ARM_VCPU_INIT`` returns ``-EPERM``, as re-initialising it would
reset the host-side copy alone.
* ``KVM_SET_VCPU_EVENTS`` returns ``-EPERM`` for external-abort injection
(``ext_dabt_pending``), and ``KVM_CAP_ARM_INJECT_EXT_DABT`` is not
advertised to a protected VM. SError injection is unaffected.
* ``KVM_SET_GUEST_DEBUG`` returns ``-EPERM``, and ``KVM_CAP_SET_GUEST_DEBUG``
is not advertised to a protected VM (see `Debug`_).
Power state
-----------
EL2 implements PSCI 1.1 for a protected guest. The calls that move a
vCPU's power state, ``CPU_ON`` and ``CPU_OFF``, are handled at EL2, and
the host cannot change the outcome: for ``CPU_ON`` the host only
schedules the target, which EL2 has already reset to the entry point
the guest chose, and for ``CPU_OFF`` it only stops scheduling it.
``AFFINITY_INFO`` is handled by the host, which returns ``OFF`` once
it has stopped scheduling every vCPU the query covers, as for a
non-protected VM. A vCPU becomes a valid ``CPU_ON`` target at its first
``KVM_RUN``, and before that EL2 returns ``INVALID_PARAMETERS``. The
platform calls, ``CPU_SUSPEND``, ``SYSTEM_OFF``, ``SYSTEM_RESET`` and
``SYSTEM_RESET2``, are forwarded to the host and behave as for a
non-protected VM, with the ``SYSTEM_*`` calls exiting to the VMM as
``KVM_EXIT_SYSTEM_EVENT``; the ``SYSTEM_RESET2`` reset type and cookie
are in the guest's registers, which ``KVM_GET_ONE_REG`` rejects once
the vCPU has run. Any other function returns ``NOT_SUPPORTED``, and
``PSCI_FEATURES`` reports the same set. Because the host handles those
forwarded calls, ``KVM_SET_ONE_REG`` on ``KVM_REG_ARM_PSCI_VERSION``
returns ``-EINVAL`` for a version below 1.1.
Once a vCPU has run, its power state follows the guest's PSCI calls, not
the VMM's. A vCPU that the guest has powered off with ``CPU_OFF``, or has
not yet brought online with ``CPU_ON``, reads as
``KVM_MP_STATE_UNINITIALIZED``, and only an in-guest ``CPU_ON`` can power
it on: ``KVM_SET_MP_STATE`` returns ``-EPERM`` for ``KVM_MP_STATE_RUNNABLE``
and ``KVM_MP_STATE_SUSPENDED`` on it, accepts ``KVM_MP_STATE_STOPPED`` and
leaves it as it is, and never takes ``KVM_MP_STATE_UNINITIALIZED`` from
the VMM. ``KVM_MP_STATE_STOPPED`` still stops a vCPU EL2 has powered on,
and ``KVM_MP_STATE_RUNNABLE`` resumes it, so the VMM can pause it.
Other interface differences
---------------------------
* ``KVM_CHECK_EXTENSION`` reports only the arm64 capabilities that pKVM
supports for protected guests, and ``KVM_ENABLE_CAP`` accepts only
those. Query them on the VM file descriptor: the system file descriptor
has no VM to filter against. The capabilities the generic KVM code
answers itself (``KVM_CAP_SYNC_MMU`` or ``KVM_CAP_DEVICE_CTRL``, for
example) are reported as for any VM, and don't mean the feature works
for a protected one. The filter does not cover every interface either:
device-fd configuration (for example the VGIC after
``KVM_CREATE_DEVICE``) and vCPU attributes are unfiltered, so a feature
the filter excludes is rejected by the attribute itself.
``KVM_ARM_VCPU_PVTIME_CTRL`` returns ``-ENXIO``, for example, as it does
where steal time isn't implemented.
* The vGIC of a protected VM remains host-managed: device creation,
configuration and interrupt injection all work as they do for a
non-protected VM.
* ``KVM_ARM_SET_COUNTER_OFFSET`` and ``KVM_ARM_GET_REG_WRITABLE_MASKS``
return ``-EINVAL``: their capabilities are not advertised to a protected
VM, whose counter offset and ID registers are set by EL2. A protected
guest sees the physical timebase.
* A protected guest's first access to each page of memory exits to the
host, since the hypervisor cannot tell memory from a device before the
page is mapped. For a store, the host sees the value of the register
the syndrome names, clamped to the access width, and nothing else from
the register file. For a load, the register the syndrome names takes
the value the host returns, as for any emulated access.
* A protected guest that uses a feature it was not given, or executes an
``SMC``, takes an undefined instruction exception from the hypervisor;
the host is not involved.
* Apart from PSCI, the hypervisor handles a protected guest's SMCCC
calls itself and does not involve the host. A function it does not
implement returns ``NOT_SUPPORTED``, and the host's SMCCC filter does
not apply: the VM device attribute ioctls (``KVM_ARM_VM_SMCCC_CTRL``)
return ``-EINVAL`` for a protected VM.
* The hypervisor can decode a trapped guest access only from the CPU's
instruction syndrome, which is provided only for a load or store of a
single general-purpose register. An access without one (for example a
load/store pair or a SIMD/FP access) cannot be decoded. For a
non-protected VM it can exit to the VMM as ``KVM_EXIT_ARM_NISV``. For a
protected VM it cannot be emulated by the VMM, so the guest takes a
synchronous external abort instead.
Debug
-----
Hardware-assisted debugging is not available to protected guests.
``MDSCR_EL1``, ``OSLAR_EL1``, ``OSLSR_EL1``, ``OSDLR_EL1``, and breakpoint
and watchpoint 0 are RAZ/WI; any other debug register access takes an
undefined instruction exception.
Resources
=========

3
Documentation/virt/kvm/devices/vcpu.rst
View File

@ -231,7 +231,8 @@ Returns:
Specifies the base address of the stolen time structure for this VCPU. The
base address must be 64 byte aligned and exist within a valid guest memory
region. See Documentation/virt/kvm/arm/pvtime.rst for more information
including the layout of the stolen time structure.
including the layout of the stolen time structure. Stolen time is not
supported for protected VMs (see Documentation/virt/kvm/arm/pkvm.rst).
4. GROUP: KVM_VCPU_TSC_CTRL
===========================

22
arch/arm64/include/asm/kvm_host.h
View File

@ -1062,11 +1062,28 @@ struct kvm_vcpu_arch {
set; \
})
#define __vcpu_copy_flag(vt, vs, flagset, f, m) \
do { \
typeof(vs->arch.flagset) tmp, val; \
\
__build_check_flag(vs, flagset, f, m); \
\
val = READ_ONCE(vs->arch.flagset); \
val &= (m); \
__vcpu_flags_preempt_disable(); \
tmp = READ_ONCE(vt->arch.flagset); \
tmp &= ~(m); \
tmp |= val; \
WRITE_ONCE(vt->arch.flagset, tmp); \
__vcpu_flags_preempt_enable(); \
} while (0)
#define vcpu_get_flag(v, ...) __vcpu_get_flag((v), __VA_ARGS__)
#define vcpu_set_flag(v, ...) __vcpu_set_flag((v), __VA_ARGS__)
#define vcpu_clear_flag(v, ...) __vcpu_clear_flag((v), __VA_ARGS__)
#define vcpu_test_and_clear_flag(v, ...) \
__vcpu_test_and_clear_flag((v), __VA_ARGS__)
#define vcpu_copy_flag(vt, vs, ...) __vcpu_copy_flag((vt), (vs), __VA_ARGS__)
/* KVM_ARM_VCPU_INIT completed */
#define VCPU_INITIALIZED __vcpu_single_flag(cflags, BIT(0))
@ -1089,6 +1106,8 @@ bool kvm_arm_vcpu_is_finalized(struct kvm_vcpu *vcpu);
#define INCREMENT_PC __vcpu_single_flag(iflags, BIT(1))
/* Target EL/MODE (not a single flag, but let's abuse the macro) */
#define EXCEPT_MASK __vcpu_single_flag(iflags, GENMASK(3, 1))
/* Cover both PENDING_EXCEPTION and EXCEPT_MASK for global operations */
#define PC_UPDATE_REQ __vcpu_single_flag(iflags, GENMASK(3, 0))
/* Host-set: the hyp flushes the non-protected vCPU state in on entry */
#define PKVM_HOST_STATE_DIRTY __vcpu_single_flag(iflags, BIT(4))
@ -1343,7 +1362,7 @@ long kvm_hypercall_pv_features(struct kvm_vcpu *vcpu);
gpa_t kvm_init_stolen_time(struct kvm_vcpu *vcpu);
void kvm_update_stolen_time(struct kvm_vcpu *vcpu);
bool kvm_arm_pvtime_supported(void);
bool kvm_arm_pvtime_supported(struct kvm *kvm);
int kvm_arm_pvtime_set_attr(struct kvm_vcpu *vcpu,
struct kvm_device_attr *attr);
int kvm_arm_pvtime_get_attr(struct kvm_vcpu *vcpu,
@ -1541,6 +1560,7 @@ static inline void kvm_hyp_reserve(void) { }
void kvm_arm_vcpu_power_off(struct kvm_vcpu *vcpu);
bool kvm_arm_vcpu_stopped(struct kvm_vcpu *vcpu);
bool kvm_pkvm_vcpu_is_powered_off(struct kvm_vcpu *vcpu);
static inline u64 *__vm_id_reg(struct kvm_arch *ka, u32 reg)
{

4
arch/arm64/include/asm/kvm_hyp.h
View File

@ -10,6 +10,7 @@
#include <linux/compiler.h>
#include <linux/kvm_host.h>
#include <asm/alternative.h>
#include <asm/spectre.h>
#include <asm/sysreg.h>
DECLARE_PER_CPU(struct kvm_cpu_context, kvm_hyp_ctxt);
@ -153,6 +154,9 @@ extern u64 kvm_nvhe_sym(id_aa64smfr0_el1_sys_val);
extern unsigned long kvm_nvhe_sym(__icache_flags);
extern unsigned int kvm_nvhe_sym(kvm_arm_vmid_bits);
extern enum mitigation_state kvm_nvhe_sym(spectre_v2_state);
extern enum mitigation_state kvm_nvhe_sym(spectre_v4_state);
extern enum mitigation_state kvm_nvhe_sym(spectre_bhb_state);
extern unsigned int kvm_nvhe_sym(kvm_host_sve_max_vl);
extern unsigned long kvm_nvhe_sym(hyp_nr_cpus);
extern unsigned int kvm_nvhe_sym(hyp_gicv3_nr_lr);

33
arch/arm64/include/asm/kvm_pkvm.h
View File

@ -42,6 +42,15 @@ static inline bool kvm_pkvm_ext_allowed(struct kvm *kvm, long ext)
case KVM_CAP_ARM_VM_IPA_SIZE:
case KVM_CAP_ARM_PTRAUTH_ADDRESS:
case KVM_CAP_ARM_PTRAUTH_GENERIC:
case KVM_CAP_ONE_REG:
case KVM_CAP_MP_STATE:
case KVM_CAP_VCPU_EVENTS:
case KVM_CAP_VCPU_ATTRIBUTES:
case KVM_CAP_IMMEDIATE_EXIT:
case KVM_CAP_IOEVENTFD:
case KVM_CAP_IRQFD_RESAMPLE:
case KVM_CAP_ARM_IRQ_LINE_LAYOUT_2:
case KVM_CAP_ARM_INJECT_SERROR_ESR:
return true;
case KVM_CAP_ARM_MTE:
case KVM_CAP_ARM_EAGER_SPLIT_CHUNK_SIZE:
@ -53,6 +62,30 @@ static inline bool kvm_pkvm_ext_allowed(struct kvm *kvm, long ext)
}
}
/*
* The vCPU features a protected VM may use: checked by the host at
* KVM_ARM_VCPU_INIT, applied by EL2 when the hyp VM is created.
*/
static inline void kvm_pkvm_vcpu_allowed_features(struct kvm *kvm,
unsigned long *allowed)
{
bitmap_zero(allowed, KVM_VCPU_MAX_FEATURES);
set_bit(KVM_ARM_VCPU_PSCI_0_2, allowed);
if (kvm_pkvm_ext_allowed(kvm, KVM_CAP_ARM_PMU_V3))
set_bit(KVM_ARM_VCPU_PMU_V3, allowed);
if (kvm_pkvm_ext_allowed(kvm, KVM_CAP_ARM_PTRAUTH_ADDRESS))
set_bit(KVM_ARM_VCPU_PTRAUTH_ADDRESS, allowed);
if (kvm_pkvm_ext_allowed(kvm, KVM_CAP_ARM_PTRAUTH_GENERIC))
set_bit(KVM_ARM_VCPU_PTRAUTH_GENERIC, allowed);
if (kvm_pkvm_ext_allowed(kvm, KVM_CAP_ARM_SVE))
set_bit(KVM_ARM_VCPU_SVE, allowed);
}
/*
* Check whether the KVM VM IOCTL is allowed in pKVM.
*

76
arch/arm64/kvm/arm.c
View File

@ -452,7 +452,7 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
r = system_supports_mte();
break;
case KVM_CAP_STEAL_TIME:
r = kvm_arm_pvtime_supported();
r = kvm_arm_pvtime_supported(kvm);
break;
case KVM_CAP_ARM_EL1_32BIT:
r = cpus_have_final_cap(ARM64_HAS_32BIT_EL1);
@ -775,23 +775,28 @@ void kvm_arch_vcpu_put(struct kvm_vcpu *vcpu)
vcpu->cpu = -1;
}
static void __kvm_arm_vcpu_power_off(struct kvm_vcpu *vcpu)
static void __kvm_arm_vcpu_power_off(struct kvm_vcpu *vcpu, u32 mp_state)
{
WRITE_ONCE(vcpu->arch.mp_state.mp_state, KVM_MP_STATE_STOPPED);
WRITE_ONCE(vcpu->arch.mp_state.mp_state, mp_state);
kvm_make_request(KVM_REQ_SLEEP, vcpu);
kvm_vcpu_kick(vcpu);
}
/* The guest's own CPU_OFF: EL2 has already powered the vCPU off. */
void kvm_arm_vcpu_power_off(struct kvm_vcpu *vcpu)
{
u32 mp_state = vcpu_is_protected(vcpu) ? KVM_MP_STATE_UNINITIALIZED : KVM_MP_STATE_STOPPED;
spin_lock(&vcpu->arch.mp_state_lock);
__kvm_arm_vcpu_power_off(vcpu);
__kvm_arm_vcpu_power_off(vcpu, mp_state);
spin_unlock(&vcpu->arch.mp_state_lock);
}
bool kvm_arm_vcpu_stopped(struct kvm_vcpu *vcpu)
{
return READ_ONCE(vcpu->arch.mp_state.mp_state) == KVM_MP_STATE_STOPPED;
u32 mp_state = READ_ONCE(vcpu->arch.mp_state.mp_state);
return mp_state == KVM_MP_STATE_STOPPED || mp_state == KVM_MP_STATE_UNINITIALIZED;
}
static void kvm_arm_vcpu_suspend(struct kvm_vcpu *vcpu)
@ -806,6 +811,12 @@ static bool kvm_arm_vcpu_suspended(struct kvm_vcpu *vcpu)
return READ_ONCE(vcpu->arch.mp_state.mp_state) == KVM_MP_STATE_SUSPENDED;
}
/* Only the guest's CPU_ON may start a vCPU EL2 holds powered off. */
bool kvm_pkvm_vcpu_is_powered_off(struct kvm_vcpu *vcpu)
{
return READ_ONCE(vcpu->arch.mp_state.mp_state) == KVM_MP_STATE_UNINITIALIZED;
}
int kvm_arch_vcpu_ioctl_get_mpstate(struct kvm_vcpu *vcpu,
struct kvm_mp_state *mp_state)
{
@ -823,12 +834,24 @@ int kvm_arch_vcpu_ioctl_set_mpstate(struct kvm_vcpu *vcpu,
switch (mp_state->mp_state) {
case KVM_MP_STATE_RUNNABLE:
if (kvm_pkvm_vcpu_is_powered_off(vcpu)) {
ret = -EPERM;
break;
}
WRITE_ONCE(vcpu->arch.mp_state, *mp_state);
break;
case KVM_MP_STATE_STOPPED:
__kvm_arm_vcpu_power_off(vcpu);
/* Keep EL2's off record: STOPPED over it would let RUNNABLE through. */
if (!kvm_pkvm_vcpu_is_powered_off(vcpu))
__kvm_arm_vcpu_power_off(vcpu, KVM_MP_STATE_STOPPED);
break;
case KVM_MP_STATE_SUSPENDED:
if (kvm_pkvm_vcpu_is_powered_off(vcpu)) {
ret = -EPERM;
break;
}
kvm_arm_vcpu_suspend(vcpu);
break;
default:
@ -1634,6 +1657,19 @@ static int kvm_vcpu_init_check_features(struct kvm_vcpu *vcpu,
if (features & ~system_supported_vcpu_features())
return -EINVAL;
/* Reject features EL2 would drop when it creates the hyp VM. */
if (vcpu_is_protected(vcpu)) {
DECLARE_BITMAP(allowed, KVM_VCPU_MAX_FEATURES);
kvm_pkvm_vcpu_allowed_features(vcpu->kvm, allowed);
if (!bitmap_subset(&features, allowed, KVM_VCPU_MAX_FEATURES))
return -EINVAL;
/* EL2 implements PSCI 1.1; the host must not dispatch as 0.1. */
if (!test_bit(KVM_ARM_VCPU_PSCI_0_2, &features))
return -EINVAL;
}
/*
* For now make sure that both address/generic pointer authentication
* features are requested by the userspace together.
@ -1791,7 +1827,7 @@ static int kvm_arch_vcpu_ioctl_vcpu_init(struct kvm_vcpu *vcpu,
spin_lock(&vcpu->arch.mp_state_lock);
if (power_off)
__kvm_arm_vcpu_power_off(vcpu);
__kvm_arm_vcpu_power_off(vcpu, KVM_MP_STATE_STOPPED);
else
WRITE_ONCE(vcpu->arch.mp_state.mp_state, KVM_MP_STATE_RUNNABLE);
@ -1868,6 +1904,24 @@ static int kvm_arm_vcpu_set_events(struct kvm_vcpu *vcpu,
return __kvm_arm_vcpu_set_events(vcpu, events);
}
/*
* Once a protected vCPU has run, the host copy holds the boot state the VMM
* wrote plus what the exit handlers copy out, and EL2 has read mp_state,
* which it does only at hyp vCPU creation.
*/
static long pkvm_filter_vcpu_ioctl(struct kvm_vcpu *vcpu, unsigned int ioctl)
{
switch (ioctl) {
case KVM_ARM_VCPU_INIT:
case KVM_SET_ONE_REG:
case KVM_GET_ONE_REG:
if (vcpu_is_protected(vcpu) && vcpu_has_run_once(vcpu))
return -EPERM;
}
return 0;
}
long kvm_arch_vcpu_ioctl(struct file *filp,
unsigned int ioctl, unsigned long arg)
{
@ -1876,6 +1930,10 @@ long kvm_arch_vcpu_ioctl(struct file *filp,
struct kvm_device_attr attr;
long r;
r = pkvm_filter_vcpu_ioctl(vcpu, ioctl);
if (r)
return r;
switch (ioctl) {
case KVM_ARM_VCPU_INIT: {
struct kvm_vcpu_init init;
@ -2638,6 +2696,10 @@ static void kvm_hyp_init_symbols(void)
kvm_nvhe_sym(__icache_flags) = __icache_flags;
kvm_nvhe_sym(kvm_arm_vmid_bits) = kvm_arm_vmid_bits;
kvm_nvhe_sym(spectre_v2_state) = arm64_get_spectre_v2_state();
kvm_nvhe_sym(spectre_v4_state) = arm64_get_spectre_v4_state();
kvm_nvhe_sym(spectre_bhb_state) = arm64_get_spectre_bhb_state();
/* Propagate the FGT state to the nVHE side */
kvm_nvhe_sym(hfgrtr_masks) = hfgrtr_masks;
kvm_nvhe_sym(hfgwtr_masks) = hfgwtr_masks;

11
arch/arm64/kvm/guest.c
View File

@ -792,6 +792,13 @@ int __kvm_arm_vcpu_set_events(struct kvm_vcpu *vcpu,
u64 esr = events->exception.serror_esr;
int ret = 0;
/*
* EL2 injects an external abort only to complete a forwarded abort.
* SError injection is forwarded.
*/
if (vcpu_is_protected(vcpu) && ext_dabt_pending)
return -EPERM;
/*
* Immediately commit the pending SEA to the vCPU's architectural
* state which is necessary since we do not return a pending SEA
@ -889,6 +896,10 @@ int kvm_arch_vcpu_ioctl_set_guest_debug(struct kvm_vcpu *vcpu,
{
trace_kvm_set_guest_debug(vcpu, dbg->control);
/* A protected guest's debug state is not exposed to the host. */
if (vcpu_is_protected(vcpu))
return -EPERM;
if (dbg->control & ~KVM_GUESTDBG_VALID_MASK)
return -EINVAL;

26
arch/arm64/kvm/hyp/exception.c
View File

@ -70,6 +70,7 @@ static void enter_exception64(struct kvm_vcpu *vcpu, unsigned long target_mode,
enum exception_type type)
{
unsigned long sctlr, vbar, old, new, mode;
struct kvm *kvm;
u64 exc_offset;
mode = *vcpu_cpsr(vcpu) & (PSR_MODE_MASK | PSR_MODE32_BIT);
@ -109,8 +110,10 @@ static void enter_exception64(struct kvm_vcpu *vcpu, unsigned long target_mode,
new |= (old & PSR_C_BIT);
new |= (old & PSR_V_BIT);
if (kvm_has_mte(kern_hyp_va(vcpu->kvm)))
kvm = vcpu_get_kvm(vcpu);
if (kvm && kvm_has_mte(kvm))
new |= PSR_TCO_BIT;
vcpu_put_kvm(vcpu, kvm);
new |= (old & PSR_DIT_BIT);
@ -350,12 +353,19 @@ static void kvm_inject_exception(struct kvm_vcpu *vcpu)
*/
void __kvm_adjust_pc(struct kvm_vcpu *vcpu)
{
if (vcpu_get_flag(vcpu, PENDING_EXCEPTION)) {
kvm_inject_exception(vcpu);
vcpu_clear_flag(vcpu, PENDING_EXCEPTION);
vcpu_clear_flag(vcpu, EXCEPT_MASK);
} else if (vcpu_get_flag(vcpu, INCREMENT_PC)) {
kvm_skip_instr(vcpu);
vcpu_clear_flag(vcpu, INCREMENT_PC);
struct kvm_vcpu *target = kvm_adjust_pc_get(vcpu);
if (!target)
return;
if (vcpu_get_flag(target, PENDING_EXCEPTION)) {
kvm_inject_exception(target);
vcpu_clear_flag(target, PENDING_EXCEPTION);
vcpu_clear_flag(target, EXCEPT_MASK);
} else if (vcpu_get_flag(target, INCREMENT_PC)) {
kvm_skip_instr(target);
vcpu_clear_flag(target, INCREMENT_PC);
}
kvm_adjust_pc_put(vcpu, target);
}

48
arch/arm64/kvm/hyp/include/hyp/adjust_pc.h
View File

@ -13,6 +13,34 @@
#include <asm/kvm_emulate.h>
#include <asm/kvm_host.h>
#ifdef __KVM_NVHE_HYPERVISOR__
#include <nvhe/pkvm.h>
/* Under pKVM a host vCPU's ->kvm is host-writable. */
static inline struct kvm *vcpu_get_kvm(struct kvm_vcpu *vcpu)
{
if (is_protected_kvm_enabled())
return pkvm_vcpu_get_kvm(vcpu);
return kern_hyp_va(vcpu->kvm);
}
static inline void vcpu_put_kvm(struct kvm_vcpu *vcpu, struct kvm *kvm)
{
if (is_protected_kvm_enabled())
pkvm_vcpu_put_kvm(vcpu, kvm);
}
#else
static inline struct kvm *vcpu_get_kvm(struct kvm_vcpu *vcpu)
{
return vcpu->kvm;
}
static inline void vcpu_put_kvm(struct kvm_vcpu *vcpu, struct kvm *kvm)
{
}
#endif
static inline void kvm_skip_instr(struct kvm_vcpu *vcpu)
{
if (vcpu_mode_is_32bit(vcpu)) {
@ -50,4 +78,24 @@ static inline void kvm_skip_host_instr(void)
write_sysreg_el2(read_sysreg_el2(SYS_ELR) + 4, SYS_ELR);
}
/*
* Under pKVM, the vCPU __kvm_adjust_pc() adjusts for @vcpu (NULL leaves the
* request on @vcpu for its next entry), and the copy of the consumed
* PC_UPDATE_REQ back to @vcpu.
*/
#ifdef __KVM_NVHE_HYPERVISOR__
struct kvm_vcpu *kvm_adjust_pc_get(struct kvm_vcpu *vcpu);
void kvm_adjust_pc_put(struct kvm_vcpu *vcpu, struct kvm_vcpu *target);
#else
static inline struct kvm_vcpu *kvm_adjust_pc_get(struct kvm_vcpu *vcpu)
{
return vcpu;
}
static inline void kvm_adjust_pc_put(struct kvm_vcpu *vcpu,
struct kvm_vcpu *target)
{
}
#endif
#endif

20
arch/arm64/kvm/hyp/include/nvhe/pkvm.h
View File

@ -27,6 +27,15 @@ struct pkvm_hyp_vcpu {
* per-cpu pointer tracking us. Otherwise, NULL if not loaded.
*/
struct pkvm_hyp_vcpu **loaded_hyp_vcpu;
/* The previous exit's ARM_EXCEPTION_* code. */
u32 exit_code;
/*
* PSCI_0_2_AFFINITY_LEVEL_{OFF, ON_PENDING, ON}. A non-protected
* vCPU is always ON.
*/
int power_state;
};
/*
@ -44,6 +53,12 @@ struct pkvm_hyp_vm {
struct hyp_pool pool;
hyp_spinlock_t lock;
/*
* The vCPU initialised RUNNABLE: claimed under vm_table_lock,
* released only if its own init fails.
*/
struct pkvm_hyp_vcpu *primary_vcpu;
/* Array of the hyp vCPU structures for this VM. */
struct pkvm_hyp_vcpu *vcpus[];
};
@ -84,6 +99,8 @@ struct pkvm_hyp_vcpu *pkvm_load_hyp_vcpu(pkvm_handle_t handle,
unsigned int vcpu_idx);
void pkvm_put_hyp_vcpu(struct pkvm_hyp_vcpu *hyp_vcpu);
struct pkvm_hyp_vcpu *pkvm_get_loaded_hyp_vcpu(void);
struct kvm *pkvm_vcpu_get_kvm(struct kvm_vcpu *vcpu);
void pkvm_vcpu_put_kvm(struct kvm_vcpu *vcpu, struct kvm *kvm);
struct pkvm_hyp_vm *get_pkvm_hyp_vm(pkvm_handle_t handle);
struct pkvm_hyp_vm *get_np_pkvm_hyp_vm(pkvm_handle_t handle);
@ -93,6 +110,9 @@ bool kvm_handle_pvm_hvc64(struct kvm_vcpu *vcpu, u64 *exit_code);
bool kvm_handle_pvm_sysreg(struct kvm_vcpu *vcpu, u64 *exit_code);
bool kvm_handle_pvm_restricted(struct kvm_vcpu *vcpu, u64 *exit_code);
void kvm_init_pvm_id_regs(struct kvm_vcpu *vcpu);
void kvm_reset_pvm_sys_regs(struct kvm_vcpu *vcpu);
int kvm_check_pvm_sysreg_table(void);
int pkvm_reset_vcpu(struct pkvm_hyp_vcpu *hyp_vcpu);
struct pkvm_hyp_vcpu *pkvm_mpidr_to_hyp_vcpu(struct pkvm_hyp_vm *vm, unsigned long mpidr);
#endif /* __ARM64_KVM_NVHE_PKVM_H__ */

597
arch/arm64/kvm/hyp/nvhe/hyp-main.c
View File

@ -4,10 +4,13 @@
* Author: Andrew Scull <ascull@google.com>
*/
#include <kvm/arm_hypercalls.h>
#include <hyp/adjust_pc.h>
#include <hyp/switch.h>
#include <linux/irqchip/arm-gic-v3.h>
#include <uapi/linux/psci.h>
#include <asm/pgtable-types.h>
#include <asm/kvm_asm.h>
@ -73,6 +76,375 @@ unsigned int hyp_gicv3_nr_lr;
void __kvm_hyp_host_forward_smc(struct kvm_cpu_context *host_ctxt);
typedef void (*hyp_entry_exit_handler_fn)(struct pkvm_hyp_vcpu *);
static bool pvm_sys64_is_write(u64 esr)
{
return (esr & ESR_ELx_SYS64_ISS_DIR_MASK) == ESR_ELx_SYS64_ISS_DIR_WRITE;
}
static void handle_pvm_entry_wfx(struct pkvm_hyp_vcpu *hyp_vcpu)
{
struct kvm_vcpu *host_vcpu = hyp_vcpu->host_vcpu;
struct kvm_vcpu *vcpu = &hyp_vcpu->vcpu;
/* Exceptions have priority; the host injects none on WFx. */
if (vcpu_get_flag(host_vcpu, PENDING_EXCEPTION))
return;
if (vcpu_get_flag(host_vcpu, INCREMENT_PC)) {
vcpu_clear_flag(vcpu, PC_UPDATE_REQ);
kvm_incr_pc(vcpu);
}
}
static void handle_pvm_entry_sys64(struct pkvm_hyp_vcpu *hyp_vcpu)
{
struct kvm_vcpu *host_vcpu = hyp_vcpu->host_vcpu;
struct kvm_vcpu *vcpu = &hyp_vcpu->vcpu;
bool pc_update;
/* Exceptions have priority over anything else */
if (vcpu_get_flag(host_vcpu, PENDING_EXCEPTION)) {
/* A host-requested exception on SYS64 is always an UNDEF. */
u32 esr = (ESR_ELx_EC_UNKNOWN << ESR_ELx_EC_SHIFT) | ESR_ELx_IL;
__vcpu_assign_sys_reg(vcpu, ESR_EL1, esr);
kvm_pend_exception(vcpu, EXCEPT_AA64_EL1_SYNC);
return;
}
/* Handle PC increment on a host-emulated access */
pc_update = vcpu_get_flag(host_vcpu, INCREMENT_PC);
if (pc_update) {
vcpu_clear_flag(vcpu, PC_UPDATE_REQ);
kvm_incr_pc(vcpu);
}
/* If the host emulated a read access, update the register */
if (pc_update && !pvm_sys64_is_write(kvm_vcpu_get_esr(vcpu))) {
/* r0 as transfer register between the guest and the host. */
u64 rt_val = READ_ONCE(vcpu_gp_regs(host_vcpu)[0]);
int rt = kvm_vcpu_sys_get_rt(vcpu);
vcpu_set_reg(vcpu, rt, rt_val);
}
}
static void handle_pvm_entry_iabt(struct pkvm_hyp_vcpu *hyp_vcpu)
{
struct kvm_vcpu *host_vcpu = hyp_vcpu->host_vcpu;
struct kvm_vcpu *vcpu = &hyp_vcpu->vcpu;
unsigned long cpsr = *vcpu_cpsr(vcpu);
u32 esr = ESR_ELx_IL;
if (!vcpu_get_flag(host_vcpu, PENDING_EXCEPTION))
return;
/* The host's only IABT injection: an external abort. */
if ((cpsr & PSR_MODE_MASK) == PSR_MODE_EL0t)
esr |= (ESR_ELx_EC_IABT_LOW << ESR_ELx_EC_SHIFT);
else
esr |= (ESR_ELx_EC_IABT_CUR << ESR_ELx_EC_SHIFT);
esr |= ESR_ELx_FSC_EXTABT;
__vcpu_assign_sys_reg(vcpu, ESR_EL1, esr);
__vcpu_assign_sys_reg(vcpu, FAR_EL1, kvm_vcpu_get_hfar(vcpu));
/* Injected by __kvm_adjust_pc() on entry. */
kvm_pend_exception(vcpu, EXCEPT_AA64_EL1_SYNC);
}
/*
* Clamp MMIO data to the access width, so a write does not leak the
* register's upper bits and a read takes no bits beyond the load. The
* host applies endianness.
*/
static inline u64 kvm_mmio_clamp_data(struct kvm_vcpu *vcpu, u64 val)
{
unsigned int len = kvm_vcpu_dabt_get_as(vcpu);
return val & GENMASK_U64(len * 8 - 1, 0);
}
/*
* Complete an MMIO load: sign-extend from EL2's own syndrome, as the
* architecture does.
*/
static inline u64 kvm_mmio_read_data(struct kvm_vcpu *vcpu, u64 val)
{
val = kvm_mmio_clamp_data(vcpu, val);
if (kvm_vcpu_dabt_issext(vcpu))
val = sign_extend64(val, kvm_vcpu_dabt_get_as(vcpu) * 8 - 1);
if (!kvm_vcpu_dabt_issf(vcpu))
val &= GENMASK_U64(31, 0);
return val;
}
static void handle_pvm_entry_dabt(struct pkvm_hyp_vcpu *hyp_vcpu)
{
struct kvm_vcpu *host_vcpu = hyp_vcpu->host_vcpu;
struct kvm_vcpu *vcpu = &hyp_vcpu->vcpu;
bool pc_update;
/* Exceptions have priority over anything else */
if (vcpu_get_flag(host_vcpu, PENDING_EXCEPTION)) {
unsigned long cpsr = *vcpu_cpsr(vcpu);
u32 esr = ESR_ELx_IL;
if ((cpsr & PSR_MODE_MASK) == PSR_MODE_EL0t)
esr |= (ESR_ELx_EC_DABT_LOW << ESR_ELx_EC_SHIFT);
else
esr |= (ESR_ELx_EC_DABT_CUR << ESR_ELx_EC_SHIFT);
esr |= ESR_ELx_FSC_EXTABT;
__vcpu_assign_sys_reg(vcpu, ESR_EL1, esr);
__vcpu_assign_sys_reg(vcpu, FAR_EL1, kvm_vcpu_get_hfar(vcpu));
/* Injected by __kvm_adjust_pc() on entry. */
kvm_pend_exception(vcpu, EXCEPT_AA64_EL1_SYNC);
/* Cancel any in-flight MMIO */
vcpu->mmio_needed = false;
return;
}
/* Handle PC increment on MMIO, or on a CMO the host skipped */
pc_update = vcpu_get_flag(host_vcpu, INCREMENT_PC) &&
(vcpu->mmio_needed || esr_dabt_is_cm(kvm_vcpu_get_esr(vcpu)));
if (pc_update) {
vcpu_clear_flag(vcpu, PC_UPDATE_REQ);
kvm_incr_pc(vcpu);
}
/* If the host emulated an MMIO read, update the register */
if (pc_update && vcpu->mmio_needed && !kvm_vcpu_dabt_iswrite(vcpu)) {
/* r0 as transfer register between the guest and the host. */
u64 rd_val = READ_ONCE(vcpu_gp_regs(host_vcpu)[0]);
int rd = kvm_vcpu_dabt_get_rd(vcpu);
rd_val = kvm_mmio_read_data(vcpu, rd_val);
vcpu_set_reg(vcpu, rd, rd_val);
}
vcpu->mmio_needed = false;
}
static void handle_pvm_entry_hvc64(struct pkvm_hyp_vcpu *hyp_vcpu)
{
struct kvm_vcpu *host_vcpu = hyp_vcpu->host_vcpu;
struct kvm_vcpu *vcpu = &hyp_vcpu->vcpu;
u64 ret = READ_ONCE(vcpu_gp_regs(host_vcpu)[0]);
u32 psci_fn = smccc_get_function(vcpu);
switch (psci_fn) {
case PSCI_0_2_FN_CPU_ON:
case PSCI_0_2_FN64_CPU_ON:
/*
* Roll back a CPU_ON the host failed, unless the target
* already reached ON: it is running, and the guest sees
* SUCCESS.
*/
if (ret != PSCI_RET_SUCCESS) {
unsigned long cpu_id = smccc_get_arg1(vcpu);
struct pkvm_hyp_vcpu *target_vcpu;
struct pkvm_hyp_vm *hyp_vm;
int prev;
hyp_vm = pkvm_hyp_vcpu_to_hyp_vm(hyp_vcpu);
target_vcpu = pkvm_mpidr_to_hyp_vcpu(hyp_vm, cpu_id);
/*
* pvm_psci_vcpu_on() resolved this MPIDR and vcpus[]
* entries are never removed, so the lookup cannot miss.
* The release orders this vCPU's reset_state writes
* before OFF, for the next CPU_ON.
*/
prev = cmpxchg_release(&target_vcpu->power_state,
PSCI_0_2_AFFINITY_LEVEL_ON_PENDING,
PSCI_0_2_AFFINITY_LEVEL_OFF);
switch (prev) {
case PSCI_0_2_AFFINITY_LEVEL_ON_PENDING:
/*
* Leave reset_state.reset set: a clear races a
* fresh CPU_ON's publish. The stale pc/r0/be are
* the guest's own. ALREADY_ON is PSCI's retry
* signal for a CPU_ON that raced the CPU_OFF.
*/
if (ret != PSCI_RET_ALREADY_ON)
ret = PSCI_RET_INTERNAL_FAILURE;
break;
case PSCI_0_2_AFFINITY_LEVEL_ON:
case PSCI_0_2_AFFINITY_LEVEL_OFF:
/* Target already ran (and may have stopped). */
ret = PSCI_RET_SUCCESS;
break;
default:
ret = PSCI_RET_INTERNAL_FAILURE;
break;
}
}
break;
default:
break;
}
vcpu_set_reg(vcpu, 0, ret);
}
/* The host's view of a syndrome: the guest register index is withheld. */
static u64 pvm_host_esr(u64 esr)
{
switch (ESR_ELx_EC(esr)) {
case ESR_ELx_EC_WFx:
return esr & ~ESR_ELx_WFx_ISS_RN;
case ESR_ELx_EC_SYS64:
return esr & ~ESR_ELx_SYS64_ISS_RT_MASK;
case ESR_ELx_EC_DABT_LOW:
return esr & ~ESR_ELx_SRT_MASK;
default:
return esr;
}
}
/*
* The host's view of PSTATE: the mode, with SErrors masked so that the host
* pends an SError through HCR_EL2.VSE rather than emulating the entry.
*/
static unsigned long pvm_host_pstate(unsigned long pstate)
{
return (pstate & PSR_MODE_MASK) | PSR_A_BIT;
}
static void handle_pvm_exit_wfx(struct pkvm_hyp_vcpu *hyp_vcpu)
{
struct kvm_vcpu *host_vcpu = hyp_vcpu->host_vcpu;
struct kvm_vcpu *vcpu = &hyp_vcpu->vcpu;
*vcpu_cpsr(host_vcpu) = pvm_host_pstate(*vcpu_cpsr(vcpu));
}
static void handle_pvm_exit_sys64(struct pkvm_hyp_vcpu *hyp_vcpu)
{
struct kvm_vcpu *host_vcpu = hyp_vcpu->host_vcpu;
struct kvm_vcpu *vcpu = &hyp_vcpu->vcpu;
u64 esr = kvm_vcpu_get_esr(vcpu);
/* The mode is required for the host to emulate some sysregs */
*vcpu_cpsr(host_vcpu) = pvm_host_pstate(*vcpu_cpsr(vcpu));
/* r0 as transfer register between the guest and the host. */
if (pvm_sys64_is_write(esr)) {
int rt = kvm_vcpu_sys_get_rt(vcpu);
u64 rt_val = vcpu_get_reg(vcpu, rt);
vcpu_set_reg(host_vcpu, 0, rt_val);
}
}
static void handle_pvm_exit_iabt(struct pkvm_hyp_vcpu *hyp_vcpu)
{
struct kvm_vcpu *host_vcpu = hyp_vcpu->host_vcpu;
struct kvm_vcpu *vcpu = &hyp_vcpu->vcpu;
host_vcpu->arch.fault.hpfar_el2 = vcpu->arch.fault.hpfar_el2;
}
static void handle_pvm_exit_dabt(struct pkvm_hyp_vcpu *hyp_vcpu)
{
struct kvm_vcpu *host_vcpu = hyp_vcpu->host_vcpu;
struct kvm_vcpu *vcpu = &hyp_vcpu->vcpu;
u64 sctlr;
/*
* EL2 has no memslot view: a decodable data abort is prepared as MMIO
* for the host to resolve. On unbacked memory the host injects an SEA
* for one with ISV clear (LDP/STP, atomics), which EL2 does not
* decode, and skips a cache maintenance operation, as for any guest.
*/
vcpu->mmio_needed = kvm_vcpu_dabt_isvalid(vcpu);
/* r0 as transfer register between the guest and the host. */
if (vcpu->mmio_needed && kvm_vcpu_dabt_iswrite(vcpu)) {
int rt = kvm_vcpu_dabt_get_rd(vcpu);
u64 rt_val = vcpu_get_reg(vcpu, rt);
rt_val = kvm_mmio_clamp_data(vcpu, rt_val);
vcpu_set_reg(host_vcpu, 0, rt_val);
}
*vcpu_cpsr(host_vcpu) = pvm_host_pstate(*vcpu_cpsr(vcpu));
host_vcpu->arch.fault.far_el2 = vcpu->arch.fault.far_el2 & GENMASK(11, 0);
host_vcpu->arch.fault.hpfar_el2 = vcpu->arch.fault.hpfar_el2;
sctlr = __vcpu_sys_reg(vcpu, SCTLR_EL1) & (SCTLR_ELx_EE | SCTLR_EL1_E0E);
__vcpu_assign_sys_reg(host_vcpu, SCTLR_EL1, sctlr);
}
static void handle_pvm_exit_hvc64(struct pkvm_hyp_vcpu *hyp_vcpu)
{
struct kvm_vcpu *host_vcpu = hyp_vcpu->host_vcpu;
struct kvm_vcpu *vcpu = &hyp_vcpu->vcpu;
int n, i;
switch (smccc_get_function(vcpu)) {
/*
* CPU_ON: the host uses only the target MPIDR (x1). EL2 resets the
* target from its own copy of the entry point and context id.
*/
case PSCI_0_2_FN_CPU_ON:
case PSCI_0_2_FN64_CPU_ON:
n = 2;
break;
case PSCI_0_2_FN_CPU_OFF:
case PSCI_0_2_FN_SYSTEM_OFF:
case PSCI_0_2_FN_SYSTEM_RESET:
case PSCI_0_2_FN_CPU_SUSPEND:
case PSCI_0_2_FN64_CPU_SUSPEND:
n = 1;
break;
case PSCI_0_2_FN_AFFINITY_INFO:
case PSCI_0_2_FN64_AFFINITY_INFO:
case PSCI_1_1_FN_SYSTEM_RESET2:
case PSCI_1_1_FN64_SYSTEM_RESET2:
n = 3;
break;
/* Unreachable: kvm_handle_pvm_hvc64() forwards only the calls above. */
default:
hyp_panic();
}
/* Pass the HVC function id (r0) and its arguments. */
for (i = 0; i < n; i++)
vcpu_set_reg(host_vcpu, i, vcpu_get_reg(vcpu, i));
}
static const hyp_entry_exit_handler_fn entry_hyp_pvm_handlers[] = {
[0 ... ESR_ELx_EC_MAX] = NULL,
[ESR_ELx_EC_WFx] = handle_pvm_entry_wfx,
[ESR_ELx_EC_SYS64] = handle_pvm_entry_sys64,
[ESR_ELx_EC_IABT_LOW] = handle_pvm_entry_iabt,
[ESR_ELx_EC_DABT_LOW] = handle_pvm_entry_dabt,
[ESR_ELx_EC_HVC64] = handle_pvm_entry_hvc64,
};
static const hyp_entry_exit_handler_fn exit_hyp_pvm_handlers[] = {
[0 ... ESR_ELx_EC_MAX] = NULL,
[ESR_ELx_EC_WFx] = handle_pvm_exit_wfx,
[ESR_ELx_EC_SYS64] = handle_pvm_exit_sys64,
[ESR_ELx_EC_IABT_LOW] = handle_pvm_exit_iabt,
[ESR_ELx_EC_DABT_LOW] = handle_pvm_exit_dabt,
[ESR_ELx_EC_HVC64] = handle_pvm_exit_hvc64,
};
static void __hyp_sve_save_guest(struct kvm_vcpu *vcpu)
{
__vcpu_assign_sys_reg(vcpu, ZCR_EL1, read_sysreg_el1(SYS_ZCR));
@ -185,6 +557,35 @@ static void sync_hyp_vgic_state(struct pkvm_hyp_vcpu *hyp_vcpu)
host_cpu_if->vgic_lr[i] = hyp_cpu_if->vgic_lr[i];
}
static void flush_hyp_timer_state(struct pkvm_hyp_vcpu *hyp_vcpu)
{
struct kvm_vcpu *vcpu = &hyp_vcpu->vcpu;
if (!pkvm_hyp_vcpu_is_protected(hyp_vcpu))
return;
/* A hyp vcpu has no offset, and sees vtime == ptime. */
write_sysreg(0, cntvoff_el2);
write_sysreg_el0(__vcpu_sys_reg(vcpu, CNTV_CVAL_EL0), SYS_CNTV_CVAL);
isb();
write_sysreg_el0(__vcpu_sys_reg(vcpu, CNTV_CTL_EL0), SYS_CNTV_CTL);
}
static void sync_hyp_timer_state(struct pkvm_hyp_vcpu *hyp_vcpu)
{
struct kvm_vcpu *vcpu = &hyp_vcpu->vcpu;
if (!pkvm_hyp_vcpu_is_protected(hyp_vcpu))
return;
/*
* Preserve the vtimer state so that it is always correct,
* even if the host tries to make a mess.
*/
__vcpu_assign_sys_reg(vcpu, CNTV_CVAL_EL0, read_sysreg_el0(SYS_CNTV_CVAL));
__vcpu_assign_sys_reg(vcpu, CNTV_CTL_EL0, read_sysreg_el0(SYS_CNTV_CTL));
}
static void __copy_vcpu_state(const struct kvm_vcpu *from_vcpu,
struct kvm_vcpu *to_vcpu)
{
@ -231,6 +632,9 @@ static void flush_debug_state(struct pkvm_hyp_vcpu *hyp_vcpu)
{
struct kvm_vcpu *host_vcpu = hyp_vcpu->host_vcpu;
if (pkvm_hyp_vcpu_is_protected(hyp_vcpu))
return;
hyp_vcpu->vcpu.arch.debug_owner = host_vcpu->arch.debug_owner;
if (kvm_guest_owns_debug_regs(&hyp_vcpu->vcpu)) {
@ -249,6 +653,9 @@ static void sync_debug_state(struct pkvm_hyp_vcpu *hyp_vcpu)
{
struct kvm_vcpu *host_vcpu = hyp_vcpu->host_vcpu;
if (pkvm_hyp_vcpu_is_protected(hyp_vcpu))
return;
if (kvm_guest_owns_debug_regs(&hyp_vcpu->vcpu))
host_vcpu->arch.vcpu_debug_state = hyp_vcpu->vcpu.arch.vcpu_debug_state;
else if (kvm_host_owns_debug_regs(&hyp_vcpu->vcpu))
@ -258,6 +665,8 @@ static void sync_debug_state(struct pkvm_hyp_vcpu *hyp_vcpu)
static void flush_hyp_vcpu(struct pkvm_hyp_vcpu *hyp_vcpu)
{
struct kvm_vcpu *host_vcpu = hyp_vcpu->host_vcpu;
hyp_entry_exit_handler_fn ec_handler;
u8 esr_ec;
fpsimd_sve_flush();
flush_debug_state(hyp_vcpu);
@ -270,42 +679,78 @@ static void flush_hyp_vcpu(struct pkvm_hyp_vcpu *hyp_vcpu)
if (!pkvm_hyp_vcpu_is_protected(hyp_vcpu)) {
if (vcpu_get_flag(host_vcpu, PKVM_HOST_STATE_DIRTY))
flush_hyp_vcpu_state(hyp_vcpu);
} else {
hyp_vcpu->vcpu.arch.ctxt = host_vcpu->arch.ctxt;
hyp_vcpu->vcpu.arch.hcr_el2 &= ~(HCR_TWI | HCR_TWE);
hyp_vcpu->vcpu.arch.hcr_el2 |= READ_ONCE(host_vcpu->arch.hcr_el2) &
(HCR_TWI | HCR_TWE);
hyp_vcpu->vcpu.arch.mdcr_el2 = host_vcpu->arch.mdcr_el2;
hyp_vcpu->vcpu.arch.iflags = host_vcpu->arch.iflags;
}
/* __hyp_running_vcpu must be NULL in a guest context. */
hyp_vcpu->vcpu.arch.ctxt.__hyp_running_vcpu = NULL;
hyp_vcpu->vcpu.arch.mdcr_el2 = host_vcpu->arch.mdcr_el2;
/*
* HCR_EL2.VSE is host-owned (a pending virtual SError to inject), not a
* trap-control bit, so it must flow to the hyp vCPU alongside TWI/TWE
* for the vSError to be delivered. sync_hyp_vcpu() reflects it back.
* A host-injected vSError is masked by the guest's own PSTATE.A, so it
* applies to protected guests too.
*/
hyp_vcpu->vcpu.arch.hcr_el2 &= ~(HCR_TWI | HCR_TWE | HCR_VSE);
hyp_vcpu->vcpu.arch.hcr_el2 |= READ_ONCE(host_vcpu->arch.hcr_el2) &
(HCR_TWI | HCR_TWE | HCR_VSE);
hyp_vcpu->vcpu.arch.iflags = host_vcpu->arch.iflags;
hyp_vcpu->vcpu.arch.vsesr_el2 = host_vcpu->arch.vsesr_el2;
hyp_vcpu->vcpu.arch.hcr_el2 &= ~HCR_VSE;
hyp_vcpu->vcpu.arch.hcr_el2 |= READ_ONCE(host_vcpu->arch.hcr_el2) & HCR_VSE;
hyp_vcpu->vcpu.arch.vsesr_el2 = host_vcpu->arch.vsesr_el2;
flush_hyp_vgic_state(hyp_vcpu);
flush_hyp_timer_state(hyp_vcpu);
hyp_vcpu->vcpu.arch.pid = host_vcpu->arch.pid;
switch (ARM_EXCEPTION_CODE(hyp_vcpu->exit_code)) {
case ARM_EXCEPTION_IRQ:
case ARM_EXCEPTION_EL1_SERROR:
case ARM_EXCEPTION_IL:
break;
case ARM_EXCEPTION_TRAP:
/* Nothing was marshalled for this trap, see sync_hyp_vcpu(). */
if (ARM_SERROR_PENDING(hyp_vcpu->exit_code))
break;
if (pkvm_hyp_vcpu_is_protected(hyp_vcpu)) {
esr_ec = ESR_ELx_EC(kvm_vcpu_get_esr(&hyp_vcpu->vcpu));
ec_handler = entry_hyp_pvm_handlers[esr_ec];
if (ec_handler)
ec_handler(hyp_vcpu);
}
break;
default:
BUG();
}
hyp_vcpu->exit_code = 0;
}
static void sync_hyp_vcpu(struct pkvm_hyp_vcpu *hyp_vcpu)
static void sync_hyp_vcpu(struct pkvm_hyp_vcpu *hyp_vcpu, u32 exit_reason)
{
struct kvm_vcpu *host_vcpu = hyp_vcpu->host_vcpu;
hyp_entry_exit_handler_fn ec_handler;
u8 esr_ec;
fpsimd_sve_sync(&hyp_vcpu->vcpu);
sync_debug_state(hyp_vcpu);
if (pkvm_hyp_vcpu_is_protected(hyp_vcpu)) {
host_vcpu->arch.ctxt = hyp_vcpu->vcpu.arch.ctxt;
/*
* Protected: the host sees ESR_EL2 as EL2 took it, register
* index withheld; the fault addresses stay withheld unless the
* EC handler below adds them.
*/
host_vcpu->arch.fault = (struct kvm_vcpu_fault_info) {
.esr_el2 = pvm_host_esr(hyp_vcpu->vcpu.arch.fault.esr_el2),
.disr_el1 = hyp_vcpu->vcpu.arch.fault.disr_el1,
};
} else {
/* Non-protected: the host gets the full fault. */
host_vcpu->arch.fault = hyp_vcpu->vcpu.arch.fault;
host_vcpu->arch.iflags = hyp_vcpu->vcpu.arch.iflags;
/*
* PC feeds trace_kvm_exit(), PSTATE.SS the host software-step
* machine, and both run before the next on-demand ctxt sync.
@ -314,11 +759,40 @@ static void sync_hyp_vcpu(struct pkvm_hyp_vcpu *hyp_vcpu)
host_vcpu->arch.ctxt.regs.pstate = hyp_vcpu->vcpu.arch.ctxt.regs.pstate;
}
host_vcpu->arch.fault = hyp_vcpu->vcpu.arch.fault;
switch (ARM_EXCEPTION_CODE(exit_reason)) {
case ARM_EXCEPTION_IRQ:
break;
case ARM_EXCEPTION_TRAP:
/* SError pending: not handled at EL2, the guest replays it. */
if (ARM_SERROR_PENDING(exit_reason))
break;
host_vcpu->arch.iflags = hyp_vcpu->vcpu.arch.iflags;
/* Per-EC marshalling is for protected guests only. */
if (pkvm_hyp_vcpu_is_protected(hyp_vcpu)) {
esr_ec = ESR_ELx_EC(kvm_vcpu_get_esr(&hyp_vcpu->vcpu));
ec_handler = exit_hyp_pvm_handlers[esr_ec];
if (ec_handler)
ec_handler(hyp_vcpu);
}
break;
case ARM_EXCEPTION_EL1_SERROR:
case ARM_EXCEPTION_IL:
break;
default:
BUG();
}
/* Cleared by hardware once the guest takes the vSError. */
host_vcpu->arch.hcr_el2 &= ~HCR_VSE;
host_vcpu->arch.hcr_el2 |= hyp_vcpu->vcpu.arch.hcr_el2 & HCR_VSE;
sync_hyp_vgic_state(hyp_vcpu);
sync_hyp_timer_state(hyp_vcpu);
if (pkvm_hyp_vcpu_is_protected(hyp_vcpu))
vcpu_clear_flag(host_vcpu, PC_UPDATE_REQ);
hyp_vcpu->exit_code = exit_reason;
}
DEFINE_KVM_HOST_HCALL(void, __pkvm_vcpu_load,
@ -331,9 +805,17 @@ DEFINE_KVM_HOST_HCALL(void, __pkvm_vcpu_load,
return;
if (pkvm_hyp_vcpu_is_protected(hyp_vcpu)) {
u64 dfr0 = read_sysreg(id_aa64dfr0_el1);
/* Propagate WFx trapping flags */
hyp_vcpu->vcpu.arch.hcr_el2 &= ~(HCR_TWE | HCR_TWI);
hyp_vcpu->vcpu.arch.hcr_el2 |= hcr_el2 & (HCR_TWE | HCR_TWI);
/* HPMN == 0 is reserved without FEAT_HPMN0. */
if (pmuv3_implemented(SYS_FIELD_GET(ID_AA64DFR0_EL1, PMUVer, dfr0)))
u64p_replace_bits(&hyp_vcpu->vcpu.arch.mdcr_el2,
FIELD_GET(ARMV8_PMU_PMCR_N, read_sysreg(pmcr_el0)),
MDCR_EL2_HPMN);
} else {
memcpy(&hyp_vcpu->vcpu.arch.fgt, hyp_vcpu->host_vcpu->arch.fgt,
sizeof(hyp_vcpu->vcpu.arch.fgt));
@ -396,16 +878,15 @@ __get_host_hyp_vcpus_from_vgic_v3_cpu_if(struct vgic_v3_cpu_if __kern *cpu_if,
return __get_host_hyp_vcpus(host_vcpu, hyp_vcpup);
}
DEFINE_KVM_HOST_HCALL(int, __kvm_vcpu_run,
struct kvm_vcpu __kern *, vcpu)
DEFINE_KVM_HOST_HCALL(int, __kvm_vcpu_run, struct kvm_vcpu __kern *, vcpu)
{
struct pkvm_hyp_vcpu *hyp_vcpu;
struct kvm_vcpu *host_vcpu;
int ret;
int ret = ARM_EXCEPTION_IL;
host_vcpu = __get_host_hyp_vcpus(kern_hyp_va_host(vcpu), &hyp_vcpu);
if (!host_vcpu)
return -EINVAL;
return ret;
if (unlikely(hyp_vcpu)) {
/*
@ -415,13 +896,29 @@ DEFINE_KVM_HOST_HCALL(int, __kvm_vcpu_run,
* is misbehaving.
*/
if (unlikely(system_supports_sme() && read_sysreg_s(SYS_SVCR)))
return -EINVAL;
return ret;
/*
* ON has a single writer, pkvm_reset_vcpu() on this CPU, so
* READ_ONCE suffices. ON_PENDING takes the reset; -ECANCELED
* is a rollback that raced it.
*/
switch (READ_ONCE(hyp_vcpu->power_state)) {
case PSCI_0_2_AFFINITY_LEVEL_ON:
break;
case PSCI_0_2_AFFINITY_LEVEL_ON_PENDING:
if (pkvm_reset_vcpu(hyp_vcpu))
return ret;
break;
default:
return ret;
}
flush_hyp_vcpu(hyp_vcpu);
ret = __kvm_vcpu_run(&hyp_vcpu->vcpu);
sync_hyp_vcpu(hyp_vcpu);
sync_hyp_vcpu(hyp_vcpu, ret);
} else {
/* The host is fully trusted, run its vCPU directly. */
fpsimd_lazy_switch_to_guest(host_vcpu);
@ -547,10 +1044,62 @@ DEFINE_KVM_HOST_HCALL(int, __pkvm_host_mkyoung_guest,
return __pkvm_host_mkyoung_guest(gfn, hyp_vcpu);
}
/*
* PKVM_HOST_STATE_DIRTY names the authoritative copy, the host's when set.
* A loaded protected vCPU takes the request at its next entry instead.
*/
struct kvm_vcpu *kvm_adjust_pc_get(struct kvm_vcpu *vcpu)
{
struct pkvm_hyp_vcpu *hyp_vcpu;
if (!is_protected_kvm_enabled())
return vcpu;
hyp_vcpu = pkvm_get_loaded_hyp_vcpu();
if (!hyp_vcpu || vcpu == &hyp_vcpu->vcpu)
return vcpu;
if (pkvm_hyp_vcpu_is_protected(hyp_vcpu))
return NULL;
if (vcpu_get_flag(vcpu, PKVM_HOST_STATE_DIRTY))
return vcpu;
vcpu_copy_flag(&hyp_vcpu->vcpu, vcpu, PC_UPDATE_REQ);
return &hyp_vcpu->vcpu;
}
/* Reflect the consumed request back, otherwise it stays pending. */
void kvm_adjust_pc_put(struct kvm_vcpu *vcpu, struct kvm_vcpu *target)
{
if (target != vcpu)
vcpu_copy_flag(vcpu, target, PC_UPDATE_REQ);
}
DEFINE_KVM_HOST_HCALL(void, __kvm_adjust_pc,
struct kvm_vcpu __kern *, vcpu)
{
__kvm_adjust_pc(kern_hyp_va_host(vcpu));
struct pkvm_hyp_vcpu *hyp_vcpu;
struct kvm_vcpu *host_vcpu;
host_vcpu = __get_host_hyp_vcpus(kern_hyp_va_host(vcpu), &hyp_vcpu);
if (host_vcpu) {
__kvm_adjust_pc(host_vcpu);
return;
}
/*
* With no hyp vCPU loaded for it, the host vCPU may be unpinned,
* and so unmapped at EL2: its first run pins it. A pin fails only
* for memory the host isn't sharing, a bad pointer, so the request
* is dropped.
*/
host_vcpu = kern_hyp_va_host(vcpu);
if (hyp_pin_shared_mem(host_vcpu, host_vcpu + 1))
return;
__kvm_adjust_pc(host_vcpu);
hyp_unpin_shared_mem(host_vcpu, host_vcpu + 1);
}
DEFINE_KVM_HOST_HCALL0(void, __kvm_flush_vm_context)

418
arch/arm64/kvm/hyp/nvhe/pkvm.c
View File

@ -5,17 +5,24 @@
*/
#include <kvm/arm_hypercalls.h>
#include <kvm/arm_psci.h>
#include <linux/kvm_host.h>
#include <linux/mm.h>
#include <asm/kvm_emulate.h>
#include <asm/spectre.h>
#include <nvhe/mem_protect.h>
#include <nvhe/memory.h>
#include <nvhe/pkvm.h>
#include <nvhe/trap_handler.h>
/* The host's Spectre mitigation state, for SMCCC_ARCH_FEATURES. */
enum mitigation_state spectre_v2_state;
enum mitigation_state spectre_v4_state;
enum mitigation_state spectre_bhb_state;
/* Used by icache_is_aliasing(). */
unsigned long __icache_flags;
@ -102,7 +109,7 @@ static void pvm_init_traps_mdcr(struct kvm_vcpu *vcpu)
if (!kvm_has_feat(kvm, ID_AA64DFR0_EL1, PMUVer, IMP)) {
val |= MDCR_EL2_TPM | MDCR_EL2_TPMCR;
val &= ~(MDCR_EL2_HPME | MDCR_EL2_MTPME | MDCR_EL2_HPMN_MASK);
val &= ~(MDCR_EL2_HPME | MDCR_EL2_MTPME);
}
if (!kvm_has_feat(kvm, ID_AA64DFR0_EL1, DebugVer, IMP))
@ -304,6 +311,43 @@ struct pkvm_hyp_vcpu *pkvm_get_loaded_hyp_vcpu(void)
}
static struct pkvm_hyp_vm *pkvm_get_loaded_hyp_vm(struct kvm_vcpu *vcpu)
{
struct pkvm_hyp_vcpu *hyp_vcpu = pkvm_get_loaded_hyp_vcpu();
if (hyp_vcpu &&
(vcpu == &hyp_vcpu->vcpu || vcpu == hyp_vcpu->host_vcpu))
return pkvm_hyp_vcpu_to_hyp_vm(hyp_vcpu);
return NULL;
}
/*
* A loaded vCPU's VM is the hyp VM. An unloaded host vCPU's is mapped at
* EL2 only while pinned, so it's read once and pinned; the pin fails for
* memory the host isn't sharing.
*/
struct kvm *pkvm_vcpu_get_kvm(struct kvm_vcpu *vcpu)
{
struct pkvm_hyp_vm *hyp_vm = pkvm_get_loaded_hyp_vm(vcpu);
struct kvm *kvm;
if (hyp_vm)
return &hyp_vm->kvm;
kvm = kern_hyp_va(READ_ONCE(vcpu->kvm));
if (hyp_pin_shared_mem(kvm, kvm + 1))
return NULL;
return kvm;
}
void pkvm_vcpu_put_kvm(struct kvm_vcpu *vcpu, struct kvm *kvm)
{
if (kvm && !pkvm_get_loaded_hyp_vm(vcpu))
hyp_unpin_shared_mem(kvm, kvm + 1);
}
struct pkvm_hyp_vm *get_pkvm_hyp_vm(pkvm_handle_t handle)
{
struct pkvm_hyp_vm *hyp_vm;
@ -366,21 +410,7 @@ static void pkvm_init_features_from_host(struct pkvm_hyp_vm *hyp_vm, const struc
if (kvm_pkvm_ext_allowed(kvm, KVM_CAP_ARM_MTE))
kvm->arch.flags |= host_arch_flags & BIT(KVM_ARCH_FLAG_MTE_ENABLED);
bitmap_zero(allowed_features, KVM_VCPU_MAX_FEATURES);
set_bit(KVM_ARM_VCPU_PSCI_0_2, allowed_features);
if (kvm_pkvm_ext_allowed(kvm, KVM_CAP_ARM_PMU_V3))
set_bit(KVM_ARM_VCPU_PMU_V3, allowed_features);
if (kvm_pkvm_ext_allowed(kvm, KVM_CAP_ARM_PTRAUTH_ADDRESS))
set_bit(KVM_ARM_VCPU_PTRAUTH_ADDRESS, allowed_features);
if (kvm_pkvm_ext_allowed(kvm, KVM_CAP_ARM_PTRAUTH_GENERIC))
set_bit(KVM_ARM_VCPU_PTRAUTH_GENERIC, allowed_features);
if (kvm_pkvm_ext_allowed(kvm, KVM_CAP_ARM_SVE))
set_bit(KVM_ARM_VCPU_SVE, allowed_features);
kvm_pkvm_vcpu_allowed_features(kvm, allowed_features);
bitmap_and(kvm->arch.vcpu_features, host_kvm->arch.vcpu_features,
allowed_features, KVM_VCPU_MAX_FEATURES);
@ -389,6 +419,40 @@ static void pkvm_init_features_from_host(struct pkvm_hyp_vm *hyp_vm, const struc
kvm_vcpu_has_feature(kvm, KVM_ARM_VCPU_SVE));
}
static int pkvm_vcpu_init_psci(struct pkvm_hyp_vcpu *hyp_vcpu, u32 mp_state)
{
struct vcpu_reset_state *reset_state = &hyp_vcpu->vcpu.arch.reset_state;
struct pkvm_hyp_vm *hyp_vm = pkvm_hyp_vcpu_to_hyp_vm(hyp_vcpu);
struct kvm_vcpu *host_vcpu;
if (!pkvm_hyp_vcpu_is_protected(hyp_vcpu)) {
/* The host manages a non-protected vCPU: always ON at EL2. */
hyp_vcpu->power_state = PSCI_0_2_AFFINITY_LEVEL_ON;
return 0;
}
if (mp_state != KVM_MP_STATE_RUNNABLE && mp_state != KVM_MP_STATE_STOPPED)
return -EINVAL;
if (mp_state == KVM_MP_STATE_STOPPED) {
reset_state->reset = false;
hyp_vcpu->power_state = PSCI_0_2_AFFINITY_LEVEL_OFF;
return 0;
}
hyp_assert_lock_held(&vm_table_lock);
if (hyp_vm->primary_vcpu)
return -EINVAL;
hyp_vm->primary_vcpu = hyp_vcpu;
host_vcpu = hyp_vcpu->host_vcpu;
reset_state->pc = READ_ONCE(*vcpu_pc(host_vcpu));
reset_state->r0 = READ_ONCE(vcpu_gp_regs(host_vcpu)[0]);
reset_state->reset = true;
hyp_vcpu->power_state = PSCI_0_2_AFFINITY_LEVEL_ON_PENDING;
return 0;
}
static void unpin_host_vcpu(struct kvm_vcpu *host_vcpu)
{
if (host_vcpu)
@ -511,10 +575,12 @@ static int init_pkvm_hyp_vcpu(struct pkvm_hyp_vcpu *hyp_vcpu,
struct kvm_vcpu *host_vcpu)
{
int ret = 0;
u32 mp_state;
if (hyp_pin_shared_mem(host_vcpu, host_vcpu + 1))
return -EBUSY;
mp_state = READ_ONCE(host_vcpu->arch.mp_state.mp_state);
hyp_vcpu->host_vcpu = host_vcpu;
hyp_vcpu->vcpu.kvm = &hyp_vm->kvm;
@ -523,7 +589,6 @@ static int init_pkvm_hyp_vcpu(struct pkvm_hyp_vcpu *hyp_vcpu,
hyp_vcpu->vcpu.arch.hw_mmu = &hyp_vm->kvm.arch.mmu;
hyp_vcpu->vcpu.arch.cflags = READ_ONCE(host_vcpu->arch.cflags);
hyp_vcpu->vcpu.arch.mp_state.mp_state = KVM_MP_STATE_STOPPED;
if (!pkvm_hyp_vcpu_is_protected(hyp_vcpu)) {
/*
@ -548,9 +613,17 @@ static int init_pkvm_hyp_vcpu(struct pkvm_hyp_vcpu *hyp_vcpu,
goto done;
ret = pkvm_vcpu_init_sve(hyp_vcpu, host_vcpu);
done:
if (ret)
goto done;
if (pkvm_hyp_vcpu_is_protected(hyp_vcpu))
kvm_reset_pvm_sys_regs(&hyp_vcpu->vcpu);
ret = pkvm_vcpu_init_psci(hyp_vcpu, mp_state);
done:
if (ret) {
unpin_host_vcpu(host_vcpu);
unpin_host_sve_state(hyp_vcpu);
}
return ret;
}
@ -927,13 +1000,22 @@ int __pkvm_init_vcpu(pkvm_handle_t handle, struct kvm_vcpu *host_vcpu,
ret = init_pkvm_hyp_vcpu(hyp_vcpu, hyp_vm, host_vcpu);
if (ret)
goto unlock;
goto unclaim;
ret = register_hyp_vcpu(hyp_vm, hyp_vcpu);
if (ret) {
unpin_host_vcpu(host_vcpu);
unpin_host_sve_state(hyp_vcpu);
goto unclaim;
}
goto unlock;
unclaim:
/*
* Under vm_table_lock, so no other claim can have landed: undo
* this one.
*/
if (hyp_vm->primary_vcpu == hyp_vcpu)
hyp_vm->primary_vcpu = NULL;
unlock:
hyp_spin_unlock(&vm_table_lock);
@ -1125,6 +1207,238 @@ static void pkvm_memunshare_call(u64 *ret, struct kvm_vcpu *vcpu)
ret[0] = SMCCC_RET_SUCCESS;
}
/*
* Reset the vCPU to its power-on state and commit ON_PENDING -> ON, on the
* target's own CPU. Returns -ECANCELED, with no side effects, if a rollback
* raced the reset.
*
* The requestor's PSCI CPU_ON, in pvm_psci_vcpu_on():
*
* cmpxchg_relaxed(power_state): OFF -> ON_PENDING
* <ctrl>
* write_reset_state();
* smp_store_release(reset_state->reset, true);
*
* The target coming online here, running, then its PSCI CPU_OFF in
* pvm_psci_vcpu_off():
*
* smp_load_acquire(reset_state->reset) == true;
* cmpxchg_relaxed(power_state): ON_PENDING -> ON;
* read_reset_state();
*
* <run guest>
*
* WRITE_ONCE(reset_state->reset, false);
* smp_store_release(power_state, OFF);
*/
int pkvm_reset_vcpu(struct pkvm_hyp_vcpu *hyp_vcpu)
{
struct vcpu_reset_state *reset_state = &hyp_vcpu->vcpu.arch.reset_state;
/*
* Pairs with smp_store_release(&reset_state->reset, true) in
* pvm_psci_vcpu_on(). The acquire must precede the cmpxchg: reversed, a
* winning cmpxchg with a false acquire would leave power_state == ON
* with the reset skipped.
*/
if (!smp_load_acquire(&reset_state->reset))
return -ECANCELED;
if (cmpxchg_relaxed(&hyp_vcpu->power_state, PSCI_0_2_AFFINITY_LEVEL_ON_PENDING,
PSCI_0_2_AFFINITY_LEVEL_ON) != PSCI_0_2_AFFINITY_LEVEL_ON_PENDING)
return -ECANCELED;
kvm_reset_vcpu_core(&hyp_vcpu->vcpu);
kvm_reset_pvm_sys_regs(&hyp_vcpu->vcpu);
/* Must be done after resetting sys registers. */
kvm_reset_vcpu_psci(&hyp_vcpu->vcpu, reset_state);
hyp_vcpu->exit_code = 0;
return 0;
}
struct pkvm_hyp_vcpu *pkvm_mpidr_to_hyp_vcpu(struct pkvm_hyp_vm *hyp_vm, unsigned long mpidr)
{
struct pkvm_hyp_vcpu *hyp_vcpu;
int i;
mpidr &= MPIDR_HWID_BITMASK;
for (i = 0; i < hyp_vm->kvm.created_vcpus; i++) {
/* Pairs with smp_store_release() in register_hyp_vcpu(). */
hyp_vcpu = smp_load_acquire(&hyp_vm->vcpus[i]);
if (!hyp_vcpu)
continue;
if (mpidr == kvm_vcpu_get_mpidr_aff(&hyp_vcpu->vcpu))
return hyp_vcpu;
}
return NULL;
}
/*
* Returns true when handled at EL2, false when the host must wake the target
* vCPU.
*/
static bool pvm_psci_vcpu_on(struct pkvm_hyp_vcpu *hyp_vcpu)
{
struct pkvm_hyp_vm *hyp_vm = pkvm_hyp_vcpu_to_hyp_vm(hyp_vcpu);
struct vcpu_reset_state *reset_state;
struct pkvm_hyp_vcpu *target;
unsigned long cpu_id, ret;
int power_state;
cpu_id = smccc_get_arg1(&hyp_vcpu->vcpu);
if (!kvm_psci_valid_affinity(&hyp_vcpu->vcpu, cpu_id)) {
ret = PSCI_RET_INVALID_PARAMS;
goto error;
}
target = pkvm_mpidr_to_hyp_vcpu(hyp_vm, cpu_id);
if (!target) {
ret = PSCI_RET_INVALID_PARAMS;
goto error;
}
/*
* vCPUs race to power on the same target. The writes below depend on
* the cmpxchg reading OFF, and that control dependency with the release
* of OFF in pvm_psci_vcpu_off() orders them after the target's accesses
* to reset_state, so relaxed suffices.
*/
power_state = cmpxchg_relaxed(&target->power_state,
PSCI_0_2_AFFINITY_LEVEL_OFF,
PSCI_0_2_AFFINITY_LEVEL_ON_PENDING);
switch (power_state) {
case PSCI_0_2_AFFINITY_LEVEL_ON_PENDING:
ret = PSCI_RET_ON_PENDING;
goto error;
case PSCI_0_2_AFFINITY_LEVEL_ON:
ret = PSCI_RET_ALREADY_ON;
goto error;
case PSCI_0_2_AFFINITY_LEVEL_OFF:
break;
default:
ret = PSCI_RET_INTERNAL_FAILURE;
goto error;
}
reset_state = &target->vcpu.arch.reset_state;
WRITE_ONCE(reset_state->pc, smccc_get_arg2(&hyp_vcpu->vcpu));
WRITE_ONCE(reset_state->r0, smccc_get_arg3(&hyp_vcpu->vcpu));
WRITE_ONCE(reset_state->be, kvm_vcpu_is_be(&hyp_vcpu->vcpu));
/*
* Publish reset_state.{pc, r0, be} to the target vCPU. Pairs with
* smp_load_acquire(&reset_state->reset) in pkvm_reset_vcpu().
*/
smp_store_release(&reset_state->reset, true);
/* The host requests KVM_REQ_VCPU_RESET and wakes the target. */
return false;
error:
smccc_set_retval(&hyp_vcpu->vcpu, ret, 0, 0, 0);
return true;
}
/*
* Returns true when handled at EL2, false when the host must stop scheduling
* the vCPU.
*/
static bool pvm_psci_vcpu_off(struct pkvm_hyp_vcpu *hyp_vcpu)
{
/* No other writer runs while this vCPU is ON and executing. */
WARN_ON(READ_ONCE(hyp_vcpu->power_state) != PSCI_0_2_AFFINITY_LEVEL_ON);
WRITE_ONCE(hyp_vcpu->vcpu.arch.reset_state.reset, false);
/*
* Orders pkvm_reset_vcpu()'s accesses to reset_state, and the clear
* above, before OFF. Pairs with the control dependency in
* pvm_psci_vcpu_on().
*/
smp_store_release(&hyp_vcpu->power_state, PSCI_0_2_AFFINITY_LEVEL_OFF);
/* Return to the host so that it can finish powering off the vcpu. */
return false;
}
static bool pvm_psci_version(struct pkvm_hyp_vcpu *hyp_vcpu)
{
/* Nothing to be handled by the host. Go back to the guest. */
smccc_set_retval(&hyp_vcpu->vcpu, KVM_ARM_PSCI_1_1, 0, 0, 0);
return true;
}
static bool pvm_psci_features(struct pkvm_hyp_vcpu *hyp_vcpu)
{
struct kvm_vcpu *vcpu = &hyp_vcpu->vcpu;
u32 feature = smccc_get_arg1(vcpu);
unsigned long val;
switch (feature) {
case PSCI_0_2_FN_PSCI_VERSION:
case PSCI_0_2_FN_CPU_SUSPEND:
case PSCI_0_2_FN64_CPU_SUSPEND:
case PSCI_0_2_FN_CPU_OFF:
case PSCI_0_2_FN_CPU_ON:
case PSCI_0_2_FN64_CPU_ON:
case PSCI_0_2_FN_AFFINITY_INFO:
case PSCI_0_2_FN64_AFFINITY_INFO:
case PSCI_0_2_FN_SYSTEM_OFF:
case PSCI_0_2_FN_SYSTEM_RESET:
case PSCI_1_0_FN_PSCI_FEATURES:
case PSCI_1_1_FN_SYSTEM_RESET2:
case PSCI_1_1_FN64_SYSTEM_RESET2:
case ARM_SMCCC_VERSION_FUNC_ID:
val = PSCI_RET_SUCCESS;
break;
default:
val = PSCI_RET_NOT_SUPPORTED;
break;
}
/* Nothing to be handled by the host. Go back to the guest. */
smccc_set_retval(vcpu, val, 0, 0, 0);
return true;
}
static bool pkvm_handle_psci(struct pkvm_hyp_vcpu *hyp_vcpu)
{
struct kvm_vcpu *vcpu = &hyp_vcpu->vcpu;
u32 psci_fn = smccc_get_function(vcpu);
switch (psci_fn) {
case PSCI_0_2_FN_CPU_ON:
kvm_psci_narrow_to_32bit(vcpu);
fallthrough;
case PSCI_0_2_FN64_CPU_ON:
return pvm_psci_vcpu_on(hyp_vcpu);
case PSCI_0_2_FN_CPU_OFF:
return pvm_psci_vcpu_off(hyp_vcpu);
case PSCI_0_2_FN_PSCI_VERSION:
return pvm_psci_version(hyp_vcpu);
case PSCI_1_0_FN_PSCI_FEATURES:
return pvm_psci_features(hyp_vcpu);
case PSCI_0_2_FN_AFFINITY_INFO:
case PSCI_0_2_FN64_AFFINITY_INFO:
case PSCI_0_2_FN_SYSTEM_RESET:
case PSCI_0_2_FN_CPU_SUSPEND:
case PSCI_0_2_FN64_CPU_SUSPEND:
case PSCI_0_2_FN_SYSTEM_OFF:
case PSCI_1_1_FN_SYSTEM_RESET2:
case PSCI_1_1_FN64_SYSTEM_RESET2:
return false; /* Handled by the host. */
default:
/* Unknown PSCI calls are handled here, not forwarded. */
smccc_set_retval(vcpu, PSCI_RET_NOT_SUPPORTED, 0, 0, 0);
return true;
}
}
/*
* Handler for protected VM HVC calls.
*
@ -1134,10 +1448,73 @@ static void pkvm_memunshare_call(u64 *ret, struct kvm_vcpu *vcpu)
*/
bool kvm_handle_pvm_hvc64(struct kvm_vcpu *vcpu, u64 *exit_code)
{
struct pkvm_hyp_vcpu *hyp_vcpu = container_of(vcpu, struct pkvm_hyp_vcpu, vcpu);
u64 val[4] = { SMCCC_RET_INVALID_PARAMETER };
bool handled = true;
u32 feature;
uuid_t uuid;
switch (smccc_get_function(vcpu)) {
case ARM_SMCCC_VERSION_FUNC_ID:
/* Nothing to be handled by the host. Go back to the guest. */
val[0] = ARM_SMCCC_VERSION_1_1;
break;
case ARM_SMCCC_ARCH_FEATURES_FUNC_ID:
/* The workaround calls themselves are handled on hyp entry. */
val[0] = SMCCC_RET_NOT_SUPPORTED;
feature = smccc_get_arg1(vcpu);
switch (feature) {
case ARM_SMCCC_VERSION_FUNC_ID:
case ARM_SMCCC_ARCH_FEATURES_FUNC_ID:
val[0] = SMCCC_RET_SUCCESS;
break;
case ARM_SMCCC_ARCH_WORKAROUND_1:
switch (spectre_v2_state) {
case SPECTRE_VULNERABLE:
break;
case SPECTRE_MITIGATED:
val[0] = SMCCC_RET_SUCCESS;
break;
case SPECTRE_UNAFFECTED:
val[0] = SMCCC_ARCH_WORKAROUND_RET_UNAFFECTED;
break;
}
break;
case ARM_SMCCC_ARCH_WORKAROUND_2:
switch (spectre_v4_state) {
case SPECTRE_VULNERABLE:
break;
case SPECTRE_MITIGATED:
/* With SSBS advertised the guest's default is safe. */
if (kvm_has_feat(vcpu->kvm, ID_AA64PFR1_EL1, SSBS, IMP))
break;
fallthrough;
case SPECTRE_UNAFFECTED:
val[0] = SMCCC_RET_NOT_REQUIRED;
break;
}
break;
case ARM_SMCCC_ARCH_WORKAROUND_3:
switch (spectre_bhb_state) {
case SPECTRE_VULNERABLE:
break;
case SPECTRE_MITIGATED:
val[0] = SMCCC_RET_SUCCESS;
break;
case SPECTRE_UNAFFECTED:
val[0] = SMCCC_ARCH_WORKAROUND_RET_UNAFFECTED;
break;
}
break;
}
break;
case ARM_SMCCC_VENDOR_HYP_CALL_UID_FUNC_ID:
uuid = ARM_SMCCC_VENDOR_HYP_UID_KVM;
val[0] = smccc_uuid_to_reg(&uuid, 0);
val[1] = smccc_uuid_to_reg(&uuid, 1);
val[2] = smccc_uuid_to_reg(&uuid, 2);
val[3] = smccc_uuid_to_reg(&uuid, 3);
break;
case ARM_SMCCC_VENDOR_HYP_KVM_FEATURES_FUNC_ID:
val[0] = BIT(ARM_SMCCC_KVM_FUNC_FEATURES);
val[0] |= BIT(ARM_SMCCC_KVM_FUNC_HYP_MEMINFO);
@ -1170,8 +1547,7 @@ bool kvm_handle_pvm_hvc64(struct kvm_vcpu *vcpu, u64 *exit_code)
pkvm_memunshare_call(val, vcpu);
break;
default:
/* Punt everything else back to the host, for now. */
handled = false;
return pkvm_handle_psci(hyp_vcpu);
}
if (handled)

29
arch/arm64/kvm/hyp/nvhe/switch.c
View File

@ -203,15 +203,36 @@ static const exit_handler_fn hyp_exit_handlers[] = {
[ESR_ELx_EC_MOPS] = kvm_hyp_handle_mops,
};
/* WFI/WFE exit to the host, which emulates them. WFxT is not advertised. */
static bool kvm_handle_pvm_wfx(struct kvm_vcpu *vcpu, u64 *exit_code)
{
if (kvm_vcpu_get_esr(vcpu) & ESR_ELx_WFx_ISS_WFxT)
return kvm_handle_pvm_restricted(vcpu, exit_code);
return false;
}
/* Lazy FP/SIMD switch, or the UNDEF that would otherwise be left to the host. */
static bool kvm_handle_pvm_fpsimd(struct kvm_vcpu *vcpu, u64 *exit_code)
{
if (kvm_hyp_handle_fpsimd(vcpu, exit_code))
return true;
return kvm_handle_pvm_restricted(vcpu, exit_code);
}
/*
* A class not listed takes an UNDEF at EL2: the host has no way to
* inject one into a protected vCPU.
*/
static const exit_handler_fn pvm_exit_handlers[] = {
[0 ... ESR_ELx_EC_MAX] = NULL,
[0 ... ESR_ELx_EC_MAX] = kvm_handle_pvm_restricted,
[ESR_ELx_EC_WFx] = kvm_handle_pvm_wfx,
[ESR_ELx_EC_HVC64] = kvm_handle_pvm_hvc64,
[ESR_ELx_EC_SYS64] = kvm_handle_pvm_sys64,
[ESR_ELx_EC_SVE] = kvm_handle_pvm_restricted,
[ESR_ELx_EC_FP_ASIMD] = kvm_hyp_handle_fpsimd,
[ESR_ELx_EC_FP_ASIMD] = kvm_handle_pvm_fpsimd,
[ESR_ELx_EC_IABT_LOW] = kvm_hyp_handle_iabt_low,
[ESR_ELx_EC_DABT_LOW] = kvm_hyp_handle_dabt_low,
[ESR_ELx_EC_WATCHPT_LOW] = kvm_hyp_handle_watchpt_low,
[ESR_ELx_EC_MOPS] = kvm_hyp_handle_mops,
};

60
arch/arm64/kvm/hyp/nvhe/sys_regs.c
View File

@ -528,6 +528,66 @@ static const struct sys_reg_desc pvm_sys_reg_descs[] = {
/* Performance Monitoring Registers are restricted. */
};
struct sys_reg_desc_reset {
int reg;
void (*reset)(struct kvm_vcpu *vcpu, const struct sys_reg_desc_reset *rd);
u64 value;
};
static void reset_mpidr(struct kvm_vcpu *vcpu, const struct sys_reg_desc_reset *r)
{
__vcpu_assign_sys_reg(vcpu, r->reg, kvm_calculate_mpidr(vcpu));
}
static void reset_value(struct kvm_vcpu *vcpu, const struct sys_reg_desc_reset *r)
{
__vcpu_assign_sys_reg(vcpu, r->reg, r->value);
}
#define RESET_VAL(REG, VAL) { REG, reset_value, VAL }
#define RESET_ZERO(REG) RESET_VAL(REG, 0)
#define RESET_UNKNOWN(REG) RESET_VAL(REG, 0x1de7ec7edbadc0deULL)
#define RESET_FUNC(REG, FN) { REG, FN, 0 }
static const struct sys_reg_desc_reset pvm_sys_reg_reset_vals[] = {
RESET_FUNC(MPIDR_EL1, reset_mpidr),
RESET_UNKNOWN(TPIDR_EL0),
RESET_UNKNOWN(TPIDRRO_EL0),
RESET_UNKNOWN(TPIDR_EL1),
RESET_ZERO(CNTKCTL_EL1),
RESET_UNKNOWN(PAR_EL1),
RESET_ZERO(DISR_EL1),
RESET_ZERO(CPACR_EL1),
RESET_VAL(CONTEXTIDR_EL1, 0x00000000dbadc0deULL),
RESET_VAL(SCTLR_EL1, 0x00C50078ULL),
RESET_ZERO(TCR_EL1),
RESET_UNKNOWN(AFSR0_EL1),
RESET_UNKNOWN(AFSR1_EL1),
RESET_UNKNOWN(ESR_EL1),
RESET_UNKNOWN(MAIR_EL1),
RESET_ZERO(AMAIR_EL1),
RESET_ZERO(MDSCR_EL1),
RESET_ZERO(CNTV_CTL_EL0),
RESET_UNKNOWN(TTBR0_EL1),
RESET_UNKNOWN(TTBR1_EL1),
RESET_UNKNOWN(FAR_EL1),
RESET_VAL(VBAR_EL1, 0x1de7ec7edbadc000ULL),
};
void kvm_reset_pvm_sys_regs(struct kvm_vcpu *vcpu)
{
unsigned long i;
for (i = 0; i < ARRAY_SIZE(pvm_sys_reg_reset_vals); i++) {
const struct sys_reg_desc_reset *r = &pvm_sys_reg_reset_vals[i];
r->reset(vcpu, r);
}
}
/*
* Initializes feature registers for protected vms.
*/

7
arch/arm64/kvm/hypercalls.c
View File

@ -585,6 +585,13 @@ int kvm_arm_set_fw_reg(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg)
wants_02 = vcpu_has_feature(vcpu, KVM_ARM_VCPU_PSCI_0_2);
/*
* EL2 advertises PSCI 1.1; the host handles the forwarded calls
* by this version.
*/
if (vcpu_is_protected(vcpu) && val < KVM_ARM_PSCI_1_1)
return -EINVAL;
switch (val) {
case KVM_ARM_PSCI_0_1:
if (wants_02)

10
arch/arm64/kvm/inject_fault.c
View File

@ -371,15 +371,15 @@ int kvm_inject_serror_esr(struct kvm_vcpu *vcpu, u64 esr)
}
/*
* Emulate the exception entry if SErrors are unmasked. This is useful if
* the vCPU is in a nested context w/ vSErrors enabled then we've already
* delegated he hardware vSError context (i.e. HCR_EL2.VSE, VSESR_EL2,
* VDISR_EL2) to the guest hypervisor.
* With NV, emulate the exception entry if SErrors are unmasked: in a
* nested context with vSErrors enabled, the hardware vSError context
* (i.e. HCR_EL2.VSE, VSESR_EL2, VDISR_EL2) is delegated to the guest
* hypervisor. Otherwise, HCR_EL2.VSE delivers it once it is unmasked.
*
* As we're emulating the SError injection we need to explicitly populate
* ESR_ELx.EC because hardware will not do it on our behalf.
*/
if (!serror_is_masked(vcpu)) {
if (vcpu_has_nv(vcpu) && !serror_is_masked(vcpu)) {
pend_serror_exception(vcpu);
esr |= FIELD_PREP(ESR_ELx_EC_MASK, ESR_ELx_EC_SERROR) | ESR_ELx_IL;
vcpu_write_sys_reg(vcpu, esr, exception_esr_elx(vcpu));

21
arch/arm64/kvm/pkvm.c
View File

@ -118,12 +118,25 @@ static int __pkvm_create_hyp_vcpu(struct kvm_vcpu *vcpu)
return -ENOMEM;
ret = kvm_call_hyp_nvhe(__pkvm_init_vcpu, handle, vcpu, hyp_vcpu);
if (!ret)
vcpu_set_flag(vcpu, VCPU_PKVM_FINALIZED);
else
if (ret) {
free_pages_exact(hyp_vcpu, hyp_vcpu_sz);
return ret;
}
return ret;
/*
* Mirror EL2's seeding of power_state from mp_state. The hyp vCPU is
* published, so take mp_state_lock against kvm_psci_vcpu_on().
*/
if (vcpu_is_protected(vcpu)) {
spin_lock(&vcpu->arch.mp_state_lock);
if (kvm_arm_vcpu_stopped(vcpu))
WRITE_ONCE(vcpu->arch.mp_state.mp_state, KVM_MP_STATE_UNINITIALIZED);
spin_unlock(&vcpu->arch.mp_state_lock);
}
vcpu_set_flag(vcpu, VCPU_PKVM_FINALIZED);
return 0;
}
/*

11
arch/arm64/kvm/psci.c
View File

@ -63,7 +63,12 @@ static unsigned long kvm_psci_vcpu_on(struct kvm_vcpu *source_vcpu)
return PSCI_RET_INVALID_PARAMS;
spin_lock(&vcpu->arch.mp_state_lock);
if (!kvm_arm_vcpu_stopped(vcpu)) {
/*
* A protected vCPU is off once the host has acted on EL2's CPU_OFF.
* STOPPED before that is the VMM's pause.
*/
if (!kvm_arm_vcpu_stopped(vcpu) ||
(vcpu_is_protected(vcpu) && !kvm_pkvm_vcpu_is_powered_off(vcpu))) {
if (kvm_psci_version(source_vcpu) != KVM_ARM_PSCI_0_1)
ret = PSCI_RET_ALREADY_ON;
else
@ -161,7 +166,9 @@ static void kvm_prepare_system_event(struct kvm_vcpu *vcpu, u32 type, u64 flags)
*/
kvm_for_each_vcpu(i, tmp, vcpu->kvm) {
spin_lock(&tmp->arch.mp_state_lock);
WRITE_ONCE(tmp->arch.mp_state.mp_state, KVM_MP_STATE_STOPPED);
/* Keep EL2's off record for the guest's next CPU_ON. */
if (!kvm_pkvm_vcpu_is_powered_off(tmp))
WRITE_ONCE(tmp->arch.mp_state.mp_state, KVM_MP_STATE_STOPPED);
spin_unlock(&tmp->arch.mp_state_lock);
}
kvm_make_all_cpus_request(vcpu->kvm, KVM_REQ_SLEEP);

10
arch/arm64/kvm/pvtime.c
View File

@ -67,9 +67,9 @@ gpa_t kvm_init_stolen_time(struct kvm_vcpu *vcpu)
return base;
}
bool kvm_arm_pvtime_supported(void)
bool kvm_arm_pvtime_supported(struct kvm *kvm)
{
return !!sched_info_on();
return !!sched_info_on() && (!kvm || !kvm_vm_is_protected(kvm));
}
int kvm_arm_pvtime_set_attr(struct kvm_vcpu *vcpu,
@ -81,7 +81,7 @@ int kvm_arm_pvtime_set_attr(struct kvm_vcpu *vcpu,
int ret = 0;
int idx;
if (!kvm_arm_pvtime_supported() ||
if (!kvm_arm_pvtime_supported(kvm) ||
attr->attr != KVM_ARM_VCPU_PVTIME_IPA)
return -ENXIO;
@ -110,7 +110,7 @@ int kvm_arm_pvtime_get_attr(struct kvm_vcpu *vcpu,
u64 __user *user = (u64 __user *)attr->addr;
u64 ipa;
if (!kvm_arm_pvtime_supported() ||
if (!kvm_arm_pvtime_supported(vcpu->kvm) ||
attr->attr != KVM_ARM_VCPU_PVTIME_IPA)
return -ENXIO;
@ -126,7 +126,7 @@ int kvm_arm_pvtime_has_attr(struct kvm_vcpu *vcpu,
{
switch (attr->attr) {
case KVM_ARM_VCPU_PVTIME_IPA:
if (kvm_arm_pvtime_supported())
if (kvm_arm_pvtime_supported(vcpu->kvm))
return 0;
}
return -ENXIO;