github-mirror/linux
1
0
Fork 0
mirror of https://github.com/torvalds/linux.git synced 2026-07-27 09:36:22 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity Graph

Arm:

Browse Source 
- Fix an accounting buglet when reclaiming pages from a protected guest
 
 - Fix a bunch of architectural compliance issues when injecting a
   synthesised exception, most of which were missing the PSTATE.IL bit
   indicating a 32bit-wide instruction
 
 - Another set of fixes addressing issues with translation of VNCR_EL2,
   including corner cases where the guest point that register at a RO
   page...
 
 - Don't warn when trapping accesses to ZCR_EL2 from an L2 guest, as
   that's not unexpected at all
 
 - Address a bunch of races with LPI migration vs LPIs being disabled
 
 - Fix a total howler of a bug combining FEAT_MOPS and NV, resulting in
   exception returning in the wrong place...
 
 - Move locking for kvm_io_bus_get_dev() into the caller, ensuring
   race-free checks that the returned object is of the correct type
 
 - Fix initialisation of the page-table walk level when relaxing
   permissions
 
 - Correctly update the XN attribute when relaxing permissions
 
 - Fix the sign extension of loads from emulated MMIO regions
 
 - Assorted collection of fixes for pKVM's FFA proxy, together with a
   couple of FFA driver adjustments
 
 - Coerce Fuad Tabba into a reviewer role, and may his Inbox catch fire!
 
 s390:
 
 - more gmap KVM memory management fixes
 
 - PCI passthru fixes
 
 x86:
 
 - Fix a bug where KVM will trigger a UAF if updating IOMMU IRTEs fails when
   registering an IRQ-bypass producer.
 
 - Ignore pending PV EOI instead of BUG()ing the host if the feature was
   disabled by the guest.
 
 - Fix nVMX bugs where KVM would run L1 with an L1-controlled CR3 after a
   failed "late" consistency check when KVM is NOT using EPT.
 
 - Disallow intra-host migration/mirroring of SNP VMs as KVM doesn't yet
   support moving/mirroring SNP state.
 
 - Fix a TOCTOU bug in KVM's handling of the "trusted" CPUID for TDX guests.
 
 - Fix a NULL pointer deref in trace_kvm_inj_exception() where a change to the
   core infrastructure missed KVM's unique (ab)use of __print_symbolic().
 
 - Put vmcs12 pages if nested VM-Enter fails due to invalid guest state
 
 - Fix TLB conflicts between two VMs if one of them VM is run on a CPU before
   and after it is hotplugged.
 -----BEGIN PGP SIGNATURE-----
 
 iQFIBAABCgAyFiEE8TM4V0tmI4mGbHaCv/vSX3jHroMFAmpYiWIUHHBib256aW5p
 QHJlZGhhdC5jb20ACgkQv/vSX3jHroPa5gf/SMRd1vAZnpsvv4MxU4plSwOzgD6k
 /tR3f7SUQEuP9fO6p89g+xqhid7zoq5sA25PdEP8A92KchomMdAmcMQeOHzVB2Tm
 Hd1JEkhsq/1zBjDwzhlCEVnlBKG9T4/iiQGIa1t4504dy8b0Faz7yG7/X8eAk7VF
 HwSmhasQClP3meBNv0C8QE+hRQVLP/W0lUis3Gng7lIlRVVcvwgSRFtQVTLNmePa
 1kEXJvzPbOJNi4OPip2e20iYdMGsi0XW7qi40TRllagucsXNzgXiOVpNgUniZqTO
 ADZhc4WGEH3lr4UQDjA3YekbHYAxhwxdwX4x7pInbtVEqri4PL2D6IKzWg==
 =Dw68
 -----END PGP SIGNATURE-----

Merge tag 'for-linus' of git://git.kernel.org/pub/scm/virt/kvm/kvm

Pull kvm fixes from Paolo Bonzini:
 "arm64:

   - Fix an accounting buglet when reclaiming pages from a protected
     guest

   - Fix a bunch of architectural compliance issues when injecting a
     synthesised exception, most of which were missing the PSTATE.IL bit
     indicating a 32bit-wide instruction

   - Another set of fixes addressing issues with translation of
     VNCR_EL2, including corner cases where the guest point that
     register at a RO page...

   - Don't warn when trapping accesses to ZCR_EL2 from an L2 guest, as
     that's not unexpected at all

   - Address a bunch of races with LPI migration vs LPIs being disabled

   - Fix a total howler of a bug combining FEAT_MOPS and NV, resulting
     in exception returning in the wrong place...

   - Move locking for kvm_io_bus_get_dev() into the caller, ensuring
     race-free checks that the returned object is of the correct type

   - Fix initialisation of the page-table walk level when relaxing
     permissions

   - Correctly update the XN attribute when relaxing permissions

   - Fix the sign extension of loads from emulated MMIO regions

   - Assorted collection of fixes for pKVM's FFA proxy, together with a
     couple of FFA driver adjustments

   - Coerce Fuad Tabba into a reviewer role, and may his Inbox catch
     fire!

  s390:

   - more gmap KVM memory management fixes

   - PCI passthru fixes

  x86:

   - Fix a bug where KVM will trigger a UAF if updating IOMMU IRTEs
     fails when registering an IRQ-bypass producer

   - Ignore pending PV EOI instead of BUG()ing the host if the feature
     was disabled by the guest

   - Fix nVMX bugs where KVM would run L1 with an L1-controlled CR3
     after a failed "late" consistency check when KVM is NOT using EPT

   - Disallow intra-host migration/mirroring of SNP VMs as KVM doesn't
     yet support moving/mirroring SNP state

   - Fix a TOCTOU bug in KVM's handling of the "trusted" CPUID for TDX
     guests

   - Fix a NULL pointer deref in trace_kvm_inj_exception() where a
     change to the core infrastructure missed KVM's unique (ab)use of
     __print_symbolic()

   - Put vmcs12 pages if nested VM-Enter fails due to invalid guest
     state

   - Fix TLB conflicts between two VMs if one of them VM is run on a CPU
     before and after it is hotplugged"

* tag 'for-linus' of git://git.kernel.org/pub/scm/virt/kvm/kvm: (48 commits)
  KVM: SVM: Bump asid_generation on CPU online to avoid ASID collision after hotplug
  KVM: nVMX: Put vmcs12 pages if nested VM-Enter fails due to invalid guest state
  KVM: x86: Fix null pointer deref due to dummy array in trace_kvm_inj_exception()
  KVM: TDX: Reject concurrent change to CPUID entry count
  KVM: selftests: Verify SNP VMs are rejected from migration and mirroring
  KVM: SEV: Do not allow intra-host migration/mirroring of SNP VMs
  KVM: s390: pci: Fix handling of AIF enable without AISB
  KVM: s390: Improve kvm_s390_vm_stop_migration()
  KVM: s390: Fix dat_crste_walk_range() early return
  KVM: s390: vsie: Avoid potential deadlock with real spaces
  KVM: s390: pci: Fix GISC refcount leak on AIF enable failure
  KVM: nVMX: Don't use vmcs01.GUEST_CR3 to snapshot L1's CR3 when EPT is disabled
  KVM: nVMX: Move vTPR vs. TPR Threshold consistency check into "normal" checks
  KVM: x86: Ignore pending PV EOI if the vCPU has since disabled PV EOIs
  KVM: x86: Nullify irqfd->producer if updating IRTE for bypass fails
  KVM: arm64: Fix propagation of TLBI level in kvm_pgtable_stage2_relax_perms()
  firmware: arm_ffa: Fix Endpoint Memory Access Descriptor offset calculation
  firmware: arm_ffa: Fix out-of-bound writes in ffa_setup_and_transmit()
  KVM: arm64: Zero out the stack initialized data in the FFA handler
  KVM: arm64: Ensure FFA ranges are page aligned
  ...
This commit is contained in:
Linus Torvalds 2026-07-16 07:25:13 -07:00
commit c270a4218b
33 changed files with 621 additions and 235 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
MAINTAINERS
View File

@ -14192,6 +14192,7 @@ F: virt/kvm/*
KERNEL VIRTUAL MACHINE FOR ARM64 (KVM/arm64)
M: Marc Zyngier <maz@kernel.org>
M: Oliver Upton <oupton@kernel.org>
R: Fuad Tabba <tabba@google.com>
R: Joey Gouly <joey.gouly@arm.com>
R: Steffen Eiden <seiden@linux.ibm.com>
R: Suzuki K Poulose <suzuki.poulose@arm.com>

8
arch/arm64/include/asm/kvm_nested.h
View File

@ -388,6 +388,14 @@ struct s1_walk_result {
bool failed;
};
static inline void fail_s1_walk(struct s1_walk_result *wr, u8 fst, bool s1ptw)
{
wr->fst = fst;
wr->ptw = s1ptw;
wr->s2 = s1ptw;
wr->failed = true;
}
int __kvm_translate_va(struct kvm_vcpu *vcpu, struct s1_walk_info *wi,
struct s1_walk_result *wr, u64 va);
int __kvm_find_s1_desc_level(struct kvm_vcpu *vcpu, u64 va, u64 ipa,

8
arch/arm64/kvm/at.c
View File

@ -11,14 +11,6 @@
#include <asm/kvm_mmu.h>
#include <asm/lsui.h>
static void fail_s1_walk(struct s1_walk_result *wr, u8 fst, bool s1ptw)
{
wr->fst = fst;
wr->ptw = s1ptw;
wr->s2 = s1ptw;
wr->failed = true;
}
#define S1_MMU_DISABLED (-127)
static int get_ia_size(struct s1_walk_info *wi)

41
arch/arm64/kvm/emulate-nested.c
View File

@ -2746,17 +2746,33 @@ static u64 kvm_check_illegal_exception_return(struct kvm_vcpu *vcpu, u64 spsr)
(spsr & PSR_MODE32_BIT) ||
(vcpu_el2_tge_is_set(vcpu) && (mode == PSR_MODE_EL1t ||
mode == PSR_MODE_EL1h))) {
/*
* The guest is playing with our nerves. Preserve EL, SP,
* masks, flags from the existing PSTATE, and set IL.
* The HW will then generate an Illegal State Exception
* immediately after ERET.
*/
spsr = *vcpu_cpsr(vcpu);
u64 mask;
spsr &= (PSR_D_BIT | PSR_A_BIT | PSR_I_BIT | PSR_F_BIT |
PSR_N_BIT | PSR_Z_BIT | PSR_C_BIT | PSR_V_BIT |
PSR_MODE_MASK | PSR_MODE32_BIT);
/*
* On an illegal exception return, the flags and masks are
* taken from the SPSR while PSTATE.{EL,SP,nRW} and, if
* FEAT_GCS, PSTATE.EXLOCK are unchanged (R_VWJHB). Set IL
* so the HW generates an Illegal State Exception right
* after ERET.
*/
mask = PSR_D_BIT | PSR_A_BIT | PSR_I_BIT | PSR_F_BIT |
PSR_N_BIT | PSR_Z_BIT | PSR_C_BIT | PSR_V_BIT;
if (kvm_has_feat(vcpu->kvm, ID_AA64MMFR1_EL1, PAN, IMP))
mask |= PSR_PAN_BIT;
if (kvm_has_feat(vcpu->kvm, ID_AA64PFR1_EL1, NMI, IMP))
mask |= ALLINT_ALLINT;
/* FEAT_SPE_EXC and FEAT_TRBE_EXC also gate PSTATE.PM one day... */
if (kvm_has_feat(vcpu->kvm, ID_AA64DFR1_EL1, EBEP, IMP))
mask |= BIT_ULL(32); /* SPSR_ELx.PM */
spsr &= mask;
mask = PSR_MODE_MASK | PSR_MODE32_BIT;
if (kvm_has_feat(vcpu->kvm, ID_AA64PFR1_EL1, GCS, IMP))
mask |= BIT_ULL(34); /* PSTATE.EXLOCK */
spsr |= *vcpu_cpsr(vcpu) & mask;
spsr |= PSR_IL_BIT;
}
@ -2784,7 +2800,7 @@ void kvm_emulate_nested_eret(struct kvm_vcpu *vcpu)
* ERET handling, and the guest will have a little surprise.
*/
if (kvm_has_pauth(vcpu->kvm, FPACCOMBINE) && !(spsr & PSR_IL_BIT)) {
esr &= ESR_ELx_ERET_ISS_ERETA;
esr &= (ESR_ELx_ERET_ISS_ERETA | ESR_ELx_IL);
esr |= FIELD_PREP(ESR_ELx_EC_MASK, ESR_ELx_EC_FPAC);
kvm_inject_nested_sync(vcpu, esr);
return;
@ -2826,6 +2842,7 @@ static void kvm_inject_el2_exception(struct kvm_vcpu *vcpu, u64 esr_el2,
break;
case except_type_serror:
kvm_pend_exception(vcpu, EXCEPT_AA64_EL2_SERR);
vcpu_write_sys_reg(vcpu, esr_el2, ESR_EL2);
break;
default:
WARN_ONCE(1, "Unsupported EL2 exception injection %d\n", type);
@ -2950,6 +2967,6 @@ int kvm_inject_nested_serror(struct kvm_vcpu *vcpu, u64 esr)
* vSError injection. Manually populate EC for an emulated SError
* exception.
*/
esr |= FIELD_PREP(ESR_ELx_EC_MASK, ESR_ELx_EC_SERROR);
esr |= FIELD_PREP(ESR_ELx_EC_MASK, ESR_ELx_EC_SERROR) | ESR_ELx_IL;
return kvm_inject_nested(vcpu, esr, except_type_serror);
}

11
arch/arm64/kvm/hyp/include/hyp/switch.h
View File

@ -448,16 +448,19 @@ static inline bool __populate_fault_info(struct kvm_vcpu *vcpu)
static inline bool kvm_hyp_handle_mops(struct kvm_vcpu *vcpu, u64 *exit_code)
{
u64 spsr;
*vcpu_pc(vcpu) = read_sysreg_el2(SYS_ELR);
arm64_mops_reset_regs(vcpu_gp_regs(vcpu), vcpu->arch.fault.esr_el2);
write_sysreg_el2(*vcpu_pc(vcpu), SYS_ELR);
/*
* Finish potential single step before executing the prologue
* instruction.
* instruction. Modify the hardware SPSR_EL2 directly, as vcpu_cpsr()
* may hold a synthetic (vEL2) value for a guest hypervisor.
*/
*vcpu_cpsr(vcpu) &= ~DBG_SPSR_SS;
write_sysreg_el2(*vcpu_cpsr(vcpu), SYS_SPSR);
spsr = read_sysreg_el2(SYS_SPSR);
write_sysreg_el2(spsr & ~DBG_SPSR_SS, SYS_SPSR);
return true;
}
@ -602,8 +605,6 @@ static inline bool kvm_hyp_handle_fpsimd(struct kvm_vcpu *vcpu, u64 *exit_code)
return false;
break;
case ESR_ELx_EC_SYS64:
if (WARN_ON_ONCE(!is_hyp_ctxt(vcpu)))
return false;
fallthrough;
case ESR_ELx_EC_SVE:
if (!sve_guest)

44
arch/arm64/kvm/hyp/nvhe/ffa.c
View File

@ -352,7 +352,7 @@ static u32 __ffa_host_share_ranges(struct ffa_mem_region_addr_range *ranges,
u64 sz = (u64)range->pg_cnt * FFA_PAGE_SIZE;
u64 pfn = hyp_phys_to_pfn(range->address);
if (!PAGE_ALIGNED(sz))
if (!PAGE_ALIGNED(sz | range->address))
break;
if (__pkvm_host_share_ffa(pfn, sz / PAGE_SIZE))
@ -372,7 +372,7 @@ static u32 __ffa_host_unshare_ranges(struct ffa_mem_region_addr_range *ranges,
u64 sz = (u64)range->pg_cnt * FFA_PAGE_SIZE;
u64 pfn = hyp_phys_to_pfn(range->address);
if (!PAGE_ALIGNED(sz))
if (!PAGE_ALIGNED(sz | range->address))
break;
if (__pkvm_host_unshare_ffa(pfn, sz / PAGE_SIZE))
@ -476,11 +476,12 @@ static void __do_ffa_mem_xfer(const u64 func_id,
DECLARE_REG(u32, fraglen, ctxt, 2);
DECLARE_REG(u64, addr_mbz, ctxt, 3);
DECLARE_REG(u32, npages_mbz, ctxt, 4);
u32 offset, nr_ranges, checked_offset, em_mem_access_off;
struct ffa_mem_region_attributes *ep_mem_access;
struct ffa_composite_mem_region *reg;
struct ffa_mem_region *buf;
u32 offset, nr_ranges, checked_offset;
int ret = 0;
size_t mem_region_len = FFA_MEM_REGION_SZ(hyp_ffa_version);
if (addr_mbz || npages_mbz || fraglen > len ||
fraglen > KVM_FFA_MBOX_NR_PAGES * PAGE_SIZE) {
@ -488,8 +489,7 @@ static void __do_ffa_mem_xfer(const u64 func_id,
goto out;
}
if (fraglen < sizeof(struct ffa_mem_region) +
sizeof(struct ffa_mem_region_attributes)) {
if (fraglen < mem_region_len + ffa_emad_size_get(hyp_ffa_version)) {
ret = FFA_RET_INVALID_PARAMETERS;
goto out;
}
@ -508,8 +508,13 @@ static void __do_ffa_mem_xfer(const u64 func_id,
buf = hyp_buffers.tx;
memcpy(buf, host_buffers.tx, fraglen);
ep_mem_access = (void *)buf +
ffa_mem_desc_offset(buf, 0, hyp_ffa_version);
em_mem_access_off = ffa_mem_desc_offset(buf, 0, hyp_ffa_version);
if ((u64)em_mem_access_off + ffa_emad_size_get(hyp_ffa_version) > fraglen) {
ret = FFA_RET_INVALID_PARAMETERS;
goto out_unlock;
}
ep_mem_access = (void *)buf + em_mem_access_off;
offset = ep_mem_access->composite_off;
if (!offset || buf->ep_count != 1 || buf->sender_id != HOST_FFA_ID) {
ret = FFA_RET_INVALID_PARAMETERS;
@ -574,9 +579,9 @@ static void do_ffa_mem_reclaim(struct arm_smccc_1_2_regs *res,
DECLARE_REG(u32, handle_lo, ctxt, 1);
DECLARE_REG(u32, handle_hi, ctxt, 2);
DECLARE_REG(u32, flags, ctxt, 3);
u32 offset, len, fraglen, fragoff, em_mem_access_off;
struct ffa_mem_region_attributes *ep_mem_access;
struct ffa_composite_mem_region *reg;
u32 offset, len, fraglen, fragoff;
struct ffa_mem_region *buf;
int ret = 0;
u64 handle;
@ -599,16 +604,22 @@ static void do_ffa_mem_reclaim(struct arm_smccc_1_2_regs *res,
len = res->a1;
fraglen = res->a2;
ep_mem_access = (void *)buf +
ffa_mem_desc_offset(buf, 0, hyp_ffa_version);
em_mem_access_off = ffa_mem_desc_offset(buf, 0, hyp_ffa_version);
if ((u64)em_mem_access_off + ffa_emad_size_get(hyp_ffa_version) > fraglen) {
ret = FFA_RET_INVALID_PARAMETERS;
ffa_rx_release(res);
goto out_unlock;
}
ep_mem_access = (void *)buf + em_mem_access_off;
offset = ep_mem_access->composite_off;
/*
* We can trust the SPMD to get this right, but let's at least
* check that we end up with something that doesn't look _completely_
* bogus.
*/
if (WARN_ON(offset > len ||
fraglen > KVM_FFA_MBOX_NR_PAGES * PAGE_SIZE)) {
if (offset + CONSTITUENTS_OFFSET(0) > len ||
fraglen > KVM_FFA_MBOX_NR_PAGES * PAGE_SIZE) {
ret = FFA_RET_ABORTED;
ffa_rx_release(res);
goto out_unlock;
@ -636,11 +647,16 @@ static void do_ffa_mem_reclaim(struct arm_smccc_1_2_regs *res,
ffa_rx_release(res);
}
reg = (void *)buf + offset;
if (offset + CONSTITUENTS_OFFSET(reg->addr_range_cnt) > len) {
ret = FFA_RET_ABORTED;
goto out_unlock;
}
ffa_mem_reclaim(res, handle_lo, handle_hi, flags);
if (res->a0 != FFA_SUCCESS)
goto out_unlock;
reg = (void *)buf + offset;
/* If the SPMD was happy, then we should be too. */
WARN_ON(ffa_host_unshare_ranges(reg->constituents,
reg->addr_range_cnt));
@ -864,7 +880,7 @@ static void do_ffa_part_get(struct arm_smccc_1_2_regs *res,
bool kvm_host_ffa_handler(struct kvm_cpu_context *host_ctxt, u32 func_id)
{
struct arm_smccc_1_2_regs res;
struct arm_smccc_1_2_regs res = {0};
/*
* There's no way we can tell what a non-standard SMC call might

3
arch/arm64/kvm/hyp/nvhe/pkvm.c
View File

@ -1056,7 +1056,8 @@ static u64 __pkvm_memshare_page_req(struct kvm_vcpu *vcpu, u64 ipa)
/* Fake up a data abort (level 3 translation fault on write) */
vcpu->arch.fault.esr_el2 = (ESR_ELx_EC_DABT_LOW << ESR_ELx_EC_SHIFT) |
ESR_ELx_WNR | ESR_ELx_FSC_FAULT |
ESR_ELx_IL | ESR_ELx_WNR |
ESR_ELx_FSC_FAULT |
FIELD_PREP(ESR_ELx_FSC_LEVEL, 3);
/* Shuffle the IPA around into the HPFAR */

3
arch/arm64/kvm/hyp/nvhe/sys_regs.c
View File

@ -268,6 +268,7 @@ static void inject_sync64(struct kvm_vcpu *vcpu, u64 esr)
write_sysreg_el1(esr, SYS_ESR);
write_sysreg_el1(read_sysreg_el2(SYS_ELR), SYS_ELR);
write_sysreg_el1(read_sysreg_el2(SYS_SPSR), SYS_SPSR);
write_sysreg_el2(*vcpu_pc(vcpu), SYS_ELR);
write_sysreg_el2(*vcpu_cpsr(vcpu), SYS_SPSR);
}
@ -278,7 +279,7 @@ static void inject_sync64(struct kvm_vcpu *vcpu, u64 esr)
*/
static void inject_undef64(struct kvm_vcpu *vcpu)
{
inject_sync64(vcpu, (ESR_ELx_EC_UNKNOWN << ESR_ELx_EC_SHIFT));
inject_sync64(vcpu, (ESR_ELx_EC_UNKNOWN << ESR_ELx_EC_SHIFT) | ESR_ELx_IL);
}
static u64 read_id_reg(const struct kvm_vcpu *vcpu,

15
arch/arm64/kvm/hyp/pgtable.c
View File

@ -1370,16 +1370,19 @@ int kvm_pgtable_stage2_relax_perms(struct kvm_pgtable *pgt, u64 addr,
if (prot & KVM_PGTABLE_PROT_W)
set |= KVM_PTE_LEAF_ATTR_LO_S2_S2AP_W;
ret = stage2_set_xn_attr(prot, &xn);
if (ret)
return ret;
if (prot & KVM_PGTABLE_PROT_X) {
ret = stage2_set_xn_attr(prot, &xn);
if (ret)
return ret;
set |= xn & KVM_PTE_LEAF_ATTR_HI_S2_XN;
clr |= ~xn & KVM_PTE_LEAF_ATTR_HI_S2_XN;
set |= xn & KVM_PTE_LEAF_ATTR_HI_S2_XN;
clr |= ~xn & KVM_PTE_LEAF_ATTR_HI_S2_XN;
}
ret = stage2_update_leaf_attrs(pgt, addr, 1, set, clr, NULL, &level, flags);
if (!ret || ret == -EAGAIN)
kvm_call_hyp(__kvm_tlb_flush_vmid_ipa_nsh, pgt->mmu, addr, level);
kvm_call_hyp(__kvm_tlb_flush_vmid_ipa_nsh, pgt->mmu, addr,
(ret == -EAGAIN) ? TLBI_TTL_UNKNOWN : level);
return ret;
}

18
arch/arm64/kvm/inject_fault.c
View File

@ -138,11 +138,10 @@ static void inject_abt64(struct kvm_vcpu *vcpu, bool is_iabt, unsigned long addr
pend_sync_exception(vcpu);
/*
* Build an {i,d}abort, depending on the level and the
* instruction set. Report an external synchronous abort.
* Build an {i,d}abort, depending on the level.
* Report an external synchronous abort.
*/
if (kvm_vcpu_trap_il_is32bit(vcpu))
esr |= ESR_ELx_IL;
esr |= ESR_ELx_IL;
/*
* Here, the guest runs in AArch64 mode when in EL1. If we get
@ -170,14 +169,7 @@ void kvm_inject_sync(struct kvm_vcpu *vcpu, u64 esr)
static void inject_undef64(struct kvm_vcpu *vcpu)
{
u64 esr = (ESR_ELx_EC_UNKNOWN << ESR_ELx_EC_SHIFT);
/*
* Build an unknown exception, depending on the instruction
* set.
*/
if (kvm_vcpu_trap_il_is32bit(vcpu))
esr |= ESR_ELx_IL;
u64 esr = (ESR_ELx_EC_UNKNOWN << ESR_ELx_EC_SHIFT) | ESR_ELx_IL;
kvm_inject_sync(vcpu, esr);
}
@ -389,7 +381,7 @@ int kvm_inject_serror_esr(struct kvm_vcpu *vcpu, u64 esr)
*/
if (!serror_is_masked(vcpu)) {
pend_serror_exception(vcpu);
esr |= FIELD_PREP(ESR_ELx_EC_MASK, ESR_ELx_EC_SERROR);
esr |= FIELD_PREP(ESR_ELx_EC_MASK, ESR_ELx_EC_SERROR) | ESR_ELx_IL;
vcpu_write_sys_reg(vcpu, esr, exception_esr_elx(vcpu));
return 1;
}

7
arch/arm64/kvm/mmio.c
View File

@ -126,6 +126,10 @@ int kvm_handle_mmio_return(struct kvm_vcpu *vcpu)
len = kvm_vcpu_dabt_get_as(vcpu);
data = kvm_mmio_read_buf(run->mmio.data, len);
trace_kvm_mmio(KVM_TRACE_MMIO_READ, len, run->mmio.phys_addr,
&data);
data = vcpu_data_host_to_guest(vcpu, data, len);
if (kvm_vcpu_dabt_issext(vcpu) &&
len < sizeof(unsigned long)) {
mask = 1U << ((len * 8) - 1);
@ -135,9 +139,6 @@ int kvm_handle_mmio_return(struct kvm_vcpu *vcpu)
if (!kvm_vcpu_dabt_issf(vcpu))
data = data & 0xffffffff;
trace_kvm_mmio(KVM_TRACE_MMIO_READ, len, run->mmio.phys_addr,
&data);
data = vcpu_data_host_to_guest(vcpu, data, len);
vcpu_set_reg(vcpu, kvm_vcpu_dabt_get_rd(vcpu), data);
}

164
arch/arm64/kvm/nested.c
View File

@ -24,6 +24,7 @@ struct vncr_tlb {
struct s1_walk_result wr;
u64 hpa;
bool hpa_writable;
/* -1 when not mapped on a CPU */
int cpu;
@ -1401,15 +1402,19 @@ static int kvm_translate_vncr(struct kvm_vcpu *vcpu, bool *is_gmem)
gfn = vt->wr.pa >> PAGE_SHIFT;
memslot = gfn_to_memslot(vcpu->kvm, gfn);
if (!memslot)
if (!memslot) {
fail_s1_walk(&vt->wr, ESR_ELx_FSC_EXTABT, false);
return -EFAULT;
}
*is_gmem = kvm_slot_has_gmem(memslot);
if (!*is_gmem) {
pfn = __kvm_faultin_pfn(memslot, gfn, write_fault ? FOLL_WRITE : 0,
&writable, &page);
if (is_error_noslot_pfn(pfn) || (write_fault && !writable))
if (is_error_noslot_pfn(pfn)) {
fail_s1_walk(&vt->wr, ESR_ELx_FSC_EXTABT, false);
return -EFAULT;
}
} else {
ret = kvm_gmem_get_pfn(vcpu->kvm, memslot, gfn, &pfn, &page, NULL);
if (ret) {
@ -1417,6 +1422,19 @@ static int kvm_translate_vncr(struct kvm_vcpu *vcpu, bool *is_gmem)
write_fault, false, false);
return ret;
}
writable = !(memslot->flags & KVM_MEM_READONLY);
}
/*
* FIXME: This check is too restrictive as KVM allows cacheable memory
* attributes for PFNMAP VMAs that have cacheable attributes in host
* stage-1.
*/
if (!pfn_is_map_memory(pfn)) {
kvm_release_faultin_page(vcpu->kvm, page, true, false);
fail_s1_walk(&vt->wr, ESR_ELx_FSC_EXTABT, false);
return -EINVAL;
}
scoped_guard(write_lock, &vcpu->kvm->mmu_lock) {
@ -1427,116 +1445,100 @@ static int kvm_translate_vncr(struct kvm_vcpu *vcpu, bool *is_gmem)
vt->gva = va;
vt->hpa = pfn << PAGE_SHIFT;
vt->hpa_writable = writable;
vt->valid = true;
vt->cpu = -1;
kvm_make_request(KVM_REQ_MAP_L1_VNCR_EL2, vcpu);
kvm_release_faultin_page(vcpu->kvm, page, false, vt->wr.pw);
kvm_release_faultin_page(vcpu->kvm, page, false, vt->wr.pw && vt->hpa_writable);
}
if (vt->wr.pw)
if (vt->wr.pw && vt->hpa_writable)
mark_page_dirty(vcpu->kvm, gfn);
return 0;
}
static void inject_vncr_perm(struct kvm_vcpu *vcpu)
static void handle_vncr_perm(struct kvm_vcpu *vcpu)
{
struct vncr_tlb *vt = vcpu->arch.vncr_tlb;
u64 esr = kvm_vcpu_get_esr(vcpu);
u64 fsc;
/*
* Promote to an external abort if the stage-1 permits writes but the
* HPA is read-only (e.g. RO memslot).
*/
if (kvm_is_write_fault(vcpu) && vt->wr.pw && !vt->hpa_writable)
fsc = ESR_ELx_FSC_EXTABT;
/*
* Otherwise, inject a permission fault using the guest's translation
* level rather than the host's.
*/
else
fsc = ESR_ELx_FSC_PERM_L(vt->wr.level);
/* Adjust the fault level to reflect that of the guest's */
esr &= ~ESR_ELx_FSC;
esr |= FIELD_PREP(ESR_ELx_FSC,
ESR_ELx_FSC_PERM_L(vt->wr.level));
esr |= FIELD_PREP(ESR_ELx_FSC, fsc);
kvm_inject_nested_sync(vcpu, esr);
}
static bool kvm_vncr_tlb_lookup(struct kvm_vcpu *vcpu)
{
struct vncr_tlb *vt = vcpu->arch.vncr_tlb;
lockdep_assert_held_read(&vcpu->kvm->mmu_lock);
if (!vt->valid)
return false;
if (read_vncr_el2(vcpu) != vt->gva)
return false;
if (vt->wr.nG)
return get_asid_by_regime(vcpu, TR_EL20) == vt->wr.asid;
return true;
}
int kvm_handle_vncr_abort(struct kvm_vcpu *vcpu)
{
struct vncr_tlb *vt = vcpu->arch.vncr_tlb;
u64 esr = kvm_vcpu_get_esr(vcpu);
bool is_gmem = false;
bool perm;
int ret;
WARN_ON_ONCE(!(esr & ESR_ELx_VNCR));
if (kvm_vcpu_abt_issea(vcpu))
return kvm_handle_guest_sea(vcpu);
if (esr_fsc_is_permission_fault(esr)) {
inject_vncr_perm(vcpu);
} else if (esr_fsc_is_translation_fault(esr)) {
bool valid, is_gmem = false;
int ret;
if (!esr_fsc_is_translation_fault(esr) && !esr_fsc_is_permission_fault(esr)) {
KVM_BUG(1, vcpu->kvm, "Unhandled VNCR abort, ESR=%llx\n", esr);
return -EIO;
}
scoped_guard(read_lock, &vcpu->kvm->mmu_lock)
valid = kvm_vncr_tlb_lookup(vcpu);
ret = kvm_translate_vncr(vcpu, &is_gmem);
switch (ret) {
case -EAGAIN:
/* Let's try again... */
return 1;
case -ENOMEM:
/*
* For guest_memfd, this indicates that it failed to
* create a folio to back the memory. Inform userspace.
*/
if (is_gmem)
return 0;
/* Otherwise, let's try again... */
break;
case -EFAULT:
case -EIO:
case -EHWPOISON:
if (is_gmem)
return 0;
fallthrough;
case -EINVAL:
case -ENOENT:
case -EACCES:
/*
* Translation failed, inject the corresponding
* exception back to EL2.
*/
esr &= ~ESR_ELx_FSC;
esr |= FIELD_PREP(ESR_ELx_FSC, vt->wr.fst);
if (!valid)
ret = kvm_translate_vncr(vcpu, &is_gmem);
else
ret = -EPERM;
switch (ret) {
case -EAGAIN:
/* Let's try again... */
break;
case -ENOMEM:
/*
* For guest_memfd, this indicates that it failed to
* create a folio to back the memory. Inform userspace.
*/
if (is_gmem)
return 0;
/* Otherwise, let's try again... */
break;
case -EFAULT:
case -EIO:
case -EHWPOISON:
if (is_gmem)
return 0;
fallthrough;
case -EINVAL:
case -ENOENT:
case -EACCES:
/*
* Translation failed, inject the corresponding
* exception back to EL2.
*/
BUG_ON(!vt->wr.failed);
esr &= ~ESR_ELx_FSC;
esr |= FIELD_PREP(ESR_ELx_FSC, vt->wr.fst);
kvm_inject_nested_sync(vcpu, esr);
break;
case -EPERM:
/* Hack to deal with POE until we get kernel support */
inject_vncr_perm(vcpu);
break;
case 0:
break;
}
} else {
WARN_ONCE(1, "Unhandled VNCR abort, ESR=%llx\n", esr);
kvm_inject_nested_sync(vcpu, esr);
break;
case 0:
perm = kvm_is_write_fault(vcpu) ? vt->wr.pw && vt->hpa_writable : vt->wr.pr;
if (!perm)
handle_vncr_perm(vcpu);
break;
}
return 1;
@ -1574,7 +1576,7 @@ static void kvm_map_l1_vncr(struct kvm_vcpu *vcpu)
vt->cpu = smp_processor_id();
if (vt->wr.pw && vt->wr.pr)
if (vt->hpa_writable && vt->wr.pw && vt->wr.pr)
prot = PAGE_KERNEL;
else if (vt->wr.pr)
prot = PAGE_KERNEL_RO;

2
arch/arm64/kvm/pkvm.c
View File

@ -352,7 +352,7 @@ static int __pkvm_pgtable_stage2_reclaim(struct kvm_pgtable *pgt, u64 start, u64
page = pfn_to_page(mapping->pfn);
WARN_ON_ONCE(mapping->nr_pages != 1);
unpin_user_pages_dirty_lock(&page, 1, true);
account_locked_vm(current->mm, 1, false);
account_locked_vm(kvm->mm, 1, false);
pkvm_mapping_remove(mapping, &pgt->pkvm_mappings);
kfree(mapping);
}

2
arch/arm64/kvm/vgic/vgic-its.c
View File

@ -508,6 +508,8 @@ static struct vgic_its *__vgic_doorbell_to_its(struct kvm *kvm, gpa_t db)
struct kvm_io_device *kvm_io_dev;
struct vgic_io_device *iodev;
guard(srcu)(&kvm->srcu);
kvm_io_dev = kvm_io_bus_get_dev(kvm, KVM_MMIO_BUS, db);
if (!kvm_io_dev)
return ERR_PTR(-EINVAL);

20
arch/arm64/kvm/vgic/vgic.c
View File

@ -203,6 +203,7 @@ void vgic_flush_pending_lpis(struct kvm_vcpu *vcpu)
list_for_each_entry_safe(irq, tmp, &vgic_cpu->ap_list_head, ap_list) {
if (irq_is_lpi(vcpu->kvm, irq->intid)) {
raw_spin_lock(&irq->irq_lock);
irq->pending_latch = false;
list_del(&irq->ap_list);
irq->vcpu = NULL;
raw_spin_unlock(&irq->irq_lock);
@ -792,7 +793,11 @@ static void vgic_prune_ap_list(struct kvm_vcpu *vcpu)
continue;
}
/* This interrupt looks like it has to be migrated. */
/*
* This interrupt looks like it has to be migrated,
* make sure it is kept alive while locks are dropped.
*/
vgic_get_irq_ref(irq);
raw_spin_unlock(&irq->irq_lock);
raw_spin_unlock(&vgic_cpu->ap_list_lock);
@ -815,15 +820,16 @@ static void vgic_prune_ap_list(struct kvm_vcpu *vcpu)
raw_spin_lock(&irq->irq_lock);
/*
* If the affinity has been preserved, move the
* interrupt around. Otherwise, it means things have
* changed while the interrupt was unlocked, and we
* need to replay this.
* If the interrupt is still ours and its affinity has
* been preserved, move it around. Otherwise, it means
* things have changed while the interrupt was unlocked
* (it may even have been taken off the list with its
* affinity left untouched), and we need to replay this.
*
* In all cases, we cannot trust the list not to have
* changed, so we restart from the beginning.
*/
if (target_vcpu == vgic_target_oracle(irq)) {
if (irq->vcpu == vcpu && target_vcpu == vgic_target_oracle(irq)) {
struct vgic_cpu *new_cpu = &target_vcpu->arch.vgic_cpu;
list_del(&irq->ap_list);
@ -836,6 +842,8 @@ static void vgic_prune_ap_list(struct kvm_vcpu *vcpu)
raw_spin_unlock(&vcpuB->arch.vgic_cpu.ap_list_lock);
raw_spin_unlock(&vcpuA->arch.vgic_cpu.ap_list_lock);
deleted_lpis |= vgic_put_irq_norelease(vcpu->kvm, irq);
if (target_vcpu_needs_kick) {
kvm_make_request(KVM_REQ_IRQ_PENDING, target_vcpu);
kvm_vcpu_kick(target_vcpu);

2
arch/s390/kvm/dat.c
View File

@ -570,6 +570,8 @@ static long dat_crste_walk_range(gfn_t start, gfn_t end, struct crst_table *tabl
else if (walk->ops->pte_entry)
rc = dat_pte_walk_range(max(start, cur), min(end, next),
dereference_pmd(crste.pmd), walk);
if (rc)
break;
}
}
return rc;

7
arch/s390/kvm/gmap.c
View File

@ -1374,8 +1374,13 @@ struct gmap *gmap_create_shadow(struct kvm_s390_mmu_cache *mc, struct gmap *pare
/* Only allow one real-space gmap shadow. */
list_for_each_entry(sg, &parent->children, list) {
if (sg->guest_asce.r) {
scoped_guard(write_lock, &parent->kvm->mmu_lock)
if (write_trylock(&parent->kvm->mmu_lock)) {
gmap_unshadow(sg);
write_unlock(&parent->kvm->mmu_lock);
} else {
gmap_put(new);
return ERR_PTR(-EAGAIN);
}
break;
}
}

6
arch/s390/kvm/kvm-s390.c
View File

@ -1280,8 +1280,10 @@ static int kvm_s390_vm_stop_migration(struct kvm *kvm)
* PGSTEs might have cmma_d set.
*/
WRITE_ONCE(kvm->arch.migration_mode, 0);
if (kvm->arch.use_cmma)
kvm_s390_sync_request_broadcast(kvm, KVM_REQ_STOP_MIGRATION);
if (!kvm->arch.use_cmma)
return 0;
kvm_s390_sync_request_broadcast(kvm, KVM_REQ_STOP_MIGRATION);
/* Clear cmma_d on all existing PGSTEs and set cmma_dirty_pages to 0. */
gmap_set_cmma_all_clean(kvm->arch.gmap);
atomic64_set(&kvm->arch.cmma_dirty_pages, 0);

12
arch/s390/kvm/pci.c
View File

@ -300,9 +300,14 @@ static int kvm_s390_pci_aif_enable(struct zpci_dev *zdev, struct zpci_fib *fib,
gaite->gisc = fib->fmt0.isc;
gaite->count++;
gaite->aisbo = fib->fmt0.aisbo;
gaite->aisb = virt_to_phys(page_address(aisb_page) + (fib->fmt0.aisb &
~PAGE_MASK));
if (fib->fmt0.sum == 1) {
gaite->aisbo = fib->fmt0.aisbo;
gaite->aisb = virt_to_phys(page_address(aisb_page) +
(fib->fmt0.aisb & ~PAGE_MASK));
} else {
gaite->aisbo = 0;
gaite->aisb = 0;
}
aift->kzdev[zdev->aisb] = zdev->kzdev;
spin_unlock_irq(&aift->gait_lock);
@ -328,6 +333,7 @@ static int kvm_s390_pci_aif_enable(struct zpci_dev *zdev, struct zpci_fib *fib,
unpin1:
unpin_user_page(aibv_page);
out:
kvm_s390_gisc_unregister(kvm, fib->fmt0.isc);
return rc;
}

4
arch/x86/kvm/irq.c
View File

@ -488,8 +488,10 @@ int kvm_arch_irq_bypass_add_producer(struct irq_bypass_consumer *cons,
if (irqfd->irq_entry.type == KVM_IRQ_ROUTING_MSI) {
ret = kvm_pi_update_irte(irqfd, &irqfd->irq_entry);
if (ret)
if (ret) {
kvm->arch.nr_possible_bypass_irqs--;
irqfd->producer = NULL;
}
}
spin_unlock_irq(&kvm->irqfds.lock);

8
arch/x86/kvm/lapic.c
View File

@ -3371,6 +3371,12 @@ static void apic_sync_pv_eoi_from_guest(struct kvm_vcpu *vcpu,
struct kvm_lapic *apic)
{
int vector;
if (unlikely(!pv_eoi_enabled(vcpu))) {
__clear_bit(KVM_APIC_PV_EOI_PENDING, &vcpu->arch.apic_attention);
return;
}
/*
* PV EOI state is derived from KVM_APIC_PV_EOI_PENDING in host
* and KVM_PV_EOI_ENABLED in guest memory as follows:
@ -3382,8 +3388,6 @@ static void apic_sync_pv_eoi_from_guest(struct kvm_vcpu *vcpu,
* KVM_APIC_PV_EOI_PENDING is set, KVM_PV_EOI_ENABLED is unset:
* -> host enabled PV EOI, guest executed EOI.
*/
BUG_ON(!pv_eoi_enabled(vcpu));
if (pv_eoi_test_and_clr_pending(vcpu))
return;
vector = apic_set_eoi(apic);

6
arch/x86/kvm/svm/sev.c
View File

@ -2129,8 +2129,9 @@ int sev_vm_move_enc_context_from(struct kvm *kvm, unsigned int source_fd)
if (ret)
return ret;
/* Do not allow SNP VM migration until additional state transfer is implemented */
if (kvm->arch.vm_type != source_kvm->arch.vm_type ||
sev_guest(kvm) || !sev_guest(source_kvm)) {
sev_guest(kvm) || !sev_guest(source_kvm) || sev_snp_guest(source_kvm)) {
ret = -EINVAL;
goto out_unlock;
}
@ -2851,8 +2852,9 @@ int sev_vm_copy_enc_context_from(struct kvm *kvm, unsigned int source_fd)
* disallow out-of-band SEV/SEV-ES init if the target is already an
* SEV guest, or if vCPUs have been created. KVM relies on vCPUs being
* created after SEV/SEV-ES initialization, e.g. to init intercepts.
* Also do not allow SNP VM mirroring until additional state transfer is implemented.
*/
if (sev_guest(kvm) || !sev_guest(source_kvm) ||
if (sev_guest(kvm) || !sev_guest(source_kvm) || sev_snp_guest(source_kvm) ||
is_mirroring_enc_context(source_kvm) || kvm->created_vcpus) {
ret = -EINVAL;
goto e_unlock;

7
arch/x86/kvm/svm/svm.c
View File

@ -571,7 +571,12 @@ static int svm_enable_virtualization_cpu(void)
return r;
sd = per_cpu_ptr(&svm_data, me);
sd->asid_generation = 1;
/*
* Bump the current asid_generation value to ensure any vCPU that
* previously ran on this CPU sees a stale generation and is forced
* to acquire a new ASID, preventing a latent ASID collision.
*/
sd->asid_generation++;
sd->max_asid = cpuid_ebx(SVM_CPUID_FUNC) - 1;
sd->next_asid = sd->max_asid + 1;
sd->min_asid = max_sev_asid + 1;

2
arch/x86/kvm/trace.h
View File

@ -490,7 +490,7 @@ TRACE_EVENT(kvm_inj_exception,
TP_printk("%s%s%s%s%s",
__print_symbolic(__entry->exception, kvm_trace_sym_exc),
!__entry->has_error ? "" : " (",
!__entry->has_error ? "" : __print_symbolic(__entry->error_code, { }),
!__entry->has_error ? "" : __print_symbolic(__entry->error_code),
!__entry->has_error ? "" : ")",
__entry->reinjected ? " [reinjected]" : "")
);

89
arch/x86/kvm/vmx/nested.c
View File

@ -582,6 +582,9 @@ static int nested_vmx_check_msr_bitmap_controls(struct kvm_vcpu *vcpu,
static int nested_vmx_check_tpr_shadow_controls(struct kvm_vcpu *vcpu,
struct vmcs12 *vmcs12)
{
gpa_t vtpr_gpa = vmcs12->virtual_apic_page_addr + APIC_TASKPRI;
u32 vtpr;
if (!nested_cpu_has(vmcs12, CPU_BASED_TPR_SHADOW))
return 0;
@ -591,6 +594,32 @@ static int nested_vmx_check_tpr_shadow_controls(struct kvm_vcpu *vcpu,
if (CC(!nested_cpu_has_vid(vmcs12) && vmcs12->tpr_threshold >> 4))
return -EINVAL;
/*
* Do the illegal vTPR vs. TPR Threshold consistency check if and only
* if KVM is configured to WARN on missed consistency checks, otherwise
* it's a waste of time. KVM needs to rely on hardware to fully detect
* an illegal combination due to the vTPR being writable by L1 at all
* times (it's an in-memory value, not a VMCS field). I.e. even if the
* check passes now, it might fail at the actual VM-Enter.
*
* If reading guest memory fails, skip the check as KVM's de facto ABI
* for VMX instruction accesses to non-existent memory is to provide
* PCI Bus Error semantics (reads return 0xFFs), in which case the vTPR
* is guaranteed to greater than or equal to the threshold.
*
* Note! Deliberately use the VM-scoped API when reading guest memory,
* to ensure the read doesn't hit SMRAM when restoring L2 state on RSM,
* and only perform the check when in KVM_RUN, to avoid a false failure
* if userspace hasn't yet configured memslots during state restore.
*/
if (warn_on_missed_cc && vcpu->wants_to_run &&
nested_cpu_has(vmcs12, CPU_BASED_TPR_SHADOW) &&
!nested_cpu_has_vid(vmcs12) &&
!nested_cpu_has2(vmcs12, SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES) &&
!kvm_read_guest(vcpu->kvm, vtpr_gpa, &vtpr, sizeof(vtpr)) &&
CC((vmcs12->tpr_threshold & GENMASK(3, 0)) > ((vtpr >> 4) & GENMASK(3, 0))))
return -EINVAL;
return 0;
}
@ -3104,38 +3133,6 @@ static int nested_vmx_check_controls(struct kvm_vcpu *vcpu,
return 0;
}
static int nested_vmx_check_controls_late(struct kvm_vcpu *vcpu,
struct vmcs12 *vmcs12)
{
void *vapic = to_vmx(vcpu)->nested.virtual_apic_map.hva;
u32 vtpr = vapic ? (*(u32 *)(vapic + APIC_TASKPRI)) >> 4 : 0;
/*
* Don't bother with the consistency checks if KVM isn't configured to
* WARN on missed consistency checks, as KVM needs to rely on hardware
* to fully detect an illegal vTPR vs. TRP Threshold combination due to
* the vTPR being writable by L1 at all times (it's an in-memory value,
* not a VMCS field). I.e. even if the check passes now, it might fail
* at the actual VM-Enter.
*
* Keying off the module param also allows treating an invalid vAPIC
* mapping as a consistency check failure without increasing the risk
* of breaking a "real" VM.
*/
if (!warn_on_missed_cc)
return 0;
if ((exec_controls_get(to_vmx(vcpu)) & CPU_BASED_TPR_SHADOW) &&
nested_cpu_has(vmcs12, CPU_BASED_TPR_SHADOW) &&
!nested_cpu_has_vid(vmcs12) &&
!nested_cpu_has2(vmcs12, SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES) &&
(CC(!vapic) ||
CC((vmcs12->tpr_threshold & GENMASK(3, 0)) > (vtpr & GENMASK(3, 0)))))
return -EINVAL;
return 0;
}
static int nested_vmx_check_address_space_size(struct kvm_vcpu *vcpu,
struct vmcs12 *vmcs12)
{
@ -3661,19 +3658,14 @@ enum nvmx_vmentry_status nested_vmx_enter_non_root_mode(struct kvm_vcpu *vcpu,
&vmx->nested.pre_vmenter_ssp_tbl);
/*
* Overwrite vmcs01.GUEST_CR3 with L1's CR3 if EPT is disabled. In the
* event of a "late" VM-Fail, i.e. a VM-Fail detected by hardware but
* not KVM, KVM must unwind its software model to the pre-VM-Entry host
* state. When EPT is disabled, GUEST_CR3 holds KVM's shadow CR3, not
* L1's "real" CR3, which causes nested_vmx_restore_host_state() to
* corrupt vcpu->arch.cr3. Stuffing vmcs01.GUEST_CR3 results in the
* unwind naturally setting arch.cr3 to the correct value. Smashing
* vmcs01.GUEST_CR3 is safe because nested VM-Exits, and the unwind,
* reset KVM's MMU, i.e. vmcs01.GUEST_CR3 is guaranteed to be
* overwritten with a shadow CR3 prior to re-entering L1.
* Stash L1's CR3, so that in the event of a "late" VM-Fail, i.e. a
* VM-Fail detected by hardware but not KVM, KVM can unwind its
* software model to the pre-VM-Entry host state. When EPT is
* disabled, GUEST_CR3 holds KVM's shadow CR3, not L1's "real" CR3,
* and so simply restoring from vmcs01.GUEST_CR3 would corrupt
* vcpu->arch.cr3.
*/
if (!enable_ept)
vmcs_writel(GUEST_CR3, vcpu->arch.cr3);
vmx->nested.pre_vmenter_cr3 = kvm_read_cr3(vcpu);
vmx_switch_vmcs(vcpu, &vmx->nested.vmcs02);
@ -3685,11 +3677,6 @@ enum nvmx_vmentry_status nested_vmx_enter_non_root_mode(struct kvm_vcpu *vcpu,
return NVMX_VMENTRY_KVM_INTERNAL_ERROR;
}
if (nested_vmx_check_controls_late(vcpu, vmcs12)) {
vmx_switch_vmcs(vcpu, &vmx->vmcs01);
return NVMX_VMENTRY_VMFAIL;
}
if (nested_vmx_check_guest_state(vcpu, vmcs12,
&entry_failure_code)) {
exit_reason.basic = EXIT_REASON_INVALID_STATE;
@ -3774,6 +3761,8 @@ enum nvmx_vmentry_status nested_vmx_enter_non_root_mode(struct kvm_vcpu *vcpu,
if (!from_vmentry)
return NVMX_VMENTRY_VMEXIT;
nested_put_vmcs12_pages(vcpu);
load_vmcs12_host_state(vcpu, vmcs12);
vmcs12->vm_exit_reason = exit_reason.full;
if (enable_shadow_vmcs || nested_vmx_is_evmptr12_valid(vmx))
@ -4990,7 +4979,7 @@ static void nested_vmx_restore_host_state(struct kvm_vcpu *vcpu)
vmx_set_cr4(vcpu, vmcs_readl(CR4_READ_SHADOW));
nested_ept_uninit_mmu_context(vcpu);
vcpu->arch.cr3 = vmcs_readl(GUEST_CR3);
vcpu->arch.cr3 = vmx->nested.pre_vmenter_cr3;
kvm_register_mark_available(vcpu, VCPU_REG_CR3);
/*

6
arch/x86/kvm/vmx/tdx.c
View File

@ -2797,7 +2797,11 @@ static int tdx_td_init(struct kvm *kvm, struct kvm_tdx_cmd *cmd)
goto out;
}
if (init_vm->cpuid.padding) {
/*
* Reject the request if userspace changes cpuid.nent between the
* initial read and the subsequent copy.
*/
if (init_vm->cpuid.padding || init_vm->cpuid.nent != nr_user_entries) {
ret = -EINVAL;
goto out;
}

7
arch/x86/kvm/vmx/vmx.h
View File

@ -159,6 +159,13 @@ struct nested_vmx {
bool has_preemption_timer_deadline;
bool preemption_timer_expired;
/*
* Used to restore L1's CR3 if hardware detects a VM-Fail Consistency
* Check that KVM does not, in which case KVM needs to unwind CR3 back
* to its pre-VM-Enter state, NOT to vmcs01.HOST_CR3.
*/
unsigned long pre_vmenter_cr3;
/*
* Used to snapshot MSRs that are conditionally loaded on VM-Enter in
* order to propagate the guest's pre-VM-Enter value into vmcs02. For

25
drivers/firmware/arm_ffa/driver.c
View File

@ -713,30 +713,39 @@ ffa_setup_and_transmit(u32 func_id, void *buffer, u32 max_fragsize,
struct ffa_composite_mem_region *composite;
struct ffa_mem_region_addr_range *constituents;
struct ffa_mem_region_attributes *ep_mem_access;
u32 idx, frag_len, length, buf_sz = 0, num_entries = sg_nents(args->sg);
u32 idx, frag_len, length, buf_sz = 0, num_entries = sg_nents(args->sg), ep_offset;
u32 emad_end, emad_size = ffa_emad_size_get(drv_info->version);
mem_region->tag = args->tag;
mem_region->flags = args->flags;
mem_region->sender_id = drv_info->vm_id;
mem_region->attributes = ffa_memory_attributes_get(func_id);
ffa_mem_region_additional_setup(drv_info->version, mem_region);
composite_offset = ffa_mem_desc_offset(buffer, args->nattrs,
drv_info->version);
if (composite_offset + sizeof(*composite) > max_fragsize)
return -ENXIO;
for (idx = 0; idx < args->nattrs; idx++) {
ep_mem_access = buffer +
ffa_mem_desc_offset(buffer, idx, drv_info->version);
ep_offset = ffa_mem_desc_offset(buffer, idx, drv_info->version);
if (check_add_overflow(ep_offset, emad_size, &emad_end))
return -ENXIO;
if (emad_end > max_fragsize)
return -ENXIO;
ep_mem_access = buffer + ep_offset;
memset(ep_mem_access, 0, emad_size);
ep_mem_access->receiver = args->attrs[idx].receiver;
ep_mem_access->attrs = args->attrs[idx].attrs;
ep_mem_access->composite_off = composite_offset;
ep_mem_access->flag = 0;
ep_mem_access->reserved = 0;
ffa_emad_impdef_value_init(drv_info->version,
ep_mem_access->impdef_val,
args->attrs[idx].impdef_val);
}
mem_region->handle = 0;
mem_region->ep_count = args->nattrs;
ffa_mem_region_additional_setup(drv_info->version, mem_region);
composite = buffer + composite_offset;
composite->total_pg_cnt = ffa_get_num_pages_sg(args->sg);
@ -769,7 +778,7 @@ ffa_setup_and_transmit(u32 func_id, void *buffer, u32 max_fragsize,
constituents = buffer;
}
if ((void *)constituents - buffer > max_fragsize) {
if ((void *)constituents + sizeof(*constituents) - buffer > max_fragsize) {
pr_err("Memory Region Fragment > Tx Buffer size\n");
return -EFAULT;
}
@ -778,7 +787,7 @@ ffa_setup_and_transmit(u32 func_id, void *buffer, u32 max_fragsize,
constituents->pg_cnt = args->sg->length / FFA_PAGE_SIZE;
constituents->reserved = 0;
constituents++;
frag_len += sizeof(struct ffa_mem_region_addr_range);
frag_len += sizeof(*constituents);
} while ((args->sg = sg_next(args->sg)));
return ffa_transmit_fragment(func_id, addr, buf_sz, frag_len,

9
include/linux/arm_ffa.h
View File

@ -421,6 +421,13 @@ struct ffa_mem_region {
#define FFA_EMAD_HAS_IMPDEF_FIELD(version) ((version) >= FFA_VERSION_1_2)
#define FFA_MEM_REGION_HAS_EP_MEM_OFFSET(version) ((version) > FFA_VERSION_1_0)
/* The layout changed from FFA_VERSION_1_0 and the region includes an
* ep_mem_offset.
*/
#define FFA_MEM_REGION_SZ(version) (!FFA_MEM_REGION_HAS_EP_MEM_OFFSET((version)) ?\
offsetof(struct ffa_mem_region, ep_mem_offset) :\
sizeof(struct ffa_mem_region))
static inline u32 ffa_emad_size_get(u32 ffa_version)
{
u32 sz;
@ -445,7 +452,7 @@ ffa_mem_desc_offset(struct ffa_mem_region *buf, int count, u32 ffa_version)
if (!FFA_MEM_REGION_HAS_EP_MEM_OFFSET(ffa_version))
offset += offsetof(struct ffa_mem_region, ep_mem_offset);
else
offset += sizeof(struct ffa_mem_region);
offset += buf->ep_mem_offset;
return offset;
}

1
tools/testing/selftests/kvm/Makefile.kvm
View File

@ -174,6 +174,7 @@ TEST_GEN_PROGS_arm64 += arm64/hello_el2
TEST_GEN_PROGS_arm64 += arm64/host_sve
TEST_GEN_PROGS_arm64 += arm64/hypercalls
TEST_GEN_PROGS_arm64 += arm64/external_aborts
TEST_GEN_PROGS_arm64 += arm64/mmio_sign_ext
TEST_GEN_PROGS_arm64 += arm64/page_fault_test
TEST_GEN_PROGS_arm64 += arm64/psci_test
TEST_GEN_PROGS_arm64 += arm64/sea_to_user

255
tools/testing/selftests/kvm/arm64/mmio_sign_ext.c Normal file
View File

@ -0,0 +1,255 @@
// SPDX-License-Identifier: GPL-2.0-only
/*
* mmio_sign_ext - Test sign-extending MMIO load emulation (LDRSB/LDRSH/LDRSW)
*
* Copyright (c) 2026 Google LLC
* Author: Fuad Tabba <fuad.tabba@linux.dev>
*/
#include <asm/ptrace.h>
#include "processor.h"
#include "test_util.h"
#define MMIO_ADDR 0x8000000ULL
/* AP[1]: allow unprivileged (EL0) access to a mapping. */
#define PTE_USER BIT(6)
/* SPSR for ERET to EL0t with DAIF masked. */
#define SPSR_EL0 (PSR_MODE_EL0t | PSR_D_BIT | PSR_A_BIT | PSR_I_BIT | PSR_F_BIT)
struct mmio_test {
const char *name;
uint64_t data; /* access-width value, host byte order */
uint8_t len;
uint64_t expected; /* sign-extended result; same for LE and BE */
};
/* Paired 1:1, in order, with the loads in guest_loads_le() and el0_be_loads. */
static const struct mmio_test tests[] = {
/* LDRSB Xt: byte sign-extended to 64 bits */
{ "LDRSB Xt 0xFF", 0xFF, 1, 0xFFFFFFFFFFFFFFFFULL },
{ "LDRSB Xt 0x7F", 0x7F, 1, 0x7FULL },
/* LDRSB Wt: byte sign-extended to 32 bits, upper 32 bits zeroed */
{ "LDRSB Wt 0xFF", 0xFF, 1, 0xFFFFFFFFULL },
{ "LDRSB Wt 0x7F", 0x7F, 1, 0x7FULL },
/* LDRSH Xt: halfword sign-extended to 64 bits */
{ "LDRSH Xt 0x8001", 0x8001, 2, 0xFFFFFFFFFFFF8001ULL },
{ "LDRSH Xt 0x7FFF", 0x7FFF, 2, 0x7FFFULL },
/* LDRSH Wt: halfword sign-extended to 32 bits, upper 32 bits zeroed */
{ "LDRSH Wt 0x8001", 0x8001, 2, 0xFFFF8001ULL },
{ "LDRSH Wt 0x7FFF", 0x7FFF, 2, 0x7FFFULL },
/* LDRSW Xt: word sign-extended to 64 bits (no Wt form) */
{ "LDRSW Xt 0x80000001", 0x80000001, 4, 0xFFFFFFFF80000001ULL },
{ "LDRSW Xt 0x7FFFFFFF", 0x7FFFFFFF, 4, 0x7FFFFFFFULL },
};
/* Issue one sign-extending load from MMIO and report the result. */
#define GUEST_LDRS(load) do { \
uint64_t val; \
\
asm volatile(load : "=r"(val) : "r"(MMIO_ADDR) : "memory"); \
GUEST_SYNC(val); \
} while (0)
/* Little-endian pass: loads issued at EL1. */
static void guest_loads_le(void)
{
GUEST_LDRS("ldrsb %0, [%1]");
GUEST_LDRS("ldrsb %0, [%1]");
GUEST_LDRS("ldrsb %w0, [%1]");
GUEST_LDRS("ldrsb %w0, [%1]");
GUEST_LDRS("ldrsh %0, [%1]");
GUEST_LDRS("ldrsh %0, [%1]");
GUEST_LDRS("ldrsh %w0, [%1]");
GUEST_LDRS("ldrsh %w0, [%1]");
GUEST_LDRS("ldrsw %0, [%1]");
GUEST_LDRS("ldrsw %0, [%1]");
}
/*
* Run the big-endian loads at EL0, where SCTLR_EL1.E0E flips only the data
* endianness; at EL1, SCTLR_EL1.EE would also flip the page-table walk and
* fault on the little-endian tables. x0 holds MMIO_ADDR; results return in
* x19..x28 (tests[] order) via a single SVC.
*/
extern char el0_be_loads[];
asm(
" .pushsection .text, \"ax\"\n"
" .global el0_be_loads\n"
"el0_be_loads:\n"
" ldrsb x19, [x0]\n"
" ldrsb x20, [x0]\n"
" ldrsb w21, [x0]\n"
" ldrsb w22, [x0]\n"
" ldrsh x23, [x0]\n"
" ldrsh x24, [x0]\n"
" ldrsh w25, [x0]\n"
" ldrsh w26, [x0]\n"
" ldrsw x27, [x0]\n"
" ldrsw x28, [x0]\n"
" svc #0\n"
" .popsection\n"
);
/* EL1 handler for the EL0 SVC: report the results, then finish. */
static void el0_svc_handler(struct ex_regs *regs)
{
int i;
for (i = 0; i < ARRAY_SIZE(tests); i++)
GUEST_SYNC(regs->regs[19 + i]);
GUEST_DONE();
}
static bool guest_mixed_endian_el0(void)
{
uint64_t mmfr0 = read_sysreg(id_aa64mmfr0_el1);
return SYS_FIELD_GET(ID_AA64MMFR0_EL1, BIGEND, mmfr0) ||
SYS_FIELD_GET(ID_AA64MMFR0_EL1, BIGENDEL0, mmfr0);
}
static void guest_code(void)
{
guest_loads_le();
if (guest_mixed_endian_el0()) {
write_sysreg(read_sysreg(sctlr_el1) | SCTLR_EL1_E0E, sctlr_el1);
isb();
asm volatile(
" msr elr_el1, %[pc]\n"
" msr spsr_el1, %[spsr]\n"
" mov x0, %[mmio]\n"
" isb\n"
" eret\n"
:
: [pc] "r"(el0_be_loads),
[spsr] "r"((uint64_t)SPSR_EL0),
[mmio] "r"(MMIO_ADDR)
: "x0", "memory");
__builtin_unreachable(); /* el0_svc_handler ends the test */
}
GUEST_DONE();
}
static void handle_mmio(struct kvm_run *run, const struct mmio_test *t, bool be)
{
int i;
TEST_ASSERT_EQ(run->mmio.phys_addr, MMIO_ADDR);
TEST_ASSERT(!run->mmio.is_write, "Expected MMIO read for %s", t->name);
TEST_ASSERT_EQ(run->mmio.len, t->len);
memset(run->mmio.data, 0, sizeof(run->mmio.data));
if (be) {
/* The guest reads the device bytes most-significant first. */
for (i = 0; i < t->len; i++)
run->mmio.data[i] = t->data >> (8 * (t->len - 1 - i));
} else {
/* Works because arm64 KVM hosts are always little-endian. */
memcpy(run->mmio.data, &t->data, t->len);
}
}
static void expect_sync(struct kvm_vcpu *vcpu, struct ucall *uc,
const struct mmio_test *t)
{
switch (get_ucall(vcpu, uc)) {
case UCALL_SYNC:
TEST_ASSERT(uc->args[1] == t->expected,
"%s: got %#lx, want %#lx", t->name,
(unsigned long)uc->args[1], (unsigned long)t->expected);
break;
case UCALL_ABORT:
REPORT_GUEST_ASSERT(*uc);
break;
default:
TEST_FAIL("Unexpected ucall for %s", t->name);
}
}
/* OR PTE_USER into the leaf descriptors covering [gva, gva + len). */
static void make_el0_accessible(struct kvm_vm *vm, uint64_t gva, uint64_t len)
{
uint64_t addr;
for (addr = gva & ~((uint64_t)vm->page_size - 1); addr < gva + len;
addr += vm->page_size)
*virt_get_pte_hva(vm, addr) |= PTE_USER;
}
static bool vcpu_mixed_endian_el0(struct kvm_vcpu *vcpu)
{
uint64_t mmfr0 = vcpu_get_reg(vcpu, KVM_ARM64_SYS_REG(SYS_ID_AA64MMFR0_EL1));
return SYS_FIELD_GET(ID_AA64MMFR0_EL1, BIGEND, mmfr0) ||
SYS_FIELD_GET(ID_AA64MMFR0_EL1, BIGENDEL0, mmfr0);
}
int main(void)
{
struct kvm_vcpu *vcpu;
struct kvm_vm *vm;
struct ucall uc;
unsigned int i;
bool be;
vm = vm_create_with_one_vcpu(&vcpu, guest_code);
virt_map(vm, MMIO_ADDR, MMIO_ADDR, 1);
vm_init_descriptor_tables(vm);
vcpu_init_descriptor_tables(vcpu);
vm_install_sync_handler(vm, VECTOR_SYNC_LOWER_64, ESR_ELx_EC_SVC64,
el0_svc_handler);
be = vcpu_mixed_endian_el0(vcpu);
if (be)
make_el0_accessible(vm, MMIO_ADDR, vm->page_size);
ksft_print_header();
ksft_set_plan(ARRAY_SIZE(tests) * (be ? 2 : 1));
/* Little-endian pass: one load and one result per iteration. */
for (i = 0; i < ARRAY_SIZE(tests); i++) {
const struct mmio_test *t = &tests[i];
vcpu_run(vcpu);
TEST_ASSERT_KVM_EXIT_REASON(vcpu, KVM_EXIT_MMIO);
handle_mmio(vcpu->run, t, false);
vcpu_run(vcpu);
expect_sync(vcpu, &uc, t);
ksft_test_result_pass("%s\n", t->name);
}
if (be) {
/* The EL0 stub issues all the loads, then reports the results. */
for (i = 0; i < ARRAY_SIZE(tests); i++) {
vcpu_run(vcpu);
TEST_ASSERT_KVM_EXIT_REASON(vcpu, KVM_EXIT_MMIO);
handle_mmio(vcpu->run, &tests[i], true);
}
for (i = 0; i < ARRAY_SIZE(tests); i++) {
vcpu_run(vcpu);
expect_sync(vcpu, &uc, &tests[i]);
ksft_test_result_pass("BE %s\n", tests[i].name);
}
}
vcpu_run(vcpu);
TEST_ASSERT(get_ucall(vcpu, &uc) == UCALL_DONE, "Expected UCALL_DONE");
kvm_vm_free(vm);
ksft_finished();
}

47
tools/testing/selftests/kvm/x86/sev_migrate_tests.c
View File

@ -313,6 +313,49 @@ static void test_sev_mirror_parameters(void)
kvm_vm_free(vm_no_vcpu);
}
static void test_sev_snp_migrate_reject(void)
{
struct kvm_vm *src_vm, *dst_vm;
int ret;
src_vm = vm_create_barebones_type(KVM_X86_SNP_VM);
snp_vm_init(src_vm);
__vm_vcpu_add(src_vm, 0);
vm_sev_launch(src_vm, snp_default_policy(), NULL);
dst_vm = vm_create_barebones_type(KVM_X86_SNP_VM);
__vm_vcpu_add(dst_vm, 0);
ret = __sev_migrate_from(dst_vm, src_vm);
TEST_ASSERT(ret == -1 && errno == EINVAL,
"SNP VM migration should be rejected. ret: %d, errno: %d",
ret, errno);
kvm_vm_free(src_vm);
kvm_vm_free(dst_vm);
}
static void test_sev_snp_mirror_reject(void)
{
struct kvm_vm *src_vm, *dst_vm;
int ret;
src_vm = vm_create_barebones_type(KVM_X86_SNP_VM);
snp_vm_init(src_vm);
__vm_vcpu_add(src_vm, 0);
vm_sev_launch(src_vm, snp_default_policy(), NULL);
dst_vm = aux_vm_create(false);
ret = __sev_mirror_create(dst_vm, src_vm);
TEST_ASSERT(ret == -1 && errno == EINVAL,
"SNP VM mirroring should be rejected. ret: %d, errno: %d",
ret, errno);
kvm_vm_free(src_vm);
kvm_vm_free(dst_vm);
}
static void test_sev_move_copy(void)
{
struct kvm_vm *dst_vm, *dst2_vm, *dst3_vm, *sev_vm, *mirror_vm,
@ -384,12 +427,16 @@ int main(int argc, char *argv[])
test_sev_migrate_parameters();
if (kvm_has_cap(KVM_CAP_VM_COPY_ENC_CONTEXT_FROM))
test_sev_move_copy();
if (kvm_cpu_has(X86_FEATURE_SEV_SNP))
test_sev_snp_migrate_reject();
}
if (kvm_has_cap(KVM_CAP_VM_COPY_ENC_CONTEXT_FROM)) {
test_sev_mirror(/* es= */ false);
if (have_sev_es)
test_sev_mirror(/* es= */ true);
test_sev_mirror_parameters();
if (kvm_cpu_has(X86_FEATURE_SEV_SNP))
test_sev_snp_mirror_reject();
}
return 0;
}

16
virt/kvm/kvm_main.c
View File

@ -6069,25 +6069,19 @@ struct kvm_io_device *kvm_io_bus_get_dev(struct kvm *kvm, enum kvm_bus bus_idx,
gpa_t addr)
{
struct kvm_io_bus *bus;
int dev_idx, srcu_idx;
struct kvm_io_device *iodev = NULL;
int dev_idx;
srcu_idx = srcu_read_lock(&kvm->srcu);
lockdep_assert_held(&kvm->srcu);
bus = kvm_get_bus_srcu(kvm, bus_idx);
if (!bus)
goto out_unlock;
return NULL;
dev_idx = kvm_io_bus_get_first_dev(bus, addr, 1);
if (dev_idx < 0)
goto out_unlock;
return NULL;
iodev = bus->range[dev_idx].dev;
out_unlock:
srcu_read_unlock(&kvm->srcu, srcu_idx);
return iodev;
return bus->range[dev_idx].dev;
}
EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_io_bus_get_dev);