github-mirror/linux
1
0
Fork 0
mirror of https://github.com/torvalds/linux.git synced 2026-09-27 11:02:03 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity Graph

Arm:

Browse Source 
- Invalidate the ITS translation cache when the guest changes the
   base address of the ITS tables (Fuad Tabba)
 
 - Skip saving ITS devices with device IDs that are out-of-bounds
   rather than failing the entire ITS save ioctl (Fuad Tabba)
 
 - Close race between VM teardown and invalidations of nested MMUs
   when handling MMU operations that are allowed to block
   (Lorenzo Stoakes)
 
 - Various fixes for the handling of the host's untrusted SVE
   configuration in pKVM (Fuad Tabba)
 
 - Make sure that empty SMCCC ranges based at 0 are rejected by the
   kvm_smccc_set_filter() (Karl Mehltretter)
 
 - Revoke the host mapping for pKVM's private stack pages, along
   with a new sanity check that all mappings in the hyp's private
   VA range have been correctly marked as hyp-owned (Fuad Tabba)
 
 - Lifetime fixes for the array of shadow stage-2 MMUs, ensuring that
   concurrent vCPU initialization cannot relocate in-use MMUs. Defer
   the freeing of shadow stage-2 MMUs to the point that no other users
   (e.g. MMU notifier) could reference them (Marc Zyngier)
 
 - Drop useless WARN when rejecting an unsupported ioctl for pKVM
   (Fuad Tabba)
 
 - Fix the steal_time selftest to install correctly-sized mappings for
   non-4K hosts (Sebastian Ott)
 
 - Correct mapping of fine-grained trap for GCSPOPX instruction
   (Mark Brown)
 
 - Fix KVM_BUG_ON() due to missing handling of DBGBXVR<n> from 32-bit
   guests (Karl Mehltretter)
 
 RISC-V:
 
 - Synchronize hrtimer during VCPU teardown.
 
 - Fix the conversion between vsip and hvip values.
 
 - Serialize IMSIC attributes with vCPU migration.
 
 - Release unused page after MMU invalidation.
 
 - Propagate interrupted G-stage faults to KVM user-space as EINTR.
 
 - Fix nested acceleration hfence entry update order.
 
 - Fix sdata leak and stale snapshot_addr in snapshot_set_shmem.
 
 - Preserve firmware counter value across PMU counter stop/start.
 
 - Report PMU snapshot write failure to the guest.
 
 - Fix perf-backed counter accounting across PMU stop and read.
 
 - Correctly propagate error of a hart status SBI call.
 
 s390:
 
 - Ensure that accesses through kvm_arch_set_irq_inatomic mark as dirty
   the pages that contain indicator and summary bits.
 
 - Fix compile warning for kvm_s390_update_cmma_dirty().
 
 - Fix incorrect propagation of ENOENT from _gaccess_shadow_fault() to
   userspace.
 
 - Move s390_kvm_mmu_commit_memory_region() into
   s390_kvm_mmu_prepare_memory_region() so that it can fail instead of WARN.
 
 - Add missing srcu in kvm_s390_set_irq_state().
 
 - Fix potential races in storage functions.
 
 - Fix race in _destroy_pages_crste().
 
 - Fix issues in the handling of KVM interrupt and page resources, when a
   queue that is assigned to a mediated device (mdev) is removed from the
   host's AP configuration.
 
 - Fix loop condition in uv_find_secrets.
 
 - Prevent potential out-of-bounds read.
 
 x86:
 
 - Fix a brown paper bag bug where KVM would incorrectly treat Intel PMU MSRs
   as valid on AMD.
 
 - Fix a regression in the hardware disable selftest where it checked the wrong
   macro when detecting glibc support (breaks at least musl).
 
 - Never clear KVM_REQ_VM_DEAD so that dead VMs stay dead, which is especially
   important for KVM_BUG_ON() flows, which often guard more dangerous bugs.
 
 - Re-pend GET_NESTED_STATE_PAGES if getting the pages fails, to fix a bug
   where KVM would let userspace run a broken setup with stale vmcs12 pages.
 
 - Fix a class of bugs where KVM would fail to fill kvm_run exit fields if
   getting nested pages failed.
 
 - Treat reserved entries in the memory attributes xarray as "no attributes",
   to fix false positives when checking for mixed attributes.
 
 - Fix memcg accounting for the memory attributes xarray (the xarray library
   subtly requires the xarray to be configured for accounting upfront; the gfp
   flags taken at runtime are used only rarely).
 
 - Don't pre-reserve xarray entries when storing empty attributes, as storing
   NULL must not require memory allocation (KVM and other subsystems heavily
   rely on this behavior).
 
 - Fix a memory leak and a cache maintenance issue related to doing intra-host
   migration on an SEV guest.
 -----BEGIN PGP SIGNATURE-----
 
 iQFIBAABCAAyFiEE8TM4V0tmI4mGbHaCv/vSX3jHroMFAmq3Ta0UHHBib256aW5p
 QHJlZGhhdC5jb20ACgkQv/vSX3jHroMcygf/Z8GovgUACbWs/PSgVxZr5++MsbnM
 /N+vUP0VhsBxWCt191fBmKSBMNyTL734CKe0fkmisIAIPxf63uA3lttfMshNXKct
 VJ3TQkbA7qiSEGf2Um1dp4k6egqDDbTS1a4NF7CM7AEL4Wm3YDfqv2h4rZicEBsd
 Spy4yV8sKoiEHMnduhF2m5r0cWkrj194e2dolJy0p1WiNeEykf/nMTmNq5JWusBh
 TDDzZf1p2BianOmQ6+fkzAfWDSIdv0OPmoH7iqvQ/xhvm4dLWq2oeOz5Q2/uWAp0
 BqU4Z5zBrGRUWyXM0cDZEqBfmNH4v7oR9PfLgvD4qpMoqONuMj1+mIO68A==
 =XtLD
 -----END PGP SIGNATURE-----

Merge tag 'for-linus' of git://git.kernel.org/pub/scm/virt/kvm/kvm

Pull kvm fixes from Paolo Bonzini:
 "Arm:

   - Invalidate the ITS translation cache when the guest changes the
     base address of the ITS tables (Fuad Tabba)

   - Skip saving ITS devices with device IDs that are out-of-bounds
     rather than failing the entire ITS save ioctl (Fuad Tabba)

   - Close race between VM teardown and invalidations of nested MMUs
     when handling MMU operations that are allowed to block (Lorenzo
     Stoakes)

   - Various fixes for the handling of the host's untrusted SVE
     configuration in pKVM (Fuad Tabba)

   - Make sure that empty SMCCC ranges based at 0 are rejected by the
     kvm_smccc_set_filter() (Karl Mehltretter)

   - Revoke the host mapping for pKVM's private stack pages, along with
     a new sanity check that all mappings in the hyp's private VA range
     have been correctly marked as hyp-owned (Fuad Tabba)

   - Lifetime fixes for the array of shadow stage-2 MMUs, ensuring that
     concurrent vCPU initialization cannot relocate in-use MMUs. Defer
     the freeing of shadow stage-2 MMUs to the point that no other users
     (e.g. MMU notifier) could reference them (Marc Zyngier)

   - Drop useless WARN when rejecting an unsupported ioctl for pKVM
     (Fuad Tabba)

   - Fix the steal_time selftest to install correctly-sized mappings for
     non-4K hosts (Sebastian Ott)

   - Correct mapping of fine-grained trap for GCSPOPX instruction (Mark
     Brown)

   - Fix KVM_BUG_ON() due to missing handling of DBGBXVR<n> from 32-bit
     guests (Karl Mehltretter)

  RISC-V:

   - Synchronize hrtimer during VCPU teardown

   - Fix the conversion between vsip and hvip values

   - Serialize IMSIC attributes with vCPU migration

   - Release unused page after MMU invalidation

   - Propagate interrupted G-stage faults to KVM user-space as EINTR

   - Fix nested acceleration hfence entry update order

   - Fix sdata leak and stale snapshot_addr in snapshot_set_shmem

   - Preserve firmware counter value across PMU counter stop/start

   - Report PMU snapshot write failure to the guest

   - Fix perf-backed counter accounting across PMU stop and read

   - Correctly propagate error of a hart status SBI call

  s390:

   - Ensure that accesses through kvm_arch_set_irq_inatomic mark as
     dirty the pages that contain indicator and summary bits

   - Fix compile warning for kvm_s390_update_cmma_dirty()

   - Fix incorrect propagation of ENOENT from _gaccess_shadow_fault() to
     userspace

   - Move s390_kvm_mmu_commit_memory_region() into
     s390_kvm_mmu_prepare_memory_region() so that it can fail instead of
     WARN

   - Add missing srcu in kvm_s390_set_irq_state()

   - Fix potential races in storage functions

   - Fix race in _destroy_pages_crste()

   - Fix issues in the handling of KVM interrupt and page resources,
     when a queue that is assigned to a mediated device (mdev) is
     removed from the host's AP configuration

   - Fix loop condition in uv_find_secrets

   - Prevent potential out-of-bounds read

  x86:

   - Fix a brown paper bag bug where KVM would incorrectly treat Intel
     PMU MSRs as valid on AMD

   - Fix a regression in the hardware disable selftest where it checked
     the wrong macro when detecting glibc support (breaks at least musl)

   - Never clear KVM_REQ_VM_DEAD so that dead VMs stay dead, which is
     especially important for KVM_BUG_ON() flows, which often guard more
     dangerous bugs

   - Re-pend GET_NESTED_STATE_PAGES if getting the pages fails, to fix a
     bug where KVM would let userspace run a broken setup with stale
     vmcs12 pages

   - Fix a class of bugs where KVM would fail to fill kvm_run exit
     fields if getting nested pages failed

   - Treat reserved entries in the memory attributes xarray as "no
     attributes", to fix false positives when checking for mixed
     attributes

   - Fix memcg accounting for the memory attributes xarray (the xarray
     library subtly requires the xarray to be configured for accounting
     upfront; the gfp flags taken at runtime are used only rarely)

   - Don't pre-reserve xarray entries when storing empty attributes, as
     storing NULL must not require memory allocation (KVM and other
     subsystems heavily rely on this behavior)

   - Fix a memory leak and a cache maintenance issue related to doing
     intra-host migration on an SEV guest"

* tag 'for-linus' of git://git.kernel.org/pub/scm/virt/kvm/kvm: (54 commits)
  KVM: SEV: Do cache maintenance on the source VM during intra-host migration
  KVM: SEV: Free have_run_cpus during VM destruction even if VM is no longer SEV
  KVM: Don't pre-reserve xarray entries when storing empty/NULL attributes
  KVM: Ensure memory attributes xarray nodes are accounted to the caller's memcg
  KVM: Don't treat reserved xarray entries as having memory attributes
  KVM: x86: Fill kvm_run exit fields in common get_nested_state_pages() error paths
  KVM: x86: Re-pend GET_NESTED_STATE_PAGES if getting said pages fails
  KVM: arm64: Fix AArch32 DBGBXVR<n> handling
  KVM: arm64: Fix FGT mapping for HFGITR_EL2.nGCSEPP
  KVM: selftests: fix steal_time for arm64 with host page size > 4K
  KVM: arm64: Don't WARN on an unknown VM ioctl in protected mode
  KVM: arm64: nv: Delay freeing of shadow S2 structures until VM destruction
  KVM: arm64: nv: Fix life cycle of the nested_mmus array
  KVM: arm64: Check every private mapping is hyp-owned at pKVM init
  KVM: arm64: Move the private VA allocation cursor to __io_map_next
  KVM: arm64: Match hyp text by physical address in fix_host_ownership()
  KVM: arm64: Transfer the hyp stack pages out of the host stage-2
  KVM: arm64: selftests: Test empty SMCCC filter range at base 0
  KVM: arm64: Return -EINVAL for an empty SMCCC filter range at base 0
  KVM: arm64: Derive GUEST_HAS_SVE from the SVE feature bit at EL2
  ...
This commit is contained in:
Linus Torvalds 2026-09-26 08:26:12 -07:00
commit eff8d2791c
53 changed files with 1032 additions and 347 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
arch/arm64/include/asm/kvm_host.h
View File

@ -322,7 +322,7 @@ struct kvm_arch {
* Stage 2 paging state for VMs with nested S2 using a virtual
* VMID.
*/
struct kvm_s2_mmu *nested_mmus;
struct kvm_s2_mmu **nested_mmus;
size_t nested_mmus_size;
int nested_mmus_next;

3
arch/arm64/include/asm/kvm_nested.h
View File

@ -66,7 +66,8 @@ static inline u64 translate_ttbr0_el2_to_ttbr0_el1(u64 ttbr0)
extern bool forward_smc_trap(struct kvm_vcpu *vcpu);
extern bool forward_debug_exception(struct kvm_vcpu *vcpu);
extern void kvm_init_nested(struct kvm *kvm);
extern int kvm_init_nested(struct kvm *kvm);
extern void kvm_destroy_nested(struct kvm *kvm);
extern int kvm_vcpu_init_nested(struct kvm_vcpu *vcpu);
extern void kvm_init_nested_s2_mmu(struct kvm_s2_mmu *mmu);
extern struct kvm_s2_mmu *lookup_s2_mmu(struct kvm_vcpu *vcpu);

3
arch/arm64/include/asm/kvm_pkvm.h
View File

@ -65,8 +65,7 @@ static inline bool kvm_pkvm_ioctl_allowed(struct kvm *kvm, unsigned int ioctl)
int r;
r = kvm_get_cap_for_kvm_ioctl(ioctl, &ext);
if (WARN_ON_ONCE(r < 0))
if (r < 0)
return false;
return kvm_pkvm_ext_allowed(kvm, ext);

10
arch/arm64/kvm/arm.c
View File

@ -236,8 +236,6 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned long type)
mutex_unlock(&kvm->lock);
#endif
kvm_init_nested(kvm);
ret = kvm_share_hyp(kvm, kvm + 1);
if (ret)
return ret;
@ -252,6 +250,10 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned long type)
if (ret)
goto err_free_cpumask;
ret = kvm_init_nested(kvm);
if (ret)
goto err_uninit_mmu;
if (is_protected_kvm_enabled()) {
/*
* If any failures occur after this is successful, make sure to
@ -280,6 +282,7 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned long type)
err_uninit_mmu:
kvm_uninit_stage2_mmu(kvm);
kvm_destroy_nested(kvm);
err_free_cpumask:
free_cpumask_var(kvm->arch.supported_cpus);
err_unshare_kvm:
@ -337,6 +340,7 @@ void kvm_arch_destroy_vm(struct kvm *kvm)
kvm_unshare_hyp(kvm, kvm + 1);
kvm_destroy_nested(kvm);
kvm_arm_teardown_hypercalls(kvm);
}
@ -1129,7 +1133,7 @@ static int kvm_vcpu_suspend(struct kvm_vcpu *vcpu)
static int check_vcpu_requests(struct kvm_vcpu *vcpu)
{
if (kvm_request_pending(vcpu)) {
if (kvm_check_request(KVM_REQ_VM_DEAD, vcpu))
if (kvm_test_request(KVM_REQ_VM_DEAD, vcpu))
return -EIO;
if (kvm_check_request(KVM_REQ_SLEEP, vcpu))

2
arch/arm64/kvm/emulate-nested.c
View File

@ -1445,7 +1445,7 @@ static const struct encoding_to_trap_config encoding_to_fgt[] __initconst = {
SR_FGT(OP_AT_S1E1A, HFGITR, ATS1E1A, 1),
SR_FGT(OP_COSP_RCTX, HFGITR, COSPRCTX, 1),
SR_FGT(OP_GCSPUSHX, HFGITR, nGCSEPP, 0),
SR_FGT(OP_GCSPOPX, HFGITR, nGCSEPP, 0),
SR_FGT(OP_GCSPOPCX, HFGITR, nGCSEPP, 0),
SR_FGT(OP_GCSPUSHM, HFGITR, nGCSPUSHM_EL1, 0),
SR_FGT(OP_BRB_IALL, HFGITR, nBRBIALL, 0),
SR_FGT(OP_BRB_INJ, HFGITR, nBRBINJ, 0),

2
arch/arm64/kvm/hyp/include/nvhe/mem_protect.h
View File

@ -52,8 +52,10 @@ int __pkvm_host_test_clear_young_guest(u64 gfn, u64 nr_pages, bool mkold, struct
int __pkvm_host_mkyoung_guest(u64 gfn, struct pkvm_hyp_vcpu *vcpu);
bool addr_is_memory(phys_addr_t phys);
bool addr_is_hyp_text(phys_addr_t phys);
int host_stage2_idmap_locked(phys_addr_t addr, u64 size, enum kvm_pgtable_prot prot);
int host_stage2_set_owner_locked(phys_addr_t addr, u64 size, u8 owner_id);
bool host_stage2_pte_is_hyp_owned(kvm_pte_t pte);
int kvm_host_prepare_stage2(void *pgt_pool_base);
int kvm_guest_prepare_stage2(struct pkvm_hyp_vm *vm, void *pgd);
void kvm_guest_destroy_stage2(struct pkvm_hyp_vm *vm);

1
arch/arm64/kvm/hyp/include/nvhe/mm.h
View File

@ -29,6 +29,7 @@ int __pkvm_create_private_mapping(phys_addr_t phys, size_t size,
enum kvm_pgtable_prot prot,
unsigned long *haddr);
int pkvm_create_stack(phys_addr_t phys, unsigned long *haddr);
int pkvm_check_host_ownership(void);
int pkvm_alloc_private_va_range(size_t size, unsigned long *haddr);
#endif /* __KVM_HYP_MM_H */

20
arch/arm64/kvm/hyp/nvhe/mem_protect.c
View File

@ -450,6 +450,14 @@ bool addr_is_memory(phys_addr_t phys)
return !!find_mem_range(phys, &range);
}
bool addr_is_hyp_text(phys_addr_t phys)
{
phys_addr_t start = ALIGN_DOWN(__hyp_pa(__hyp_text_start), PAGE_SIZE);
phys_addr_t end = PAGE_ALIGN(__hyp_pa(__hyp_text_end));
return phys >= start && phys < end;
}
static bool is_in_mem_range(u64 addr, struct kvm_mem_range *range)
{
return range->start <= addr && addr < range->end;
@ -633,6 +641,18 @@ int host_stage2_set_owner_locked(phys_addr_t addr, u64 size, u8 owner_id)
return ret;
}
bool host_stage2_pte_is_hyp_owned(kvm_pte_t pte)
{
if (kvm_pte_valid(pte))
return false;
if (FIELD_GET(KVM_INVALID_PTE_TYPE_MASK, pte) !=
KVM_HOST_INVALID_PTE_TYPE_DONATION)
return false;
return FIELD_GET(KVM_HOST_DONATION_PTE_OWNER_MASK, pte) == PKVM_ID_HYP;
}
#define KVM_HOST_PTE_OWNER_GUEST_HANDLE_MASK GENMASK(15, 0)
/* We need 40 bits for the GFN to cover a 52-bit IPA with 4k pages and LPA2 */
#define KVM_HOST_PTE_OWNER_GUEST_GFN_MASK GENMASK(55, 16)

79
arch/arm64/kvm/hyp/nvhe/mm.c
View File

@ -25,6 +25,7 @@ struct memblock_region hyp_memory[HYP_MEMBLOCK_REGIONS];
unsigned int hyp_memblock_nr;
static u64 __io_map_base;
static u64 __io_map_next;
struct hyp_fixmap_slot {
u64 addr;
@ -50,7 +51,7 @@ static int __pkvm_alloc_private_va_range(unsigned long start, size_t size)
hyp_assert_lock_held(&pkvm_pgd_lock);
if (!start || start < __io_map_base)
if (!start || start < __io_map_next)
return -EINVAL;
/* The allocated size is always a multiple of PAGE_SIZE */
@ -60,7 +61,7 @@ static int __pkvm_alloc_private_va_range(unsigned long start, size_t size)
if (cur > __hyp_vmemmap)
return -ENOMEM;
__io_map_base = cur;
__io_map_next = cur;
return 0;
}
@ -70,7 +71,7 @@ static int __pkvm_alloc_private_va_range(unsigned long start, size_t size)
* @size: The size of the VA range to reserve.
* @haddr: The hypervisor virtual start address of the allocation.
*
* The private virtual address (VA) range is allocated above __io_map_base
* The private virtual address (VA) range is allocated above __io_map_next
* and aligned based on the order of @size.
*
* Return: 0 on success or negative error code on failure.
@ -81,7 +82,7 @@ int pkvm_alloc_private_va_range(size_t size, unsigned long *haddr)
int ret;
hyp_spin_lock(&pkvm_pgd_lock);
addr = __io_map_base;
addr = __io_map_next;
ret = __pkvm_alloc_private_va_range(addr, size);
hyp_spin_unlock(&pkvm_pgd_lock);
@ -341,7 +342,7 @@ static int create_fixblock(void)
return -EINVAL;
hyp_spin_lock(&pkvm_pgd_lock);
addr = ALIGN(__io_map_base, PMD_SIZE);
addr = ALIGN(__io_map_next, PMD_SIZE);
ret = __pkvm_alloc_private_va_range(addr, PMD_SIZE);
if (ret)
goto unlock;
@ -426,6 +427,7 @@ int hyp_create_idmap(u32 hyp_va_bits)
*/
__io_map_base = start & BIT(hyp_va_bits - 2);
__io_map_base ^= BIT(hyp_va_bits - 2);
__io_map_next = __io_map_base;
__hyp_vmemmap = __io_map_base | BIT(hyp_va_bits - 3);
return __pkvm_create_mappings(start, end - start, start, PAGE_HYP_EXEC);
@ -433,19 +435,19 @@ int hyp_create_idmap(u32 hyp_va_bits)
int pkvm_create_stack(phys_addr_t phys, unsigned long *haddr)
{
unsigned long addr, prev_base;
unsigned long addr, prev_next;
size_t size;
int ret;
hyp_spin_lock(&pkvm_pgd_lock);
prev_base = __io_map_base;
prev_next = __io_map_next;
/*
* Efficient stack verification using the NVHE_STACK_SHIFT bit implies
* an alignment of our allocation on the order of the size.
*/
size = NVHE_STACK_SIZE * 2;
addr = ALIGN(__io_map_base, size);
addr = ALIGN(__io_map_next, size);
ret = __pkvm_alloc_private_va_range(addr, size);
if (!ret) {
@ -461,7 +463,7 @@ int pkvm_create_stack(phys_addr_t phys, unsigned long *haddr)
ret = kvm_pgtable_hyp_map(&pkvm_pgtable, addr + NVHE_STACK_SIZE,
NVHE_STACK_SIZE, phys, PAGE_HYP);
if (ret)
__io_map_base = prev_base;
__io_map_next = prev_next;
}
hyp_spin_unlock(&pkvm_pgd_lock);
@ -470,6 +472,65 @@ int pkvm_create_stack(phys_addr_t phys, unsigned long *haddr)
return ret;
}
static int check_page_ownership(phys_addr_t phys)
{
kvm_pte_t pte;
bool host_ok;
int ret;
if (addr_is_memory(phys)) {
struct hyp_page *page = hyp_phys_to_page(phys);
if (get_hyp_state(page) != PKVM_PAGE_OWNED ||
get_host_state(page) != PKVM_NOPAGE)
return -EPERM;
}
ret = kvm_pgtable_get_leaf(&host_mmu.pgt, phys, &pte, NULL);
if (ret)
return ret;
/* Hyp text may stay host-readable, see fix_host_ownership_walker(). */
if (kvm_pte_valid(pte) && addr_is_hyp_text(phys))
host_ok = !(kvm_pgtable_stage2_pte_prot(pte) & KVM_PGTABLE_PROT_W);
else
host_ok = host_stage2_pte_is_hyp_owned(pte);
return host_ok ? 0 : -EPERM;
}
static int check_host_ownership_walker(const struct kvm_pgtable_visit_ctx *ctx,
enum kvm_pgtable_walk_flags visit)
{
phys_addr_t phys, end;
int ret;
if (!kvm_pte_valid(ctx->old))
return 0;
phys = kvm_pte_to_phys(ctx->old);
end = phys + kvm_granule_size(ctx->level);
for (; phys < end; phys += PAGE_SIZE) {
ret = check_page_ownership(phys);
if (ret)
return ret;
}
return 0;
}
int pkvm_check_host_ownership(void)
{
struct kvm_pgtable_walker walker = {
.cb = check_host_ownership_walker,
.flags = KVM_PGTABLE_WALK_LEAF,
};
/* The private range and the vmemmap share one quarter of the VA space. */
return kvm_pgtable_walk(&pkvm_pgtable, __io_map_base,
BIT(pkvm_pgtable.ia_bits - 2), &walker);
}
static void *admit_host_page(void *arg)
{
struct kvm_hyp_memcache *host_mc = arg;

29
arch/arm64/kvm/hyp/nvhe/pkvm.c
View File

@ -360,7 +360,7 @@ static void pkvm_init_features_from_host(struct pkvm_hyp_vm *hyp_vm, const struc
if (test_bit(KVM_ARCH_FLAG_WRITABLE_IMP_ID_REGS, &host_arch_flags))
hyp_vm->kvm.arch.midr_el1 = host_kvm->arch.midr_el1;
return;
goto out;
}
if (kvm_pkvm_ext_allowed(kvm, KVM_CAP_ARM_MTE))
@ -379,13 +379,14 @@ static void pkvm_init_features_from_host(struct pkvm_hyp_vm *hyp_vm, const struc
if (kvm_pkvm_ext_allowed(kvm, KVM_CAP_ARM_PTRAUTH_GENERIC))
set_bit(KVM_ARM_VCPU_PTRAUTH_GENERIC, allowed_features);
if (kvm_pkvm_ext_allowed(kvm, KVM_CAP_ARM_SVE)) {
if (kvm_pkvm_ext_allowed(kvm, KVM_CAP_ARM_SVE))
set_bit(KVM_ARM_VCPU_SVE, allowed_features);
kvm->arch.flags |= host_arch_flags & BIT(KVM_ARCH_FLAG_GUEST_HAS_SVE);
}
bitmap_and(kvm->arch.vcpu_features, host_kvm->arch.vcpu_features,
allowed_features, KVM_VCPU_MAX_FEATURES);
out:
__assign_bit(KVM_ARCH_FLAG_GUEST_HAS_SVE, &kvm->arch.flags,
kvm_vcpu_has_feature(kvm, KVM_ARM_VCPU_SVE));
}
static void unpin_host_vcpu(struct kvm_vcpu *host_vcpu)
@ -398,10 +399,10 @@ static void unpin_host_sve_state(struct pkvm_hyp_vcpu *hyp_vcpu)
{
void *sve_state;
if (!vcpu_has_feature(&hyp_vcpu->vcpu, KVM_ARM_VCPU_SVE))
sve_state = hyp_vcpu->vcpu.arch.sve_state;
if (!sve_state)
return;
sve_state = hyp_vcpu->vcpu.arch.sve_state;
hyp_unpin_shared_mem(sve_state,
sve_state + vcpu_sve_state_size(&hyp_vcpu->vcpu));
}
@ -450,7 +451,7 @@ static int pkvm_vcpu_init_sve(struct pkvm_hyp_vcpu *hyp_vcpu, struct kvm_vcpu *h
unsigned int sve_max_vl;
size_t sve_state_size;
void *sve_state;
int ret = 0;
int ret;
if (!vcpu_has_feature(vcpu, KVM_ARM_VCPU_SVE)) {
vcpu_clear_flag(vcpu, VCPU_SVE_FINALIZED);
@ -459,25 +460,21 @@ static int pkvm_vcpu_init_sve(struct pkvm_hyp_vcpu *hyp_vcpu, struct kvm_vcpu *h
/* Limit guest vector length to the maximum supported by the host. */
sve_max_vl = min(READ_ONCE(host_vcpu->arch.sve_max_vl), kvm_host_sve_max_vl);
sve_state_size = sve_state_size_from_vl(sve_max_vl);
sve_state = kern_hyp_va(READ_ONCE(host_vcpu->arch.sve_state));
if (!sve_state || !sve_state_size) {
ret = -EINVAL;
goto err;
}
if (!sve_vl_valid(sve_max_vl) || !sve_state)
return -EINVAL;
sve_state_size = sve_state_size_from_vl(sve_max_vl);
ret = hyp_pin_shared_mem(sve_state, sve_state + sve_state_size);
if (ret)
goto err;
return ret;
vcpu->arch.sve_state = sve_state;
vcpu->arch.sve_max_vl = sve_max_vl;
return 0;
err:
clear_bit(KVM_ARM_VCPU_SVE, vcpu->kvm->arch.vcpu_features);
return ret;
}
static int vm_copy_id_regs(struct pkvm_hyp_vcpu *hyp_vcpu)

16
arch/arm64/kvm/hyp/nvhe/setup.c
View File

@ -217,7 +217,7 @@ static int fix_host_ownership_walker(const struct kvm_pgtable_visit_ctx *ctx,
case PKVM_PAGE_OWNED:
set_hyp_state(page, PKVM_PAGE_OWNED);
/* hyp text is RO in the host stage-2 to be inspected on panic. */
if (prot == PAGE_HYP_EXEC) {
if (addr_is_hyp_text(phys)) {
set_host_state(page, PKVM_NOPAGE);
return host_stage2_idmap_locked(phys, PAGE_SIZE, KVM_PGTABLE_PROT_R);
} else {
@ -269,6 +269,16 @@ static int fix_host_ownership(void)
return ret;
}
/* The stacks sit in the private VA range, not the linear map. */
for (i = 0; i < hyp_nr_cpus; i++) {
struct kvm_nvhe_init_params *params = per_cpu_ptr(&kvm_init_params, i);
u64 start = params->stack_hyp_va - NVHE_STACK_SIZE;
ret = kvm_pgtable_walk(&pkvm_pgtable, start, NVHE_STACK_SIZE, &walker);
if (ret)
return ret;
}
return 0;
}
@ -324,6 +334,10 @@ void __noreturn __pkvm_init_finalise(void)
if (ret)
goto out;
ret = pkvm_check_host_ownership();
if (ret)
goto out;
ret = hyp_ffa_init(ffa_proxy_pages);
if (ret)
goto out;

3
arch/arm64/kvm/hypercalls.c
View File

@ -185,7 +185,8 @@ static int kvm_smccc_set_filter(struct kvm *kvm, struct kvm_smccc_filter __user
start = filter.base;
end = start + filter.nr_functions - 1;
if (end < start || filter.action >= NR_SMCCC_FILTER_ACTIONS)
if (!filter.nr_functions || end < start ||
filter.action >= NR_SMCCC_FILTER_ACTIONS)
return -EINVAL;
mutex_lock(&kvm->arch.config_lock);

15
arch/arm64/kvm/mmu.c
View File

@ -59,27 +59,36 @@ static phys_addr_t stage2_range_addr_end(phys_addr_t addr, phys_addr_t end)
* long will also starve other vCPUs. We have to also make sure that the page
* tables are not freed while we released the lock.
*/
static int stage2_apply_range(struct kvm_s2_mmu *mmu, phys_addr_t addr,
static int stage2_apply_range(struct kvm_s2_mmu *mmu, phys_addr_t start,
phys_addr_t end,
int (*fn)(struct kvm_pgtable *, u64, u64),
bool resched)
{
struct kvm *kvm = kvm_s2_mmu_to_kvm(mmu);
bool lock_dropped = false;
phys_addr_t addr = start;
int ret;
u64 next;
do {
struct kvm_pgtable *pgt = mmu->pgt;
/*
* We may be raced on PGT teardown when we release the
* kvm->mmu_lock. That's fine as the PGT is legitimately no
* longer present.
*/
if (!pgt)
return -EINVAL;
return lock_dropped ? 0 : -EINVAL;
next = stage2_range_addr_end(addr, end);
ret = fn(pgt, addr, next - addr);
if (ret)
break;
if (resched && next != end)
if (resched && next != end) {
cond_resched_rwlock_write(&kvm->mmu_lock);
lock_dropped = true;
}
} while (addr = next, addr != end);
return ret;

115
arch/arm64/kvm/nested.c
View File

@ -45,11 +45,24 @@ struct vncr_tlb {
*/
#define S2_MMU_PER_VCPU 2
void kvm_init_nested(struct kvm *kvm)
int kvm_init_nested(struct kvm *kvm)
{
kvm->arch.nested_mmus = NULL;
kvm->arch.nested_mmus = kvmalloc_objs(struct kvm_s2_mmu *,
KVM_MAX_VCPUS * S2_MMU_PER_VCPU,
GFP_KERNEL_ACCOUNT);
kvm->arch.nested_mmus_size = 0;
atomic_set(&kvm->arch.vncr_tlb_count, 0);
return kvm->arch.nested_mmus ? 0 : -ENOMEM;
}
void kvm_destroy_nested(struct kvm *kvm)
{
for (int i = 0; i < kvm->arch.nested_mmus_size; i+= S2_MMU_PER_VCPU)
kvfree(kvm->arch.nested_mmus[i]);
kvm->arch.nested_mmus_size = 0;
kvfree(kvm->arch.nested_mmus);
}
static int init_nested_s2_mmu(struct kvm *kvm, struct kvm_s2_mmu *mmu)
@ -70,8 +83,9 @@ static int init_nested_s2_mmu(struct kvm *kvm, struct kvm_s2_mmu *mmu)
int kvm_vcpu_init_nested(struct kvm_vcpu *vcpu)
{
struct kvm *kvm = vcpu->kvm;
struct kvm_s2_mmu *tmp;
int num_mmus, ret = 0;
int num_mmus;
lockdep_assert_held(&kvm->arch.config_lock);
if (test_bit(KVM_ARM_VCPU_HAS_EL2_E2H0, kvm->arch.vcpu_features) &&
!cpus_have_final_cap(ARM64_HAS_HCR_NV1))
@ -84,51 +98,40 @@ int kvm_vcpu_init_nested(struct kvm_vcpu *vcpu)
if (!vcpu->arch.ctxt.vncr_array)
return -ENOMEM;
/*
* Let's treat memory allocation failures as benign: If we fail to
* allocate anything, return an error and keep the allocated array
* alive. Userspace may try to recover by initializing the vcpu
* again, and there is no reason to affect the whole VM for this.
*/
num_mmus = atomic_read(&kvm->online_vcpus) * S2_MMU_PER_VCPU;
if (num_mmus > kvm->arch.nested_mmus_size) {
tmp = kvzalloc_objs(*tmp, num_mmus, GFP_KERNEL_ACCOUNT);
struct kvm_s2_mmu *tmp;
int i, ret = 0;
tmp = kvzalloc_objs(*tmp, S2_MMU_PER_VCPU, GFP_KERNEL_ACCOUNT);
if (!tmp)
return -ENOMEM;
ret = -ENOMEM;
write_lock(&kvm->mmu_lock);
if (kvm->arch.nested_mmus_size) {
memcpy(tmp, kvm->arch.nested_mmus,
size_mul(sizeof(*tmp), kvm->arch.nested_mmus_size));
for (int i = 0; i < kvm->arch.nested_mmus_size; i++)
tmp[i].pgt->mmu = &tmp[i];
for (i = 0; !ret && i < S2_MMU_PER_VCPU; i++) {
ret = init_nested_s2_mmu(kvm, &tmp[i]);
if (ret)
break;
}
swap(kvm->arch.nested_mmus, tmp);
if (ret) {
while (--i >= 0)
kvm_free_stage2_pgd(&tmp[i]);
write_unlock(&kvm->mmu_lock);
kvfree(tmp);
free_page((unsigned long)vcpu->arch.ctxt.vncr_array);
vcpu->arch.ctxt.vncr_array = NULL;
return ret;
}
kvfree(tmp);
guard(write_lock)(&kvm->mmu_lock);
for (i = 0; i < S2_MMU_PER_VCPU; i++)
kvm->arch.nested_mmus[i + kvm->arch.nested_mmus_size] = &tmp[i];
kvm->arch.nested_mmus_size += S2_MMU_PER_VCPU;
}
for (int i = kvm->arch.nested_mmus_size; !ret && i < num_mmus; i++)
ret = init_nested_s2_mmu(kvm, &kvm->arch.nested_mmus[i]);
if (ret) {
for (int i = kvm->arch.nested_mmus_size; i < num_mmus; i++)
kvm_free_stage2_pgd(&kvm->arch.nested_mmus[i]);
free_page((unsigned long)vcpu->arch.ctxt.vncr_array);
vcpu->arch.ctxt.vncr_array = NULL;
return ret;
}
kvm->arch.nested_mmus_size = num_mmus;
return 0;
}
@ -742,7 +745,7 @@ void kvm_s2_mmu_iterate_by_vmid(struct kvm *kvm, u16 vmid,
write_lock(&kvm->mmu_lock);
for (int i = 0; i < kvm->arch.nested_mmus_size; i++) {
struct kvm_s2_mmu *mmu = &kvm->arch.nested_mmus[i];
struct kvm_s2_mmu *mmu = kvm->arch.nested_mmus[i];
if (!kvm_s2_mmu_valid(mmu))
continue;
@ -784,7 +787,7 @@ struct kvm_s2_mmu *lookup_s2_mmu(struct kvm_vcpu *vcpu)
* if S2 translation is disabled.
*/
for (int i = 0; i < kvm->arch.nested_mmus_size; i++) {
struct kvm_s2_mmu *mmu = &kvm->arch.nested_mmus[i];
struct kvm_s2_mmu *mmu = kvm->arch.nested_mmus[i];
if (!kvm_s2_mmu_valid(mmu))
continue;
@ -823,7 +826,7 @@ static struct kvm_s2_mmu *get_s2_mmu_nested(struct kvm_vcpu *vcpu)
for (i = kvm->arch.nested_mmus_next;
i < (kvm->arch.nested_mmus_size + kvm->arch.nested_mmus_next);
i++) {
s2_mmu = &kvm->arch.nested_mmus[i % kvm->arch.nested_mmus_size];
s2_mmu = kvm->arch.nested_mmus[i % kvm->arch.nested_mmus_size];
if (atomic_read(&s2_mmu->refcnt) == 0)
break;
@ -1260,6 +1263,17 @@ void kvm_handle_s1e2_tlbi(struct kvm_vcpu *vcpu, u32 inst, u64 val)
invalidate_vncr_va(vcpu->kvm, &scope);
}
static void kvm_invalidate_vncr_ipa_all(struct kvm *kvm)
{
struct kvm_pgtable *pgt = kvm->arch.mmu.pgt;
lockdep_assert_held_write(&kvm->mmu_lock);
/* if the mmu lock was dropped, pgt teardown may have raced. */
if (pgt)
kvm_invalidate_vncr_ipa(kvm, 0, BIT(pgt->ia_bits));
}
void kvm_nested_s2_wp(struct kvm *kvm)
{
int i;
@ -1270,13 +1284,13 @@ void kvm_nested_s2_wp(struct kvm *kvm)
return;
for (i = 0; i < kvm->arch.nested_mmus_size; i++) {
struct kvm_s2_mmu *mmu = &kvm->arch.nested_mmus[i];
struct kvm_s2_mmu *mmu = kvm->arch.nested_mmus[i];
if (kvm_s2_mmu_valid(mmu))
kvm_stage2_wp_range(mmu, 0, kvm_phys_size(mmu));
}
kvm_invalidate_vncr_ipa(kvm, 0, BIT(kvm->arch.mmu.pgt->ia_bits));
kvm_invalidate_vncr_ipa_all(kvm);
}
void kvm_nested_s2_unmap(struct kvm *kvm, bool may_block)
@ -1289,13 +1303,13 @@ void kvm_nested_s2_unmap(struct kvm *kvm, bool may_block)
return;
for (i = 0; i < kvm->arch.nested_mmus_size; i++) {
struct kvm_s2_mmu *mmu = &kvm->arch.nested_mmus[i];
struct kvm_s2_mmu *mmu = kvm->arch.nested_mmus[i];
if (kvm_s2_mmu_valid(mmu))
kvm_stage2_unmap_range(mmu, 0, kvm_phys_size(mmu), may_block);
}
kvm_invalidate_vncr_ipa(kvm, 0, BIT(kvm->arch.mmu.pgt->ia_bits));
kvm_invalidate_vncr_ipa_all(kvm);
}
void kvm_nested_s2_flush(struct kvm *kvm)
@ -1308,7 +1322,7 @@ void kvm_nested_s2_flush(struct kvm *kvm)
return;
for (i = 0; i < kvm->arch.nested_mmus_size; i++) {
struct kvm_s2_mmu *mmu = &kvm->arch.nested_mmus[i];
struct kvm_s2_mmu *mmu = kvm->arch.nested_mmus[i];
if (kvm_s2_mmu_valid(mmu))
kvm_stage2_flush_range(mmu, 0, kvm_phys_size(mmu));
@ -1317,17 +1331,12 @@ void kvm_nested_s2_flush(struct kvm *kvm)
void kvm_arch_flush_shadow_all(struct kvm *kvm)
{
int i;
for (i = 0; i < kvm->arch.nested_mmus_size; i++) {
struct kvm_s2_mmu *mmu = &kvm->arch.nested_mmus[i];
for (int i = 0; i < kvm->arch.nested_mmus_size; i++) {
struct kvm_s2_mmu *mmu = kvm->arch.nested_mmus[i];
if (!WARN_ON(atomic_read(&mmu->refcnt)))
kvm_free_stage2_pgd(mmu);
}
kvfree(kvm->arch.nested_mmus);
kvm->arch.nested_mmus = NULL;
kvm->arch.nested_mmus_size = 0;
kvm_uninit_stage2_mmu(kvm);
}

1
arch/arm64/kvm/sys_regs.c
View File

@ -926,6 +926,7 @@ static u64 *demux_wb_reg(struct kvm_vcpu *vcpu, const struct sys_reg_desc *rd)
struct kvm_guest_debug_arch *dbg = &vcpu->arch.vcpu_debug_state;
switch (rd->Op2) {
case 0b001:
case 0b100:
return &dbg->dbg_bvr[rd->CRm];
case 0b101:

46
arch/arm64/kvm/vgic/vgic-its.c
View File

@ -1658,7 +1658,7 @@ static void vgic_mmio_write_its_baser(struct kvm *kvm,
unsigned long val)
{
const struct vgic_its_abi *abi = vgic_its_get_abi(its);
u64 entry_size, table_type;
u64 old, entry_size, table_type;
u64 reg, *regptr, clearbits = 0;
/* When GITS_CTLR.Enable is 1, we ignore write accesses. */
@ -1681,7 +1681,9 @@ static void vgic_mmio_write_its_baser(struct kvm *kvm,
return;
}
reg = update_64bit_reg(*regptr, addr & 7, len, val);
old = *regptr;
reg = update_64bit_reg(old, addr & 7, len, val);
reg &= ~GITS_BASER_RO_MASK;
reg &= ~clearbits;
@ -1691,7 +1693,8 @@ static void vgic_mmio_write_its_baser(struct kvm *kvm,
*regptr = reg;
if (!(reg & GITS_BASER_VALID)) {
/* The ITS driver rewrites an unchanged GITS_BASER<n> on resume. */
if (reg != old) {
/* Take the its_lock to prevent a race with a save/restore */
mutex_lock(&its->its_lock);
switch (table_type) {
@ -1702,6 +1705,8 @@ static void vgic_mmio_write_its_baser(struct kvm *kvm,
vgic_its_free_collection_list(kvm, its);
break;
}
/* A concurrent injection may have cached a translation. */
vgic_its_invalidate_cache(its);
mutex_unlock(&its->its_lock);
}
}
@ -2019,18 +2024,22 @@ static int vgic_its_attr_regs_access(struct kvm_device *dev,
return ret;
}
static u32 compute_next_devid_offset(struct list_head *h,
static u32 compute_next_devid_offset(struct vgic_its *its, u64 baser,
struct its_device *dev)
{
struct its_device *next;
u32 next_offset;
struct its_device *next = dev;
if (list_is_last(&dev->dev_list, h))
return 0;
next = list_next_entry(dev, dev_list);
next_offset = next->device_id - dev->device_id;
/*
* Point at the next device vgic_its_save_device_tables() saves. It
* sorts device_list first, so the subtraction cannot underflow.
*/
list_for_each_entry_continue(next, &its->device_list, dev_list) {
if (vgic_its_check_id(its, baser, next->device_id, NULL))
return min_t(u32, next->device_id - dev->device_id,
VITS_DTE_MAX_DEVID_OFFSET);
}
return min_t(u32, next_offset, VITS_DTE_MAX_DEVID_OFFSET);
return 0;
}
static u32 compute_next_eventid_offset(struct list_head *h, struct its_ite *ite)
@ -2271,17 +2280,18 @@ static int vgic_its_restore_itt(struct vgic_its *its, struct its_device *dev)
* vgic_its_save_dte - Save a device table entry at a given GPA
*
* @its: ITS handle
* @baser: GITS_BASER<dev> the caller is saving against
* @dev: ITS device
* @ptr: GPA
*/
static int vgic_its_save_dte(struct vgic_its *its, struct its_device *dev,
gpa_t ptr)
static int vgic_its_save_dte(struct vgic_its *its, u64 baser,
struct its_device *dev, gpa_t ptr)
{
u64 val, itt_addr_field;
u32 next_offset;
itt_addr_field = dev->itt_addr >> 8;
next_offset = compute_next_devid_offset(&its->device_list, dev);
next_offset = compute_next_devid_offset(its, baser, dev);
val = (1ULL << KVM_ITS_DTE_VALID_SHIFT |
((u64)next_offset << KVM_ITS_DTE_NEXT_SHIFT) |
(itt_addr_field << KVM_ITS_DTE_ITTADDR_SHIFT) |
@ -2380,15 +2390,16 @@ static int vgic_its_save_device_tables(struct vgic_its *its)
int ret;
gpa_t eaddr;
/* Don't fail a save that userspace must be able to issue. */
if (!vgic_its_check_id(its, baser,
dev->device_id, &eaddr))
return -EINVAL;
continue;
ret = vgic_its_save_itt(its, dev);
if (ret)
return ret;
ret = vgic_its_save_dte(its, dev, eaddr);
ret = vgic_its_save_dte(its, baser, dev, eaddr);
if (ret)
return ret;
}
@ -2541,9 +2552,6 @@ static int vgic_its_save_collection_table(struct vgic_its *its)
max_size = GITS_BASER_NR_PAGES(baser) * SZ_64K;
list_for_each_entry(collection, &its->collection_list, coll_list) {
if (!vgic_its_check_id(its, baser, collection->collection_id, NULL))
return -EINVAL;
ret = vgic_its_save_cte(its, collection, gpa);
if (ret)
return ret;

20
arch/riscv/include/asm/csr.h
View File

@ -188,12 +188,24 @@
#define HGATP_MODE_SHIFT HGATP32_MODE_SHIFT
#endif
/* VSIP & HVIP relation */
/*
* VSIP & HVIP relation
*
* The bit positions are same between VSIP and HVIP for interrupt
* numbers 13-63, where there's a shift for the SSI, STI and SEI.
*/
#define VSIP_TO_HVIP_SHIFT (IRQ_VS_SOFT - IRQ_S_SOFT)
#define VSIP_VALID_MASK ((_AC(1, UL) << IRQ_S_SOFT) | \
#define VSIP_BIAS_MASK ((_AC(1, UL) << IRQ_S_SOFT) | \
(_AC(1, UL) << IRQ_S_TIMER) | \
(_AC(1, UL) << IRQ_S_EXT) | \
(_AC(1, UL) << IRQ_PMU_OVF))
(_AC(1, UL) << IRQ_S_EXT))
#define VSIP_NO_BIAS_MASK (_AC(1, UL) << IRQ_PMU_OVF)
#define VSIP_VALID_MASK (VSIP_BIAS_MASK | VSIP_NO_BIAS_MASK)
#define vsip_to_hvip(_vsip) ((((_vsip) & VSIP_BIAS_MASK) << \
VSIP_TO_HVIP_SHIFT) | \
((_vsip) & VSIP_NO_BIAS_MASK))
#define hvip_to_vsip(_hvip) ((((_hvip) >> VSIP_TO_HVIP_SHIFT) & \
VSIP_BIAS_MASK) | \
((_hvip) & VSIP_NO_BIAS_MASK))
/* AIA CSR bits */
#define TOPI_IID_SHIFT 16

11
arch/riscv/kvm/aia_imsic.c
View File

@ -965,9 +965,14 @@ int kvm_riscv_aia_imsic_rw_attr(struct kvm *kvm, unsigned long type,
if (!vcpu)
return -ENODEV;
if (mutex_lock_killable(&vcpu->mutex))
return -EINTR;
imsic = vcpu->arch.aia_context.imsic_state;
if (!imsic)
return -ENODEV;
if (!imsic) {
rc = -ENODEV;
goto out_unlock;
}
isel = KVM_DEV_RISCV_AIA_IMSIC_GET_ISEL(type);
read_lock_irqsave(&imsic->vsfile_lock, flags);
@ -991,6 +996,8 @@ int kvm_riscv_aia_imsic_rw_attr(struct kvm *kvm, unsigned long type,
rc = imsic_vsfile_rw(vsfile_hgei, vsfile_cpu, imsic->nr_eix,
isel, write, val);
out_unlock:
mutex_unlock(&vcpu->mutex);
return rc;
}

10
arch/riscv/kvm/mmu.c
View File

@ -627,6 +627,7 @@ int kvm_riscv_mmu_map(struct kvm_vcpu *vcpu, struct kvm_memory_slot *memslot,
int ret;
kvm_pfn_t hfn;
bool is_hugetlb;
bool unused = false;
bool writable;
unsigned int vma_pageshift;
gfn_t gfn = gpa >> PAGE_SHIFT;
@ -707,6 +708,8 @@ int kvm_riscv_mmu_map(struct kvm_vcpu *vcpu, struct kvm_memory_slot *memslot,
vma_pageshift, current);
return 0;
}
if (is_sigpending_pfn(hfn))
return -EINTR;
if (is_error_noslot_pfn(hfn))
return -EFAULT;
@ -719,8 +722,10 @@ int kvm_riscv_mmu_map(struct kvm_vcpu *vcpu, struct kvm_memory_slot *memslot,
write_lock(&kvm->mmu_lock);
if (mmu_invalidate_retry(kvm, mmu_seq))
if (mmu_invalidate_retry(kvm, mmu_seq)) {
unused = true;
goto out_unlock;
}
/*
* Check if we are backed by a THP and thus use block mapping if
@ -743,7 +748,8 @@ int kvm_riscv_mmu_map(struct kvm_vcpu *vcpu, struct kvm_memory_slot *memslot,
kvm_err("Failed to map in G-stage\n");
out_unlock:
kvm_release_faultin_page(kvm, page, ret && ret != -EEXIST, writable);
kvm_release_faultin_page(kvm, page,
unused || (ret && ret != -EEXIST), writable);
write_unlock(&kvm->mmu_lock);
return ret;
}

20
arch/riscv/kvm/nacl.c
View File

@ -42,12 +42,24 @@ void __kvm_riscv_nacl_hfence(void *shmem,
}
}
entp = shmem + SBI_NACL_SHMEM_HFENCE_ENTRY_CONFIG(i);
*entp = cpu_to_lelong(control);
/*
* Per SBI v3.0 section 15.1.2, the Page_Number and Page_Count
* words must be updated before the Config word with its Pending
* bit set. WRITE_ONCE() stops the compiler from reordering the
* stores and smp_wmb() makes the parameter words globally
* visible to the SBI implementation (or NACL hardware) before
* the Pending bit is set.
*/
entp = shmem + SBI_NACL_SHMEM_HFENCE_ENTRY_PNUM(i);
*entp = cpu_to_lelong(page_num);
WRITE_ONCE(*entp, cpu_to_lelong(page_num));
entp = shmem + SBI_NACL_SHMEM_HFENCE_ENTRY_PCOUNT(i);
*entp = cpu_to_lelong(page_count);
WRITE_ONCE(*entp, cpu_to_lelong(page_count));
/* Ensure the parameter words are visible before the Pending bit */
smp_wmb();
entp = shmem + SBI_NACL_SHMEM_HFENCE_ENTRY_CONFIG(i);
WRITE_ONCE(*entp, cpu_to_lelong(control));
}
int kvm_riscv_nacl_enable(void)

3
arch/riscv/kvm/vcpu.c
View File

@ -491,8 +491,7 @@ bool kvm_riscv_vcpu_has_interrupts(struct kvm_vcpu *vcpu, u64 mask)
bool ret;
raw_spin_lock_irqsave(&vcpu->arch.irqs_pending_lock, flags);
ie = ((vcpu->arch.guest_csr.vsie & VSIP_VALID_MASK)
<< VSIP_TO_HVIP_SHIFT) & (unsigned long)mask;
ie = vsip_to_hvip(vcpu->arch.guest_csr.vsie) & (unsigned long)mask;
ie |= vcpu->arch.guest_csr.vsie & ~IRQ_LOCAL_MASK &
(unsigned long)mask;
ret = vcpu->arch.irqs_pending[0] & ie;

2
arch/riscv/kvm/vcpu_exit.c
View File

@ -283,7 +283,7 @@ int kvm_riscv_vcpu_exit(struct kvm_vcpu *vcpu, struct kvm_run *run,
}
/* Print details in-case of error */
if (ret < 0) {
if (ret < 0 && ret != -EINTR) {
kvm_err("VCPU exit error %d\n", ret);
kvm_err("SEPC=0x%lx SSTATUS=0x%lx HSTATUS=0x%lx\n",
vcpu->arch.guest_context.sepc,

8
arch/riscv/kvm/vcpu_onereg.c
View File

@ -272,7 +272,7 @@ static int kvm_riscv_vcpu_general_get_csr(struct kvm_vcpu *vcpu,
if (reg_num == KVM_REG_RISCV_CSR_REG(sip)) {
kvm_riscv_vcpu_flush_interrupts(vcpu);
*out_val = (csr->hvip >> VSIP_TO_HVIP_SHIFT) & VSIP_VALID_MASK;
*out_val = hvip_to_vsip(csr->hvip);
*out_val |= csr->hvip & ~IRQ_LOCAL_MASK;
} else
*out_val = ((unsigned long *)csr)[reg_num];
@ -293,10 +293,8 @@ static int kvm_riscv_vcpu_general_set_csr(struct kvm_vcpu *vcpu,
reg_num = array_index_nospec(reg_num, regs_max);
if (reg_num == KVM_REG_RISCV_CSR_REG(sip)) {
reg_val &= VSIP_VALID_MASK;
reg_val <<= VSIP_TO_HVIP_SHIFT;
}
if (reg_num == KVM_REG_RISCV_CSR_REG(sip))
reg_val = vsip_to_hvip(reg_val);
((unsigned long *)csr)[reg_num] = reg_val;

41
arch/riscv/kvm/vcpu_pmu.c
View File

@ -270,12 +270,13 @@ static int pmu_ctr_read(struct kvm_vcpu *vcpu, unsigned long cidx,
return -EINVAL;
pmc->counter_val = kvpmu->fw_event[fevent_code].value;
*out_val = pmc->counter_val;
} else if (pmc->perf_event) {
pmc->counter_val += perf_event_read_value(pmc->perf_event, &enabled, &running);
*out_val = pmc->counter_val +
perf_event_read_value(pmc->perf_event, &enabled, &running);
} else {
return -EINVAL;
}
*out_val = pmc->counter_val;
return 0;
}
@ -454,6 +455,14 @@ int kvm_riscv_vcpu_pmu_snapshot_set_shmem(struct kvm_vcpu *vcpu, unsigned long s
}
}
/*
* Clear any previously installed snapshot area to avoid leaking
* the old sdata and to keep sdata/snapshot_addr consistent if
* the re-install fails below.
*/
if (kvpmu->snapshot_addr != INVALID_GPA)
kvm_pmu_clear_snapshot_area(vcpu);
kvpmu->sdata = kzalloc(snapshot_area_size, GFP_ATOMIC | __GFP_ACCOUNT);
if (!kvpmu->sdata) {
sbiret = SBI_ERR_FAILURE;
@ -645,7 +654,6 @@ int kvm_riscv_vcpu_pmu_ctr_stop(struct kvm_vcpu *vcpu, unsigned long ctr_base,
{
struct kvm_pmu *kvpmu = vcpu_to_pmu(vcpu);
int i, pmc_index, sbiret = 0;
u64 enabled, running;
struct kvm_pmc *pmc;
int fevent_code;
bool snap_flag_set = flags & SBI_PMU_STOP_FLAG_TAKE_SNAPSHOT;
@ -675,14 +683,19 @@ int kvm_riscv_vcpu_pmu_ctr_stop(struct kvm_vcpu *vcpu, unsigned long ctr_base,
goto out;
}
if (!kvpmu->fw_event[fevent_code].started)
if (!kvpmu->fw_event[fevent_code].started) {
sbiret = SBI_ERR_ALREADY_STOPPED;
kvpmu->fw_event[fevent_code].started = false;
} else {
kvpmu->fw_event[fevent_code].started = false;
pmc->counter_val = kvpmu->fw_event[fevent_code].value;
}
} else if (pmc->perf_event) {
if (pmc->started) {
/* Stop counting the counter */
perf_event_disable(pmc->perf_event);
/*
* Stop the counter and fold the live count into counter_val.
* Reset the event value to avoid redundant accumulation.
*/
pmc->counter_val += perf_event_pause(pmc->perf_event, true);
pmc->started = false;
} else {
sbiret = SBI_ERR_ALREADY_STOPPED;
@ -696,11 +709,6 @@ int kvm_riscv_vcpu_pmu_ctr_stop(struct kvm_vcpu *vcpu, unsigned long ctr_base,
}
if (snap_flag_set && !sbiret) {
if (pmc->cinfo.type == SBI_PMU_CTR_TYPE_FW)
pmc->counter_val = kvpmu->fw_event[fevent_code].value;
else if (pmc->perf_event)
pmc->counter_val += perf_event_read_value(pmc->perf_event,
&enabled, &running);
/*
* The counter and overflow indices in the snapshot region are w.r.to
* cbase. Modify the set bit in the counter mask instead of the pmc_index
@ -727,9 +735,10 @@ int kvm_riscv_vcpu_pmu_ctr_stop(struct kvm_vcpu *vcpu, unsigned long ctr_base,
}
}
if (shmem_needs_update)
kvm_vcpu_write_guest(vcpu, kvpmu->snapshot_addr, kvpmu->sdata,
sizeof(struct riscv_pmu_snapshot_data));
if (shmem_needs_update &&
kvm_vcpu_write_guest(vcpu, kvpmu->snapshot_addr, kvpmu->sdata,
sizeof(struct riscv_pmu_snapshot_data)))
sbiret = SBI_ERR_FAILURE;
out:
retdata->err_val = sbiret;

4
arch/riscv/kvm/vcpu_sbi_hsm.c
View File

@ -95,9 +95,9 @@ static int kvm_sbi_ext_hsm_handler(struct kvm_vcpu *vcpu, struct kvm_run *run,
ret = kvm_sbi_hsm_vcpu_get_status(vcpu);
if (ret >= 0) {
retdata->out_val = ret;
retdata->err_val = 0;
ret = 0;
}
return 0;
break;
case SBI_EXT_HSM_HART_SUSPEND:
switch (lower_32_bits(cp->a0)) {
case SBI_HSM_SUSPEND_RET_DEFAULT:

5
arch/riscv/kvm/vcpu_timer.c
View File

@ -61,10 +61,13 @@ static enum hrtimer_restart kvm_riscv_vcpu_hrtimer_expired(struct hrtimer *h)
static int kvm_riscv_vcpu_timer_cancel(struct kvm_vcpu_timer *t)
{
if (!t->init_done || !t->next_set)
if (!t->init_done)
return -EINVAL;
hrtimer_cancel(&t->hrt);
if (!t->next_set)
return -EINVAL;
t->next_set = false;
return 0;

8
arch/s390/kernel/uv.c
View File

@ -825,7 +825,7 @@ static int find_secret_in_page(const u8 secret_id[UV_SECRET_ID_LEN],
{
u16 i;
for (i = 0; i < list->total_num_secrets; i++) {
for (i = 0; i < list->num_secr_stored; i++) {
if (memcmp(secret_id, list->secrets[i].id, UV_SECRET_ID_LEN) == 0) {
*secret = list->secrets[i].hdr;
return 0;
@ -846,11 +846,14 @@ int uv_find_secret(const u8 secret_id[UV_SECRET_ID_LEN],
struct uv_secret_list *list,
struct uv_secret_list_item_hdr *secret)
{
u16 start_idx = 0;
u16 start_idx;
u16 list_rc;
int ret;
list->next_secret_idx = 0;
do {
start_idx = list->next_secret_idx;
uv_list_secrets(list, start_idx, &list_rc, NULL);
if (list_rc != UVC_RC_EXECUTED && list_rc != UVC_RC_MORE_DATA) {
if (list_rc == UVC_RC_INV_CMD)
@ -861,7 +864,6 @@ int uv_find_secret(const u8 secret_id[UV_SECRET_ID_LEN],
ret = find_secret_in_page(secret_id, list, secret);
if (ret == 0)
return ret;
start_idx = list->next_secret_idx;
} while (list_rc == UVC_RC_MORE_DATA && start_idx < list->next_secret_idx);
return -ENOENT;

65
arch/s390/kvm/gmap/dat.c
View File

@ -621,17 +621,20 @@ int dat_get_storage_key(union asce asce, gfn_t gfn, union skey *skey)
union pte *ptep;
int rc;
again:
skey->skey = 0;
rc = dat_entry_walk(NULL, gfn, asce, DAT_WALK_ANY, TABLE_TYPE_PAGE_TABLE, &crstep, &ptep);
if (rc)
return rc;
if (!ptep) {
union crste crste;
union crste crste = READ_ONCE(*crstep);
crste = READ_ONCE(*crstep);
if (!crste.h.fc || !crste.s.fc1.pr)
if (!crste_leaf(crste) && !crste.h.i)
goto again;
if (!crste.s.fc1.pr)
return 0;
skey->skey = page_get_storage_key(large_crste_to_phys(crste, gfn));
return 0;
}
@ -662,13 +665,20 @@ int dat_set_storage_key(struct kvm_s390_mmu_cache *mc, union asce asce, gfn_t gf
union pte *ptep;
int rc;
again:
rc = dat_entry_walk(mc, gfn, asce, DAT_WALK_LEAF_ALLOC, TABLE_TYPE_PAGE_TABLE,
&crstep, &ptep);
if (rc)
return rc;
if (!ptep) {
page_set_storage_key(large_crste_to_phys(*crstep, gfn), skey.skey, !nq);
union crste crste = READ_ONCE(*crstep);
/* A large page has been split concurrently, try again */
if (!crste_leaf(crste))
goto again;
page_set_storage_key(large_crste_to_phys(crste, gfn), skey.skey, !nq);
return 0;
}
@ -718,15 +728,22 @@ int dat_cond_set_storage_key(struct kvm_s390_mmu_cache *mmc, union asce asce, gf
union pte *ptep;
int rc;
again:
rc = dat_entry_walk(mmc, gfn, asce, DAT_WALK_LEAF_ALLOC, TABLE_TYPE_PAGE_TABLE,
&crstep, &ptep);
if (rc)
return rc;
if (!ptep) {
union crste crste = READ_ONCE(*crstep);
/* A large page has been split concurrently, try again */
if (!crste_leaf(crste))
goto again;
if (!oldkey)
oldkey = &prev;
return page_cond_set_storage_key(large_crste_to_phys(*crstep, gfn), skey, oldkey,
return page_cond_set_storage_key(large_crste_to_phys(crste, gfn), skey, oldkey,
nq, mr, mc);
}
@ -768,7 +785,7 @@ int dat_reset_reference_bit(union asce asce, gfn_t gfn, union skey *skey)
int rc;
skey->skey = 0;
again:
rc = dat_entry_walk(NULL, gfn, asce, DAT_WALK_ANY, TABLE_TYPE_PAGE_TABLE, &crstep, &ptep);
if (rc)
return rc;
@ -776,9 +793,12 @@ int dat_reset_reference_bit(union asce asce, gfn_t gfn, union skey *skey)
if (!ptep) {
union crste crste = READ_ONCE(*crstep);
if (!crste.h.fc || !crste.s.fc1.pr)
/* A large page has been split concurrently, try again */
if (!crste_leaf(crste) && !crste.h.i)
goto again;
if (!crste.s.fc1.pr)
return 0;
skey->skey = page_reset_referenced(large_crste_to_phys(*crstep, gfn)) << 1;
skey->skey = page_reset_referenced(large_crste_to_phys(crste, gfn)) << 1;
return 0;
}
old = pgste_get_lock(ptep);
@ -846,19 +866,12 @@ long dat_reset_skeys(union asce asce, gfn_t start)
}
#endif /* KVM_S390_MANAGES_S390_GUEST */
struct slot_priv {
unsigned long token;
struct kvm_s390_mmu_cache *mc;
};
static long _dat_slot_pte(union pte *ptep, gfn_t gfn, gfn_t next, struct dat_walk *walk)
{
struct slot_priv *p = walk->priv;
union crste dummy = { .val = p->token };
union pte new_pte, pte = READ_ONCE(*ptep);
union pgste pgste;
new_pte = _PTE_TOK(dummy.tok.type, dummy.tok.par);
new_pte = walk->priv ? _PTE_EMPTY : _PTE_TOK(_DAT_TOKEN_PIC, PGM_ADDRESSING);
/* Table entry already in the desired state. */
if (pte.val == new_pte.val)
@ -875,10 +888,9 @@ static long _dat_slot_pte(union pte *ptep, gfn_t gfn, gfn_t next, struct dat_wal
static long _dat_slot_crste(union crste *crstep, gfn_t gfn, gfn_t next, struct dat_walk *walk)
{
union crste new_crste, crste = READ_ONCE(*crstep);
struct slot_priv *p = walk->priv;
struct kvm_s390_mmu_cache *mc = walk->priv;
new_crste.val = p->token;
new_crste.h.tt = crste.h.tt;
new_crste = mc ? _CRSTE_EMPTY(crste.h.tt) : _CRSTE_HOLE(crste.h.tt);
/* Table entry already in the desired state. */
if (crste.val == new_crste.val)
@ -902,7 +914,10 @@ static long _dat_slot_crste(union crste *crstep, gfn_t gfn, gfn_t next, struct d
if (!crste.h.fc && !crste.h.i)
return 0;
/* Split (install a lower level table), and handle things there. */
return dat_split_crste(p->mc, crstep, gfn, walk->asce, false);
if (mc)
return dat_split_crste(mc, crstep, gfn, walk->asce, false);
/* A large page should never cross memslots boundaries */
return -EINVAL;
}
static const struct dat_walk_ops dat_slot_ops = {
@ -910,16 +925,10 @@ static const struct dat_walk_ops dat_slot_ops = {
.crste_ops = { _dat_slot_crste, _dat_slot_crste, _dat_slot_crste, _dat_slot_crste, },
};
int dat_set_slot(struct kvm_s390_mmu_cache *mc, union asce asce, gfn_t start, gfn_t end,
u16 type, u16 param)
int dat_set_slot(struct kvm_s390_mmu_cache *mc, union asce asce, gfn_t start, gfn_t end)
{
struct slot_priv priv = {
.token = _CRSTE_TOK(0, type, param).val,
.mc = mc,
};
return _dat_walk_gfn_range(start, end, asce, &dat_slot_ops,
DAT_WALK_IGN_HOLES | DAT_WALK_ANY, &priv);
DAT_WALK_IGN_HOLES | DAT_WALK_ANY, mc);
}
static void pgste_set_unlock_multiple(union pte *first, int n, union pgste *pgstes)

10
arch/s390/kvm/gmap/dat.h
View File

@ -547,8 +547,7 @@ long dat_reset_skeys(union asce asce, gfn_t start);
unsigned long dat_get_ptval(struct page_table *table, struct ptval_param param);
void dat_set_ptval(struct page_table *table, struct ptval_param param, unsigned long val);
int dat_set_slot(struct kvm_s390_mmu_cache *mc, union asce asce, gfn_t start, gfn_t end,
u16 type, u16 param);
int dat_set_slot(struct kvm_s390_mmu_cache *mc, union asce asce, gfn_t start, gfn_t end);
#if KVM_S390_MANAGES_S390_GUEST
int dat_set_prefix_notif_bit(union asce asce, gfn_t gfn);
@ -973,16 +972,15 @@ static inline int get_level(union crste *crstep, union pte *ptep)
return ptep ? TABLE_TYPE_PAGE_TABLE : crstep->h.tt;
}
static inline int dat_delete_slot(struct kvm_s390_mmu_cache *mc, union asce asce, gfn_t start,
unsigned long npages)
static inline int dat_delete_slot(union asce asce, gfn_t start, unsigned long npages)
{
return dat_set_slot(mc, asce, start, start + npages, _DAT_TOKEN_PIC, PGM_ADDRESSING);
return dat_set_slot(NULL, asce, start, start + npages);
}
static inline int dat_create_slot(struct kvm_s390_mmu_cache *mc, union asce asce, gfn_t start,
unsigned long npages)
{
return dat_set_slot(mc, asce, start, start + npages, _DAT_TOKEN_NONE, 0);
return dat_set_slot(mc, asce, start, start + npages);
}
static inline bool crste_is_ucas(union crste crste)

6
arch/s390/kvm/gmap/gmap.c
View File

@ -994,11 +994,13 @@ static long _destroy_pages_pte(union pte *ptep, gfn_t gfn, gfn_t next, struct da
static long _destroy_pages_crste(union crste *crstep, gfn_t gfn, gfn_t next, struct dat_walk *walk)
{
phys_addr_t origin, cur, end;
union crste crste;
if (!crstep->h.fc || !crstep->s.fc1.pr)
crste = READ_ONCE(*crstep);
if (!crste.h.fc || !crste.s.fc1.pr)
return 0;
origin = crste_origin_large(*crstep);
origin = crste_origin_large(crste);
cur = ((max(gfn, walk->start) - gfn) << PAGE_SHIFT) + origin;
end = ((min(next, walk->end) - gfn) << PAGE_SHIFT) + origin;
for ( ; cur < end; cur += PAGE_SIZE)

91
arch/s390/kvm/gmap/kvm_mmu.c
View File

@ -47,6 +47,9 @@ int s390_kvm_mmu_prepare_memory_region(struct kvm *kvm,
struct kvm_memory_slot *new,
enum kvm_mr_change change)
{
struct kvm_s390_mmu_cache *mc __free(kvm_s390_mmu_cache) = NULL;
int rc = 0;
if (kvm_is_ucontrol(kvm) && new && new->id < KVM_USER_MEM_SLOTS)
return -EINVAL;
@ -61,6 +64,10 @@ int s390_kvm_mmu_prepare_memory_region(struct kvm *kvm,
* and munmap() stuff in this slot after doing this call at any
* time.
*/
if (change != KVM_MR_MOVE && change != KVM_MR_CREATE) {
WARN(1, "Unknown KVM MR CHANGE: %d\n", change);
return -EINVAL;
}
if (new->userspace_addr & ~PAGE_MASK)
return -EINVAL;
if ((new->base_gfn + new->npages) * PAGE_SIZE > kvm->arch.mem_limit)
@ -69,65 +76,41 @@ int s390_kvm_mmu_prepare_memory_region(struct kvm *kvm,
return -EINVAL;
}
if (!kvm_s390_is_migration_mode(kvm))
return 0;
/*
* Turn off migration mode when:
* - userspace creates a new memslot with dirty logging off,
* - userspace modifies an existing memslot (MOVE or FLAGS_ONLY) and
* dirty logging is turned off.
* Migration mode expects dirty page logging being enabled to store
* its dirty bitmap.
*/
if (change != KVM_MR_DELETE &&
!(new->flags & KVM_MEM_LOG_DIRTY_PAGES))
WARN(kvm_s390_vm_stop_migration(kvm),
"Failed to stop migration mode");
return 0;
}
void s390_kvm_mmu_commit_memory_region(struct kvm *kvm,
struct kvm_memory_slot *old,
const struct kvm_memory_slot *new,
enum kvm_mr_change change)
{
struct kvm_s390_mmu_cache *mc __free(kvm_s390_mmu_cache) = NULL;
int rc = 0;
guard(mutex)(&kvm->slots_arch_lock);
if (kvm->arch.migration_mode) {
/*
* Turn off migration mode when:
* - userspace creates a new memslot with dirty logging off,
* - userspace modifies an existing memslot (MOVE or FLAGS_ONLY)
* and dirty logging is turned off.
* Migration mode expects dirty page logging being enabled to
* store its dirty bitmap.
*/
if (change != KVM_MR_DELETE &&
!(new->flags & KVM_MEM_LOG_DIRTY_PAGES))
WARN(kvm_s390_vm_stop_migration(kvm),
"Failed to stop migration mode");
}
if (change == KVM_MR_FLAGS_ONLY)
return;
mc = kvm_s390_new_mmu_cache();
if (!mc) {
rc = -ENOMEM;
goto out;
return 0;
if (change != KVM_MR_DELETE) {
/* Enough capacity to add a new memslot */
mc = kvm_s390_new_mmu_cache();
if (!mc)
return -ENOMEM;
}
scoped_guard(write_lock, &kvm->mmu_lock) {
kvm_s390_update_cmma_dirty(kvm, old);
switch (change) {
case KVM_MR_DELETE:
rc = dat_delete_slot(mc, kvm->arch.gmap->asce, old->base_gfn, old->npages);
break;
case KVM_MR_MOVE:
rc = dat_delete_slot(mc, kvm->arch.gmap->asce, old->base_gfn, old->npages);
if (rc)
break;
fallthrough;
case KVM_MR_CREATE:
if (change == KVM_MR_DELETE || change == KVM_MR_MOVE)
rc = dat_delete_slot(kvm->arch.gmap->asce, old->base_gfn, old->npages);
if (!rc && (change == KVM_MR_MOVE || change == KVM_MR_CREATE))
rc = dat_create_slot(mc, kvm->arch.gmap->asce, new->base_gfn, new->npages);
break;
case KVM_MR_FLAGS_ONLY:
break;
default:
WARN(1, "Unknown KVM MR CHANGE: %d\n", change);
}
}
out:
if (rc)
pr_warn("failed to commit memory region\n");
/*
* Can only be triggered if dat_{create,delete}_slot() found an
* internal inconsistency or if the mmu cache ran out of memory;
* both should be impossible.
*/
KVM_BUG_ON(rc, kvm);
return rc;
}

4
arch/s390/kvm/gmap/kvm_mmu.h
View File

@ -10,9 +10,5 @@ int s390_kvm_mmu_prepare_memory_region(struct kvm *kvm,
const struct kvm_memory_slot *old,
struct kvm_memory_slot *new,
enum kvm_mr_change change);
void s390_kvm_mmu_commit_memory_region(struct kvm *kvm,
struct kvm_memory_slot *old,
const struct kvm_memory_slot *new,
enum kvm_mr_change change);
#endif /* ARCH_KVM_GMAP_KVM_MMU_H */

13
arch/s390/kvm/s390/gaccess.c
View File

@ -1589,12 +1589,25 @@ static inline int ___gaccess_shadow_fault(struct kvm_vcpu *vcpu, struct gmap *sg
parent = READ_ONCE(sg->parent);
if (!parent)
return -EAGAIN;
retry:
scoped_guard(spinlock, &parent->children_lock) {
if (READ_ONCE(sg->parent) != parent)
return -EAGAIN;
sg->invalidated = false;
rc = _gaccess_do_shadow(vcpu->arch.mc, sg, saddr, walk);
}
if (rc == -ENOENT) {
struct kvm_memory_slot *slot;
struct guest_fault *entries;
entries = get_entries(walk);
slot = kvm_vcpu_gfn_to_memslot(vcpu, entries[LEVEL_MEM].gfn);
if (!slot)
return PGM_ADDRESSING;
rc = gmap_link(vcpu->arch.mc, parent, entries + LEVEL_MEM, slot);
if (!rc)
goto retry;
}
if (!rc)
kvm_s390_release_faultin_array(vcpu->kvm, walk->raw_entries, false);
return rc;

81
arch/s390/kvm/s390/interrupt.c
View File

@ -2984,61 +2984,58 @@ static int adapter_indicators_set(struct kvm *kvm,
struct s390_io_adapter *adapter,
struct kvm_s390_adapter_int *adapter_int)
{
unsigned long bit;
int summary_set, idx;
struct s390_map_info *ind_info, *summary_info;
void *map;
struct page *ind_page, *summary_page;
unsigned long flags;
unsigned long bit;
int summary_set;
void *map;
ind_page = NULL;
spin_lock_irqsave(&adapter->maps_lock, flags);
ind_info = get_map_info(adapter, adapter_int->ind_addr);
scoped_guard(spinlock_irqsave, &adapter->maps_lock) {
ind_info = get_map_info(adapter, adapter_int->ind_addr);
if (ind_info) {
map = page_address(ind_info->page);
bit = get_ind_bit(ind_info->addr, adapter_int->ind_offset, adapter->swap);
set_bit(bit, map);
}
}
if (!ind_info) {
spin_unlock_irqrestore(&adapter->maps_lock, flags);
ind_page = pin_map_page(kvm, adapter_int->ind_addr, 0);
if (!ind_page)
return -1;
idx = srcu_read_lock(&kvm->srcu);
map = page_address(ind_page);
bit = get_ind_bit(adapter_int->ind_addr,
adapter_int->ind_offset, adapter->swap);
set_bit(bit, map);
mark_page_dirty(kvm, adapter_int->ind_gaddr >> PAGE_SHIFT);
set_page_dirty_lock(ind_page);
srcu_read_unlock(&kvm->srcu, idx);
unpin_user_page(ind_page);
} else {
map = page_address(ind_info->page);
bit = get_ind_bit(ind_info->addr, adapter_int->ind_offset, adapter->swap);
set_bit(bit, map);
spin_unlock_irqrestore(&adapter->maps_lock, flags);
}
scoped_guard(srcu, &kvm->srcu)
mark_page_dirty(kvm, gpa_to_gfn(adapter_int->ind_gaddr));
spin_lock_irqsave(&adapter->maps_lock, flags);
summary_info = get_map_info(adapter, adapter_int->summary_addr);
scoped_guard(spinlock_irqsave, &adapter->maps_lock) {
summary_info = get_map_info(adapter, adapter_int->summary_addr);
if (summary_info) {
map = page_address(summary_info->page);
bit = get_ind_bit(summary_info->addr, adapter_int->summary_offset,
adapter->swap);
summary_set = test_and_set_bit(bit, map);
}
}
if (!summary_info) {
spin_unlock_irqrestore(&adapter->maps_lock, flags);
summary_page = pin_map_page(kvm, adapter_int->summary_addr, 0);
if (!summary_page)
return -1;
idx = srcu_read_lock(&kvm->srcu);
map = page_address(summary_page);
bit = get_ind_bit(adapter_int->summary_addr,
adapter_int->summary_offset, adapter->swap);
summary_set = test_and_set_bit(bit, map);
mark_page_dirty(kvm, adapter_int->summary_gaddr >> PAGE_SHIFT);
set_page_dirty_lock(summary_page);
srcu_read_unlock(&kvm->srcu, idx);
unpin_user_page(summary_page);
} else {
map = page_address(summary_info->page);
bit = get_ind_bit(summary_info->addr, adapter_int->summary_offset,
adapter->swap);
summary_set = test_and_set_bit(bit, map);
spin_unlock_irqrestore(&adapter->maps_lock, flags);
}
scoped_guard(srcu, &kvm->srcu)
mark_page_dirty(kvm, gpa_to_gfn(adapter_int->summary_gaddr));
return summary_set ? 0 : 1;
}
@ -3048,26 +3045,29 @@ static int adapter_indicators_set_fast(struct kvm *kvm,
struct kvm_s390_adapter_int *adapter_int,
int setbit)
{
struct s390_map_info *ind_info, *summary_info;
unsigned long bit;
int summary_set;
struct s390_map_info *ind_info, *summary_info;
void *map;
spin_lock(&adapter->maps_lock);
guard(srcu)(&kvm->srcu);
guard(spinlock)(&adapter->maps_lock);
ind_info = get_map_info(adapter, adapter_int->ind_addr);
if (!ind_info) {
spin_unlock(&adapter->maps_lock);
if (!ind_info)
return -EWOULDBLOCK;
}
map = page_address(ind_info->page);
bit = get_ind_bit(ind_info->addr, adapter_int->ind_offset, adapter->swap);
if (setbit)
if (setbit) {
set_bit(bit, map);
summary_info = get_map_info(adapter, adapter_int->summary_addr);
if (!summary_info) {
spin_unlock(&adapter->maps_lock);
return -EWOULDBLOCK;
mark_page_dirty(kvm, gpa_to_gfn(adapter_int->ind_gaddr));
}
summary_info = get_map_info(adapter, adapter_int->summary_addr);
if (!summary_info)
return -EWOULDBLOCK;
map = page_address(summary_info->page);
bit = get_ind_bit(summary_info->addr, adapter_int->summary_offset,
adapter->swap);
@ -3077,7 +3077,8 @@ static int adapter_indicators_set_fast(struct kvm *kvm,
summary_set = test_and_set_bit(bit, map);
else
summary_set = test_and_clear_bit(bit, map);
spin_unlock(&adapter->maps_lock);
mark_page_dirty(kvm, gpa_to_gfn(adapter_int->summary_gaddr));
return summary_set ? 0 : 1;
}
@ -3228,9 +3229,9 @@ int kvm_s390_set_irq_state(struct kvm_vcpu *vcpu, void __user *irqstate, int len
break;
}
}
if (storestatus) {
n = kvm_s390_store_status_unloaded(vcpu, KVM_S390_STORE_STATUS_NOADDR);
scoped_guard(srcu, &vcpu->kvm->srcu)
n = kvm_s390_store_status_unloaded(vcpu, KVM_S390_STORE_STATUS_NOADDR);
return r ? r : n;
}

3
arch/s390/kvm/s390/s390.c
View File

@ -5766,7 +5766,7 @@ static long cmma_d_count_pte(union pte *ptep, gfn_t gfn, gfn_t next, struct dat_
return 0;
}
void kvm_s390_update_cmma_dirty(struct kvm *kvm, struct kvm_memory_slot *old)
void kvm_s390_update_cmma_dirty(struct kvm *kvm, const struct kvm_memory_slot *old)
{
const struct dat_walk_ops ops = { .pte_entry = cmma_d_count_pte, };
@ -5781,7 +5781,6 @@ void kvm_arch_commit_memory_region(struct kvm *kvm, struct kvm_memory_slot *old,
const struct kvm_memory_slot *new,
enum kvm_mr_change change)
{
s390_kvm_mmu_commit_memory_region(kvm, old, new, change);
}
/**

2
arch/s390/kvm/s390/s390.h
View File

@ -472,7 +472,7 @@ int __kvm_s390_mprotect_many(struct gmap *gmap, gpa_t gpa, u8 npages, unsigned i
unsigned long bits);
bool kvm_arch_setup_async_pf(struct kvm_vcpu *vcpu);
void kvm_s390_update_cmma_dirty(struct kvm *kvm, struct kvm_memory_slot *old);
void kvm_s390_update_cmma_dirty(struct kvm *kvm, const struct kvm_memory_slot *old);
int kvm_s390_vm_stop_migration(struct kvm *kvm);

2
arch/x86/kvm/mmu/mmu.c
View File

@ -5058,7 +5058,7 @@ static int kvm_tdp_page_prefault(struct kvm_vcpu *vcpu, gpa_t gpa,
if (signal_pending(current))
return -EINTR;
if (kvm_check_request(KVM_REQ_VM_DEAD, vcpu))
if (kvm_test_request(KVM_REQ_VM_DEAD, vcpu))
return -EIO;
cond_resched();

8
arch/x86/kvm/pmu.c
View File

@ -823,14 +823,6 @@ void kvm_pmu_deliver_pmi(struct kvm_vcpu *vcpu)
bool kvm_pmu_is_valid_msr(struct kvm_vcpu *vcpu, u32 msr)
{
switch (msr) {
case MSR_CORE_PERF_GLOBAL_STATUS:
case MSR_CORE_PERF_GLOBAL_CTRL:
case MSR_CORE_PERF_GLOBAL_OVF_CTRL:
return kvm_pmu_has_perf_global_ctrl(vcpu_to_pmu(vcpu));
default:
break;
}
return kvm_pmu_call(msr_idx_to_pmc)(vcpu, msr) ||
kvm_pmu_call(is_valid_msr)(vcpu, msr);
}

7
arch/x86/kvm/svm/nested.c
View File

@ -2125,13 +2125,8 @@ static bool svm_get_nested_state_pages(struct kvm_vcpu *vcpu)
return false;
}
if (!nested_svm_merge_msrpm(vcpu)) {
vcpu->run->exit_reason = KVM_EXIT_INTERNAL_ERROR;
vcpu->run->internal.suberror =
KVM_INTERNAL_ERROR_EMULATION;
vcpu->run->internal.ndata = 0;
if (!nested_svm_merge_msrpm(vcpu))
return false;
}
if (kvm_hv_verify_vp_assist(vcpu))
return false;

18
arch/x86/kvm/svm/sev.c
View File

@ -2048,6 +2048,12 @@ static void sev_migrate_from(struct kvm *dst_kvm, struct kvm *src_kvm)
src->pages_locked = 0;
src->es_active = false;
/*
* Do cache maintenance on the source VM as it is no longer an SEV VM,
* i.e. memory reclaim flows won't trigger cache maintenance on the VM.
*/
sev_writeback_caches(src_kvm);
list_cut_before(&dst->regions_list, &src->regions_list, &src->regions_list);
mutex_lock(&sev_mirror_lock);
@ -2187,6 +2193,10 @@ int sev_vm_move_enc_context_from(struct kvm *kvm, unsigned int source_fd)
* the set of CPUs from the source. If a CPU was used to run a vCPU in
* the source VM but is never used for the destination VM, then the CPU
* can only have cached memory that was accessible to the source VM.
* Furthermore, KVM *must* perform cache maintenance on the source VM,
* as the source VM may have access to memory that the destination VM
* does not, i.e. KVM could skip flushes if memory is reclaimed from
* the old VM but not the new VM.
*/
if (!zalloc_cpumask_var(&dst_sev->have_run_cpus, GFP_KERNEL_ACCOUNT)) {
ret = -ENOMEM;
@ -2980,13 +2990,17 @@ void sev_vm_destroy(struct kvm *kvm)
struct list_head *head = &sev->regions_list;
struct list_head *pos, *q;
/*
* Free the mask even if the VM is not *currently* an SEV VM, as it may
* have been an SEV VM prior to intra-host migration.
*/
free_cpumask_var(sev->have_run_cpus);
if (!sev_guest(kvm))
return;
WARN_ON(!list_empty(&sev->mirror_vms));
free_cpumask_var(sev->have_run_cpus);
/*
* If this is a mirror VM, remove it from the owner's list of a mirrors
* and skip ASID cleanup (the ASID is tied to the lifetime of the owner).

15
arch/x86/kvm/vmx/nested.c
View File

@ -3465,10 +3465,6 @@ static bool nested_get_vmcs12_pages(struct kvm_vcpu *vcpu)
} else {
pr_debug_ratelimited("%s: no backing for APIC-access address in vmcs12\n",
__func__);
vcpu->run->exit_reason = KVM_EXIT_INTERNAL_ERROR;
vcpu->run->internal.suberror =
KVM_INTERNAL_ERROR_EMULATION;
vcpu->run->internal.ndata = 0;
return false;
}
}
@ -3539,11 +3535,6 @@ static bool vmx_get_nested_state_pages(struct kvm_vcpu *vcpu)
if (!nested_get_evmcs_page(vcpu)) {
pr_debug_ratelimited("%s: enlightened vmptrld failed\n",
__func__);
vcpu->run->exit_reason = KVM_EXIT_INTERNAL_ERROR;
vcpu->run->internal.suberror =
KVM_INTERNAL_ERROR_EMULATION;
vcpu->run->internal.ndata = 0;
return false;
}
#endif
@ -3915,8 +3906,12 @@ static int nested_vmx_run(struct kvm_vcpu *vcpu, bool launch)
vmentry_failed:
vcpu->arch.nested_run_pending = 0;
if (status == NVMX_VMENTRY_KVM_INTERNAL_ERROR)
if (status == NVMX_VMENTRY_KVM_INTERNAL_ERROR) {
vcpu->run->exit_reason = KVM_EXIT_INTERNAL_ERROR;
vcpu->run->internal.suberror = KVM_INTERNAL_ERROR_EMULATION;
vcpu->run->internal.ndata = 0;
return 0;
}
if (status == NVMX_VMENTRY_VMEXIT)
return 1;
WARN_ON_ONCE(status != NVMX_VMENTRY_VMFAIL);

3
arch/x86/kvm/vmx/pmu_intel.c
View File

@ -187,6 +187,9 @@ static bool intel_is_valid_msr(struct kvm_vcpu *vcpu, u32 msr)
int ret;
switch (msr) {
case MSR_CORE_PERF_GLOBAL_STATUS:
case MSR_CORE_PERF_GLOBAL_CTRL:
case MSR_CORE_PERF_GLOBAL_OVF_CTRL:
case MSR_CORE_PERF_FIXED_CTR_CTRL:
return kvm_pmu_has_perf_global_ctrl(pmu);
case MSR_IA32_PEBS_ENABLE:

2
arch/x86/kvm/vmx/tdx.c
View File

@ -1995,7 +1995,7 @@ static int tdx_handle_ept_violation(struct kvm_vcpu *vcpu)
if (kvm_vcpu_has_events(vcpu) || signal_pending(current))
break;
if (kvm_check_request(KVM_REQ_VM_DEAD, vcpu)) {
if (kvm_test_request(KVM_REQ_VM_DEAD, vcpu)) {
ret = -EIO;
break;
}

6
arch/x86/kvm/x86.c
View File

@ -8060,7 +8060,7 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)
bool req_immediate_exit = false;
if (kvm_request_pending(vcpu)) {
if (kvm_check_request(KVM_REQ_VM_DEAD, vcpu)) {
if (kvm_test_request(KVM_REQ_VM_DEAD, vcpu)) {
r = -EIO;
goto out;
}
@ -8072,6 +8072,10 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)
if (kvm_check_request(KVM_REQ_GET_NESTED_STATE_PAGES, vcpu)) {
if (unlikely(!kvm_nested_call(get_nested_state_pages)(vcpu))) {
vcpu->run->exit_reason = KVM_EXIT_INTERNAL_ERROR;
vcpu->run->internal.suberror = KVM_INTERNAL_ERROR_EMULATION;
vcpu->run->internal.ndata = 0;
kvm_make_request(KVM_REQ_GET_NESTED_STATE_PAGES, vcpu);
r = 0;
goto out;
}

18
drivers/s390/crypto/vfio_ap_ops.c
View File

@ -277,9 +277,9 @@ static void vfio_ap_free_aqic_resources(struct vfio_ap_queue *q)
{
if (!q)
return;
if (q->saved_isc != VFIO_AP_ISC_INVALID &&
!WARN_ON(!(q->matrix_mdev && q->matrix_mdev->kvm))) {
kvm_s390_gisc_unregister(q->matrix_mdev->kvm, q->saved_isc);
if (q->saved_isc != VFIO_AP_ISC_INVALID) {
if (!WARN_ON(!q->matrix_mdev) && q->matrix_mdev->kvm)
kvm_s390_gisc_unregister(q->matrix_mdev->kvm, q->saved_isc);
q->saved_isc = VFIO_AP_ISC_INVALID;
}
if (q->saved_iova && !WARN_ON(!q->matrix_mdev)) {
@ -1935,6 +1935,8 @@ static int apq_status_check(int apqn, struct ap_queue_status *status)
* a value indicating a reset needs to be performed again.
*/
return -EAGAIN;
case AP_RESPONSE_Q_NOT_AVAIL:
return -ENODEV;
default:
WARN(true,
"failed to verify reset of queue %02x.%04x: TAPQ rc=%u\n",
@ -1961,6 +1963,10 @@ static void apq_reset_check(struct work_struct *reset_work)
ret = apq_status_check(q->apqn, &status);
if (ret == -EIO)
return;
if (ret == -ENODEV) {
vfio_ap_free_aqic_resources(q);
return;
}
if (ret == -EBUSY) {
pr_notice_ratelimited(WAIT_MSG, elapsed,
AP_QID_CARD(q->apqn),
@ -2004,6 +2010,7 @@ static void vfio_ap_mdev_reset_queue(struct vfio_ap_queue *q)
break;
case AP_RESPONSE_DECONFIGURED:
case AP_RESPONSE_CHECKSTOPPED:
case AP_RESPONSE_Q_NOT_AVAIL:
vfio_ap_free_aqic_resources(q);
break;
default:
@ -2528,12 +2535,15 @@ void vfio_ap_mdev_remove_queue(struct ap_device *apdev)
/*
* If the queue is not in the host's AP configuration, then resetting
* it will fail with response code 01, (APQN not valid); so, let's make
* sure it is in the host's config.
* sure it is in the host's config. If it is not, free the KVM GISC
* resources.
*/
if (test_bit_inv(apid, (unsigned long *)matrix_dev->info.apm) &&
test_bit_inv(apqi, (unsigned long *)matrix_dev->info.aqm)) {
vfio_ap_mdev_reset_queue(q);
flush_work(&q->reset_work);
} else {
vfio_ap_free_aqic_resources(q);
}
done:

9
include/linux/kvm_host.h
View File

@ -2324,13 +2324,18 @@ static inline bool kvm_test_request(int req, struct kvm_vcpu *vcpu)
return test_bit(req & KVM_REQUEST_MASK, (void *)&vcpu->requests);
}
static inline void kvm_clear_request(int req, struct kvm_vcpu *vcpu)
static __always_inline void kvm_clear_request(int req, struct kvm_vcpu *vcpu)
{
BUILD_BUG_ON(req == KVM_REQ_VM_DEAD);
clear_bit(req & KVM_REQUEST_MASK, (void *)&vcpu->requests);
}
static inline bool kvm_check_request(int req, struct kvm_vcpu *vcpu)
static __always_inline bool kvm_check_request(int req, struct kvm_vcpu *vcpu)
{
/* Once a VM is dead, it needs to stay dead. */
BUILD_BUG_ON(req == KVM_REQ_VM_DEAD);
if (kvm_test_request(req, vcpu)) {
kvm_clear_request(req, vcpu);

20
tools/arch/riscv/include/asm/csr.h
View File

@ -163,12 +163,24 @@
#define HGATP_MODE_SHIFT HGATP32_MODE_SHIFT
#endif
/* VSIP & HVIP relation */
/*
* VSIP & HVIP relation
*
* The bit positions are same between VSIP and HVIP for interrupt
* numbers 13-63, where there's a shift for the SSI, STI and SEI.
*/
#define VSIP_TO_HVIP_SHIFT (IRQ_VS_SOFT - IRQ_S_SOFT)
#define VSIP_VALID_MASK ((_AC(1, UL) << IRQ_S_SOFT) | \
#define VSIP_BIAS_MASK ((_AC(1, UL) << IRQ_S_SOFT) | \
(_AC(1, UL) << IRQ_S_TIMER) | \
(_AC(1, UL) << IRQ_S_EXT) | \
(_AC(1, UL) << IRQ_PMU_OVF))
(_AC(1, UL) << IRQ_S_EXT))
#define VSIP_NO_BIAS_MASK (_AC(1, UL) << IRQ_PMU_OVF)
#define VSIP_VALID_MASK (VSIP_BIAS_MASK | VSIP_NO_BIAS_MASK)
#define vsip_to_hvip(_vsip) ((((_vsip) & VSIP_BIAS_MASK) << \
VSIP_TO_HVIP_SHIFT) | \
((_vsip) & VSIP_NO_BIAS_MASK))
#define hvip_to_vsip(_hvip) ((((_hvip) >> VSIP_TO_HVIP_SHIFT) & \
VSIP_BIAS_MASK) | \
((_hvip) & VSIP_NO_BIAS_MASK))
/* AIA CSR bits */
#define TOPI_IID_SHIFT 16

1
tools/testing/selftests/kvm/Makefile.kvm
View File

@ -189,6 +189,7 @@ TEST_GEN_PROGS_arm64 += arm64/stage2_block_transitions
TEST_GEN_PROGS_arm64 += arm64/vcpu_width_config
TEST_GEN_PROGS_arm64 += arm64/vgic_init
TEST_GEN_PROGS_arm64 += arm64/vgic_irq
TEST_GEN_PROGS_arm64 += arm64/vgic_its_save
TEST_GEN_PROGS_arm64 += arm64/vgic_lpi_stress
TEST_GEN_PROGS_arm64 += arm64/vgic_v5
TEST_GEN_PROGS_arm64 += arm64/vpmu_counter_access

4
tools/testing/selftests/kvm/arm64/smccc_filter.c
View File

@ -140,6 +140,10 @@ static void test_invalid_nr_functions(void)
TEST_ASSERT(r < 0 && errno == EINVAL,
"Attempt to filter 0 functions should return EINVAL");
r = __set_smccc_filter(vm, 0, 0, KVM_SMCCC_FILTER_DENY);
TEST_ASSERT(r < 0 && errno == EINVAL,
"Attempt to filter 0 functions at base 0 should return EINVAL");
kvm_vm_free(vm);
}

441
tools/testing/selftests/kvm/arm64/vgic_its_save.c Normal file
View File

@ -0,0 +1,441 @@
// SPDX-License-Identifier: GPL-2.0-only
/*
* vgic_its_save - KVM_DEV_ARM_ITS_SAVE_TABLES against tables a guest broke.
*
* Both cases are reachable by a guest on its own, and neither may fail a save
* that userspace has to be able to issue:
*
* - Changing GITS_BASER<coll> drops the collections it described, so the save
* writes nothing but the terminating invalid entry.
* - A device the device table can no longer address is skipped, and the saved
* DTE chain skips it too rather than pointing at an entry never written.
*
* Both cases then reset and restore, which is what the save exists for.
*
* Copyright (c) 2026 Google LLC
* Author: Fuad Tabba <fuad.tabba@linux.dev>
*/
#include <endian.h>
#include <linux/align.h>
#include <linux/bitfield.h>
#include <linux/sizes.h>
#include "kvm_util.h"
#include "gic.h"
#include "gic_v3.h"
#include "gic_v3_its.h"
#include "processor.h"
#include "ucall.h"
#include "vgic.h"
#define TEST_MEMSLOT_INDEX 1
/* All three ITS table entry sizes are 8 bytes in ABI 0. */
#define ESZ 8
#define ENTRIES_PER_PAGE (SZ_64K / ESZ)
/* CTE and DTE layout, mirroring KVM's KVM_ITS_* in arch/arm64/kvm/vgic/vgic.h */
#define CTE_VALID_MASK BIT_ULL(63)
#define DTE_VALID_MASK BIT_ULL(63)
#define DTE_NEXT_SHIFT 49
#define DTE_NEXT_MASK GENMASK_ULL(62, 49)
/* L1 entry of an indirect table: valid bit plus a 64K aligned L2 address. */
#define L1E_VALID_MASK BIT_ULL(63)
#define L1E_ADDR_MASK GENMASK_ULL(51, 16)
#define GITS_BASER_PAGES_MASK GENMASK_ULL(7, 0)
#define POISON 0xdeadbeefdeadbeefULL
/* The collection table starts at two pages and is shrunk to one. */
#define COLL_TBL_PAGES 2
#define COLL_TBL_SZ (COLL_TBL_PAGES * SZ_64K)
/* One more collection than the shrunken table can hold. */
#define NR_COLLECTIONS (ENTRIES_PER_PAGE + 1)
/* Two devices, one per L2 block of the indirect device table. */
#define DEVICE_A_ID 0
#define DEVICE_B_ID ENTRIES_PER_PAGE
/*
* its_send_mapd_cmd() encodes ilog2(itt_size) - 1 as num_eventid_bits, and
* vgic_its_restore_itt() scans BIT_ULL(num_eventid_bits) * ESZ, so the size
* handed to MAPD has to match the ITT allocated for it.
*/
#define ITT_EVENTID_BITS 13
#define ITT_MAPD_SIZE BIT_ULL(ITT_EVENTID_BITS + 1)
#define ITT_SZ (BIT_ULL(ITT_EVENTID_BITS) * ESZ)
static struct kvm_vm *vm;
static struct kvm_vcpu *vcpu;
static int its_fd;
static gpa_t gpa_base;
static struct test_data {
gpa_t device_table;
gpa_t collection_table;
gpa_t cmdq_base;
void *cmdq_base_va;
gpa_t lpi_prop_table;
gpa_t lpi_pend_table;
void *device_l1_va;
gpa_t device_l2[2];
gpa_t itt_tables;
} test_data;
static unsigned long its_baser_offset(unsigned int type)
{
int i;
for (i = 0; i < GITS_BASER_NR_REGS; i++) {
unsigned long offset = GITS_BASER + (i * sizeof(u64));
u64 baser = readq_relaxed(GITS_BASE_GVA + offset);
if (GITS_BASER_TYPE(baser) == type)
return offset;
}
GUEST_FAIL("Couldn't find an ITS BASER of type %u", type);
return -1;
}
static void its_set_enable(bool enable)
{
u32 ctlr = readl_relaxed(GITS_BASE_GVA + GITS_CTLR);
if (enable)
ctlr |= GITS_CTLR_ENABLE;
else
ctlr &= ~GITS_CTLR_ENABLE;
writel_relaxed(ctlr, GITS_BASE_GVA + GITS_CTLR);
}
/*
* Shrink the collection table to a single page, leaving VALID set. BASER
* writes are ignored while the ITS is enabled.
*/
static void guest_shrink_coll_table(void)
{
unsigned long offset = its_baser_offset(GITS_BASER_TYPE_COLLECTION);
u64 baser;
its_set_enable(false);
baser = readq_relaxed(GITS_BASE_GVA + offset);
baser &= ~GITS_BASER_PAGES_MASK;
writeq_relaxed(baser, GITS_BASE_GVA + offset);
}
static void guest_baser_change(void)
{
u32 coll_id;
gic_init(GIC_V3, 1);
gic_rdist_enable_lpis(test_data.lpi_prop_table, SZ_64K,
test_data.lpi_pend_table);
its_init(test_data.collection_table, COLL_TBL_SZ,
test_data.device_table, SZ_64K,
test_data.cmdq_base, SZ_64K);
for (coll_id = 0; coll_id < NR_COLLECTIONS; coll_id++)
its_send_mapc_cmd(test_data.cmdq_base_va, 0, coll_id, true);
guest_shrink_coll_table();
GUEST_DONE();
}
/* Turn the already installed device table into an indirect one. */
static void guest_make_device_table_indirect(void)
{
unsigned long offset = its_baser_offset(GITS_BASER_TYPE_DEVICE);
u64 baser;
its_set_enable(false);
baser = readq_relaxed(GITS_BASE_GVA + offset);
writeq_relaxed(baser | GITS_BASER_INDIRECT, GITS_BASE_GVA + offset);
its_set_enable(true);
}
static void guest_unreachable_device(void)
{
u64 *l1;
gic_init(GIC_V3, 1);
gic_rdist_enable_lpis(test_data.lpi_prop_table, SZ_64K,
test_data.lpi_pend_table);
its_init(test_data.collection_table, SZ_64K,
test_data.device_table, SZ_64K,
test_data.cmdq_base, SZ_64K);
guest_make_device_table_indirect();
/* Both L2 blocks present, so both MAPDs are in range. */
l1 = test_data.device_l1_va;
l1[0] = L1E_VALID_MASK | (test_data.device_l2[0] & L1E_ADDR_MASK);
l1[1] = L1E_VALID_MASK | (test_data.device_l2[1] & L1E_ADDR_MASK);
its_send_mapd_cmd(test_data.cmdq_base_va, DEVICE_A_ID,
test_data.itt_tables, ITT_MAPD_SIZE, true);
its_send_mapd_cmd(test_data.cmdq_base_va, DEVICE_B_ID,
test_data.itt_tables + ITT_SZ, ITT_MAPD_SIZE, true);
/*
* Drop the block holding device B. No ITS command and no GITS_BASER
* write is involved, so nothing tells KVM the device is now
* unreachable.
*/
l1[1] = 0;
GUEST_DONE();
}
static void run_guest(void)
{
struct ucall uc;
vcpu_run(vcpu);
switch (get_ucall(vcpu, &uc)) {
case UCALL_DONE:
break;
case UCALL_ABORT:
REPORT_GUEST_ASSERT(uc);
break;
default:
TEST_FAIL("Unexpected ucall: %lu", uc.cmd);
}
}
static int save_tables(void)
{
return __kvm_device_attr_set(its_fd, KVM_DEV_ARM_VGIC_GRP_CTRL,
KVM_DEV_ARM_ITS_SAVE_TABLES, NULL);
}
static u64 its_reg_get(unsigned long offset)
{
u64 val;
kvm_device_attr_get(its_fd, KVM_DEV_ARM_VGIC_GRP_ITS_REGS, offset,
&val);
return val;
}
static void its_reg_set(unsigned long offset, u64 val)
{
kvm_device_attr_set(its_fd, KVM_DEV_ARM_VGIC_GRP_ITS_REGS, offset,
&val);
}
/*
* What a migration target does with the saved tables, in the order
* Documentation/virt/kvm/devices/arm-vgic-its.rst gives: the GITS_ registers
* first, then the tables. The reset in between clears GITS_BASER<n>.Valid,
* which is why the registers have to be written back before the restore.
*/
static void reset_and_restore_tables(void)
{
u64 baser[GITS_BASER_NR_REGS];
int ret, i;
for (i = 0; i < GITS_BASER_NR_REGS; i++)
baser[i] = its_reg_get(GITS_BASER + (i * sizeof(u64)));
ret = __kvm_device_attr_set(its_fd, KVM_DEV_ARM_VGIC_GRP_CTRL,
KVM_DEV_ARM_ITS_CTRL_RESET, NULL);
TEST_ASSERT(!ret, "Expected the reset to succeed, got ret %d errno %d",
ret, errno);
for (i = 0; i < GITS_BASER_NR_REGS; i++)
its_reg_set(GITS_BASER + (i * sizeof(u64)), baser[i]);
ret = __kvm_device_attr_set(its_fd, KVM_DEV_ARM_VGIC_GRP_CTRL,
KVM_DEV_ARM_ITS_RESTORE_TABLES, NULL);
TEST_ASSERT(!ret, "Expected the restore to succeed, got ret %d errno %d",
ret, errno);
}
static void poison_range(gpa_t base, size_t size)
{
u64 *entry = addr_gpa2hva(vm, base);
size_t i;
for (i = 0; i < size / ESZ; i++)
entry[i] = POISON;
}
static void setup_memslot(size_t sz)
{
size_t pages = sz / vm->page_size;
gpa_base = ((vm_compute_max_gfn(vm) + 1) * vm->page_size) - sz;
vm_userspace_mem_region_add(vm, VM_MEM_SRC_ANONYMOUS, gpa_base,
TEST_MEMSLOT_INDEX, pages, 0);
}
static gpa_t alloc_64k(size_t nr)
{
size_t pages_per_64k = vm_calc_num_guest_pages(vm->mode, SZ_64K);
gpa_t gpa = vm_phy_pages_alloc(vm, nr * pages_per_64k, gpa_base,
TEST_MEMSLOT_INDEX);
TEST_ASSERT(IS_ALIGNED(gpa, SZ_64K),
"Allocation at 0x%lx is not 64K aligned, GITS_BASER cannot address it",
gpa);
return gpa;
}
static void map_to_guest(gpa_t gpa, size_t nr)
{
size_t pages_per_64k = vm_calc_num_guest_pages(vm->mode, SZ_64K);
virt_map(vm, gpa, gpa, nr * pages_per_64k);
}
static void setup_common(void)
{
test_data.cmdq_base = alloc_64k(1);
map_to_guest(test_data.cmdq_base, 1);
test_data.cmdq_base_va = (void *)test_data.cmdq_base;
test_data.lpi_prop_table = alloc_64k(1);
test_data.lpi_pend_table = alloc_64k(1);
}
static void teardown(void)
{
close(its_fd);
kvm_vm_free(vm);
memset(&test_data, 0, sizeof(test_data));
}
/*
* A GITS_BASER<coll> write that changes the table drops the collections it
* described. The save then has an empty list, so it writes the terminating
* invalid entry and nothing else.
*/
static void test_baser_change_drops_collections(void)
{
u64 *cte;
int ret, i;
pr_info("Testing that a GITS_BASER change drops the collections\n");
vm = vm_create_with_one_vcpu(&vcpu, guest_baser_change);
setup_memslot((4 + COLL_TBL_PAGES) * SZ_64K);
its_fd = vgic_its_setup(vm);
test_data.device_table = alloc_64k(1);
test_data.collection_table = alloc_64k(COLL_TBL_PAGES);
setup_common();
sync_global_to_guest(vm, test_data);
run_guest();
/* Anything KVM writes is then the only thing that changed. */
poison_range(test_data.collection_table, COLL_TBL_SZ);
ret = save_tables();
TEST_ASSERT(!ret, "Expected the save to succeed, got %d errno %d",
ret, errno);
cte = addr_gpa2hva(vm, test_data.collection_table);
/*
* Finding the terminator at the head of the table is also what proves
* the reads below landed in the saved table rather than elsewhere.
*/
TEST_ASSERT(le64toh(cte[0]) == 0,
"CTE 0: expected the terminating invalid entry, got 0x%llx",
(unsigned long long)le64toh(cte[0]));
for (i = 1; i < COLL_TBL_SZ / ESZ; i++)
TEST_ASSERT(cte[i] == POISON,
"CTE %d: expected it untouched, got 0x%llx",
i, (unsigned long long)cte[i]);
reset_and_restore_tables();
teardown();
}
/*
* A device whose L2 block the guest dropped is skipped by the save, and the
* DTE chain skips it too: left alone, the surviving device would point at an
* entry the save never wrote.
*/
static void test_unreachable_device_skipped(void)
{
u64 dte;
int ret;
pr_info("Testing that an unreachable device is skipped by the save\n");
vm = vm_create_with_one_vcpu(&vcpu, guest_unreachable_device);
setup_memslot(9 * SZ_64K);
its_fd = vgic_its_setup(vm);
test_data.device_table = alloc_64k(1);
test_data.collection_table = alloc_64k(1);
test_data.device_l2[0] = alloc_64k(1);
test_data.device_l2[1] = alloc_64k(1);
test_data.itt_tables = alloc_64k(2);
setup_common();
map_to_guest(test_data.device_table, 1);
test_data.device_l1_va = (void *)test_data.device_table;
sync_global_to_guest(vm, test_data);
run_guest();
poison_range(test_data.device_l2[0], SZ_64K);
poison_range(test_data.device_l2[1], SZ_64K);
ret = save_tables();
TEST_ASSERT(!ret, "Expected the save to succeed, got %d errno %d",
ret, errno);
dte = le64toh(*(u64 *)addr_gpa2hva(vm, test_data.device_l2[0]));
/* Device A is still reachable, so it is saved. */
TEST_ASSERT(dte & DTE_VALID_MASK,
"Device A: expected a valid DTE, got 0x%llx",
(unsigned long long)dte);
/*
* Device B is the only device after it and was skipped, so nothing
* follows A in the saved chain.
*/
TEST_ASSERT(FIELD_GET(DTE_NEXT_MASK, dte) == 0,
"Device A: expected no next device, got offset %llu",
(unsigned long long)FIELD_GET(DTE_NEXT_MASK, dte));
/* And nothing was written into the block the guest dropped. */
TEST_ASSERT(*(u64 *)addr_gpa2hva(vm, test_data.device_l2[1]) == POISON,
"Device B: expected its entry untouched");
reset_and_restore_tables();
teardown();
}
int main(void)
{
TEST_REQUIRE(kvm_supports_vgic_v3());
test_baser_change_drops_collections();
test_unreachable_device_skipped();
pr_info("All ok!\n");
return 0;
}

6
tools/testing/selftests/kvm/hardware_disable_test.c
View File

@ -37,7 +37,7 @@ static void *run_vcpu(void *arg)
struct kvm_vcpu *vcpu = arg;
struct kvm_run *run = vcpu->run;
#ifndef _GNU_SOURCE
#ifndef __GLIBC__
kvm_sched_setaffinity(0, sizeof(cpu_set_t), &threads_cpu_set);
#endif
@ -51,7 +51,7 @@ static void *sleeping_thread(void *arg)
{
int fd;
#ifndef _GNU_SOURCE
#ifndef __GLIBC__
kvm_sched_setaffinity(0, sizeof(cpu_set_t), &threads_cpu_set);
#endif
@ -71,7 +71,7 @@ static void run_test(u32 run)
u32 i, j;
TEST_ASSERT_EQ(pthread_attr_init(&attr), 0);
#ifdef _GNU_SOURCE
#ifdef __GLIBC__
TEST_ASSERT_EQ(pthread_attr_setaffinity_np(&attr, sizeof(cpu_set_t), &threads_cpu_set), 0);
#endif

30
tools/testing/selftests/kvm/steal_time.c
View File

@ -27,6 +27,9 @@
static void *st_gva[NR_VCPUS];
static u64 guest_stolen_time[NR_VCPUS];
static struct kvm_vm *vm_create_steal_time(u32 nr_vcpus, void *guest_code,
struct kvm_vcpu *vcpus[]);
#if defined(__x86_64__)
/* steal_time must have 64-byte alignment */
@ -210,17 +213,14 @@ static void check_steal_time_uapi(void)
u64 st_ipa;
int ret;
vm = vm_create_with_one_vcpu(&vcpu, NULL);
struct kvm_device_attr dev = {
.group = KVM_ARM_VCPU_PVTIME_CTRL,
.attr = KVM_ARM_VCPU_PVTIME_IPA,
.addr = (u64)&st_ipa,
};
vm = vm_create_steal_time(1, NULL, &vcpu);
vcpu_ioctl(vcpu, KVM_HAS_DEVICE_ATTR, &dev);
vm_userspace_mem_region_add(vm, VM_MEM_SRC_ANONYMOUS, ST_GPA_BASE, 1, 1, 0);
virt_map(vm, ST_GPA_BASE, ST_GPA_BASE, 1);
st_ipa = (ulong)ST_GPA_BASE | 1;
ret = __vcpu_ioctl(vcpu, KVM_SET_DEVICE_ATTR, &dev);
@ -500,13 +500,27 @@ static void run_vcpu(struct kvm_vcpu *vcpu)
}
}
static struct kvm_vm *vm_create_steal_time(u32 nr_vcpus, void *guest_code,
struct kvm_vcpu *vcpus[])
{
unsigned int gpages;
struct kvm_vm *vm;
/* Create a VM and an identity mapped memslot for the steal time structure */
vm = vm_create_with_vcpus(nr_vcpus, guest_code, vcpus);
gpages = vm_calc_num_guest_pages(VM_MODE_DEFAULT, STEAL_TIME_SIZE * nr_vcpus);
vm_userspace_mem_region_add(vm, VM_MEM_SRC_ANONYMOUS, ST_GPA_BASE, 1, gpages, 0);
virt_map(vm, ST_GPA_BASE, ST_GPA_BASE, gpages);
return vm;
}
int main(int ac, char **av)
{
struct kvm_vcpu *vcpus[NR_VCPUS];
struct kvm_vm *vm;
pthread_t thread;
cpu_set_t cpuset;
unsigned int gpages;
long stolen_time;
long run_delay;
bool verbose;
@ -517,11 +531,7 @@ int main(int ac, char **av)
/* Set CPU affinity so we can force preemption of the VCPU */
cpu = pin_self_to_any_cpu();
/* Create a VM and an identity mapped memslot for the steal time structure */
vm = vm_create_with_vcpus(NR_VCPUS, guest_code, vcpus);
gpages = vm_calc_num_guest_pages(VM_MODE_DEFAULT, STEAL_TIME_SIZE * NR_VCPUS);
vm_userspace_mem_region_add(vm, VM_MEM_SRC_ANONYMOUS, ST_GPA_BASE, 1, gpages, 0);
virt_map(vm, ST_GPA_BASE, ST_GPA_BASE, gpages);
vm = vm_create_steal_time(NR_VCPUS, guest_code, vcpus);
ksft_print_header();
TEST_REQUIRE(is_steal_time_supported(vcpus[0]));

35
virt/kvm/kvm_main.c
View File

@ -1117,7 +1117,7 @@ static struct kvm *kvm_create_vm(unsigned long type, const char *fdname)
rcuwait_init(&kvm->mn_memslots_update_rcuwait);
xa_init(&kvm->vcpu_array);
#ifdef CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES
xa_init(&kvm->mem_attr_array);
xa_init_flags(&kvm->mem_attr_array, XA_FLAGS_ACCOUNT);
#endif
INIT_LIST_HEAD(&kvm->gpc_list);
@ -2447,14 +2447,36 @@ bool kvm_range_has_memory_attributes(struct kvm *kvm, gfn_t start, gfn_t end,
return (kvm_get_memory_attributes(kvm, start) & mask) == attrs;
guard(rcu)();
if (!attrs)
return !xas_find(&xas, end - 1);
/*
* Lookup the entry for each index instead of iterating over the xarray
* as KVM deletes/nullifies entries to represent "no attributes", and
* the xas index is effectively invalid when no entry is found. I.e.
* matching non-zero attributes for *every* entry effectively requires
* a manually lookup for each index.
*
* Skip pre-allocated, reserved entries, or restart the lookup if the
* xarray was concurrently modified, via xas_retry() ("retry" means the
* entry holds an internal xarray value, i.e. is either invalid or NULL
* from the caller's perspective).
*
* Use xas_next() when looking for non-zero attributes to optimize for
* the case where the start of the range (or the entire range) doesn't
* have any attributes, as xas_next() returns literally the next entry,
* whereas xas_next_entry() returns the next non-NULL entry (bounded by
* a maximum index).
*/
for (index = start; index < end; index++) {
do {
entry = xas_next(&xas);
entry = attrs ? xas_next(&xas) :
xas_next_entry(&xas, end - 1);
} while (xas_retry(&xas, entry));
if (!entry)
return !attrs;
WARN_ON_ONCE(!xa_to_value(entry));
if (xas.xa_index != index ||
(xa_to_value(entry) & mask) != attrs)
return false;
@ -2571,9 +2593,10 @@ static int kvm_vm_set_mem_attributes(struct kvm *kvm, gfn_t start, gfn_t end,
/*
* Reserve memory ahead of time to avoid having to deal with failures
* partway through setting the new attributes.
* partway through setting the new attributes. Storing NULL never
* allocates, so no reservations are needed when clearing.
*/
for (i = start; i < end; i++) {
for (i = start; entry && i < end; i++) {
r = xa_reserve(&kvm->mem_attr_array, i, GFP_KERNEL_ACCOUNT);
if (r)
goto out_unlock;