github-mirror/linux
1
0
Fork 0
mirror of https://github.com/torvalds/linux.git synced 2026-07-27 01:32:21 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity Graph

Arm:

Browse Source 
"This is a bit of an odd merge window on the KVM/arm64 front. There is
 absolutely no new feature in the pull request. It is purely fixes,
 because it is simply becoming too hard to review new stuff when so
 many AI-fuelled fixes hit the list".
 
 - Significant cleanup of the vgic-v5 PPI support which was merged in
   7.1. This makes the code more maintainable, and squashes a couple
   of bugs in the meantime.
 
 - Set of fixes for the handling of the MMU in an NV context,
   particularly VNCR-triggered faults. S1POE support is fixed
   as well.
 
 - Large set of pKVM fixes, mostly addressing recurring issues
   around hypervisor tracking of donated pages in obscure cases
   where the donation could fail and leave things in a bizarre
   state.
 
 - Fixes for the so-called "lazy vgic init", which resulted in
   sleeping operations in non-preemptible sections. This turned
   out to be far more invasive than initially expected...
 
 - Reduce the overhead of L1/L2 context switch by not touching
   the FP registers.
 
 - Fix the way non-implemented page sizes are dealt with when
   a guest insist on using them for S2 translation.
 
 - The usual set of low-impact fixes and cleanups all over the map.
 
 Loongarch:
 
 - On a request for lazy FPU load, load all FPU state that the VM
   supports instead of enabling only the part (FPU, LSX or LASX)
   that caused the FPU load request.
 
 - Some enhancements about interrupt injection.
 
 - Some bug fixes and other small changes.
 
 RISC-V:
 
 - Batch G-stage TLB flushes for GPA range based page table updates
 
 - Convert HGEI line management to fully per-HART
 
 - Fix missing CSR dirty marking when FWFT state updated via ONE_REG
 
 - Fix stale FWFT feature exposure to Guest/VM
 
 - Speed up dirty logging write faults using MMU rwlock and atomic
   PTE updates using cmpxchg() for permission-only changes
 
 - Use flexible array for APLIC IRQ state
 
 - Use kvm_slot_dirty_track_enabled() for logging enable check on
   a memslot
 
 - Avoid skipping valid pages in kvm_riscv_gstage_wp_range()
 
 - Avoid skipping valid pages in kvm_riscv_gstage_unmap_range()
 
 - Use endian-specific __lelong for NACL shared memory
 
 S390:
 
 - KVM_PRE_FAULT_MEMORY support
 
 - Support for 2G hugepages
 
 - Support for the ASTFLEIE 2 facility
 
 - Support for fast inject using kvm_arch_set_irq_inatomic
 
 - Fix potential leak of uninitialized bytes
 
 - A few more misc gmap fixes.
 
 x86:
 
 - Generic support for the more granular permissions allowed by EPT,
   namely "read" (which was previously usurping the U bit) and
   separate execution bits for kernel and userspace.
 
 - Do not assume that all page tables start with U=1/W=1/NX=0 at the
   root, as AMD GMET needs to have U=0 at the root.
 
 - Introduce common assembly macros for use within Intel and AMD
   vendor-specific vmentry code.  This touches the SPEC_CTRL handling,
   which is now entirely done in assembly for Intel (by reusing the
   AMD code that already existed), and register save/restore which
   uses some macro magic to compute the offsets in the struct.  Both
   of these are preparatory changes for upcoming APX support.
 
 - Clean up KVM's register tracking and storage, primarily to prepare for
   APX support, which expands the maximum number of GPRs from 16 to 32.
 
 - Keep a single copy of the PDPTRs rather than two, since architecturally
   there is just one.
 
 - Handle EXIT_FASTPATH_EXIT_USERSPACE in vendor code to ensure vendor code
   gets a chance to handle things like reaping the PML buffer.
 
 - Update KVM's view of PV async enabling if and only if the MSR write fully
   succeeds.
 
 - Fix a variety of issues where the emulator doesn't honor guest-debug state,
   and clean up related code along the way.
 
 - Synthesize EPT Violation and #NPF "error code" bits when injecting faults
   into L1 that didn't originate in hardware (in which case the VMCS/VMCB
   doesn't hold relevant information).
 
 - Add support for virtualizing (well, emulating) AMD's flavor of CPL>0 CPUID
   faulting.
 
 - Clean up the GPR APIs so that KVM's use of "raw" is consistent, and fix a
   variety of minor bugs along the way.
 
 - Fix an OOB memory access due to not checking the VP ID when handling a
   Hyper-V PV TLB flush for L2.
 
 - Fix a bug in the mediated PMU's handling of fixed counters that allowed the
   guest to bypass the PMU event filter.
 
 - Allow userspace to return EAGAIN when handling SNP and TDX hypercalls, so
   the KVM can forward a "retry" status code to the guest, and reserve all
   unused error codes for future usage.
 
 - Overhaul the TDP MMU => S-EPT code to move as much S-EPT specific logic as
   possible into the TDX code, and to funnel (almost) all S-EPT updates into
   a single chokepoint.  The motivation is largely to prepare for upcoming
   Dynamic PAMT support, but the cleanups are nice to have on their own.
 
 - Plug a hole in shadow page table handling, where KVM fails to recursively
   zap nested EPT/NPT shadow page tables when the nested hypervisor tears
   down its own EPT/NPT page tables from the bottom up
 
 x86 (Intel):
 
 - Support for nested MBEC (Mode-Based Execute Control), see above in the
   generic section; also run with MBEC enabled even for non-nested mode.
 
 - Use the kernel's "enum pg_level" in the TDX APIs instead of the TDX-Module's
   level definitions (which are 0-based).
 
 - Rework the TDX memory APIs to not require/assume that guest memory is
   backed by "struct page" (in prepartion for guest_memfd hugepage support).
 
 - Fix a largely benign bug where KVM TDX would incorrectly state it could
   emulate several x2APIC MSRs.
 
 - Use the "safe" WRMSR API when proxying LBR MSR writes as the to-be-written
   value is guest controlled and completely unvalidated.
 
 x86 (AMD):
 
 - Support for nested GMET (Guest Mode Execution Trap), see above in the
   generic section; also run with GMET enabled even for non-nested mode.
 
 - Fixes and minor cleanups to GHCB handling, on top of the earlier work
   already merged into 7.1-rc.
 
 - Ensure KVM's copy of CR0 and CR3 are up-to-date prior to invoking
   fastpath handlers.
 
 - Add support for virtualizing gPAT (KVM previously just used L1's PAT when
   running L2).
 
 - Fix goofs where KVM mishandles side effects (e.g. single-step and PMC
   updates) when emulating VMRUN.
 
 - Fix a variety of bugs in AVIC's handling of x2APIC MSR interception, most
   notably where KVM didn't disable interception of IRR, ISR, and TMR regs.
 
 - Add support for virtualizing Host-Only/Guest-Only bits in the mediated PMU.
 
 - Don't advertise support for unusable VM types, and account for VM types
   that are disabled by firmware, e.g. to mitigate security vulnerabilities.
 
 - Rewrite the SEV {en,de}crypt debug ioctls as they were riddle with bugs and
   unnecessarily complicated, and add comprehensive tests.
 
 - Clean up and deduplicate the SEV page pinning code.
 
 - Fix minor goofs related to writing back CPUID information after firmware
   rejects a CPUID page for an SNP vCPU.
 
 Generic:
 
 - Rename invalidate_begin() to invalidate_start() throughout KVM to follow
   the kernel's nomenclature, e.g. for mmu_notifiers.
 
 - Use guard() to cleanup up various KVM+VFIO flows.
 
 - Minor cleanups.
 
 guest_memfd:
 
 - Return -EEXIST instead of -EINVAL if userspace attempts to bind a gmem
   range to multiple memslots, and fix the test that was supposed to ensure
   KVM returns -EEXIST.
 
 - Treat memslot binding offsets and sizes as unsigned values to fix a bug
   where KVM interprets a large "offset + size" as a negative value and allows
   a nonsensical offset.
 
 - Use the inode number instead of the page offset for the NUMA interleaving
   index to fix a bug where the effective index would jump by two for
   consecutive pages (the caller also adds in the page offset).
 
 Selftests:
 
 - Randomize the dirty log test's delay when reaping the bitmap on the first
   pass, as always waiting only 1ms hid a KVM RISC-V bug as the test reaped the
   bitmap before KVM could build up enough state to hit the bug.
 
 - A pile of one-off fixes and cleanups.
 -----BEGIN PGP SIGNATURE-----
 
 iQFIBAABCgAyFiEE8TM4V0tmI4mGbHaCv/vSX3jHroMFAmoypI0UHHBib256aW5p
 QHJlZGhhdC5jb20ACgkQv/vSX3jHroNCSAf8CyuvicojnZGFsNcoY4DZ0whBj5l8
 IwVkHaptuyGLSwPAebC/DwtU2NBY4CQdbtLjGmxenf+fSQOtcz1YWKbdY4Nr0hJG
 euAb/wO47RqKp1C4WPi7lL2P4XHgmUgD5XZ1qDnRV9gnGddZi3QLLmKgAtNq9Z4m
 +pAdGD2/RyZeI0Dx61+5ufE/z8YBlH6GrzgBTxfvvRNKqNDFykBenxYHI7seEfHC
 BPz0LxjHDD8BMDUo86qaylZnaeQSmFTGO/eTgvCvabxG6WbM1s0w3K1owmUmP7o3
 p9o8274Xh53iK2JYD/qGOtAJXejIwbYKT5bnvUQX0O1EH2+ouvSFapLQ9Q==
 =sSqM
 -----END PGP SIGNATURE-----

Merge tag 'for-linus' of git://git.kernel.org/pub/scm/virt/kvm/kvm

Pull kvm updates from Paolo Bonzini:
 "arm64:

     This is a bit of an odd merge window on the KVM/arm64 front. There
     is absolutely no new feature in the pull request. It is purely
     fixes, because it is simply becoming too hard to review new stuff
     when so many AI-fuelled fixes hit the list.

   - Significant cleanup of the vgic-v5 PPI support which was merged in
     7.1. This makes the code more maintainable, and squashes a couple
     of bugs in the meantime

   - Set of fixes for the handling of the MMU in an NV context,
     particularly VNCR-triggered faults. S1POE support is fixed as well

   - Large set of pKVM fixes, mostly addressing recurring issues around
     hypervisor tracking of donated pages in obscure cases where the
     donation could fail and leave things in a bizarre state

   - Fixes for the so-called "lazy vgic init", which resulted in
     sleeping operations in non-preemptible sections. This turned out to
     be far more invasive than initially expected..

   - Reduce the overhead of L1/L2 context switch by not touching the FP
     registers

   - Fix the way non-implemented page sizes are dealt with when a guest
     insist on using them for S2 translation

   - The usual set of low-impact fixes and cleanups all over the map

  Loongarch:

   - On a request for lazy FPU load, load all FPU state that the VM
     supports instead of enabling only the part (FPU, LSX or LASX) that
     caused the FPU load request

   - Some enhancements about interrupt injection

   - Some bug fixes and other small changes

  RISC-V:

   - Batch G-stage TLB flushes for GPA range based page table updates

   - Convert HGEI line management to fully per-HART

   - Fix missing CSR dirty marking when FWFT state updated via ONE_REG

   - Fix stale FWFT feature exposure to Guest/VM

   - Speed up dirty logging write faults using MMU rwlock and atomic PTE
     updates using cmpxchg() for permission-only changes

   - Use flexible array for APLIC IRQ state

   - Use kvm_slot_dirty_track_enabled() for logging enable check on a
     memslot

   - Avoid skipping valid pages in kvm_riscv_gstage_wp_range()

   - Avoid skipping valid pages in kvm_riscv_gstage_unmap_range()

   - Use endian-specific __lelong for NACL shared memory

  S390:

   - KVM_PRE_FAULT_MEMORY support

   - Support for 2G hugepages

   - Support for the ASTFLEIE 2 facility

   - Support for fast inject using kvm_arch_set_irq_inatomic

   - Fix potential leak of uninitialized bytes

   - A few more misc gmap fixes

  x86:

   - Generic support for the more granular permissions allowed by EPT,
     namely "read" (which was previously usurping the U bit) and
     separate execution bits for kernel and userspace

   - Do not assume that all page tables start with U=1/W=1/NX=0 at the
     root, as AMD GMET needs to have U=0 at the root

   - Introduce common assembly macros for use within Intel and AMD
     vendor-specific vmentry code. This touches the SPEC_CTRL handling,
     which is now entirely done in assembly for Intel (by reusing the
     AMD code that already existed), and register save/restore which
     uses some macro magic to compute the offsets in the struct. Both of
     these are preparatory changes for upcoming APX support

   - Clean up KVM's register tracking and storage, primarily to prepare
     for APX support, which expands the maximum number of GPRs from 16
     to 32

   - Keep a single copy of the PDPTRs rather than two, since
     architecturally there is just one

   - Handle EXIT_FASTPATH_EXIT_USERSPACE in vendor code to ensure vendor
     code gets a chance to handle things like reaping the PML buffer

   - Update KVM's view of PV async enabling if and only if the MSR write
     fully succeeds

   - Fix a variety of issues where the emulator doesn't honor
     guest-debug state, and clean up related code along the way

   - Synthesize EPT Violation and #NPF "error code" bits when injecting
     faults into L1 that didn't originate in hardware (in which case the
     VMCS/VMCB doesn't hold relevant information)

   - Add support for virtualizing (well, emulating) AMD's flavor of
     CPL>0 CPUID faulting

   - Clean up the GPR APIs so that KVM's use of "raw" is consistent, and
     fix a variety of minor bugs along the way

   - Fix an OOB memory access due to not checking the VP ID when
     handling a Hyper-V PV TLB flush for L2

   - Fix a bug in the mediated PMU's handling of fixed counters that
     allowed the guest to bypass the PMU event filter

   - Allow userspace to return EAGAIN when handling SNP and TDX
     hypercalls, so the KVM can forward a "retry" status code to the
     guest, and reserve all unused error codes for future usage

   - Overhaul the TDP MMU => S-EPT code to move as much S-EPT specific
     logic as possible into the TDX code, and to funnel (almost) all
     S-EPT updates into a single chokepoint. The motivation is largely
     to prepare for upcoming Dynamic PAMT support, but the cleanups are
     nice to have on their own

   - Plug a hole in shadow page table handling, where KVM fails to
     recursively zap nested EPT/NPT shadow page tables when the nested
     hypervisor tears down its own EPT/NPT page tables from the bottom
     up

  x86 (Intel):

   - Support for nested MBEC (Mode-Based Execute Control), see above in
     the generic section; also run with MBEC enabled even for non-nested
     mode

   - Use the kernel's "enum pg_level" in the TDX APIs instead of the
     TDX-Module's level definitions (which are 0-based)

   - Rework the TDX memory APIs to not require/assume that guest memory
     is backed by "struct page" (in prepartion for guest_memfd hugepage
     support)

   - Fix a largely benign bug where KVM TDX would incorrectly state it
     could emulate several x2APIC MSRs

   - Use the "safe" WRMSR API when proxying LBR MSR writes as the
     to-be-written value is guest controlled and completely unvalidated

  x86 (AMD):

   - Support for nested GMET (Guest Mode Execution Trap), see above in
     the generic section; also run with GMET enabled even for non-nested
     mode

   - Fixes and minor cleanups to GHCB handling, on top of the earlier
     work already merged into 7.1-rc

   - Ensure KVM's copy of CR0 and CR3 are up-to-date prior to invoking
     fastpath handlers

   - Add support for virtualizing gPAT (KVM previously just used L1's
     PAT when running L2)

   - Fix goofs where KVM mishandles side effects (e.g. single-step and
     PMC updates) when emulating VMRUN

   - Fix a variety of bugs in AVIC's handling of x2APIC MSR
     interception, most notably where KVM didn't disable interception of
     IRR, ISR, and TMR regs

   - Add support for virtualizing Host-Only/Guest-Only bits in the
     mediated PMU

   - Don't advertise support for unusable VM types, and account for VM
     types that are disabled by firmware, e.g. to mitigate security
     vulnerabilities

   - Rewrite the SEV {en,de}crypt debug ioctls as they were riddle with
     bugs and unnecessarily complicated, and add comprehensive tests

   - Clean up and deduplicate the SEV page pinning code

   - Fix minor goofs related to writing back CPUID information after
     firmware rejects a CPUID page for an SNP vCPU

  Generic:

   - Rename invalidate_begin() to invalidate_start() throughout KVM to
     follow the kernel's nomenclature, e.g. for mmu_notifiers

   - Use guard() to cleanup up various KVM+VFIO flows

   - Minor cleanups

  guest_memfd:

   - Return -EEXIST instead of -EINVAL if userspace attempts to bind a
     gmem range to multiple memslots, and fix the test that was supposed
     to ensure KVM returns -EEXIST

   - Treat memslot binding offsets and sizes as unsigned values to fix a
     bug where KVM interprets a large "offset + size" as a negative
     value and allows a nonsensical offset

   - Use the inode number instead of the page offset for the NUMA
     interleaving index to fix a bug where the effective index would
     jump by two for consecutive pages (the caller also adds in the page
     offset)

  Selftests:

   - Randomize the dirty log test's delay when reaping the bitmap on the
     first pass, as always waiting only 1ms hid a KVM RISC-V bug as the
     test reaped the bitmap before KVM could build up enough state to
     hit the bug

   - A pile of one-off fixes and cleanups"

* tag 'for-linus' of git://git.kernel.org/pub/scm/virt/kvm/kvm: (326 commits)
  KVM: x86/mmu: Ensure hugepage is in by slot before checking max mapping level
  KVM: x86: Fix shadow paging use-after-free due to unexpected role
  KVM: s390: Introducing kvm_arch_set_irq_inatomic fast inject
  KVM: s390: Enable adapter_indicators_set to use mapped pages
  KVM: s390: Add map/unmap ioctl and clean mappings post-guest
  riscv: kvm: Use endian-specific __lelong for NACL shared memory
  KVM: selftests: access_tracking_perf_test: bump number of NUMA nodes to 32
  KVM: s390: vsie: Implement ASTFLEIE facility 2
  KVM: s390: vsie: Refactor handle_stfle
  s390/sclp: Detect ASTFLEIE 2 facility
  KVM: s390: Minor refactor of base/ext facility lists
  KVM: x86/mmu: move pdptrs out of the MMU
  KVM: x86: check that kvm_handle_invpcid is only invoked with shadow paging
  KVM: nSVM: invalidate cached PDPTRs across nested NPT transitions
  KVM: nVMX: remove unnecessary code in prepare_vmcs02_rare
  KVM: x86: remove nested_mmu from mmu_is_nested()
  KVM: arm64: vgic-its: Make ABI commit helpers return void
  KVM: s390: Initialize KVM_S390_GET_CMMA_BITS memory
  LoongArch: KVM: Add missing slots_lock for device register/unregister
  LoongArch: KVM: Validate irqchip index in irqfd routing
  ...
This commit is contained in:
Linus Torvalds 2026-06-19 08:56:49 -07:00
commit c98d767b34
187 changed files with 6174 additions and 3559 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

51
Documentation/virt/kvm/api.rst
View File

@ -4944,10 +4944,13 @@ Errors:
#define KVM_STATE_NESTED_FORMAT_SVM 1
#define KVM_STATE_NESTED_VMX_VMCS_SIZE 0x1000
#define KVM_STATE_NESTED_SVM_VMCB_SIZE 0x1000
#define KVM_STATE_NESTED_VMX_SMM_GUEST_MODE 0x00000001
#define KVM_STATE_NESTED_VMX_SMM_VMXON 0x00000002
#define KVM_STATE_NESTED_GIF_SET 0x00000100
#define KVM_STATE_VMX_PREEMPTION_TIMER_DEADLINE 0x00000001
struct kvm_vmx_nested_state_hdr {
@ -4962,11 +4965,20 @@ Errors:
__u64 preemption_timer_deadline;
};
struct kvm_svm_nested_state_hdr {
__u64 vmcb_pa;
__u64 gpat;
};
struct kvm_vmx_nested_state_data {
__u8 vmcs12[KVM_STATE_NESTED_VMX_VMCS_SIZE];
__u8 shadow_vmcs12[KVM_STATE_NESTED_VMX_VMCS_SIZE];
};
struct kvm_svm_nested_state_data {
__u8 vmcb12[KVM_STATE_NESTED_SVM_VMCB_SIZE];
};
This ioctl copies the vcpu's nested virtualization state from the kernel to
userspace.
@ -6471,7 +6483,8 @@ Errors:
========== ===============================================================
EINVAL The specified `gpa` and `size` were invalid (e.g. not
page aligned, causes an overflow, or size is zero).
page aligned, causes an overflow, or size is zero), or the VM
is UCONTROL (s390).
ENOENT The specified `gpa` is outside defined memslots.
EINTR An unmasked signal is pending and no page was processed.
EFAULT The parameter address was invalid.
@ -6494,7 +6507,7 @@ Errors:
KVM_PRE_FAULT_MEMORY populates KVM's stage-2 page tables used to map memory
for the current vCPU state. KVM maps memory as if the vCPU generated a
stage-2 read page fault, e.g. faults in memory as needed, but doesn't break
CoW. However, KVM does not mark any newly created stage-2 PTE as Accessed.
CoW. On x86, KVM does not mark any newly created stage-2 PTE as Accessed.
In the case of confidential VM types where there is an initial set up of
private guest memory before the guest is 'finalized'/measured, this ioctl
@ -8553,6 +8566,20 @@ KVM_X86_QUIRK_VMCS12_ALLOW_FREEZE_IN_SMM By default, KVM relaxes the consisten
bit to be cleared. Note that the vmcs02
bit is still completely controlled by the
host, regardless of the quirk setting.
KVM_X86_QUIRK_NESTED_SVM_SHARED_PAT By default, KVM for nested SVM guests
shares the IA32_PAT MSR between L1 and
L2. This is legacy behavior and does
not match the AMD architecture
specification. When this quirk is
disabled and nested paging (NPT) is
enabled for L2, KVM correctly
virtualizes a separate guest PAT
register for L2, using the g_pat
field in the VMCB. When NPT is
disabled for L2, L1 and L2 continue
to share the IA32_PAT MSR regardless
of the quirk setting.
======================================== ================================================
7.32 KVM_CAP_MAX_VCPU_ID
@ -8818,6 +8845,9 @@ block sizes is exposed in KVM_CAP_ARM_SUPPORTED_BLOCK_SIZES as a
This capability, if enabled, will cause KVM to exit to userspace
with KVM_EXIT_HYPERCALL exit reason to process some hypercalls.
Userspace may fail the hypercall by setting hypercall.ret to EINVAL
or may request the hypercall to be retried the next time the guest run
by setting hypercall.ret to EAGAIN.
Calling KVM_CHECK_EXTENSION for this capability will return a bitmask
of hypercalls that can be configured to exit to userspace.
@ -8904,6 +8934,21 @@ helpful if user space wants to emulate instructions which are not
This capability can be enabled dynamically even if VCPUs were already
created and are running.
7.47 KVM_CAP_S390_HPAGE_2G
--------------------------
:Architectures: s390
:Parameters: none
:Returns: 0 on success; -EINVAL if hpage_2g module parameter was not set,
cmma is enabled, or the VM has the KVM_VM_S390_UCONTROL
flag set; -EBUSY if vCPUs were already created for the VM.
With this capability the KVM support for memory backing with 2g pages
through hugetlbfs can be enabled for a VM. After the capability is
enabled, cmma can't be enabled anymore and pfmfi and the storage key
interpretation are disabled. If cmma has already been enabled or the
hpage_2g module parameter is not set to 1, -EINVAL is returned.
8. Other capabilities.
======================
@ -9363,6 +9408,8 @@ means the VM type with value @n is supported. Possible values of @n are::
#define KVM_X86_SW_PROTECTED_VM 1
#define KVM_X86_SEV_VM 2
#define KVM_X86_SEV_ES_VM 3
#define KVM_X86_SNP_VM 4
#define KVM_X86_TDX_VM 5
Note, KVM_X86_SW_PROTECTED_VM is currently only for development and testing.
Do not use KVM_X86_SW_PROTECTED_VM for "real" VMs, and especially not in

6
Documentation/virt/kvm/devices/arm-vgic-v5.rst
View File

@ -12,8 +12,8 @@ Only one VGIC instance may be instantiated through this API. The created VGIC
will act as the VM interrupt controller, requiring emulated user-space devices
to inject interrupts to the VGIC instead of directly to CPUs.
Creating a guest GICv5 device requires a host GICv5 host. The current VGICv5
device only supports PPI interrupts. These can either be injected from emulated
Creating a guest GICv5 device requires a GICv5 host. The current VGICv5 device
only supports PPI interrupts. These can either be injected from emulated
in-kernel devices (such as the Arch Timer, or PMU), or via the KVM_IRQ_LINE
ioctl.
@ -25,7 +25,7 @@ Groups:
request the initialization of the VGIC, no additional parameter in
kvm_device_attr.addr. Must be called after all VCPUs have been created.
KVM_DEV_ARM_VGIC_USERPSPACE_PPIs
KVM_DEV_ARM_VGIC_USERSPACE_PPIS
request the mask of userspace-drivable PPIs. Only a subset of the PPIs can
be directly driven from userspace with GICv5, and the returned mask
informs userspace of which it is allowed to drive via KVM_IRQ_LINE.

7
Documentation/virt/kvm/devices/vcpu.rst
View File

@ -37,8 +37,11 @@ Returns:
A value describing the PMUv3 (Performance Monitor Unit v3) overflow interrupt
number for this vcpu. This interrupt could be a PPI or SPI, but the interrupt
type must be same for each vcpu. As a PPI, the interrupt number is the same for
all vcpus, while as an SPI it must be a separate number per vcpu. For
GICv5-based guests, the architected PPI (23) must be used.
all vcpus, while as an SPI it must be a separate number per vcpu.
For GICv5-based guests, the architected PPI (23) must be used, and must be
communicated as the full GICv5-style Interrupt ID, i.e., 0x20000017. This ioctl
can be omitted altogether for a GICv5-based guest.
1.2 ATTRIBUTE: KVM_ARM_VCPU_PMU_V3_INIT
---------------------------------------

10
Documentation/virt/kvm/x86/mmu.rst
View File

@ -184,10 +184,8 @@ Shadow pages contain the following information:
Contains the value of efer.nx for which the page is valid.
role.cr0_wp:
Contains the value of cr0.wp for which the page is valid.
role.smep_andnot_wp:
Contains the value of cr4.smep && !cr0.wp for which the page is valid
(pages for which this is true are different from other pages; see the
treatment of cr0.wp=0 below).
role.cr4_smep:
Contains the value of cr4.smep for which the page is valid.
role.smap_andnot_wp:
Contains the value of cr4.smap && !cr0.wp for which the page is valid
(pages for which this is true are different from other pages; see the
@ -435,8 +433,8 @@ from being written by the kernel after cr0.wp has changed to 1, we make
the value of cr0.wp part of the page role. This means that an spte created
with one value of cr0.wp cannot be used when cr0.wp has a different value -
it will simply be missed by the shadow page lookup code. A similar issue
exists when an spte created with cr0.wp=0 and cr4.smep=0 is used after
changing cr4.smep to 1. To avoid this, the value of !cr0.wp && cr4.smep
exists when an spte created with cr0.wp=0 and cr4.smap=0 is used after
changing cr4.smap to 1. To avoid this, the value of !cr0.wp && cr4.smap
is also made a part of the page role.
Large pages

1
MAINTAINERS
View File

@ -14253,6 +14253,7 @@ L: kvm@vger.kernel.org
S: Supported
P: Documentation/process/maintainer-kvm-x86.rst
T: git git://git.kernel.org/pub/scm/virt/kvm/kvm.git
F: Documentation/process/maintainer-kvm-x86.rst
F: arch/x86/include/asm/kvm*
F: arch/x86/include/asm/svm.h
F: arch/x86/include/asm/vmx*.h

8
arch/arm64/include/asm/kvm_host.h
View File

@ -1095,7 +1095,8 @@ struct kvm_vcpu_arch {
#define IN_NESTED_ERET __vcpu_single_flag(sflags, BIT(7))
/* SError pending for nested guest */
#define NESTED_SERROR_PENDING __vcpu_single_flag(sflags, BIT(8))
/* KVM is currently emulating an L2 to L1 exception */
#define IN_NESTED_EXCEPTION __vcpu_single_flag(sflags, BIT(9))
#define vcpu_sve_max_vq(vcpu) sve_vq_from_vl((vcpu)->arch.sve_max_vl)
@ -1252,13 +1253,14 @@ void kvm_arm_resume_guest(struct kvm *kvm);
#define vcpu_has_run_once(vcpu) (!!READ_ONCE((vcpu)->pid))
#ifndef __KVM_NVHE_HYPERVISOR__
#define kvm_call_hyp_nvhe(f, ...) \
#define kvm_call_hyp_nvhe(f, ...) \
({ \
struct arm_smccc_res res; \
\
arm_smccc_1_1_hvc(KVM_HOST_SMCCC_FUNC(f), \
##__VA_ARGS__, &res); \
WARN_ON(res.a0 != SMCCC_RET_SUCCESS); \
if (WARN_ON(res.a0 != SMCCC_RET_SUCCESS)) \
res.a1 = -EOPNOTSUPP; \
\
res.a1; \
})

1
arch/arm64/include/asm/kvm_hyp.h
View File

@ -152,5 +152,6 @@ extern unsigned long kvm_nvhe_sym(__icache_flags);
extern unsigned int kvm_nvhe_sym(kvm_arm_vmid_bits);
extern unsigned int kvm_nvhe_sym(kvm_host_sve_max_vl);
extern unsigned long kvm_nvhe_sym(hyp_nr_cpus);
extern unsigned int kvm_nvhe_sym(hyp_gicv3_nr_lr);
#endif /* __ARM64_KVM_HYP_H__ */

3
arch/arm64/include/asm/kvm_mmu.h
View File

@ -318,8 +318,7 @@ static __always_inline u64 kvm_get_vttbr(struct kvm_s2_mmu *mmu)
* Must be called from hyp code running at EL2 with an updated VTTBR
* and interrupts disabled.
*/
static __always_inline void __load_stage2(struct kvm_s2_mmu *mmu,
struct kvm_arch *arch)
static __always_inline void __load_stage2(struct kvm_s2_mmu *mmu)
{
write_sysreg(mmu->vtcr, vtcr_el2);
write_sysreg(kvm_get_vttbr(mmu), vttbr_el2);

1
arch/arm64/kernel/cpufeature.c
View File

@ -266,6 +266,7 @@ static const struct arm64_ftr_bits ftr_id_aa64isar1[] = {
};
static const struct arm64_ftr_bits ftr_id_aa64isar2[] = {
ARM64_FTR_BITS(FTR_HIDDEN, FTR_STRICT, FTR_LOWER_SAFE, ID_AA64ISAR2_EL1_ATS1A_SHIFT, 4, 0),
ARM64_FTR_BITS(FTR_VISIBLE, FTR_NONSTRICT, FTR_LOWER_SAFE, ID_AA64ISAR2_EL1_LUT_SHIFT, 4, 0),
ARM64_FTR_BITS(FTR_VISIBLE, FTR_NONSTRICT, FTR_LOWER_SAFE, ID_AA64ISAR2_EL1_CSSC_SHIFT, 4, 0),
ARM64_FTR_BITS(FTR_VISIBLE, FTR_NONSTRICT, FTR_LOWER_SAFE, ID_AA64ISAR2_EL1_RPRFM_SHIFT, 4, 0),

4
arch/arm64/kernel/hyp-stub.S
View File

@ -104,11 +104,9 @@ SYM_CODE_START_LOCAL(__finalise_el2)
mov_q x0, HCR_HOST_VHE_FLAGS
msr_hcr_el2 x0
// Use the EL1 allocated stack, per-cpu offset
// Use the EL1 allocated stack
mrs x0, sp_el1
mov sp, x0
mrs x0, tpidr_el1
msr tpidr_el2, x0
// FP configuration, vectors
mrs_s x0, SYS_CPACR_EL12

137
arch/arm64/kvm/arch_timer.c
View File

@ -39,10 +39,9 @@ static const u8 default_ppi[] = {
[TIMER_HVTIMER] = 28,
};
static bool kvm_timer_irq_can_fire(struct arch_timer_context *timer_ctx);
static void kvm_timer_update_irq(struct kvm_vcpu *vcpu, bool new_level,
struct arch_timer_context *timer_ctx);
static bool kvm_timer_should_fire(struct arch_timer_context *timer_ctx);
static bool kvm_timer_pending(struct arch_timer_context *timer_ctx);
static void kvm_arm_timer_write(struct kvm_vcpu *vcpu,
struct arch_timer_context *timer,
enum kvm_arch_timer_regs treg,
@ -52,11 +51,17 @@ static u64 kvm_arm_timer_read(struct kvm_vcpu *vcpu,
enum kvm_arch_timer_regs treg);
static bool kvm_arch_timer_get_input_level(int vintid);
static struct irq_ops arch_timer_irq_ops = {
static unsigned long kvm_arch_timer_get_irq_flags(void)
{
return kvm_vgic_global_state.no_hw_deactivation ? VGIC_IRQ_SW_RESAMPLE : 0;
}
static const struct irq_ops arch_timer_irq_ops = {
.get_flags = kvm_arch_timer_get_irq_flags,
.get_input_level = kvm_arch_timer_get_input_level,
};
static struct irq_ops arch_timer_irq_ops_vgic_v5 = {
static const struct irq_ops arch_timer_irq_ops_vgic_v5 = {
.get_input_level = kvm_arch_timer_get_input_level,
.queue_irq_unlock = vgic_v5_ppi_queue_irq_unlock,
.set_direct_injection = vgic_v5_set_ppi_dvi,
@ -224,7 +229,7 @@ static irqreturn_t kvm_arch_timer_handler(int irq, void *dev_id)
else
ctx = map.direct_ptimer;
if (kvm_timer_should_fire(ctx))
if (kvm_timer_pending(ctx))
kvm_timer_update_irq(vcpu, true, ctx);
if (userspace_irqchip(vcpu->kvm) &&
@ -257,7 +262,7 @@ static u64 kvm_timer_compute_delta(struct arch_timer_context *timer_ctx)
return kvm_counter_compute_delta(timer_ctx, timer_get_cval(timer_ctx));
}
static bool kvm_timer_irq_can_fire(struct arch_timer_context *timer_ctx)
static bool kvm_timer_enabled(struct arch_timer_context *timer_ctx)
{
WARN_ON(timer_ctx && timer_ctx->loaded);
return timer_ctx &&
@ -294,7 +299,7 @@ static u64 kvm_timer_earliest_exp(struct kvm_vcpu *vcpu)
struct arch_timer_context *ctx = &vcpu->arch.timer_cpu.timers[i];
WARN(ctx->loaded, "timer %d loaded\n", i);
if (kvm_timer_irq_can_fire(ctx))
if (kvm_timer_enabled(ctx))
min_delta = min(min_delta, kvm_timer_compute_delta(ctx));
}
@ -358,7 +363,7 @@ static enum hrtimer_restart kvm_hrtimer_expire(struct hrtimer *hrt)
return HRTIMER_NORESTART;
}
static bool kvm_timer_should_fire(struct arch_timer_context *timer_ctx)
static bool kvm_timer_pending(struct arch_timer_context *timer_ctx)
{
enum kvm_arch_timers index;
u64 cval, now;
@ -391,7 +396,7 @@ static bool kvm_timer_should_fire(struct arch_timer_context *timer_ctx)
!(cnt_ctl & ARCH_TIMER_CTRL_IT_MASK);
}
if (!kvm_timer_irq_can_fire(timer_ctx))
if (!kvm_timer_enabled(timer_ctx))
return false;
cval = timer_get_cval(timer_ctx);
@ -405,22 +410,30 @@ int kvm_cpu_has_pending_timer(struct kvm_vcpu *vcpu)
return vcpu_has_wfit_active(vcpu) && wfit_delay_ns(vcpu) == 0;
}
static u64 kvm_timer_needs_notify(struct kvm_vcpu *vcpu)
{
u64 v = vcpu->run->s.regs.device_irq_level;
v ^= kvm_timer_pending(vcpu_vtimer(vcpu)) ? KVM_ARM_DEV_EL1_VTIMER : 0;
v ^= kvm_timer_pending(vcpu_ptimer(vcpu)) ? KVM_ARM_DEV_EL1_PTIMER : 0;
return v & (KVM_ARM_DEV_EL1_VTIMER | KVM_ARM_DEV_EL1_PTIMER);
}
bool kvm_timer_should_notify_user(struct kvm_vcpu *vcpu)
{
return !!kvm_timer_needs_notify(vcpu);
}
/*
* Reflect the timer output level into the kvm_run structure
*/
void kvm_timer_update_run(struct kvm_vcpu *vcpu)
bool kvm_timer_update_run(struct kvm_vcpu *vcpu)
{
struct arch_timer_context *vtimer = vcpu_vtimer(vcpu);
struct arch_timer_context *ptimer = vcpu_ptimer(vcpu);
struct kvm_sync_regs *regs = &vcpu->run->s.regs;
/* Populate the device bitmap with the timer states */
regs->device_irq_level &= ~(KVM_ARM_DEV_EL1_VTIMER |
KVM_ARM_DEV_EL1_PTIMER);
if (kvm_timer_should_fire(vtimer))
regs->device_irq_level |= KVM_ARM_DEV_EL1_VTIMER;
if (kvm_timer_should_fire(ptimer))
regs->device_irq_level |= KVM_ARM_DEV_EL1_PTIMER;
u64 mask = kvm_timer_needs_notify(vcpu);
if (mask)
vcpu->run->s.regs.device_irq_level ^= mask;
return !!mask;
}
static void kvm_timer_update_status(struct arch_timer_context *ctx, bool level)
@ -446,9 +459,8 @@ static void kvm_timer_update_irq(struct kvm_vcpu *vcpu, bool new_level,
{
kvm_timer_update_status(timer_ctx, new_level);
timer_ctx->irq.level = new_level;
trace_kvm_timer_update_irq(vcpu->vcpu_id, timer_irq(timer_ctx),
timer_ctx->irq.level);
new_level);
if (userspace_irqchip(vcpu->kvm))
return;
@ -466,28 +478,25 @@ static void kvm_timer_update_irq(struct kvm_vcpu *vcpu, bool new_level,
kvm_vgic_inject_irq(vcpu->kvm, vcpu,
timer_irq(timer_ctx),
timer_ctx->irq.level,
new_level,
timer_ctx);
}
/* Only called for a fully emulated timer */
static void timer_emulate(struct arch_timer_context *ctx)
{
bool should_fire = kvm_timer_should_fire(ctx);
bool pending = kvm_timer_pending(ctx);
trace_kvm_timer_emulate(ctx, should_fire);
trace_kvm_timer_emulate(ctx, pending);
if (should_fire != ctx->irq.level)
kvm_timer_update_irq(timer_context_to_vcpu(ctx), should_fire, ctx);
kvm_timer_update_status(ctx, should_fire);
kvm_timer_update_irq(timer_context_to_vcpu(ctx), pending, ctx);
/*
* If the timer can fire now, we don't need to have a soft timer
* scheduled for the future. If the timer cannot fire at all,
* then we also don't need a soft timer.
* If the timer is pending, we don't need to have a soft timer
* scheduled for the future. If the timer is disabled, then
* we don't need a soft timer either.
*/
if (should_fire || !kvm_timer_irq_can_fire(ctx))
if (pending || !kvm_timer_enabled(ctx))
return;
soft_timer_start(&ctx->hrtimer, kvm_timer_compute_delta(ctx));
@ -594,10 +603,10 @@ static void kvm_timer_blocking(struct kvm_vcpu *vcpu)
* If no timers are capable of raising interrupts (disabled or
* masked), then there's no more work for us to do.
*/
if (!kvm_timer_irq_can_fire(map.direct_vtimer) &&
!kvm_timer_irq_can_fire(map.direct_ptimer) &&
!kvm_timer_irq_can_fire(map.emul_vtimer) &&
!kvm_timer_irq_can_fire(map.emul_ptimer) &&
if (!kvm_timer_enabled(map.direct_vtimer) &&
!kvm_timer_enabled(map.direct_ptimer) &&
!kvm_timer_enabled(map.emul_vtimer) &&
!kvm_timer_enabled(map.emul_ptimer) &&
!vcpu_has_wfit_active(vcpu))
return;
@ -677,6 +686,7 @@ static inline void set_timer_irq_phys_active(struct arch_timer_context *ctx, boo
static void kvm_timer_vcpu_load_gic(struct arch_timer_context *ctx)
{
struct kvm_vcpu *vcpu = timer_context_to_vcpu(ctx);
bool pending = kvm_timer_pending(ctx);
bool phys_active = false;
/*
@ -685,12 +695,12 @@ static void kvm_timer_vcpu_load_gic(struct arch_timer_context *ctx)
* this point and the register restoration, we'll take the
* interrupt anyway.
*/
kvm_timer_update_irq(vcpu, kvm_timer_should_fire(ctx), ctx);
kvm_timer_update_irq(vcpu, pending, ctx);
if (irqchip_in_kernel(vcpu->kvm))
phys_active = kvm_vgic_map_is_active(vcpu, timer_irq(ctx));
phys_active |= ctx->irq.level;
phys_active |= pending;
phys_active |= vgic_is_v5(vcpu->kvm);
set_timer_irq_phys_active(ctx, phys_active);
@ -699,6 +709,7 @@ static void kvm_timer_vcpu_load_gic(struct arch_timer_context *ctx)
static void kvm_timer_vcpu_load_nogic(struct kvm_vcpu *vcpu)
{
struct arch_timer_context *vtimer = vcpu_vtimer(vcpu);
bool pending = kvm_timer_pending(vtimer);
/*
* Update the timer output so that it is likely to match the
@ -706,7 +717,7 @@ static void kvm_timer_vcpu_load_nogic(struct kvm_vcpu *vcpu)
* this point and the register restoration, we'll take the
* interrupt anyway.
*/
kvm_timer_update_irq(vcpu, kvm_timer_should_fire(vtimer), vtimer);
kvm_timer_update_irq(vcpu, pending, vtimer);
/*
* When using a userspace irqchip with the architected timers and a
@ -718,7 +729,7 @@ static void kvm_timer_vcpu_load_nogic(struct kvm_vcpu *vcpu)
* being de-asserted, we unmask the interrupt again so that we exit
* from the guest when the timer fires.
*/
if (vtimer->irq.level)
if (pending)
disable_percpu_irq(host_vtimer_irq);
else
enable_percpu_irq(host_vtimer_irq, host_vtimer_irq_flags);
@ -904,23 +915,6 @@ void kvm_timer_vcpu_load(struct kvm_vcpu *vcpu)
timer_set_traps(vcpu, &map);
}
bool kvm_timer_should_notify_user(struct kvm_vcpu *vcpu)
{
struct arch_timer_context *vtimer = vcpu_vtimer(vcpu);
struct arch_timer_context *ptimer = vcpu_ptimer(vcpu);
struct kvm_sync_regs *sregs = &vcpu->run->s.regs;
bool vlevel, plevel;
if (likely(irqchip_in_kernel(vcpu->kvm)))
return false;
vlevel = sregs->device_irq_level & KVM_ARM_DEV_EL1_VTIMER;
plevel = sregs->device_irq_level & KVM_ARM_DEV_EL1_PTIMER;
return kvm_timer_should_fire(vtimer) != vlevel ||
kvm_timer_should_fire(ptimer) != plevel;
}
void kvm_timer_vcpu_put(struct kvm_vcpu *vcpu)
{
struct arch_timer_cpu *timer = vcpu_timer(vcpu);
@ -1006,7 +1000,7 @@ static void unmask_vtimer_irq_user(struct kvm_vcpu *vcpu)
{
struct arch_timer_context *vtimer = vcpu_vtimer(vcpu);
if (!kvm_timer_should_fire(vtimer)) {
if (!kvm_timer_pending(vtimer)) {
kvm_timer_update_irq(vcpu, false, vtimer);
if (static_branch_likely(&has_gic_active_state))
set_timer_irq_phys_active(vtimer, false);
@ -1288,7 +1282,12 @@ static int timer_irq_set_vcpu_affinity(struct irq_data *d, void *vcpu)
static int timer_irq_set_irqchip_state(struct irq_data *d,
enum irqchip_irq_state which, bool val)
{
if (which != IRQCHIP_STATE_ACTIVE || !irqd_is_forwarded_to_vcpu(d))
bool passthrough = which != IRQCHIP_STATE_ACTIVE ||
!irqd_is_forwarded_to_vcpu(d) ||
(kvm_vgic_global_state.type == VGIC_V5 &&
vgic_is_v3(kvm_get_running_vcpu()->kvm));
if (passthrough)
return irq_chip_set_parent_state(d, which, val);
if (val)
@ -1301,15 +1300,7 @@ static int timer_irq_set_irqchip_state(struct irq_data *d,
static void timer_irq_eoi(struct irq_data *d)
{
/*
* On a GICv5 host, we still need to call EOI on the parent for
* PPIs. The host driver already handles irqs which are forwarded to
* vcpus, and skips the GIC CDDI while still doing the GIC CDEOI. This
* is required to emulate the EOIMode=1 on GICv5 hardware. Failure to
* call EOI unsurprisingly results in *BAD* lock-ups.
*/
if (!irqd_is_forwarded_to_vcpu(d) ||
kvm_vgic_global_state.type == VGIC_V5)
if (!irqd_is_forwarded_to_vcpu(d))
irq_chip_eoi_parent(d);
}
@ -1392,8 +1383,6 @@ static int kvm_irq_init(struct arch_timer_kvm_info *info)
return -ENOMEM;
}
if (kvm_vgic_global_state.no_hw_deactivation)
arch_timer_irq_ops.flags |= VGIC_IRQ_SW_RESAMPLE;
WARN_ON(irq_domain_push_irq(domain, host_vtimer_irq,
(void *)TIMER_VTIMER));
}
@ -1579,7 +1568,7 @@ static bool kvm_arch_timer_get_input_level(int vintid)
ctx = vcpu_get_timer(vcpu, i);
if (timer_irq(ctx) == vintid)
return kvm_timer_should_fire(ctx);
return kvm_timer_pending(ctx);
}
/* A timer IRQ has fired, but no matching timer was found? */
@ -1591,8 +1580,8 @@ static bool kvm_arch_timer_get_input_level(int vintid)
int kvm_timer_enable(struct kvm_vcpu *vcpu)
{
struct arch_timer_cpu *timer = vcpu_timer(vcpu);
const struct irq_ops *ops;
struct timer_map map;
struct irq_ops *ops;
int ret;
if (timer->enabled)

41
arch/arm64/kvm/arm.c
View File

@ -52,6 +52,7 @@
#include <linux/irqchip/arm-gic-v5.h>
#include "vgic/vgic.h"
#include "sys_regs.h"
static enum kvm_mode kvm_mode = KVM_MODE_DEFAULT;
@ -1166,6 +1167,15 @@ static bool vcpu_mode_is_bad_32bit(struct kvm_vcpu *vcpu)
return !kvm_supports_32bit_el0();
}
static bool kvm_irq_update_run(struct kvm_vcpu *vcpu)
{
bool r;
r = kvm_timer_update_run(vcpu);
r |= kvm_pmu_update_run(vcpu);
return r;
}
/**
* kvm_vcpu_exit_request - returns true if the VCPU should *not* enter the guest
* @vcpu: The VCPU pointer
@ -1187,13 +1197,11 @@ static bool kvm_vcpu_exit_request(struct kvm_vcpu *vcpu, int *ret)
/*
* If we're using a userspace irqchip, then check if we need
* to tell a userspace irqchip about timer or PMU level
* changes and if so, exit to userspace (the actual level
* state gets updated in kvm_timer_update_run and
* kvm_pmu_update_run below).
* changes and if so, exit to userspace while updating the run
* state.
*/
if (unlikely(!irqchip_in_kernel(vcpu->kvm))) {
if (kvm_timer_should_notify_user(vcpu) ||
kvm_pmu_should_notify_user(vcpu)) {
if (unlikely(kvm_irq_update_run(vcpu))) {
*ret = -EINTR;
run->exit_reason = KVM_EXIT_INTR;
return true;
@ -1408,11 +1416,8 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu)
ret = handle_exit(vcpu, ret);
}
/* Tell userspace about in-kernel device output levels */
if (unlikely(!irqchip_in_kernel(vcpu->kvm))) {
kvm_timer_update_run(vcpu);
kvm_pmu_update_run(vcpu);
}
if (unlikely(!irqchip_in_kernel(vcpu->kvm)))
kvm_irq_update_run(vcpu);
kvm_sigset_deactivate(vcpu);
@ -1496,8 +1501,13 @@ int kvm_vm_ioctl_irq_line(struct kvm *kvm, struct kvm_irq_level *irq_level,
return vcpu_interrupt_line(vcpu, irq_num, level);
case KVM_ARM_IRQ_TYPE_PPI:
if (!irqchip_in_kernel(kvm))
if (irqchip_in_kernel(kvm)) {
int ret = vgic_lazy_init(kvm);
if (ret)
return ret;
} else {
return -ENXIO;
}
vcpu = kvm_get_vcpu_by_id(kvm, vcpu_id);
if (!vcpu)
@ -1524,8 +1534,13 @@ int kvm_vm_ioctl_irq_line(struct kvm *kvm, struct kvm_irq_level *irq_level,
return kvm_vgic_inject_irq(kvm, vcpu, irq_num, level, NULL);
case KVM_ARM_IRQ_TYPE_SPI:
if (!irqchip_in_kernel(kvm))
if (irqchip_in_kernel(kvm)) {
int ret = vgic_lazy_init(kvm);
if (ret)
return ret;
} else {
return -ENXIO;
}
if (vgic_is_v5(kvm)) {
/* Build a GICv5-style IntID here */
@ -2426,6 +2441,8 @@ static int __init init_subsystems(void)
switch (err) {
case 0:
vgic_present = true;
if (static_branch_unlikely(&kvm_vgic_global_state.gicv3_cpuif))
kvm_nvhe_sym(hyp_gicv3_nr_lr) = kvm_vgic_global_state.nr_lr;
break;
case -ENODEV:
case -ENXIO:

146
arch/arm64/kvm/at.c
View File

@ -136,14 +136,106 @@ static void compute_s1poe(struct kvm_vcpu *vcpu, struct s1_walk_info *wi)
wi->e0poe = (wi->regime != TR_EL2) && (val & TCR2_EL1_E0POE);
}
#define _has_tgran(__r, __sz) \
({ \
u64 _s1, _mmfr0 = __r; \
\
_s1 = SYS_FIELD_GET(ID_AA64MMFR0_EL1, \
TGRAN##__sz, _mmfr0); \
\
_s1 != ID_AA64MMFR0_EL1_TGRAN##__sz##_NI; \
})
static bool has_tgran(u64 mmfr0, unsigned int shift)
{
switch (shift) {
case 12:
return _has_tgran(mmfr0, 4);
case 14:
return _has_tgran(mmfr0, 16);
case 16:
return _has_tgran(mmfr0, 64);
default:
BUG();
}
}
static unsigned int tcr_to_tg0_pgshift(u64 tcr)
{
u64 tg0 = tcr & TCR_TG0_MASK;
switch (tg0) {
case TCR_TG0_4K:
return 12;
case TCR_TG0_16K:
return 14;
case TCR_TG0_64K:
default: /* IMPDEF: treat any other value as 64k */
return 16;
}
}
static unsigned int tcr_to_tg1_pgshift(u64 tcr)
{
u64 tg1 = tcr & TCR_TG1_MASK;
switch (tg1) {
case TCR_TG1_4K:
return 12;
case TCR_TG1_16K:
return 14;
case TCR_TG1_64K:
default: /* IMPDEF: treat any other value as 64k */
return 16;
}
}
static unsigned int fallback_tgran_shift(u64 mmfr0)
{
if (has_tgran(mmfr0, PAGE_SHIFT))
return PAGE_SHIFT;
else if (has_tgran(mmfr0, 12))
return 12;
else if (has_tgran(mmfr0, 14))
return 14;
else if (has_tgran(mmfr0, 16))
return 16;
else /* Should be unreacheable */
return PAGE_SHIFT;
}
static unsigned int tcr_tg_pgshift(struct kvm *kvm, u64 tcr, bool upper_range)
{
u64 mmfr0 = kvm_read_vm_id_reg(kvm, SYS_ID_AA64MMFR0_EL1);
unsigned int shift;
/* Someone was silly enough to encode TG0/TG1 differently */
if (upper_range)
shift = tcr_to_tg1_pgshift(tcr);
else
shift = tcr_to_tg0_pgshift(tcr);
/*
* If TGx is programmed to an unimplemented value (not advertised in
* ID_AA64MMFR0_EL1), we should treat it as if an implemented value is
* written, as per the architecture. Choose an available one while
* prioritizing PAGE_SIZE.
*/
if (!has_tgran(mmfr0, shift))
return fallback_tgran_shift(mmfr0);
return shift;
}
static int setup_s1_walk(struct kvm_vcpu *vcpu, struct s1_walk_info *wi,
struct s1_walk_result *wr, u64 va)
{
u64 hcr, sctlr, tcr, tg, ps, ia_bits, ttbr;
u64 hcr, sctlr, tcr, ps, ia_bits, ttbr;
unsigned int stride, x;
bool va55, tbi, lva;
bool va55, tbi, lva, upper_range;
va55 = va & BIT(55);
upper_range = va55 && wi->regime != TR_EL2;
if (vcpu_has_nv(vcpu)) {
hcr = __vcpu_sys_reg(vcpu, HCR_EL2);
@ -174,35 +266,12 @@ static int setup_s1_walk(struct kvm_vcpu *vcpu, struct s1_walk_info *wi,
BUG();
}
/* Someone was silly enough to encode TG0/TG1 differently */
if (va55 && wi->regime != TR_EL2) {
if (upper_range)
wi->txsz = FIELD_GET(TCR_T1SZ_MASK, tcr);
tg = FIELD_GET(TCR_TG1_MASK, tcr);
switch (tg << TCR_TG1_SHIFT) {
case TCR_TG1_4K:
wi->pgshift = 12; break;
case TCR_TG1_16K:
wi->pgshift = 14; break;
case TCR_TG1_64K:
default: /* IMPDEF: treat any other value as 64k */
wi->pgshift = 16; break;
}
} else {
else
wi->txsz = FIELD_GET(TCR_T0SZ_MASK, tcr);
tg = FIELD_GET(TCR_TG0_MASK, tcr);
switch (tg << TCR_TG0_SHIFT) {
case TCR_TG0_4K:
wi->pgshift = 12; break;
case TCR_TG0_16K:
wi->pgshift = 14; break;
case TCR_TG0_64K:
default: /* IMPDEF: treat any other value as 64k */
wi->pgshift = 16; break;
}
}
wi->pgshift = tcr_tg_pgshift(vcpu->kvm, tcr, upper_range);
wi->pa52bit = has_52bit_pa(vcpu, wi, tcr);
ia_bits = get_ia_size(wi);
@ -423,6 +492,9 @@ static int walk_s1(struct kvm_vcpu *vcpu, struct s1_walk_info *wi,
if (wi->s2) {
ret = kvm_walk_nested_s2(vcpu, ipa, &s2_trans);
if (ret == -EAGAIN)
return ret;
if (ret) {
fail_s1_walk(wr,
(s2_trans.esr & ~ESR_ELx_FSC_LEVEL) | level,
@ -492,15 +564,18 @@ static int walk_s1(struct kvm_vcpu *vcpu, struct s1_walk_info *wi,
/* Block mapping, check the validity of the level */
if (!(desc & BIT(1))) {
bool valid_block = false;
bool lpa = kvm_has_feat_enum(vcpu->kvm, ID_AA64MMFR0_EL1, PARANGE, 52);
switch (BIT(wi->pgshift)) {
case SZ_4K:
valid_block = level == 1 || level == 2 || (wi->pa52bit && level == 0);
break;
case SZ_16K:
case SZ_64K:
valid_block = level == 2 || (wi->pa52bit && level == 1);
break;
case SZ_64K:
valid_block = level == 2 || (lpa && level == 1);
break;
}
if (!valid_block)
@ -521,8 +596,12 @@ static int walk_s1(struct kvm_vcpu *vcpu, struct s1_walk_info *wi,
}
ret = kvm_swap_s1_desc(vcpu, ipa, desc, new_desc, wi);
if (ret)
if (ret == -EAGAIN)
return ret;
if (ret) {
fail_s1_walk(wr, ESR_ELx_FSC_SEA_TTW(level), false);
return ret;
}
desc = new_desc;
}
@ -1380,7 +1459,7 @@ static u64 __kvm_at_s1e01_fast(struct kvm_vcpu *vcpu, u32 op, u64 vaddr)
}
}
write_sysreg_el1(vcpu_read_sys_reg(vcpu, SCTLR_EL1), SYS_SCTLR);
__load_stage2(mmu, mmu->arch);
__load_stage2(mmu);
skip_mmu_switch:
/* Temporarily switch back to guest context */
@ -1553,7 +1632,10 @@ int __kvm_at_s12(struct kvm_vcpu *vcpu, u32 op, u64 vaddr)
return 0;
}
__kvm_at_s1e01(vcpu, op, vaddr);
ret = __kvm_at_s1e01(vcpu, op, vaddr);
if (ret)
return ret;
par = vcpu_read_sys_reg(vcpu, PAR_EL1);
if (par & SYS_PAR_EL1_F)
return 0;

12
arch/arm64/kvm/emulate-nested.c
View File

@ -2631,6 +2631,14 @@ bool triage_sysreg_trap(struct kvm_vcpu *vcpu, int *sr_index)
fgtreg = HFGITR2_EL2;
break;
case ICH_HFGRTR_GROUP:
fgtreg = is_read ? ICH_HFGRTR_EL2 : ICH_HFGWTR_EL2;
break;
case ICH_HFGITR_GROUP:
fgtreg = ICH_HFGITR_EL2;
break;
default:
/* Something is really wrong, bail out */
WARN_ONCE(1, "Bad FGT group (encoding %08x, config %016llx)\n",
@ -2862,6 +2870,8 @@ static int kvm_inject_nested(struct kvm_vcpu *vcpu, u64 esr_el2,
preempt_disable();
vcpu_set_flag(vcpu, IN_NESTED_EXCEPTION);
/*
* We may have an exception or PC update in the EL0/EL1 context.
* Commit it before entering EL2.
@ -2884,6 +2894,8 @@ static int kvm_inject_nested(struct kvm_vcpu *vcpu, u64 esr_el2,
__kvm_adjust_pc(vcpu);
kvm_arch_vcpu_load(vcpu, smp_processor_id());
vcpu_clear_flag(vcpu, IN_NESTED_EXCEPTION);
preempt_enable();
if (kvm_vcpu_has_pmu(vcpu))

26
arch/arm64/kvm/fpsimd.c
View File

@ -28,6 +28,20 @@ void kvm_arch_vcpu_load_fp(struct kvm_vcpu *vcpu)
if (!system_supports_fpsimd())
return;
/*
* Avoid needless save/restore of the guest's common
* FPSIMD/SVE/SME regs during transitions between L1/L2.
*
* These transitions only happens in a non-preemptible context
* where the host regs have already been saved and unbound. The
* live registers are either free or owned by the guest.
*/
if (vcpu_get_flag(vcpu, IN_NESTED_ERET) ||
vcpu_get_flag(vcpu, IN_NESTED_EXCEPTION)) {
WARN_ON_ONCE(host_owns_fp_regs());
return;
}
/*
* Ensure that any host FPSIMD/SVE/SME state is saved and unbound such
* that the host kernel is responsible for restoring this state upon
@ -102,6 +116,18 @@ void kvm_arch_vcpu_put_fp(struct kvm_vcpu *vcpu)
{
unsigned long flags;
/*
* See comment in kvm_arch_vcpu_load_fp(). Note that we also rely on
* the guest's max VL to have been set by fpsimd_lazy_switch_to_host()
* so that any intervening kernel-mode SIMD (NEON or otherwise)
* operation sees the full guest state that needs saving.
*/
if (vcpu_get_flag(vcpu, IN_NESTED_ERET) ||
vcpu_get_flag(vcpu, IN_NESTED_EXCEPTION)) {
WARN_ON_ONCE(host_owns_fp_regs());
return;
}
local_irq_save(flags);
if (guest_owns_fp_regs()) {

2
arch/arm64/kvm/hyp/include/hyp/switch.h
View File

@ -141,7 +141,7 @@ static inline void __activate_cptr_traps_vhe(struct kvm_vcpu *vcpu)
if (!(SYS_FIELD_GET(CPACR_EL1, ZEN, cptr) & BIT(0)))
val &= ~CPACR_EL1_ZEN;
if (kvm_has_feat(vcpu->kvm, ID_AA64MMFR3_EL1, S2POE, IMP))
if (kvm_has_feat(vcpu->kvm, ID_AA64MMFR3_EL1, S1POE, IMP))
val |= cptr & CPACR_EL1_E0POE;
val |= cptr & CPTR_EL2_TCPAC;

3
arch/arm64/kvm/hyp/include/nvhe/mem_protect.h
View File

@ -56,6 +56,7 @@ int host_stage2_idmap_locked(phys_addr_t addr, u64 size, enum kvm_pgtable_prot p
int host_stage2_set_owner_locked(phys_addr_t addr, u64 size, u8 owner_id);
int kvm_host_prepare_stage2(void *pgt_pool_base);
int kvm_guest_prepare_stage2(struct pkvm_hyp_vm *vm, void *pgd);
void kvm_guest_destroy_stage2(struct pkvm_hyp_vm *vm);
void handle_host_mem_abort(struct kvm_cpu_context *host_ctxt);
int hyp_pin_shared_mem(void *from, void *to);
@ -67,7 +68,7 @@ int refill_memcache(struct kvm_hyp_memcache *mc, unsigned long min_pages,
static __always_inline void __load_host_stage2(void)
{
if (static_branch_likely(&kvm_protected_mode_initialized))
__load_stage2(&host_mmu.arch.mmu, &host_mmu.arch);
__load_stage2(&host_mmu.arch.mmu);
else
write_sysreg(0, vttbr_el2);
}

21
arch/arm64/kvm/hyp/nvhe/hyp-main.c
View File

@ -24,6 +24,9 @@
DEFINE_PER_CPU(struct kvm_nvhe_init_params, kvm_init_params);
/* Number of implemented GICv3 LRs. Used by flush_hyp_vcpu(). */
unsigned int hyp_gicv3_nr_lr;
void __kvm_hyp_host_forward_smc(struct kvm_cpu_context *host_ctxt);
static void __hyp_sve_save_guest(struct kvm_vcpu *vcpu)
@ -130,10 +133,18 @@ static void flush_hyp_vcpu(struct pkvm_hyp_vcpu *hyp_vcpu)
hyp_vcpu->vcpu.arch.ctxt = host_vcpu->arch.ctxt;
/* __hyp_running_vcpu must be NULL in a guest context. */
hyp_vcpu->vcpu.arch.ctxt.__hyp_running_vcpu = NULL;
hyp_vcpu->vcpu.arch.mdcr_el2 = host_vcpu->arch.mdcr_el2;
hyp_vcpu->vcpu.arch.hcr_el2 &= ~(HCR_TWI | HCR_TWE);
/*
* HCR_EL2.VSE is host-owned (a pending virtual SError to inject), not a
* trap-control bit, so it must flow to the hyp vCPU alongside TWI/TWE
* for the vSError to be delivered. sync_hyp_vcpu() reflects it back.
*/
hyp_vcpu->vcpu.arch.hcr_el2 &= ~(HCR_TWI | HCR_TWE | HCR_VSE);
hyp_vcpu->vcpu.arch.hcr_el2 |= READ_ONCE(host_vcpu->arch.hcr_el2) &
(HCR_TWI | HCR_TWE);
(HCR_TWI | HCR_TWE | HCR_VSE);
hyp_vcpu->vcpu.arch.iflags = host_vcpu->arch.iflags;
@ -141,6 +152,12 @@ static void flush_hyp_vcpu(struct pkvm_hyp_vcpu *hyp_vcpu)
hyp_vcpu->vcpu.arch.vgic_cpu.vgic_v3 = host_vcpu->arch.vgic_cpu.vgic_v3;
/* Bound used_lrs by the number of implemented list registers. */
hyp_vcpu->vcpu.arch.vgic_cpu.vgic_v3.used_lrs =
min_t(unsigned int,
hyp_vcpu->vcpu.arch.vgic_cpu.vgic_v3.used_lrs,
hyp_gicv3_nr_lr);
hyp_vcpu->vcpu.arch.pid = host_vcpu->arch.pid;
}

37
arch/arm64/kvm/hyp/nvhe/mem_protect.c
View File

@ -217,7 +217,6 @@ static void *guest_s2_zalloc_page(void *mc)
memset(addr, 0, PAGE_SIZE);
p = hyp_virt_to_page(addr);
p->refcount = 1;
p->order = 0;
return addr;
}
@ -306,23 +305,27 @@ int kvm_guest_prepare_stage2(struct pkvm_hyp_vm *vm, void *pgd)
return 0;
}
void kvm_guest_destroy_stage2(struct pkvm_hyp_vm *vm)
{
guest_lock_component(vm);
kvm_pgtable_stage2_destroy(&vm->pgt);
vm->kvm.arch.mmu.pgd_phys = 0ULL;
guest_unlock_component(vm);
}
void reclaim_pgtable_pages(struct pkvm_hyp_vm *vm, struct kvm_hyp_memcache *mc)
{
struct hyp_page *page;
void *addr;
/* Dump all pgtable pages in the hyp_pool */
guest_lock_component(vm);
kvm_pgtable_stage2_destroy(&vm->pgt);
vm->kvm.arch.mmu.pgd_phys = 0ULL;
guest_unlock_component(vm);
kvm_guest_destroy_stage2(vm);
/* Drain the hyp_pool into the memcache */
addr = hyp_alloc_pages(&vm->pool, 0);
while (addr) {
page = hyp_virt_to_page(addr);
page->refcount = 0;
page->order = 0;
push_hyp_memcache(mc, addr, hyp_virt_to_phys);
WARN_ON(__pkvm_hyp_donate_host(hyp_virt_to_pfn(addr), 1));
addr = hyp_alloc_pages(&vm->pool, 0);
@ -352,7 +355,7 @@ int __pkvm_prot_finalize(void)
kvm_flush_dcache_to_poc(params, sizeof(*params));
write_sysreg_hcr(params->hcr_el2);
__load_stage2(&host_mmu.arch.mmu, &host_mmu.arch);
__load_stage2(&host_mmu.arch.mmu);
/*
* Make sure to have an ISB before the TLB maintenance below but only
@ -851,6 +854,16 @@ static int __hyp_check_page_state_range(phys_addr_t phys, u64 size, enum pkvm_pa
return 0;
}
static int __hyp_check_page_count_range(phys_addr_t phys, u64 size)
{
for_each_hyp_page(page, phys, size) {
if (page->refcount)
return -EBUSY;
}
return 0;
}
static bool guest_pte_is_poisoned(kvm_pte_t pte)
{
if (kvm_pte_valid(pte))
@ -1049,7 +1062,6 @@ int __pkvm_guest_unshare_host(struct pkvm_hyp_vcpu *vcpu, u64 gfn)
int __pkvm_host_unshare_hyp(u64 pfn)
{
u64 phys = hyp_pfn_to_phys(pfn);
u64 virt = (u64)__hyp_va(phys);
u64 size = PAGE_SIZE;
int ret;
@ -1062,10 +1074,9 @@ int __pkvm_host_unshare_hyp(u64 pfn)
ret = __hyp_check_page_state_range(phys, size, PKVM_PAGE_SHARED_BORROWED);
if (ret)
goto unlock;
if (hyp_page_count((void *)virt)) {
ret = -EBUSY;
ret = __hyp_check_page_count_range(phys, size);
if (ret)
goto unlock;
}
__hyp_set_page_state_range(phys, size, PKVM_NOPAGE);
WARN_ON(__host_set_page_state_range(phys, size, PKVM_PAGE_OWNED));
@ -1128,6 +1139,10 @@ int __pkvm_hyp_donate_host(u64 pfn, u64 nr_pages)
if (ret)
goto unlock;
ret = __hyp_check_page_count_range(phys, size);
if (ret)
goto unlock;
__hyp_set_page_state_range(phys, size, PKVM_NOPAGE);
WARN_ON(kvm_pgtable_hyp_unmap(&pkvm_pgtable, virt, size) != size);
WARN_ON(host_stage2_set_owner_locked(phys, size, PKVM_ID_HOST));

21
arch/arm64/kvm/hyp/nvhe/page_alloc.c
View File

@ -94,13 +94,22 @@ static void __hyp_attach_page(struct hyp_pool *pool,
struct hyp_page *p)
{
phys_addr_t phys = hyp_page_to_phys(p);
u8 order = p->order;
struct hyp_page *buddy;
bool coalesce = true;
u8 order = p->order;
memset(hyp_page_to_virt(p), 0, PAGE_SIZE << p->order);
/*
* 'external' pages are never coalesced and their ->order field
* untrusted as they bypass hyp_pool_init(). Enforce order-0.
*/
if (phys < pool->range_start || phys >= pool->range_end) {
order = 0;
coalesce = false;
}
/* Skip coalescing for 'external' pages being freed into the pool. */
if (phys < pool->range_start || phys >= pool->range_end)
memset(hyp_page_to_virt(p), 0, PAGE_SIZE << order);
if (!coalesce)
goto insert;
/*
@ -237,8 +246,10 @@ int hyp_pool_init(struct hyp_pool *pool, u64 pfn, unsigned int nr_pages,
/* Init the vmemmap portion */
p = hyp_phys_to_page(phys);
for (i = 0; i < nr_pages; i++)
for (i = 0; i < nr_pages; i++) {
hyp_set_page_refcounted(&p[i]);
p[i].order = 0;
}
/* Attach the unused pages to the buddy tree */
for (i = reserved_pages; i < nr_pages; i++)

4
arch/arm64/kvm/hyp/nvhe/pkvm.c
View File

@ -853,10 +853,12 @@ int __pkvm_init_vm(struct kvm *host_kvm, unsigned long vm_hva,
/* Must be called last since this publishes the VM. */
ret = insert_vm_table_entry(handle, hyp_vm);
if (ret)
goto err_remove_mappings;
goto err_destroy_stage2;
return 0;
err_destroy_stage2:
kvm_guest_destroy_stage2(hyp_vm);
err_remove_mappings:
unmap_donated_memory(hyp_vm, vm_size);
unmap_donated_memory(pgd, pgd_size);

2
arch/arm64/kvm/hyp/nvhe/switch.c
View File

@ -315,7 +315,7 @@ int __kvm_vcpu_run(struct kvm_vcpu *vcpu)
__sysreg_restore_state_nvhe(guest_ctxt);
mmu = kern_hyp_va(vcpu->arch.hw_mmu);
__load_stage2(mmu, kern_hyp_va(mmu->arch));
__load_stage2(mmu);
__activate_traps(vcpu);
__hyp_vgic_restore_state(vcpu);

4
arch/arm64/kvm/hyp/nvhe/tlb.c
View File

@ -110,7 +110,7 @@ static void enter_vmid_context(struct kvm_s2_mmu *mmu,
if (vcpu)
__load_host_stage2();
else
__load_stage2(mmu, kern_hyp_va(mmu->arch));
__load_stage2(mmu);
asm(ALTERNATIVE("isb", "nop", ARM64_WORKAROUND_SPECULATIVE_AT));
}
@ -128,7 +128,7 @@ static void exit_vmid_context(struct tlb_inv_context *cxt)
return;
if (vcpu)
__load_stage2(mmu, kern_hyp_va(mmu->arch));
__load_stage2(mmu);
else
__load_host_stage2();

78
arch/arm64/kvm/hyp/vgic-v5-sr.c
View File

@ -30,10 +30,9 @@ void __vgic_v5_save_ppi_state(struct vgic_v5_cpu_if *cpu_if)
{
/*
* The following code assumes that the bitmap storage that we have for
* PPIs is either 64 (architected PPIs, only) or 128 bits (architected &
* impdef PPIs).
* PPIs is either 64 (architected PPIs, only).
*/
BUILD_BUG_ON(VGIC_V5_NR_PRIVATE_IRQS % 64);
BUILD_BUG_ON(VGIC_V5_NR_PRIVATE_IRQS != 64);
bitmap_write(host_data_ptr(vgic_v5_ppi_state)->activer_exit,
read_sysreg_s(SYS_ICH_PPI_ACTIVER0_EL2), 0, 64);
@ -49,22 +48,6 @@ void __vgic_v5_save_ppi_state(struct vgic_v5_cpu_if *cpu_if)
cpu_if->vgic_ppi_priorityr[6] = read_sysreg_s(SYS_ICH_PPI_PRIORITYR6_EL2);
cpu_if->vgic_ppi_priorityr[7] = read_sysreg_s(SYS_ICH_PPI_PRIORITYR7_EL2);
if (VGIC_V5_NR_PRIVATE_IRQS == 128) {
bitmap_write(host_data_ptr(vgic_v5_ppi_state)->activer_exit,
read_sysreg_s(SYS_ICH_PPI_ACTIVER1_EL2), 64, 64);
bitmap_write(host_data_ptr(vgic_v5_ppi_state)->pendr,
read_sysreg_s(SYS_ICH_PPI_PENDR1_EL2), 64, 64);
cpu_if->vgic_ppi_priorityr[8] = read_sysreg_s(SYS_ICH_PPI_PRIORITYR8_EL2);
cpu_if->vgic_ppi_priorityr[9] = read_sysreg_s(SYS_ICH_PPI_PRIORITYR9_EL2);
cpu_if->vgic_ppi_priorityr[10] = read_sysreg_s(SYS_ICH_PPI_PRIORITYR10_EL2);
cpu_if->vgic_ppi_priorityr[11] = read_sysreg_s(SYS_ICH_PPI_PRIORITYR11_EL2);
cpu_if->vgic_ppi_priorityr[12] = read_sysreg_s(SYS_ICH_PPI_PRIORITYR12_EL2);
cpu_if->vgic_ppi_priorityr[13] = read_sysreg_s(SYS_ICH_PPI_PRIORITYR13_EL2);
cpu_if->vgic_ppi_priorityr[14] = read_sysreg_s(SYS_ICH_PPI_PRIORITYR14_EL2);
cpu_if->vgic_ppi_priorityr[15] = read_sysreg_s(SYS_ICH_PPI_PRIORITYR15_EL2);
}
/* Now that we are done, disable DVI */
write_sysreg_s(0, SYS_ICH_PPI_DVIR0_EL2);
write_sysreg_s(0, SYS_ICH_PPI_DVIR1_EL2);
@ -74,9 +57,6 @@ void __vgic_v5_restore_ppi_state(struct vgic_v5_cpu_if *cpu_if)
{
DECLARE_BITMAP(pendr, VGIC_V5_NR_PRIVATE_IRQS);
/* We assume 64 or 128 PPIs - see above comment */
BUILD_BUG_ON(VGIC_V5_NR_PRIVATE_IRQS % 64);
/* Enable DVI so that the guest's interrupt config takes over */
write_sysreg_s(bitmap_read(cpu_if->vgic_ppi_dvir, 0, 64),
SYS_ICH_PPI_DVIR0_EL2);
@ -108,50 +88,20 @@ void __vgic_v5_restore_ppi_state(struct vgic_v5_cpu_if *cpu_if)
write_sysreg_s(cpu_if->vgic_ppi_priorityr[7],
SYS_ICH_PPI_PRIORITYR7_EL2);
if (VGIC_V5_NR_PRIVATE_IRQS == 128) {
/* Enable DVI so that the guest's interrupt config takes over */
write_sysreg_s(bitmap_read(cpu_if->vgic_ppi_dvir, 64, 64),
SYS_ICH_PPI_DVIR1_EL2);
write_sysreg_s(0, SYS_ICH_PPI_DVIR1_EL2);
write_sysreg_s(bitmap_read(cpu_if->vgic_ppi_activer, 64, 64),
SYS_ICH_PPI_ACTIVER1_EL2);
write_sysreg_s(bitmap_read(cpu_if->vgic_ppi_enabler, 64, 64),
SYS_ICH_PPI_ENABLER1_EL2);
write_sysreg_s(bitmap_read(pendr, 64, 64),
SYS_ICH_PPI_PENDR1_EL2);
write_sysreg_s(0, SYS_ICH_PPI_ACTIVER1_EL2);
write_sysreg_s(0, SYS_ICH_PPI_ENABLER1_EL2);
write_sysreg_s(0, SYS_ICH_PPI_PENDR1_EL2);
write_sysreg_s(cpu_if->vgic_ppi_priorityr[8],
SYS_ICH_PPI_PRIORITYR8_EL2);
write_sysreg_s(cpu_if->vgic_ppi_priorityr[9],
SYS_ICH_PPI_PRIORITYR9_EL2);
write_sysreg_s(cpu_if->vgic_ppi_priorityr[10],
SYS_ICH_PPI_PRIORITYR10_EL2);
write_sysreg_s(cpu_if->vgic_ppi_priorityr[11],
SYS_ICH_PPI_PRIORITYR11_EL2);
write_sysreg_s(cpu_if->vgic_ppi_priorityr[12],
SYS_ICH_PPI_PRIORITYR12_EL2);
write_sysreg_s(cpu_if->vgic_ppi_priorityr[13],
SYS_ICH_PPI_PRIORITYR13_EL2);
write_sysreg_s(cpu_if->vgic_ppi_priorityr[14],
SYS_ICH_PPI_PRIORITYR14_EL2);
write_sysreg_s(cpu_if->vgic_ppi_priorityr[15],
SYS_ICH_PPI_PRIORITYR15_EL2);
} else {
write_sysreg_s(0, SYS_ICH_PPI_DVIR1_EL2);
write_sysreg_s(0, SYS_ICH_PPI_ACTIVER1_EL2);
write_sysreg_s(0, SYS_ICH_PPI_ENABLER1_EL2);
write_sysreg_s(0, SYS_ICH_PPI_PENDR1_EL2);
write_sysreg_s(0, SYS_ICH_PPI_PRIORITYR8_EL2);
write_sysreg_s(0, SYS_ICH_PPI_PRIORITYR9_EL2);
write_sysreg_s(0, SYS_ICH_PPI_PRIORITYR10_EL2);
write_sysreg_s(0, SYS_ICH_PPI_PRIORITYR11_EL2);
write_sysreg_s(0, SYS_ICH_PPI_PRIORITYR12_EL2);
write_sysreg_s(0, SYS_ICH_PPI_PRIORITYR13_EL2);
write_sysreg_s(0, SYS_ICH_PPI_PRIORITYR14_EL2);
write_sysreg_s(0, SYS_ICH_PPI_PRIORITYR15_EL2);
}
write_sysreg_s(0, SYS_ICH_PPI_PRIORITYR8_EL2);
write_sysreg_s(0, SYS_ICH_PPI_PRIORITYR9_EL2);
write_sysreg_s(0, SYS_ICH_PPI_PRIORITYR10_EL2);
write_sysreg_s(0, SYS_ICH_PPI_PRIORITYR11_EL2);
write_sysreg_s(0, SYS_ICH_PPI_PRIORITYR12_EL2);
write_sysreg_s(0, SYS_ICH_PPI_PRIORITYR13_EL2);
write_sysreg_s(0, SYS_ICH_PPI_PRIORITYR14_EL2);
write_sysreg_s(0, SYS_ICH_PPI_PRIORITYR15_EL2);
}
void __vgic_v5_save_state(struct vgic_v5_cpu_if *cpu_if)

2
arch/arm64/kvm/hyp/vhe/switch.c
View File

@ -219,7 +219,7 @@ void kvm_vcpu_load_vhe(struct kvm_vcpu *vcpu)
__vcpu_load_switch_sysregs(vcpu);
__vcpu_load_activate_traps(vcpu);
__load_stage2(vcpu->arch.hw_mmu, vcpu->arch.hw_mmu->arch);
__load_stage2(vcpu->arch.hw_mmu);
}
void kvm_vcpu_put_vhe(struct kvm_vcpu *vcpu)

4
arch/arm64/kvm/hyp/vhe/tlb.c
View File

@ -60,7 +60,7 @@ static void enter_vmid_context(struct kvm_s2_mmu *mmu,
* place before clearing TGE. __load_stage2() already
* has an ISB in order to deal with this.
*/
__load_stage2(mmu, mmu->arch);
__load_stage2(mmu);
val = read_sysreg(hcr_el2);
val &= ~HCR_TGE;
write_sysreg_hcr(val);
@ -78,7 +78,7 @@ static void exit_vmid_context(struct tlb_inv_context *cxt)
/* ... and the stage-2 MMU context that we switched away from */
if (cxt->mmu)
__load_stage2(cxt->mmu, cxt->mmu->arch);
__load_stage2(cxt->mmu);
if (cpus_have_final_cap(ARM64_WORKAROUND_SPECULATIVE_AT)) {
/* Restore the registers to what they were */

39
arch/arm64/kvm/mmu.c
View File

@ -501,6 +501,10 @@ static int share_pfn_hyp(u64 pfn)
rb_link_node(&this->node, parent, node);
rb_insert_color(&this->node, &hyp_shared_pfns);
ret = kvm_call_hyp_nvhe(__pkvm_host_share_hyp, pfn);
if (ret) {
rb_erase(&this->node, &hyp_shared_pfns);
kfree(this);
}
unlock:
mutex_unlock(&hyp_shared_pfns_lock);
@ -520,13 +524,17 @@ static int unshare_pfn_hyp(u64 pfn)
goto unlock;
}
this->count--;
if (this->count)
if (this->count > 1) {
this->count--;
goto unlock;
}
ret = kvm_call_hyp_nvhe(__pkvm_host_unshare_hyp, pfn);
if (ret)
goto unlock;
rb_erase(&this->node, &hyp_shared_pfns);
kfree(this);
ret = kvm_call_hyp_nvhe(__pkvm_host_unshare_hyp, pfn);
unlock:
mutex_unlock(&hyp_shared_pfns_lock);
@ -536,8 +544,8 @@ static int unshare_pfn_hyp(u64 pfn)
int kvm_share_hyp(void *from, void *to)
{
phys_addr_t start, end, cur;
int ret = 0;
u64 pfn;
int ret;
if (is_kernel_in_hyp_mode())
return 0;
@ -559,10 +567,24 @@ int kvm_share_hyp(void *from, void *to)
pfn = __phys_to_pfn(cur);
ret = share_pfn_hyp(pfn);
if (ret)
return ret;
break;
}
return 0;
if (!ret)
return 0;
/*
* Roll back the pages shared by this call. A failed unshare leaks
* the page (it stays shared with the hypervisor and is no longer
* reusable for pKVM) but breaks no isolation guarantee, so warn and
* continue. Not expected in practice.
*/
for (end = cur, cur = start; cur < end; cur += PAGE_SIZE) {
pfn = __phys_to_pfn(cur);
WARN_ON(unshare_pfn_hyp(pfn));
}
return ret;
}
void kvm_unshare_hyp(void *from, void *to)
@ -577,6 +599,11 @@ void kvm_unshare_hyp(void *from, void *to)
end = PAGE_ALIGN(__pa(to));
for (cur = start; cur < end; cur += PAGE_SIZE) {
pfn = __phys_to_pfn(cur);
/*
* A failed unshare leaks the page: it stays shared with the
* hypervisor and is no longer reusable for pKVM. No isolation
* guarantee is broken, and this is not expected in practice.
*/
WARN_ON(unshare_pfn_hyp(pfn));
}
}

236
arch/arm64/kvm/nested.c
View File

@ -359,8 +359,13 @@ static int walk_nested_s2_pgd(struct kvm_vcpu *vcpu, phys_addr_t ipa,
if (new_desc != desc) {
ret = swap_guest_s2_desc(vcpu, paddr, desc, new_desc, wi);
if (ret)
if (ret == -EAGAIN)
return ret;
if (ret) {
out->esr = ESR_ELx_FSC_SEA_TTW(level);
out->desc = desc;
return 1;
}
desc = new_desc;
}
@ -385,32 +390,104 @@ static int walk_nested_s2_pgd(struct kvm_vcpu *vcpu, phys_addr_t ipa,
return 0;
}
static void vtcr_to_walk_info(u64 vtcr, struct s2_walk_info *wi)
{
wi->t0sz = vtcr & TCR_EL2_T0SZ_MASK;
#define _has_tgran_2(__r, __sz) \
({ \
u64 _s1, _s2, _mmfr0 = __r; \
\
_s2 = SYS_FIELD_GET(ID_AA64MMFR0_EL1, \
TGRAN##__sz##_2, _mmfr0); \
\
_s1 = SYS_FIELD_GET(ID_AA64MMFR0_EL1, \
TGRAN##__sz, _mmfr0); \
\
((_s2 != ID_AA64MMFR0_EL1_TGRAN##__sz##_2_NI && \
_s2 != ID_AA64MMFR0_EL1_TGRAN##__sz##_2_TGRAN##__sz) || \
(_s2 == ID_AA64MMFR0_EL1_TGRAN##__sz##_2_TGRAN##__sz && \
_s1 != ID_AA64MMFR0_EL1_TGRAN##__sz##_NI)); \
})
switch (FIELD_GET(VTCR_EL2_TG0_MASK, vtcr)) {
static bool has_tgran_2(u64 mmfr0, unsigned int shift)
{
switch (shift) {
case 12:
return _has_tgran_2(mmfr0, 4);
case 14:
return _has_tgran_2(mmfr0, 16);
case 16:
return _has_tgran_2(mmfr0, 64);
default:
BUG();
}
}
static unsigned int fallback_tgran2_shift(u64 mmfr0)
{
if (has_tgran_2(mmfr0, PAGE_SHIFT))
return PAGE_SHIFT;
else if (has_tgran_2(mmfr0, 12))
return 12;
else if (has_tgran_2(mmfr0, 14))
return 14;
else if (has_tgran_2(mmfr0, 16))
return 16;
else
return PAGE_SHIFT;
}
static unsigned int vtcr_to_tg0_pgshift(struct kvm *kvm, u64 vtcr)
{
u64 tg0 = FIELD_GET(VTCR_EL2_TG0_MASK, vtcr);
u64 mmfr0 = kvm_read_vm_id_reg(kvm, SYS_ID_AA64MMFR0_EL1);
unsigned int shift;
switch (tg0) {
case VTCR_EL2_TG0_4K:
wi->pgshift = 12; break;
shift = 12;
break;
case VTCR_EL2_TG0_16K:
wi->pgshift = 14; break;
shift = 14;
break;
case VTCR_EL2_TG0_64K:
default: /* IMPDEF: treat any other value as 64k */
wi->pgshift = 16; break;
/* IMPDEF: treat any other value as 64k, subject to fallback */
default:
shift = 16;
}
/*
* If TGx is programmed to an unimplemented value (not advertised in
* ID_AA64MMFR0_EL1), we should treat it as if an implemented value is
* written, as per the architecture. Choose an available one while
* prioritizing PAGE_SIZE.
*/
if (!has_tgran_2(mmfr0, shift))
return fallback_tgran2_shift(mmfr0);
return shift;
}
static size_t vtcr_to_tg0_pgsize(struct kvm *kvm, u64 vtcr)
{
return BIT(vtcr_to_tg0_pgshift(kvm, vtcr));
}
static void setup_s2_walk(struct kvm_vcpu *vcpu, struct s2_walk_info *wi)
{
u64 vtcr = vcpu_read_sys_reg(vcpu, VTCR_EL2);
wi->baddr = vcpu_read_sys_reg(vcpu, VTTBR_EL2);
wi->t0sz = vtcr & VTCR_EL2_T0SZ_MASK;
wi->pgshift = vtcr_to_tg0_pgshift(vcpu->kvm, vtcr);
wi->sl = FIELD_GET(VTCR_EL2_SL0_MASK, vtcr);
/* Global limit for now, should eventually be per-VM */
wi->max_oa_bits = min(get_kvm_ipa_limit(),
ps_to_output_size(FIELD_GET(VTCR_EL2_PS_MASK, vtcr), false));
wi->ha = vtcr & VTCR_EL2_HA;
wi->be = vcpu_read_sys_reg(vcpu, SCTLR_EL2) & SCTLR_ELx_EE;
}
int kvm_walk_nested_s2(struct kvm_vcpu *vcpu, phys_addr_t gipa,
struct kvm_s2_trans *result)
{
u64 vtcr = vcpu_read_sys_reg(vcpu, VTCR_EL2);
struct s2_walk_info wi;
int ret;
@ -419,11 +496,7 @@ int kvm_walk_nested_s2(struct kvm_vcpu *vcpu, phys_addr_t gipa,
if (!vcpu_has_nv(vcpu))
return 0;
wi.baddr = vcpu_read_sys_reg(vcpu, VTTBR_EL2);
vtcr_to_walk_info(vtcr, &wi);
wi.be = vcpu_read_sys_reg(vcpu, SCTLR_EL2) & SCTLR_ELx_EE;
setup_s2_walk(vcpu, &wi);
ret = walk_nested_s2_pgd(vcpu, gipa, &wi, result);
if (ret)
@ -519,20 +592,21 @@ static u8 pgshift_level_to_ttl(u16 shift, u8 level)
*/
static u8 get_guest_mapping_ttl(struct kvm_s2_mmu *mmu, u64 addr)
{
u64 tmp, sz = 0, vtcr = mmu->tlb_vtcr;
size_t tg0_size = vtcr_to_tg0_pgsize(kvm_s2_mmu_to_kvm(mmu), mmu->tlb_vtcr);
u64 tmp, sz = 0;
kvm_pte_t pte;
u8 ttl, level;
lockdep_assert_held_write(&kvm_s2_mmu_to_kvm(mmu)->mmu_lock);
switch (FIELD_GET(VTCR_EL2_TG0_MASK, vtcr)) {
case VTCR_EL2_TG0_4K:
switch (tg0_size) {
case SZ_4K:
ttl = (TLBI_TTL_TG_4K << 2);
break;
case VTCR_EL2_TG0_16K:
case SZ_16K:
ttl = (TLBI_TTL_TG_16K << 2);
break;
case VTCR_EL2_TG0_64K:
case SZ_64K:
default: /* IMPDEF: treat any other value as 64k */
ttl = (TLBI_TTL_TG_64K << 2);
break;
@ -542,19 +616,19 @@ static u8 get_guest_mapping_ttl(struct kvm_s2_mmu *mmu, u64 addr)
again:
/* Iteratively compute the block sizes for a particular granule size */
switch (FIELD_GET(VTCR_EL2_TG0_MASK, vtcr)) {
case VTCR_EL2_TG0_4K:
switch (tg0_size) {
case SZ_4K:
if (sz < SZ_4K) sz = SZ_4K;
else if (sz < SZ_2M) sz = SZ_2M;
else if (sz < SZ_1G) sz = SZ_1G;
else sz = 0;
break;
case VTCR_EL2_TG0_16K:
case SZ_16K:
if (sz < SZ_16K) sz = SZ_16K;
else if (sz < SZ_32M) sz = SZ_32M;
else sz = 0;
break;
case VTCR_EL2_TG0_64K:
case SZ_64K:
default: /* IMPDEF: treat any other value as 64k */
if (sz < SZ_64K) sz = SZ_64K;
else if (sz < SZ_512M) sz = SZ_512M;
@ -605,14 +679,14 @@ unsigned long compute_tlb_inval_range(struct kvm_s2_mmu *mmu, u64 val)
if (!max_size) {
/* Compute the maximum extent of the invalidation */
switch (FIELD_GET(VTCR_EL2_TG0_MASK, mmu->tlb_vtcr)) {
case VTCR_EL2_TG0_4K:
switch (vtcr_to_tg0_pgsize(kvm, mmu->tlb_vtcr)) {
case SZ_4K:
max_size = SZ_1G;
break;
case VTCR_EL2_TG0_16K:
case SZ_16K:
max_size = SZ_32M;
break;
case VTCR_EL2_TG0_64K:
case SZ_64K:
default: /* IMPDEF: treat any other value as 64k */
/*
* No, we do not support 52bit IPA in nested yet. Once
@ -804,18 +878,24 @@ void kvm_vcpu_load_hw_mmu(struct kvm_vcpu *vcpu)
}
}
static void this_cpu_reset_vncr_fixmap(struct kvm_vcpu *vcpu)
{
if (!host_data_test_flag(L1_VNCR_MAPPED))
return;
BUG_ON(vcpu->arch.vncr_tlb->cpu != smp_processor_id());
BUG_ON(is_hyp_ctxt(vcpu));
clear_fixmap(vncr_fixmap(vcpu->arch.vncr_tlb->cpu));
vcpu->arch.vncr_tlb->cpu = -1;
host_data_clear_flag(L1_VNCR_MAPPED);
atomic_dec(&vcpu->kvm->arch.vncr_map_count);
}
void kvm_vcpu_put_hw_mmu(struct kvm_vcpu *vcpu)
{
/* Unconditionally drop the VNCR mapping if we have one */
if (host_data_test_flag(L1_VNCR_MAPPED)) {
BUG_ON(vcpu->arch.vncr_tlb->cpu != smp_processor_id());
BUG_ON(is_hyp_ctxt(vcpu));
clear_fixmap(vncr_fixmap(vcpu->arch.vncr_tlb->cpu));
vcpu->arch.vncr_tlb->cpu = -1;
host_data_clear_flag(L1_VNCR_MAPPED);
atomic_dec(&vcpu->kvm->arch.vncr_map_count);
}
this_cpu_reset_vncr_fixmap(vcpu);
/*
* Keep a reference on the associated stage-2 MMU if the vCPU is
@ -904,9 +984,21 @@ static void invalidate_vncr(struct vncr_tlb *vt)
clear_fixmap(vncr_fixmap(vt->cpu));
}
/*
* VNCR TLB invalidation occurs from MMU notifiers or TLBI instructions, and
* either can race against a vcpu not being onlined yet (no pseudo-TLB
* allocated). Similarly, the TLB might be invalid. Skip those, as they
* obviously don't participate in the invalidation at this stage.
*/
#define kvm_for_each_vncr_tlb(idx, vcpup, tlbp, kvm) \
kvm_for_each_vcpu(idx, vcpup, kvm) \
if (((tlbp) = vcpup->arch.vncr_tlb) && \
(tlbp)->valid)
static void kvm_invalidate_vncr_ipa(struct kvm *kvm, u64 start, u64 end)
{
struct kvm_vcpu *vcpu;
struct vncr_tlb *vt;
unsigned long i;
lockdep_assert_held_write(&kvm->mmu_lock);
@ -914,24 +1006,9 @@ static void kvm_invalidate_vncr_ipa(struct kvm *kvm, u64 start, u64 end)
if (!kvm_has_feat(kvm, ID_AA64MMFR4_EL1, NV_frac, NV2_ONLY))
return;
kvm_for_each_vcpu(i, vcpu, kvm) {
struct vncr_tlb *vt = vcpu->arch.vncr_tlb;
kvm_for_each_vncr_tlb(i, vcpu, vt, kvm) {
u64 ipa_start, ipa_end, ipa_size;
/*
* Careful here: We end-up here from an MMU notifier,
* and this can race against a vcpu not being onlined
* yet, without the pseudo-TLB being allocated.
*
* Skip those, as they obviously don't participate in
* the invalidation at this stage.
*/
if (!vt)
continue;
if (!vt->valid)
continue;
ipa_size = ttl_to_size(pgshift_level_to_ttl(vt->wi.pgshift,
vt->wr.level));
ipa_start = vt->wr.pa & ~(ipa_size - 1);
@ -961,17 +1038,14 @@ static void invalidate_vncr_va(struct kvm *kvm,
struct s1e2_tlbi_scope *scope)
{
struct kvm_vcpu *vcpu;
struct vncr_tlb *vt;
unsigned long i;
lockdep_assert_held_write(&kvm->mmu_lock);
kvm_for_each_vcpu(i, vcpu, kvm) {
struct vncr_tlb *vt = vcpu->arch.vncr_tlb;
kvm_for_each_vncr_tlb(i, vcpu, vt, kvm) {
u64 va_start, va_end, va_size;
if (!vt->valid)
continue;
va_size = ttl_to_size(pgshift_level_to_ttl(vt->wi.pgshift,
vt->wr.level));
va_start = vt->gva & ~(va_size - 1);
@ -1255,8 +1329,20 @@ int kvm_vcpu_allocate_vncr_tlb(struct kvm_vcpu *vcpu)
if (!kvm_has_feat(vcpu->kvm, ID_AA64MMFR4_EL1, NV_frac, NV2_ONLY))
return 0;
vcpu->arch.vncr_tlb = kzalloc_obj(*vcpu->arch.vncr_tlb,
GFP_KERNEL_ACCOUNT);
if (!vcpu->arch.vncr_tlb) {
struct vncr_tlb *vt = kzalloc_obj(*vcpu->arch.vncr_tlb,
GFP_KERNEL_ACCOUNT);
/*
* Taking the lock on assignment ensures that the TLB is
* seen as initialised when following the pointer (release
* semantics of the unlock), and avoids having acquires on
* each user which already take the lock.
*/
scoped_guard(write_lock, &vcpu->kvm->mmu_lock)
vcpu->arch.vncr_tlb = vt;
}
if (!vcpu->arch.vncr_tlb)
return -ENOMEM;
@ -1289,7 +1375,8 @@ static int kvm_translate_vncr(struct kvm_vcpu *vcpu, bool *is_gmem)
* We also prepare the next walk wilst we're at it.
*/
scoped_guard(write_lock, &vcpu->kvm->mmu_lock) {
invalidate_vncr(vt);
this_cpu_reset_vncr_fixmap(vcpu);
vt->valid = false;
vt->wi = (struct s1_walk_info) {
.regime = TR_EL20,
@ -1333,8 +1420,10 @@ static int kvm_translate_vncr(struct kvm_vcpu *vcpu, bool *is_gmem)
}
scoped_guard(write_lock, &vcpu->kvm->mmu_lock) {
if (mmu_invalidate_retry(vcpu->kvm, mmu_seq))
if (mmu_invalidate_retry(vcpu->kvm, mmu_seq)) {
kvm_release_faultin_page(vcpu->kvm, page, true, false);
return -EAGAIN;
}
vt->gva = va;
vt->hpa = pfn << PAGE_SHIFT;
@ -1505,21 +1594,6 @@ static void kvm_map_l1_vncr(struct kvm_vcpu *vcpu)
}
}
#define has_tgran_2(__r, __sz) \
({ \
u64 _s1, _s2, _mmfr0 = __r; \
\
_s2 = SYS_FIELD_GET(ID_AA64MMFR0_EL1, \
TGRAN##__sz##_2, _mmfr0); \
\
_s1 = SYS_FIELD_GET(ID_AA64MMFR0_EL1, \
TGRAN##__sz, _mmfr0); \
\
((_s2 != ID_AA64MMFR0_EL1_TGRAN##__sz##_2_NI && \
_s2 != ID_AA64MMFR0_EL1_TGRAN##__sz##_2_TGRAN##__sz) || \
(_s2 == ID_AA64MMFR0_EL1_TGRAN##__sz##_2_TGRAN##__sz && \
_s1 != ID_AA64MMFR0_EL1_TGRAN##__sz##_NI)); \
})
/*
* Our emulated CPU doesn't support all the possible features. For the
* sake of simplicity (and probably mental sanity), wipe out a number
@ -1606,15 +1680,15 @@ u64 limit_nv_id_reg(struct kvm *kvm, u32 reg, u64 val)
*/
switch (PAGE_SIZE) {
case SZ_4K:
if (has_tgran_2(orig_val, 4))
if (_has_tgran_2(orig_val, 4))
val |= SYS_FIELD_PREP_ENUM(ID_AA64MMFR0_EL1, TGRAN4_2, IMP);
fallthrough;
case SZ_16K:
if (has_tgran_2(orig_val, 16))
if (_has_tgran_2(orig_val, 16))
val |= SYS_FIELD_PREP_ENUM(ID_AA64MMFR0_EL1, TGRAN16_2, IMP);
fallthrough;
case SZ_64K:
if (has_tgran_2(orig_val, 64))
if (_has_tgran_2(orig_val, 64))
val |= SYS_FIELD_PREP_ENUM(ID_AA64MMFR0_EL1, TGRAN64_2, IMP);
break;
}

31
arch/arm64/kvm/pmu-emul.c
View File

@ -396,44 +396,31 @@ static bool kvm_pmu_overflow_status(struct kvm_vcpu *vcpu)
static void kvm_pmu_update_state(struct kvm_vcpu *vcpu)
{
struct kvm_pmu *pmu = &vcpu->arch.pmu;
bool overflow;
overflow = kvm_pmu_overflow_status(vcpu);
if (pmu->irq_level == overflow)
if (unlikely(!irqchip_in_kernel(vcpu->kvm)))
return;
pmu->irq_level = overflow;
if (likely(irqchip_in_kernel(vcpu->kvm))) {
int ret = kvm_vgic_inject_irq(vcpu->kvm, vcpu,
pmu->irq_num, overflow, pmu);
WARN_ON(ret);
}
WARN_ON(kvm_vgic_inject_irq(vcpu->kvm, vcpu, pmu->irq_num,
kvm_pmu_overflow_status(vcpu), pmu));
}
bool kvm_pmu_should_notify_user(struct kvm_vcpu *vcpu)
{
struct kvm_pmu *pmu = &vcpu->arch.pmu;
struct kvm_sync_regs *sregs = &vcpu->run->s.regs;
bool run_level = sregs->device_irq_level & KVM_ARM_DEV_PMU;
if (likely(irqchip_in_kernel(vcpu->kvm)))
return false;
return pmu->irq_level != run_level;
return kvm_pmu_overflow_status(vcpu) != run_level;
}
/*
* Reflect the PMU overflow interrupt output level into the kvm_run structure
*/
void kvm_pmu_update_run(struct kvm_vcpu *vcpu)
bool kvm_pmu_update_run(struct kvm_vcpu *vcpu)
{
struct kvm_sync_regs *regs = &vcpu->run->s.regs;
/* Populate the timer bitmap for user space */
regs->device_irq_level &= ~KVM_ARM_DEV_PMU;
if (vcpu->arch.pmu.irq_level)
regs->device_irq_level |= KVM_ARM_DEV_PMU;
bool update = kvm_pmu_should_notify_user(vcpu);
if (update)
vcpu->run->s.regs.device_irq_level ^= KVM_ARM_DEV_PMU;
return update;
}
/**

20
arch/arm64/kvm/sys_regs.c
View File

@ -724,6 +724,7 @@ static bool access_gicv5_ppi_enabler(struct kvm_vcpu *vcpu,
{
unsigned long *mask = vcpu->kvm->arch.vgic.gicv5_vm.vgic_ppi_mask;
struct vgic_v5_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.vgic_v5;
unsigned long reg = p->regval;
int i;
/* We never expect to get here with a read! */
@ -731,27 +732,23 @@ static bool access_gicv5_ppi_enabler(struct kvm_vcpu *vcpu,
return undef_access(vcpu, p, r);
/*
* If we're only handling architected PPIs and the guest writes to the
* enable for the non-architected PPIs, we just return as there's
* nothing to do at all. We don't even allocate the storage for them in
* this case.
* As we're only handling architected PPIs, the guest writes to the
* enable for the non-architected PPIs just return as there's
* nothing to do at all. We don't even allocate the storage for them.
*/
if (VGIC_V5_NR_PRIVATE_IRQS == 64 && p->Op2 % 2)
if (p->Op2 % 2)
return true;
/*
* Merge the raw guest write into out bitmap at an offset of either 0 or
* 64, then and it with our PPI mask.
* Merge the raw guest write into out bitmap, anded with our PPI mask.
*/
bitmap_write(cpu_if->vgic_ppi_enabler, p->regval, 64 * (p->Op2 % 2), 64);
bitmap_and(cpu_if->vgic_ppi_enabler, cpu_if->vgic_ppi_enabler, mask,
VGIC_V5_NR_PRIVATE_IRQS);
bitmap_and(cpu_if->vgic_ppi_enabler, &reg, mask, VGIC_V5_NR_PRIVATE_IRQS);
/*
* Sync the change in enable states to the vgic_irqs. We consider all
* PPIs as we don't expose many to the guest.
*/
for_each_set_bit(i, mask, VGIC_V5_NR_PRIVATE_IRQS) {
for_each_visible_v5_ppi(i, vcpu->kvm) {
u32 intid = vgic_v5_make_ppi(i);
struct vgic_irq *irq;
@ -4212,6 +4209,7 @@ static struct sys_reg_desc sys_insn_descs[] = {
SYS_INSN(AT_S1E0W, handle_at_s1e01),
SYS_INSN(AT_S1E1RP, handle_at_s1e01),
SYS_INSN(AT_S1E1WP, handle_at_s1e01),
SYS_INSN(AT_S1E1A, handle_at_s1e01),
{ SYS_DESC(SYS_DC_CSW), access_dcsw },
{ SYS_DESC(SYS_DC_CGSW), access_dcgsw },

45
arch/arm64/kvm/vgic/vgic-init.c
View File

@ -271,18 +271,12 @@ int kvm_vgic_vcpu_nv_init(struct kvm_vcpu *vcpu)
return ret;
}
static void vgic_allocate_private_irq(struct kvm_vcpu *vcpu, int i, u32 type)
static void vgic_setup_private_irq(struct kvm_vcpu *vcpu, struct vgic_irq *irq,
u32 type)
{
struct vgic_irq *irq = &vcpu->arch.vgic_cpu.private_irqs[i];
irq->intid = irq - &vcpu->arch.vgic_cpu.private_irqs[0];
INIT_LIST_HEAD(&irq->ap_list);
raw_spin_lock_init(&irq->irq_lock);
irq->vcpu = NULL;
irq->target_vcpu = vcpu;
refcount_set(&irq->refcount, 0);
irq->intid = i;
if (vgic_irq_is_sgi(i)) {
if (vgic_irq_is_sgi(irq->intid)) {
/* SGIs */
irq->enabled = 1;
irq->config = VGIC_CONFIG_EDGE;
@ -303,18 +297,11 @@ static void vgic_allocate_private_irq(struct kvm_vcpu *vcpu, int i, u32 type)
}
}
static void vgic_v5_allocate_private_irq(struct kvm_vcpu *vcpu, int i, u32 type)
static void vgic_v5_setup_private_irq(struct kvm_vcpu *vcpu, struct vgic_irq *irq)
{
struct vgic_irq *irq = &vcpu->arch.vgic_cpu.private_irqs[i];
u32 intid = vgic_v5_make_ppi(i);
int i = irq - &vcpu->arch.vgic_cpu.private_irqs[0];
INIT_LIST_HEAD(&irq->ap_list);
raw_spin_lock_init(&irq->irq_lock);
irq->vcpu = NULL;
irq->target_vcpu = vcpu;
refcount_set(&irq->refcount, 0);
irq->intid = intid;
irq->intid = vgic_v5_make_ppi(i);
/* The only Edge architected PPI is the SW_PPI */
if (i == GICV5_ARCH_PPI_SW_PPI)
@ -323,7 +310,7 @@ static void vgic_v5_allocate_private_irq(struct kvm_vcpu *vcpu, int i, u32 type)
irq->config = VGIC_CONFIG_LEVEL;
/* Register the GICv5-specific PPI ops */
vgic_v5_set_ppi_ops(vcpu, intid);
vgic_v5_set_ppi_ops(vcpu, irq->intid);
}
static int vgic_allocate_private_irqs_locked(struct kvm_vcpu *vcpu, u32 type)
@ -349,15 +336,19 @@ static int vgic_allocate_private_irqs_locked(struct kvm_vcpu *vcpu, u32 type)
if (!vgic_cpu->private_irqs)
return -ENOMEM;
/*
* Enable and configure all SGIs to be edge-triggered and
* configure all PPIs as level-triggered.
*/
for (i = 0; i < num_private_irqs; i++) {
struct vgic_irq *irq = &vcpu->arch.vgic_cpu.private_irqs[i];
INIT_LIST_HEAD(&irq->ap_list);
raw_spin_lock_init(&irq->irq_lock);
irq->vcpu = NULL;
irq->target_vcpu = vcpu;
refcount_set(&irq->refcount, 0);
if (vgic_is_v5(vcpu->kvm))
vgic_v5_allocate_private_irq(vcpu, i, type);
vgic_v5_setup_private_irq(vcpu, irq);
else
vgic_allocate_private_irq(vcpu, i, type);
vgic_setup_private_irq(vcpu, irq, type);
}
return 0;

6
arch/arm64/kvm/vgic/vgic-irqfd.c
View File

@ -20,9 +20,15 @@ static int vgic_irqfd_set_irq(struct kvm_kernel_irq_routing_entry *e,
int level, bool line_status)
{
unsigned int spi_id = e->irqchip.pin + VGIC_NR_PRIVATE_IRQS;
int ret;
if (!vgic_valid_spi(kvm, spi_id))
return -EINVAL;
ret = vgic_lazy_init(kvm);
if (ret)
return ret;
return kvm_vgic_inject_irq(kvm, NULL, spi_id, level, NULL);
}

21
arch/arm64/kvm/vgic/vgic-its.c
View File

@ -27,7 +27,7 @@ static struct kvm_device_ops kvm_arm_vgic_its_ops;
static int vgic_its_save_tables_v0(struct vgic_its *its);
static int vgic_its_restore_tables_v0(struct vgic_its *its);
static int vgic_its_commit_v0(struct vgic_its *its);
static void vgic_its_commit_v0(struct vgic_its *its);
static int update_lpi_config(struct kvm *kvm, struct vgic_irq *irq,
struct kvm_vcpu *filter_vcpu, bool needs_inv);
@ -168,7 +168,7 @@ struct vgic_its_abi {
int ite_esz;
int (*save_tables)(struct vgic_its *its);
int (*restore_tables)(struct vgic_its *its);
int (*commit)(struct vgic_its *its);
void (*commit)(struct vgic_its *its);
};
#define ABI_0_ESZ 8
@ -192,13 +192,13 @@ inline const struct vgic_its_abi *vgic_its_get_abi(struct vgic_its *its)
return &its_table_abi_versions[its->abi_rev];
}
static int vgic_its_set_abi(struct vgic_its *its, u32 rev)
static void vgic_its_set_abi(struct vgic_its *its, u32 rev)
{
const struct vgic_its_abi *abi;
its->abi_rev = rev;
abi = vgic_its_get_abi(its);
return abi->commit(its);
abi->commit(its);
}
/*
@ -472,7 +472,8 @@ static int vgic_mmio_uaccess_write_its_iidr(struct kvm *kvm,
if (rev >= NR_ITS_ABIS)
return -EINVAL;
return vgic_its_set_abi(its, rev);
vgic_its_set_abi(its, rev);
return 0;
}
static unsigned long vgic_mmio_read_its_idregs(struct kvm *kvm,
@ -1890,14 +1891,11 @@ static int vgic_its_create(struct kvm_device *dev, u32 type)
its->baser_coll_table = INITIAL_BASER_VALUE |
((u64)GITS_BASER_TYPE_COLLECTION << GITS_BASER_TYPE_SHIFT);
dev->kvm->arch.vgic.propbaser = INITIAL_PROPBASER_VALUE;
dev->private = its;
ret = vgic_its_set_abi(its, NR_ITS_ABIS - 1);
vgic_its_set_abi(its, NR_ITS_ABIS - 1);
mutex_unlock(&dev->kvm->arch.config_lock);
return ret;
return 0;
}
static void vgic_its_destroy(struct kvm_device *kvm_dev)
@ -2612,7 +2610,7 @@ static int vgic_its_restore_tables_v0(struct vgic_its *its)
return ret;
}
static int vgic_its_commit_v0(struct vgic_its *its)
static void vgic_its_commit_v0(struct vgic_its *its)
{
const struct vgic_its_abi *abi;
@ -2625,7 +2623,6 @@ static int vgic_its_commit_v0(struct vgic_its *its)
its->baser_device_table |= (GIC_ENCODE_SZ(abi->dte_esz, 5)
<< GITS_BASER_ENTRY_SIZE_SHIFT);
return 0;
}
static void vgic_its_reset(struct kvm *kvm, struct vgic_its *its)

9
arch/arm64/kvm/vgic/vgic-kvm-device.c
View File

@ -730,18 +730,15 @@ static int vgic_v5_get_userspace_ppis(struct kvm_device *dev,
guard(mutex)(&dev->kvm->arch.config_lock);
/*
* We either support 64 or 128 PPIs. In the former case, we need to
* return 0s for the second 64 bits as we have no storage backing those.
* We only support 64 PPIs, so, we need to return 0s for the
* second 64 bits as we have no storage backing those.
*/
ret = put_user(bitmap_read(gicv5_vm->userspace_ppis, 0, 64), uaddr);
if (ret)
return ret;
uaddr++;
if (VGIC_V5_NR_PRIVATE_IRQS == 128)
ret = put_user(bitmap_read(gicv5_vm->userspace_ppis, 64, 128), uaddr);
else
ret = put_user(0, uaddr);
ret = put_user(0, uaddr);
return ret;
}

51
arch/arm64/kvm/vgic/vgic-v5.c
View File

@ -10,7 +10,7 @@
#include "vgic.h"
static struct vgic_v5_ppi_caps ppi_caps;
#define ppi_caps kvm_vgic_global_state.vgic_v5_ppi_caps
/*
* Not all PPIs are guaranteed to be implemented for GICv5. Deterermine which
@ -18,20 +18,17 @@ static struct vgic_v5_ppi_caps ppi_caps;
*/
static void vgic_v5_get_implemented_ppis(void)
{
if (!cpus_have_final_cap(ARM64_HAS_GICV5_CPUIF))
return;
/*
* If we have KVM, we have EL2, which means that we have support for the
* EL1 and EL2 Physical & Virtual timers.
*/
__assign_bit(GICV5_ARCH_PPI_CNTHP, ppi_caps.impl_ppi_mask, 1);
__assign_bit(GICV5_ARCH_PPI_CNTV, ppi_caps.impl_ppi_mask, 1);
__assign_bit(GICV5_ARCH_PPI_CNTHV, ppi_caps.impl_ppi_mask, 1);
__assign_bit(GICV5_ARCH_PPI_CNTP, ppi_caps.impl_ppi_mask, 1);
__set_bit(GICV5_ARCH_PPI_CNTHP, ppi_caps.impl_ppi_mask);
__set_bit(GICV5_ARCH_PPI_CNTV, ppi_caps.impl_ppi_mask);
__set_bit(GICV5_ARCH_PPI_CNTHV, ppi_caps.impl_ppi_mask);
__set_bit(GICV5_ARCH_PPI_CNTP, ppi_caps.impl_ppi_mask);
/* The SW_PPI should be available */
__assign_bit(GICV5_ARCH_PPI_SW_PPI, ppi_caps.impl_ppi_mask, 1);
__set_bit(GICV5_ARCH_PPI_SW_PPI, ppi_caps.impl_ppi_mask);
/* The PMUIRQ is available if we have the PMU */
__assign_bit(GICV5_ARCH_PPI_PMUIRQ, ppi_caps.impl_ppi_mask, system_supports_pmuv3());
@ -146,9 +143,7 @@ int vgic_v5_init(struct kvm *kvm)
/* We only allow userspace to drive the SW_PPI, if it is implemented. */
bitmap_zero(kvm->arch.vgic.gicv5_vm.userspace_ppis,
VGIC_V5_NR_PRIVATE_IRQS);
__assign_bit(GICV5_ARCH_PPI_SW_PPI,
kvm->arch.vgic.gicv5_vm.userspace_ppis,
VGIC_V5_NR_PRIVATE_IRQS);
__set_bit(GICV5_ARCH_PPI_SW_PPI, kvm->arch.vgic.gicv5_vm.userspace_ppis);
bitmap_and(kvm->arch.vgic.gicv5_vm.userspace_ppis,
kvm->arch.vgic.gicv5_vm.userspace_ppis,
ppi_caps.impl_ppi_mask, VGIC_V5_NR_PRIVATE_IRQS);
@ -197,7 +192,7 @@ int vgic_v5_finalize_ppi_state(struct kvm *kvm)
/* Expose PPIs with an owner or the SW_PPI, only */
scoped_guard(raw_spinlock_irqsave, &irq->irq_lock) {
if (irq->owner || i == GICV5_ARCH_PPI_SW_PPI) {
__assign_bit(i, kvm->arch.vgic.gicv5_vm.vgic_ppi_mask, 1);
__set_bit(i, kvm->arch.vgic.gicv5_vm.vgic_ppi_mask);
__assign_bit(i, kvm->arch.vgic.gicv5_vm.vgic_ppi_hmr,
irq->config == VGIC_CONFIG_LEVEL);
}
@ -243,9 +238,9 @@ static u32 vgic_v5_get_effective_priority_mask(struct kvm_vcpu *vcpu)
/*
* For GICv5, the PPIs are mostly directly managed by the hardware. We (the
* hypervisor) handle the pending, active, enable state save/restore, but don't
* need the PPIs to be queued on a per-VCPU AP list. Therefore, sanity check the
* state, unlock, and return.
* hypervisor) handle the pending, active, enable state save/restore, but
* don't need the PPIs to be queued on a per-VCPU AP list. Therefore,
* unlock, kick the vcpu and return.
*/
bool vgic_v5_ppi_queue_irq_unlock(struct kvm *kvm, struct vgic_irq *irq,
unsigned long flags)
@ -255,12 +250,7 @@ bool vgic_v5_ppi_queue_irq_unlock(struct kvm *kvm, struct vgic_irq *irq,
lockdep_assert_held(&irq->irq_lock);
if (WARN_ON_ONCE(!__irq_is_ppi(KVM_DEV_TYPE_ARM_VGIC_V5, irq->intid)))
goto out_unlock_fail;
vcpu = irq->target_vcpu;
if (WARN_ON_ONCE(!vcpu))
goto out_unlock_fail;
raw_spin_unlock_irqrestore(&irq->irq_lock, flags);
@ -269,11 +259,6 @@ bool vgic_v5_ppi_queue_irq_unlock(struct kvm *kvm, struct vgic_irq *irq,
kvm_vcpu_kick(vcpu);
return true;
out_unlock_fail:
raw_spin_unlock_irqrestore(&irq->irq_lock, flags);
return false;
}
/*
@ -287,10 +272,10 @@ void vgic_v5_set_ppi_dvi(struct kvm_vcpu *vcpu, struct vgic_irq *irq, bool dvi)
lockdep_assert_held(&irq->irq_lock);
ppi = vgic_v5_get_hwirq_id(irq->intid);
__assign_bit(ppi, cpu_if->vgic_ppi_dvir, dvi);
assign_bit(ppi, cpu_if->vgic_ppi_dvir, dvi);
}
static struct irq_ops vgic_v5_ppi_irq_ops = {
static const struct irq_ops vgic_v5_ppi_irq_ops = {
.queue_irq_unlock = vgic_v5_ppi_queue_irq_unlock,
.set_direct_injection = vgic_v5_set_ppi_dvi,
};
@ -316,7 +301,7 @@ static void vgic_v5_sync_ppi_priorities(struct kvm_vcpu *vcpu)
* those actually exposed to the guest by first iterating over the mask
* of exposed PPIs.
*/
for_each_set_bit(i, vcpu->kvm->arch.vgic.gicv5_vm.vgic_ppi_mask, VGIC_V5_NR_PRIVATE_IRQS) {
for_each_visible_v5_ppi(i, vcpu->kvm) {
u32 intid = vgic_v5_make_ppi(i);
struct vgic_irq *irq;
int pri_idx, pri_reg, pri_bit;
@ -358,7 +343,7 @@ bool vgic_v5_has_pending_ppi(struct kvm_vcpu *vcpu)
if (!priority_mask)
return false;
for_each_set_bit(i, vcpu->kvm->arch.vgic.gicv5_vm.vgic_ppi_mask, VGIC_V5_NR_PRIVATE_IRQS) {
for_each_visible_v5_ppi(i, vcpu->kvm) {
u32 intid = vgic_v5_make_ppi(i);
bool has_pending = false;
struct vgic_irq *irq;
@ -391,8 +376,7 @@ void vgic_v5_fold_ppi_state(struct kvm_vcpu *vcpu)
activer = host_data_ptr(vgic_v5_ppi_state)->activer_exit;
pendr = host_data_ptr(vgic_v5_ppi_state)->pendr;
for_each_set_bit(i, vcpu->kvm->arch.vgic.gicv5_vm.vgic_ppi_mask,
VGIC_V5_NR_PRIVATE_IRQS) {
for_each_visible_v5_ppi(i, vcpu->kvm) {
u32 intid = vgic_v5_make_ppi(i);
struct vgic_irq *irq;
@ -429,8 +413,7 @@ void vgic_v5_flush_ppi_state(struct kvm_vcpu *vcpu)
* ICC_PPI_PENDRx_EL1, however.
*/
bitmap_zero(pendr, VGIC_V5_NR_PRIVATE_IRQS);
for_each_set_bit(i, vcpu->kvm->arch.vgic.gicv5_vm.vgic_ppi_mask,
VGIC_V5_NR_PRIVATE_IRQS) {
for_each_visible_v5_ppi(i, vcpu->kvm) {
u32 intid = vgic_v5_make_ppi(i);
struct vgic_irq *irq;

33
arch/arm64/kvm/vgic/vgic.c
View File

@ -106,24 +106,23 @@ struct vgic_irq *vgic_get_irq(struct kvm *kvm, u32 intid)
struct vgic_irq *vgic_get_vcpu_irq(struct kvm_vcpu *vcpu, u32 intid)
{
enum kvm_device_type type;
if (WARN_ON(!vcpu))
return NULL;
if (vgic_is_v5(vcpu->kvm)) {
u32 int_num, hwirq_id;
type = vcpu->kvm->arch.vgic.vgic_model;
if (!__irq_is_ppi(KVM_DEV_TYPE_ARM_VGIC_V5, intid))
return NULL;
if (__irq_is_sgi(type, intid) || __irq_is_ppi(type, intid)) {
switch (type) {
case KVM_DEV_TYPE_ARM_VGIC_V5:
intid = vgic_v5_get_hwirq_id(intid);
intid = array_index_nospec(intid, VGIC_V5_NR_PRIVATE_IRQS);
break;
default:
intid = array_index_nospec(intid, VGIC_NR_PRIVATE_IRQS);
}
hwirq_id = FIELD_GET(GICV5_HWIRQ_ID, intid);
int_num = array_index_nospec(hwirq_id, VGIC_V5_NR_PRIVATE_IRQS);
return &vcpu->arch.vgic_cpu.private_irqs[int_num];
}
/* SGIs and PPIs */
if (intid < VGIC_NR_PRIVATE_IRQS) {
intid = array_index_nospec(intid, VGIC_NR_PRIVATE_IRQS);
return &vcpu->arch.vgic_cpu.private_irqs[intid];
}
@ -534,11 +533,9 @@ int kvm_vgic_inject_irq(struct kvm *kvm, struct kvm_vcpu *vcpu,
{
struct vgic_irq *irq;
unsigned long flags;
int ret;
ret = vgic_lazy_init(kvm);
if (ret)
return ret;
if (unlikely(!vgic_initialized(kvm)))
return 0;
if (!vcpu && irq_is_private(kvm, intid))
return -EINVAL;
@ -573,7 +570,7 @@ int kvm_vgic_inject_irq(struct kvm *kvm, struct kvm_vcpu *vcpu,
}
void kvm_vgic_set_irq_ops(struct kvm_vcpu *vcpu, u32 vintid,
struct irq_ops *ops)
const struct irq_ops *ops)
{
struct vgic_irq *irq = vgic_get_vcpu_irq(vcpu, vintid);

3
arch/arm64/kvm/vgic/vgic.h
View File

@ -378,6 +378,9 @@ void vgic_v5_get_vmcr(struct kvm_vcpu *vcpu, struct vgic_vmcr *vmcr);
void vgic_v5_restore_state(struct kvm_vcpu *vcpu);
void vgic_v5_save_state(struct kvm_vcpu *vcpu);
#define for_each_visible_v5_ppi(__i, __k) \
for_each_set_bit(__i, (__k)->arch.vgic.gicv5_vm.vgic_ppi_mask, VGIC_V5_NR_PRIVATE_IRQS)
static inline int vgic_v3_max_apr_idx(struct kvm_vcpu *vcpu)
{
struct vgic_cpu *cpu_if = &vcpu->arch.vgic_cpu;

14
arch/loongarch/include/asm/kvm_host.h
View File

@ -38,7 +38,8 @@
#define KVM_REQ_TLB_FLUSH_GPA KVM_ARCH_REQ(0)
#define KVM_REQ_STEAL_UPDATE KVM_ARCH_REQ(1)
#define KVM_REQ_PMU KVM_ARCH_REQ(2)
#define KVM_REQ_AUX_LOAD KVM_ARCH_REQ(3)
#define KVM_REQ_FPU_LOAD KVM_ARCH_REQ(3)
#define KVM_REQ_LBT_LOAD KVM_ARCH_REQ(4)
#define KVM_GUESTDBG_SW_BP_MASK \
(KVM_GUESTDBG_ENABLE | KVM_GUESTDBG_USE_SW_BP)
@ -157,12 +158,10 @@ enum emulation_result {
};
#define KVM_LARCH_FPU (0x1 << 0)
#define KVM_LARCH_LSX (0x1 << 1)
#define KVM_LARCH_LASX (0x1 << 2)
#define KVM_LARCH_LBT (0x1 << 3)
#define KVM_LARCH_PMU (0x1 << 4)
#define KVM_LARCH_SWCSR_LATEST (0x1 << 5)
#define KVM_LARCH_HWCSR_USABLE (0x1 << 6)
#define KVM_LARCH_LBT (0x1 << 1)
#define KVM_LARCH_PMU (0x1 << 2)
#define KVM_LARCH_SWCSR_LATEST (0x1 << 3)
#define KVM_LARCH_HWCSR_USABLE (0x1 << 4)
#define LOONGARCH_PV_FEAT_UPDATED BIT_ULL(63)
#define LOONGARCH_PV_FEAT_MASK (BIT(KVM_FEATURE_IPI) | \
@ -203,7 +202,6 @@ struct kvm_vcpu_arch {
/* Which auxiliary state is loaded (KVM_LARCH_*) */
unsigned int aux_inuse;
unsigned int aux_ldtype;
/* FPU state */
struct loongarch_fpu fpu FPU_ALIGN;

43
arch/loongarch/include/asm/kvm_vcpu.h
View File

@ -10,22 +10,37 @@
#include <asm/loongarch.h>
/* Controlled by 0x5 guest estat */
#define CPU_SIP0 (_ULCAST_(1))
#define CPU_SIP1 (_ULCAST_(1) << 1)
#define CPU_PMU (_ULCAST_(1) << 10)
#define CPU_TIMER (_ULCAST_(1) << 11)
#define CPU_IPI (_ULCAST_(1) << 12)
#define CPU_AVEC (_ULCAST_(1) << 14)
#define CPU_SIP0 BIT(INT_SWI0)
#define CPU_SIP1 BIT(INT_SWI1)
#define CPU_HWI0 BIT(INT_HWI0)
#define CPU_HWI1 BIT(INT_HWI1)
#define CPU_HWI2 BIT(INT_HWI2)
#define CPU_HWI3 BIT(INT_HWI3)
#define CPU_HWI4 BIT(INT_HWI4)
#define CPU_HWI5 BIT(INT_HWI5)
#define CPU_HWI6 BIT(INT_HWI6)
#define CPU_HWI7 BIT(INT_HWI7)
#define CPU_PMU BIT(INT_PCOV)
#define CPU_TIMER BIT(INT_TI)
#define CPU_IPI BIT(INT_IPI)
#define CPU_AVEC BIT(INT_AVEC)
#define KVM_ESTAT_INTI_MASK (CPU_SIP0 | CPU_SIP1 | CPU_PMU | CPU_TIMER \
| CPU_IPI | CPU_AVEC)
#define KVM_ESTAT_EXTI_MASK (CPU_HWI0 | CPU_HWI1 | CPU_HWI2 | CPU_HWI3 \
| CPU_HWI4 | CPU_HWI5 | CPU_HWI6 | CPU_HWI7)
/* Controlled by 0x52 guest exception VIP aligned to estat bit 5~12 */
#define CPU_IP0 (_ULCAST_(1))
#define CPU_IP1 (_ULCAST_(1) << 1)
#define CPU_IP2 (_ULCAST_(1) << 2)
#define CPU_IP3 (_ULCAST_(1) << 3)
#define CPU_IP4 (_ULCAST_(1) << 4)
#define CPU_IP5 (_ULCAST_(1) << 5)
#define CPU_IP6 (_ULCAST_(1) << 6)
#define CPU_IP7 (_ULCAST_(1) << 7)
#define VIP_DELTA (INT_HWI0 - CSR_GINTC_VIP_SHIFT)
#define CPU_IP0 BIT(INT_HWI0 - VIP_DELTA)
#define CPU_IP1 BIT(INT_HWI1 - VIP_DELTA)
#define CPU_IP2 BIT(INT_HWI2 - VIP_DELTA)
#define CPU_IP3 BIT(INT_HWI3 - VIP_DELTA)
#define CPU_IP4 BIT(INT_HWI4 - VIP_DELTA)
#define CPU_IP5 BIT(INT_HWI5 - VIP_DELTA)
#define CPU_IP6 BIT(INT_HWI6 - VIP_DELTA)
#define CPU_IP7 BIT(INT_HWI7 - VIP_DELTA)
#define KVM_GINTC_IRQ_MASK (CPU_IP0 | CPU_IP1 | CPU_IP2 | CPU_IP3 \
| CPU_IP4 | CPU_IP5 | CPU_IP6 | CPU_IP7)
#define MNSEC_PER_SEC (NSEC_PER_SEC >> 20)

22
arch/loongarch/kvm/exit.c
View File

@ -103,7 +103,6 @@ static unsigned long kvm_emu_xchg_csr(struct kvm_vcpu *vcpu, int csrid,
old = kvm_read_sw_gcsr(csr, csrid);
val = (old & ~csr_mask) | (val & csr_mask);
kvm_write_sw_gcsr(csr, csrid, val);
old = old & csr_mask;
} else
pr_warn_once("Unsupported csrxchg 0x%x with pc %lx\n", csrid, vcpu->arch.pc);
@ -755,8 +754,7 @@ static int kvm_handle_fpu_disabled(struct kvm_vcpu *vcpu, int ecode)
return RESUME_HOST;
}
vcpu->arch.aux_ldtype = KVM_LARCH_FPU;
kvm_make_request(KVM_REQ_AUX_LOAD, vcpu);
kvm_make_request(KVM_REQ_FPU_LOAD, vcpu);
return RESUME_GUEST;
}
@ -796,10 +794,8 @@ static int kvm_handle_lsx_disabled(struct kvm_vcpu *vcpu, int ecode)
{
if (!kvm_guest_has_lsx(&vcpu->arch))
kvm_queue_exception(vcpu, EXCCODE_INE, 0);
else {
vcpu->arch.aux_ldtype = KVM_LARCH_LSX;
kvm_make_request(KVM_REQ_AUX_LOAD, vcpu);
}
else
kvm_make_request(KVM_REQ_FPU_LOAD, vcpu);
return RESUME_GUEST;
}
@ -816,10 +812,8 @@ static int kvm_handle_lasx_disabled(struct kvm_vcpu *vcpu, int ecode)
{
if (!kvm_guest_has_lasx(&vcpu->arch))
kvm_queue_exception(vcpu, EXCCODE_INE, 0);
else {
vcpu->arch.aux_ldtype = KVM_LARCH_LASX;
kvm_make_request(KVM_REQ_AUX_LOAD, vcpu);
}
else
kvm_make_request(KVM_REQ_FPU_LOAD, vcpu);
return RESUME_GUEST;
}
@ -828,10 +822,8 @@ static int kvm_handle_lbt_disabled(struct kvm_vcpu *vcpu, int ecode)
{
if (!kvm_guest_has_lbt(&vcpu->arch))
kvm_queue_exception(vcpu, EXCCODE_INE, 0);
else {
vcpu->arch.aux_ldtype = KVM_LARCH_LBT;
kvm_make_request(KVM_REQ_AUX_LOAD, vcpu);
}
else
kvm_make_request(KVM_REQ_LBT_LOAD, vcpu);
return RESUME_GUEST;
}

6
arch/loongarch/kvm/intc/eiointc.c
View File

@ -645,10 +645,14 @@ static int kvm_eiointc_create(struct kvm_device *dev, u32 type)
device = &s->device_vext;
kvm_iodevice_init(device, &kvm_eiointc_virt_ops);
mutex_lock(&kvm->slots_lock);
ret = kvm_io_bus_register_dev(kvm, KVM_IOCSR_BUS,
EIOINTC_VIRT_BASE, EIOINTC_VIRT_SIZE, device);
mutex_unlock(&kvm->slots_lock);
if (ret < 0) {
mutex_lock(&kvm->slots_lock);
kvm_io_bus_unregister_dev(kvm, KVM_IOCSR_BUS, &s->device);
mutex_unlock(&kvm->slots_lock);
kfree(s);
return ret;
}
@ -667,8 +671,10 @@ static void kvm_eiointc_destroy(struct kvm_device *dev)
kvm = dev->kvm;
eiointc = kvm->arch.eiointc;
mutex_lock(&kvm->slots_lock);
kvm_io_bus_unregister_dev(kvm, KVM_IOCSR_BUS, &eiointc->device);
kvm_io_bus_unregister_dev(kvm, KVM_IOCSR_BUS, &eiointc->device_vext);
mutex_unlock(&kvm->slots_lock);
kfree(eiointc);
kfree(dev);
}

2
arch/loongarch/kvm/intc/ipi.c
View File

@ -447,7 +447,9 @@ static void kvm_ipi_destroy(struct kvm_device *dev)
kvm = dev->kvm;
ipi = kvm->arch.ipi;
mutex_lock(&kvm->slots_lock);
kvm_io_bus_unregister_dev(kvm, KVM_IOCSR_BUS, &ipi->device);
mutex_unlock(&kvm->slots_lock);
kfree(ipi);
kfree(dev);
}

2
arch/loongarch/kvm/intc/pch_pic.c
View File

@ -481,7 +481,9 @@ static void kvm_pch_pic_destroy(struct kvm_device *dev)
kvm = dev->kvm;
s = kvm->arch.pch_pic;
/* unregister pch pic device and free it's memory */
mutex_lock(&kvm->slots_lock);
kvm_io_bus_unregister_dev(kvm, KVM_MMIO_BUS, &s->device);
mutex_unlock(&kvm->slots_lock);
kfree(s);
kfree(dev);
}

98
arch/loongarch/kvm/interrupt.c
View File

@ -9,41 +9,16 @@
#include <asm/kvm_vcpu.h>
#include <asm/kvm_dmsintc.h>
static unsigned int priority_to_irq[EXCCODE_INT_NUM] = {
[INT_TI] = CPU_TIMER,
[INT_IPI] = CPU_IPI,
[INT_SWI0] = CPU_SIP0,
[INT_SWI1] = CPU_SIP1,
[INT_HWI0] = CPU_IP0,
[INT_HWI1] = CPU_IP1,
[INT_HWI2] = CPU_IP2,
[INT_HWI3] = CPU_IP3,
[INT_HWI4] = CPU_IP4,
[INT_HWI5] = CPU_IP5,
[INT_HWI6] = CPU_IP6,
[INT_HWI7] = CPU_IP7,
[INT_AVEC] = CPU_AVEC,
};
static int kvm_irq_deliver(struct kvm_vcpu *vcpu, unsigned int priority)
static void kvm_irq_deliver(struct kvm_vcpu *vcpu, unsigned long mask)
{
unsigned int irq = 0;
unsigned long irq;
unsigned long old, new;
clear_bit(priority, &vcpu->arch.irq_pending);
if (priority < EXCCODE_INT_NUM)
irq = priority_to_irq[priority];
switch (priority) {
case INT_AVEC:
if (!kvm_guest_has_msgint(&vcpu->arch))
break;
if (mask & CPU_AVEC)
dmsintc_inject_irq(vcpu);
fallthrough;
case INT_TI:
case INT_IPI:
case INT_SWI0:
case INT_SWI1:
irq = mask & KVM_ESTAT_INTI_MASK;
if (irq) {
old = kvm_read_hw_gcsr(LOONGARCH_CSR_TVAL);
set_gcsr_estat(irq);
new = kvm_read_hw_gcsr(LOONGARCH_CSR_TVAL);
@ -51,37 +26,20 @@ static int kvm_irq_deliver(struct kvm_vcpu *vcpu, unsigned int priority)
/* Inject TI if TVAL inverted */
if (new > old)
set_gcsr_estat(CPU_TIMER);
break;
case INT_HWI0 ... INT_HWI7:
set_csr_gintc(irq);
break;
default:
break;
}
return 1;
irq = (mask >> VIP_DELTA) & KVM_GINTC_IRQ_MASK;
if (irq)
set_csr_gintc(irq);
}
static int kvm_irq_clear(struct kvm_vcpu *vcpu, unsigned int priority)
static void kvm_irq_clear(struct kvm_vcpu *vcpu, unsigned long mask)
{
unsigned int irq = 0;
unsigned long irq;
unsigned long old, new;
clear_bit(priority, &vcpu->arch.irq_clear);
if (priority < EXCCODE_INT_NUM)
irq = priority_to_irq[priority];
switch (priority) {
case INT_AVEC:
if (!kvm_guest_has_msgint(&vcpu->arch))
break;
fallthrough;
case INT_TI:
case INT_IPI:
case INT_SWI0:
case INT_SWI1:
irq = mask & KVM_ESTAT_INTI_MASK;
if (irq) {
old = kvm_read_hw_gcsr(LOONGARCH_CSR_TVAL);
clear_gcsr_estat(irq);
new = kvm_read_hw_gcsr(LOONGARCH_CSR_TVAL);
@ -89,30 +47,28 @@ static int kvm_irq_clear(struct kvm_vcpu *vcpu, unsigned int priority)
/* Inject TI if TVAL inverted */
if (new > old)
set_gcsr_estat(CPU_TIMER);
break;
case INT_HWI0 ... INT_HWI7:
clear_csr_gintc(irq);
break;
default:
break;
}
return 1;
irq = (mask >> VIP_DELTA) & KVM_GINTC_IRQ_MASK;
if (irq)
clear_csr_gintc(irq);
}
void kvm_deliver_intr(struct kvm_vcpu *vcpu)
{
unsigned int priority;
unsigned long *pending = &vcpu->arch.irq_pending;
unsigned long *pending_clr = &vcpu->arch.irq_clear;
unsigned long mask;
for_each_set_bit(priority, pending_clr, EXCCODE_INT_NUM)
kvm_irq_clear(vcpu, priority);
mask = READ_ONCE(vcpu->arch.irq_clear);
if (mask) {
mask = xchg_relaxed(&vcpu->arch.irq_clear, 0);
kvm_irq_clear(vcpu, mask);
}
for_each_set_bit(priority, pending, EXCCODE_INT_NUM)
kvm_irq_deliver(vcpu, priority);
mask = READ_ONCE(vcpu->arch.irq_pending);
if (mask) {
mask = xchg_relaxed(&vcpu->arch.irq_pending, 0);
kvm_irq_deliver(vcpu, mask);
}
}
int kvm_pending_timer(struct kvm_vcpu *vcpu)

3
arch/loongarch/kvm/irqfd.c
View File

@ -51,7 +51,8 @@ int kvm_set_routing_entry(struct kvm *kvm,
e->irqchip.irqchip = ue->u.irqchip.irqchip;
e->irqchip.pin = ue->u.irqchip.pin;
if (e->irqchip.pin >= KVM_IRQCHIP_NUM_PINS)
if (e->irqchip.pin >= KVM_IRQCHIP_NUM_PINS ||
e->irqchip.irqchip >= KVM_NR_IRQCHIPS)
return -EINVAL;
return 0;

3
arch/loongarch/kvm/timer.c
View File

@ -30,8 +30,7 @@ enum hrtimer_restart kvm_swtimer_wakeup(struct hrtimer *timer)
struct kvm_vcpu *vcpu;
vcpu = container_of(timer, struct kvm_vcpu, arch.swtimer);
kvm_queue_irq(vcpu, INT_TI);
rcuwait_wake_up(&vcpu->wait);
kvm_vcpu_wake_up(vcpu);
return HRTIMER_NORESTART;
}

118
arch/loongarch/kvm/vcpu.c
View File

@ -239,26 +239,17 @@ static void kvm_late_check_requests(struct kvm_vcpu *vcpu)
vcpu->arch.flush_gpa = INVALID_GPA;
}
if (kvm_check_request(KVM_REQ_AUX_LOAD, vcpu)) {
switch (vcpu->arch.aux_ldtype) {
case KVM_LARCH_FPU:
kvm_own_fpu(vcpu);
break;
case KVM_LARCH_LSX:
kvm_own_lsx(vcpu);
break;
case KVM_LARCH_LASX:
if (kvm_check_request(KVM_REQ_FPU_LOAD, vcpu)) {
if (kvm_guest_has_lasx(&vcpu->arch))
kvm_own_lasx(vcpu);
break;
case KVM_LARCH_LBT:
kvm_own_lbt(vcpu);
break;
default:
break;
}
vcpu->arch.aux_ldtype = 0;
else if (kvm_guest_has_lsx(&vcpu->arch))
kvm_own_lsx(vcpu);
else if (kvm_guest_has_fpu(&vcpu->arch))
kvm_own_fpu(vcpu);
}
if (kvm_check_request(KVM_REQ_LBT_LOAD, vcpu))
kvm_own_lbt(vcpu);
}
/*
@ -308,10 +299,10 @@ static int kvm_pre_enter_guest(struct kvm_vcpu *vcpu)
* check vmid before vcpu enter guest
*/
local_irq_disable();
kvm_deliver_intr(vcpu);
kvm_deliver_exception(vcpu);
/* Make sure the vcpu mode has been written */
smp_store_mb(vcpu->mode, IN_GUEST_MODE);
kvm_deliver_intr(vcpu);
kvm_check_vpid(vcpu);
/*
@ -348,7 +339,7 @@ static int kvm_handle_exit(struct kvm_run *run, struct kvm_vcpu *vcpu)
u32 intr = estat & CSR_ESTAT_IS;
u32 ecode = (estat & CSR_ESTAT_EXC) >> CSR_ESTAT_EXC_SHIFT;
vcpu->mode = OUTSIDE_GUEST_MODE;
smp_store_mb(vcpu->mode, OUTSIDE_GUEST_MODE);
/* Set a default exit reason */
run->exit_reason = KVM_EXIT_UNKNOWN;
@ -602,7 +593,7 @@ struct kvm_vcpu *kvm_get_vcpu_by_cpuid(struct kvm *kvm, int cpuid)
static int _kvm_getcsr(struct kvm_vcpu *vcpu, unsigned int id, u64 *val)
{
unsigned long gintc;
unsigned long estat, gintc;
struct loongarch_csrs *csr = vcpu->arch.csr;
if (get_gcsr_flag(id) & INVALID_GCSR)
@ -621,8 +612,9 @@ static int _kvm_getcsr(struct kvm_vcpu *vcpu, unsigned int id, u64 *val)
preempt_enable();
/* ESTAT IP0~IP7 get from GINTC */
gintc = kvm_read_sw_gcsr(csr, LOONGARCH_CSR_GINTC) & 0xff;
*val = kvm_read_sw_gcsr(csr, LOONGARCH_CSR_ESTAT) | (gintc << 2);
gintc = kvm_read_sw_gcsr(csr, LOONGARCH_CSR_GINTC) & KVM_GINTC_IRQ_MASK;
estat = kvm_read_sw_gcsr(csr, LOONGARCH_CSR_ESTAT) & ~KVM_ESTAT_EXTI_MASK;
*val = estat | (gintc << VIP_DELTA);
return 0;
}
@ -637,7 +629,8 @@ static int _kvm_getcsr(struct kvm_vcpu *vcpu, unsigned int id, u64 *val)
static int _kvm_setcsr(struct kvm_vcpu *vcpu, unsigned int id, u64 val)
{
int ret = 0, gintc;
int ret = 0;
unsigned long estat, gintc;
struct loongarch_csrs *csr = vcpu->arch.csr;
if (get_gcsr_flag(id) & INVALID_GCSR)
@ -648,11 +641,16 @@ static int _kvm_setcsr(struct kvm_vcpu *vcpu, unsigned int id, u64 val)
if (id == LOONGARCH_CSR_ESTAT) {
/* ESTAT IP0~IP7 inject through GINTC */
gintc = (val >> 2) & 0xff;
kvm_set_sw_gcsr(csr, LOONGARCH_CSR_GINTC, gintc);
gintc = (val >> VIP_DELTA) & KVM_GINTC_IRQ_MASK;
gintc |= kvm_read_sw_gcsr(csr, LOONGARCH_CSR_GINTC) & ~KVM_GINTC_IRQ_MASK;
kvm_write_sw_gcsr(csr, LOONGARCH_CSR_GINTC, gintc);
gintc = val & ~(0xffUL << 2);
kvm_set_sw_gcsr(csr, LOONGARCH_CSR_ESTAT, gintc);
/* Only set valid ESTAT bits */
estat = val & ~KVM_ESTAT_EXTI_MASK;
estat &= CSR_ESTAT_IS | CSR_ESTAT_EXC | CSR_ESTAT_ESUBCODE;
if (!kvm_guest_has_msgint(&vcpu->arch))
estat &= ~CPU_AVEC;
kvm_write_sw_gcsr(csr, LOONGARCH_CSR_ESTAT, estat);
return ret;
}
@ -1108,7 +1106,8 @@ static int kvm_loongarch_cpucfg_get_attr(struct kvm_vcpu *vcpu,
return -ENXIO;
}
put_user(val, uaddr);
if (put_user(val, uaddr))
return -EFAULT;
return ret;
}
@ -1312,7 +1311,7 @@ int kvm_arch_vcpu_ioctl_get_fpu(struct kvm_vcpu *vcpu, struct kvm_fpu *fpu)
fpu->fcc = vcpu->arch.fpu.fcc;
fpu->fcsr = vcpu->arch.fpu.fcsr;
for (i = 0; i < NUM_FPU_REGS; i++)
memcpy(&fpu->fpr[i], &vcpu->arch.fpu.fpr[i], FPU_REG_WIDTH / 64);
memcpy(&fpu->fpr[i], &vcpu->arch.fpu.fpr[i], sizeof(union fpureg));
return 0;
}
@ -1324,7 +1323,7 @@ int kvm_arch_vcpu_ioctl_set_fpu(struct kvm_vcpu *vcpu, struct kvm_fpu *fpu)
vcpu->arch.fpu.fcc = fpu->fcc;
vcpu->arch.fpu.fcsr = fpu->fcsr;
for (i = 0; i < NUM_FPU_REGS; i++)
memcpy(&vcpu->arch.fpu.fpr[i], &fpu->fpr[i], FPU_REG_WIDTH / 64);
memcpy(&vcpu->arch.fpu.fpr[i], &fpu->fpr[i], sizeof(union fpureg));
return 0;
}
@ -1398,24 +1397,10 @@ int kvm_own_lsx(struct kvm_vcpu *vcpu)
/* Enable LSX for guest */
kvm_check_fcsr(vcpu, vcpu->arch.fpu.fcsr);
set_csr_euen(CSR_EUEN_LSXEN | CSR_EUEN_FPEN);
switch (vcpu->arch.aux_inuse & KVM_LARCH_FPU) {
case KVM_LARCH_FPU:
/*
* Guest FPU state already loaded,
* only restore upper LSX state
*/
_restore_lsx_upper(&vcpu->arch.fpu);
break;
default:
/* Neither FP or LSX already active,
* restore full LSX state
*/
kvm_restore_lsx(&vcpu->arch.fpu);
break;
}
kvm_restore_lsx(&vcpu->arch.fpu);
vcpu->arch.aux_inuse |= KVM_LARCH_FPU;
trace_kvm_aux(vcpu, KVM_TRACE_AUX_RESTORE, KVM_TRACE_AUX_LSX);
vcpu->arch.aux_inuse |= KVM_LARCH_LSX | KVM_LARCH_FPU;
return 0;
}
@ -1427,25 +1412,10 @@ int kvm_own_lasx(struct kvm_vcpu *vcpu)
{
kvm_check_fcsr(vcpu, vcpu->arch.fpu.fcsr);
set_csr_euen(CSR_EUEN_FPEN | CSR_EUEN_LSXEN | CSR_EUEN_LASXEN);
switch (vcpu->arch.aux_inuse & (KVM_LARCH_FPU | KVM_LARCH_LSX)) {
case KVM_LARCH_LSX:
case KVM_LARCH_LSX | KVM_LARCH_FPU:
/* Guest LSX state already loaded, only restore upper LASX state */
_restore_lasx_upper(&vcpu->arch.fpu);
break;
case KVM_LARCH_FPU:
/* Guest FP state already loaded, only restore upper LSX & LASX state */
_restore_lsx_upper(&vcpu->arch.fpu);
_restore_lasx_upper(&vcpu->arch.fpu);
break;
default:
/* Neither FP or LSX already active, restore full LASX state */
kvm_restore_lasx(&vcpu->arch.fpu);
break;
}
kvm_restore_lasx(&vcpu->arch.fpu);
vcpu->arch.aux_inuse |= KVM_LARCH_FPU;
trace_kvm_aux(vcpu, KVM_TRACE_AUX_RESTORE, KVM_TRACE_AUX_LASX);
vcpu->arch.aux_inuse |= KVM_LARCH_LASX | KVM_LARCH_LSX | KVM_LARCH_FPU;
return 0;
}
@ -1456,29 +1426,32 @@ void kvm_lose_fpu(struct kvm_vcpu *vcpu)
{
preempt_disable();
if (!(vcpu->arch.aux_inuse & KVM_LARCH_FPU))
goto done;
kvm_check_fcsr_alive(vcpu);
if (vcpu->arch.aux_inuse & KVM_LARCH_LASX) {
if (kvm_guest_has_lasx(&vcpu->arch)) {
kvm_save_lasx(&vcpu->arch.fpu);
vcpu->arch.aux_inuse &= ~(KVM_LARCH_LSX | KVM_LARCH_FPU | KVM_LARCH_LASX);
trace_kvm_aux(vcpu, KVM_TRACE_AUX_SAVE, KVM_TRACE_AUX_LASX);
/* Disable LASX & LSX & FPU */
clear_csr_euen(CSR_EUEN_FPEN | CSR_EUEN_LSXEN | CSR_EUEN_LASXEN);
} else if (vcpu->arch.aux_inuse & KVM_LARCH_LSX) {
} else if (kvm_guest_has_lsx(&vcpu->arch)) {
kvm_save_lsx(&vcpu->arch.fpu);
vcpu->arch.aux_inuse &= ~(KVM_LARCH_LSX | KVM_LARCH_FPU);
trace_kvm_aux(vcpu, KVM_TRACE_AUX_SAVE, KVM_TRACE_AUX_LSX);
/* Disable LSX & FPU */
clear_csr_euen(CSR_EUEN_FPEN | CSR_EUEN_LSXEN);
} else if (vcpu->arch.aux_inuse & KVM_LARCH_FPU) {
kvm_save_fpu(&vcpu->arch.fpu);
vcpu->arch.aux_inuse &= ~KVM_LARCH_FPU;
trace_kvm_aux(vcpu, KVM_TRACE_AUX_SAVE, KVM_TRACE_AUX_FPU);
/* Disable FPU */
clear_csr_euen(CSR_EUEN_FPEN);
}
vcpu->arch.aux_inuse &= ~KVM_LARCH_FPU;
done:
kvm_lose_lbt(vcpu);
preempt_enable();
@ -1487,6 +1460,13 @@ void kvm_lose_fpu(struct kvm_vcpu *vcpu)
int kvm_vcpu_ioctl_interrupt(struct kvm_vcpu *vcpu, struct kvm_interrupt *irq)
{
int intr = (int)irq->irq;
unsigned int vector = abs(intr);
if (vector >= EXCCODE_INT_NUM)
return -EINVAL;
if (!kvm_guest_has_msgint(&vcpu->arch) && (vector == INT_AVEC))
return -EINVAL;
if (intr > 0)
kvm_queue_irq(vcpu, intr);

2
arch/riscv/include/asm/kvm_aia.h
View File

@ -79,7 +79,7 @@ struct kvm_vcpu_aia {
#define irqchip_in_kernel(k) ((k)->arch.aia.in_kernel)
extern unsigned int kvm_riscv_aia_nr_hgei;
extern atomic_t kvm_riscv_aia_nr_hgei;
extern unsigned int kvm_riscv_aia_max_ids;
DECLARE_STATIC_KEY_FALSE(kvm_riscv_aia_available);
#define kvm_riscv_aia_available() \

10
arch/riscv/include/asm/kvm_gstage.h
View File

@ -54,6 +54,10 @@ int kvm_riscv_gstage_set_pte(struct kvm_gstage *gstage,
struct kvm_mmu_memory_cache *pcache,
const struct kvm_gstage_mapping *map);
bool kvm_riscv_gstage_try_update_pte(struct kvm_gstage *gstage, u32 level,
gpa_t addr, pte_t *ptep,
pte_t old_pte, pte_t new_pte);
int kvm_riscv_gstage_map_page(struct kvm_gstage *gstage,
struct kvm_mmu_memory_cache *pcache,
gpa_t gpa, phys_addr_t hpa, unsigned long page_size,
@ -70,13 +74,13 @@ enum kvm_riscv_gstage_op {
GSTAGE_OP_WP, /* Write-protect */
};
void kvm_riscv_gstage_op_pte(struct kvm_gstage *gstage, gpa_t addr,
bool kvm_riscv_gstage_op_pte(struct kvm_gstage *gstage, gpa_t addr,
pte_t *ptep, u32 ptep_level, enum kvm_riscv_gstage_op op);
void kvm_riscv_gstage_unmap_range(struct kvm_gstage *gstage,
bool kvm_riscv_gstage_unmap_range(struct kvm_gstage *gstage,
gpa_t start, gpa_t size, bool may_block);
void kvm_riscv_gstage_wp_range(struct kvm_gstage *gstage, gpa_t start, gpa_t end);
bool kvm_riscv_gstage_wp_range(struct kvm_gstage *gstage, gpa_t start, gpa_t end);
void kvm_riscv_gstage_mode_detect(void);

2
arch/riscv/include/asm/kvm_host.h
View File

@ -48,6 +48,8 @@
#define __KVM_HAVE_ARCH_FLUSH_REMOTE_TLBS_RANGE
#define KVM_HAVE_MMU_RWLOCK
#define KVM_DIRTY_LOG_MANUAL_CAPS (KVM_DIRTY_LOG_MANUAL_PROTECT_ENABLE | \
KVM_DIRTY_LOG_INITIALLY_SET)

14
arch/riscv/include/asm/kvm_nacl.h
View File

@ -60,9 +60,11 @@ int kvm_riscv_nacl_init(void);
#ifdef CONFIG_32BIT
#define lelong_to_cpu(__x) le32_to_cpu(__x)
#define cpu_to_lelong(__x) cpu_to_le32(__x)
#define __lelong __le32
#else
#define lelong_to_cpu(__x) le64_to_cpu(__x)
#define cpu_to_lelong(__x) cpu_to_le64(__x)
#define __lelong __le64
#endif
#define nacl_shmem() \
@ -70,7 +72,7 @@ int kvm_riscv_nacl_init(void);
#define nacl_scratch_read_long(__shmem, __offset) \
({ \
unsigned long *__p = (__shmem) + \
__lelong *__p = (__shmem) + \
SBI_NACL_SHMEM_SCRATCH_OFFSET + \
(__offset); \
lelong_to_cpu(*__p); \
@ -78,7 +80,7 @@ int kvm_riscv_nacl_init(void);
#define nacl_scratch_write_long(__shmem, __offset, __val) \
do { \
unsigned long *__p = (__shmem) + \
__lelong *__p = (__shmem) + \
SBI_NACL_SHMEM_SCRATCH_OFFSET + \
(__offset); \
*__p = cpu_to_lelong(__val); \
@ -87,7 +89,7 @@ do { \
#define nacl_scratch_write_longs(__shmem, __offset, __array, __count) \
do { \
unsigned int __i; \
unsigned long *__p = (__shmem) + \
__lelong *__p = (__shmem) + \
SBI_NACL_SHMEM_SCRATCH_OFFSET + \
(__offset); \
for (__i = 0; __i < (__count); __i++) \
@ -168,7 +170,7 @@ __kvm_riscv_nacl_hfence(__shmem, \
#define nacl_csr_read(__shmem, __csr) \
({ \
unsigned long *__a = (__shmem) + SBI_NACL_SHMEM_CSR_OFFSET; \
__lelong *__a = (__shmem) + SBI_NACL_SHMEM_CSR_OFFSET; \
lelong_to_cpu(__a[SBI_NACL_SHMEM_CSR_INDEX(__csr)]); \
})
@ -176,7 +178,7 @@ __kvm_riscv_nacl_hfence(__shmem, \
do { \
void *__s = (__shmem); \
unsigned int __i = SBI_NACL_SHMEM_CSR_INDEX(__csr); \
unsigned long *__a = (__s) + SBI_NACL_SHMEM_CSR_OFFSET; \
__lelong *__a = (__s) + SBI_NACL_SHMEM_CSR_OFFSET; \
u8 *__b = (__s) + SBI_NACL_SHMEM_DBITMAP_OFFSET; \
__a[__i] = cpu_to_lelong(__val); \
__b[__i >> 3] |= 1U << (__i & 0x7); \
@ -186,7 +188,7 @@ do { \
({ \
void *__s = (__shmem); \
unsigned int __i = SBI_NACL_SHMEM_CSR_INDEX(__csr); \
unsigned long *__a = (__s) + SBI_NACL_SHMEM_CSR_OFFSET; \
__lelong *__a = (__s) + SBI_NACL_SHMEM_CSR_OFFSET; \
u8 *__b = (__s) + SBI_NACL_SHMEM_DBITMAP_OFFSET; \
unsigned long __r = lelong_to_cpu(__a[__i]); \
__a[__i] = cpu_to_lelong(__val); \

2
arch/riscv/include/asm/uaccess.h
View File

@ -112,7 +112,7 @@ do { \
_ASM_EXTABLE_UACCESS_ERR(1b, %l2, %0) \
: "=&r" (__tmp) \
: "m" (*(ptr)) : : label); \
(x) = (__typeof__(x))(unsigned long)__tmp; \
(x) = (__force __typeof__(x))(unsigned long)__tmp; \
} while (0)
#else /* !CONFIG_CC_HAS_ASM_GOTO_OUTPUT */
#define __get_user_asm(insn, x, ptr, label) \

88
arch/riscv/kvm/aia.c
View File

@ -21,13 +21,15 @@
struct aia_hgei_control {
raw_spinlock_t lock;
bool free_bitmap_initialized;
unsigned long free_bitmap;
struct kvm_vcpu *owners[BITS_PER_LONG];
unsigned int nr_hgei;
};
static DEFINE_PER_CPU(struct aia_hgei_control, aia_hgei);
static int hgei_parent_irq;
unsigned int kvm_riscv_aia_nr_hgei;
atomic_t kvm_riscv_aia_nr_hgei;
unsigned int kvm_riscv_aia_max_ids;
DEFINE_STATIC_KEY_FALSE(kvm_riscv_aia_available);
@ -452,7 +454,7 @@ void kvm_riscv_aia_free_hgei(int cpu, int hgei)
raw_spin_lock_irqsave(&hgctrl->lock, flags);
if (hgei > 0 && hgei <= kvm_riscv_aia_nr_hgei) {
if (hgei > 0 && hgei <= hgctrl->nr_hgei) {
if (!(hgctrl->free_bitmap & BIT(hgei))) {
hgctrl->free_bitmap |= BIT(hgei);
hgctrl->owners[hgei] = NULL;
@ -486,26 +488,18 @@ static irqreturn_t hgei_interrupt(int irq, void *dev_id)
static int aia_hgei_init(void)
{
int cpu, rc;
struct irq_domain *domain;
struct aia_hgei_control *hgctrl;
struct irq_domain *domain;
int cpu, rc;
/* Initialize per-CPU guest external interrupt line management */
for_each_possible_cpu(cpu) {
hgctrl = per_cpu_ptr(&aia_hgei, cpu);
raw_spin_lock_init(&hgctrl->lock);
if (kvm_riscv_aia_nr_hgei) {
hgctrl->free_bitmap =
BIT(kvm_riscv_aia_nr_hgei + 1) - 1;
hgctrl->free_bitmap &= ~BIT(0);
} else
hgctrl->free_bitmap = 0;
hgctrl->free_bitmap_initialized = false;
hgctrl->free_bitmap = 0;
}
/* Skip SGEI interrupt setup for zero guest external interrupts */
if (!kvm_riscv_aia_nr_hgei)
goto skip_sgei_interrupt;
/* Find INTC irq domain */
domain = irq_find_matching_fwnode(riscv_get_intc_hwnode(),
DOMAIN_BUS_ANY);
@ -529,25 +523,60 @@ static int aia_hgei_init(void)
return rc;
}
skip_sgei_interrupt:
return 0;
}
static void aia_hgei_exit(void)
{
/* Do nothing for zero guest external interrupts */
if (!kvm_riscv_aia_nr_hgei)
return;
/* Free per-CPU SGEI interrupt */
free_percpu_irq(hgei_parent_irq, &aia_hgei);
}
void kvm_riscv_aia_enable(void)
{
const struct imsic_global_config *gc;
const struct imsic_local_config *lc;
struct aia_hgei_control *hgctrl;
unsigned long flags;
int aia_nr_hgei;
if (!kvm_riscv_aia_available())
return;
gc = imsic_get_global_config();
lc = (gc) ? this_cpu_ptr(gc->local) : NULL;
hgctrl = this_cpu_ptr(&aia_hgei);
/* Figure-out number of bits in HGEIE */
csr_write(CSR_HGEIE, -1UL);
hgctrl->nr_hgei = fls_long(csr_read(CSR_HGEIE));
csr_write(CSR_HGEIE, 0);
if (hgctrl->nr_hgei)
hgctrl->nr_hgei--;
/*
* Number of usable per-HART HGEI lines should be minimum of
* per-HART IMSIC guest files and number of bits in HGEIE.
*/
if (lc)
hgctrl->nr_hgei = min((ulong)hgctrl->nr_hgei, lc->nr_guest_files);
else
hgctrl->nr_hgei = 0;
/* Update the number of IMSIC guest files across all HARTs */
aia_nr_hgei = atomic_read(&kvm_riscv_aia_nr_hgei);
do {
if (aia_nr_hgei <= hgctrl->nr_hgei)
break;
} while (!atomic_try_cmpxchg(&kvm_riscv_aia_nr_hgei, &aia_nr_hgei, hgctrl->nr_hgei));
raw_spin_lock_irqsave(&hgctrl->lock, flags);
if (!hgctrl->free_bitmap_initialized) {
hgctrl->free_bitmap = (hgctrl->nr_hgei) ? GENMASK_ULL(hgctrl->nr_hgei, 1) : 0;
hgctrl->free_bitmap_initialized = true;
}
raw_spin_unlock_irqrestore(&hgctrl->lock, flags);
csr_write(CSR_HVICTL, aia_hvictl_value(false));
csr_write(CSR_HVIPRIO1, 0x0);
csr_write(CSR_HVIPRIO2, 0x0);
@ -588,7 +617,7 @@ void kvm_riscv_aia_disable(void)
raw_spin_lock_irqsave(&hgctrl->lock, flags);
for (i = 0; i <= kvm_riscv_aia_nr_hgei; i++) {
for (i = 0; i <= hgctrl->nr_hgei; i++) {
vcpu = hgctrl->owners[i];
if (!vcpu)
continue;
@ -628,26 +657,15 @@ int kvm_riscv_aia_init(void)
return -ENODEV;
gc = imsic_get_global_config();
/* Figure-out number of bits in HGEIE */
csr_write(CSR_HGEIE, -1UL);
kvm_riscv_aia_nr_hgei = fls_long(csr_read(CSR_HGEIE));
csr_write(CSR_HGEIE, 0);
if (kvm_riscv_aia_nr_hgei)
kvm_riscv_aia_nr_hgei--;
/*
* Number of usable HGEI lines should be minimum of per-HART
* IMSIC guest files and number of bits in HGEIE
*/
/* Set initial value of IMSIC guest files across all HARTs */
if (gc)
kvm_riscv_aia_nr_hgei = min((ulong)kvm_riscv_aia_nr_hgei,
gc->nr_guest_files);
atomic_set(&kvm_riscv_aia_nr_hgei, gc->nr_guest_files);
else
kvm_riscv_aia_nr_hgei = 0;
atomic_set(&kvm_riscv_aia_nr_hgei, 0);
/* Find number of guest MSI IDs */
kvm_riscv_aia_max_ids = IMSIC_MAX_ID;
if (gc && kvm_riscv_aia_nr_hgei)
if (gc)
kvm_riscv_aia_max_ids = gc->nr_guest_ids + 1;
/* Initialize guest external interrupt line management */

15
arch/riscv/kvm/aia_aplic.c
View File

@ -35,7 +35,7 @@ struct aplic {
u32 nr_irqs;
u32 nr_words;
struct aplic_irq *irqs;
struct aplic_irq irqs[] __counted_by(nr_irqs);
};
static u32 aplic_read_sourcecfg(struct aplic *aplic, u32 irq)
@ -581,7 +581,7 @@ int kvm_riscv_aia_aplic_init(struct kvm *kvm)
return 0;
/* Allocate APLIC global state */
aplic = kzalloc_obj(*aplic);
aplic = kzalloc_flex(*aplic, irqs, kvm->arch.aia.nr_sources + 1);
if (!aplic)
return -ENOMEM;
kvm->arch.aia.aplic_state = aplic;
@ -589,11 +589,6 @@ int kvm_riscv_aia_aplic_init(struct kvm *kvm)
/* Setup APLIC IRQs */
aplic->nr_irqs = kvm->arch.aia.nr_sources + 1;
aplic->nr_words = DIV_ROUND_UP(aplic->nr_irqs, 32);
aplic->irqs = kzalloc_objs(*aplic->irqs, aplic->nr_irqs);
if (!aplic->irqs) {
ret = -ENOMEM;
goto fail_free_aplic;
}
for (i = 0; i < aplic->nr_irqs; i++)
raw_spin_lock_init(&aplic->irqs[i].lock);
@ -606,7 +601,7 @@ int kvm_riscv_aia_aplic_init(struct kvm *kvm)
&aplic->iodev);
mutex_unlock(&kvm->slots_lock);
if (ret)
goto fail_free_aplic_irqs;
goto fail_free_aplic;
/* Setup default IRQ routing */
ret = kvm_riscv_setup_default_irq_routing(kvm, aplic->nr_irqs);
@ -619,8 +614,6 @@ int kvm_riscv_aia_aplic_init(struct kvm *kvm)
mutex_lock(&kvm->slots_lock);
kvm_io_bus_unregister_dev(kvm, KVM_MMIO_BUS, &aplic->iodev);
mutex_unlock(&kvm->slots_lock);
fail_free_aplic_irqs:
kfree(aplic->irqs);
fail_free_aplic:
kvm->arch.aia.aplic_state = NULL;
kfree(aplic);
@ -638,8 +631,6 @@ void kvm_riscv_aia_aplic_cleanup(struct kvm *kvm)
kvm_io_bus_unregister_dev(kvm, KVM_MMIO_BUS, &aplic->iodev);
mutex_unlock(&kvm->slots_lock);
kfree(aplic->irqs);
kvm->arch.aia.aplic_state = NULL;
kfree(aplic);
}

4
arch/riscv/kvm/aia_device.c
View File

@ -71,7 +71,7 @@ static int aia_config(struct kvm *kvm, unsigned long type,
* external interrupts (i.e. non-zero
* VS-level IMSIC pages).
*/
if (!kvm_riscv_aia_nr_hgei)
if (!atomic_read(&kvm_riscv_aia_nr_hgei))
return -EINVAL;
break;
default:
@ -628,7 +628,7 @@ void kvm_riscv_aia_init_vm(struct kvm *kvm)
*/
/* Initialize default values in AIA global context */
aia->mode = (kvm_riscv_aia_nr_hgei) ?
aia->mode = (atomic_read(&kvm_riscv_aia_nr_hgei)) ?
KVM_DEV_RISCV_AIA_MODE_AUTO : KVM_DEV_RISCV_AIA_MODE_EMUL;
aia->nr_ids = kvm_riscv_aia_max_ids - 1;
aia->nr_sources = 0;

9
arch/riscv/kvm/aia_imsic.c
View File

@ -683,6 +683,9 @@ bool kvm_riscv_vcpu_aia_imsic_has_interrupt(struct kvm_vcpu *vcpu)
unsigned long flags;
bool ret = false;
if (!imsic)
return false;
/*
* The IMSIC SW-file directly injects interrupt via hvip so
* only check for interrupt when IMSIC VS-file is being used.
@ -722,6 +725,9 @@ void kvm_riscv_vcpu_aia_imsic_put(struct kvm_vcpu *vcpu)
struct imsic *imsic = vcpu->arch.aia_context.imsic_state;
unsigned long flags;
if (!imsic)
return;
if (!kvm_vcpu_is_blocking(vcpu))
return;
@ -738,6 +744,9 @@ void kvm_riscv_vcpu_aia_imsic_release(struct kvm_vcpu *vcpu)
int old_vsfile_hgei, old_vsfile_cpu;
struct imsic *imsic = vcpu->arch.aia_context.imsic_state;
if (!imsic)
return;
/* Read and clear IMSIC VS-file details */
write_lock_irqsave(&imsic->vsfile_lock, flags);
old_vsfile_hgei = imsic->vsfile_hgei;

98
arch/riscv/kvm/gstage.c
View File

@ -123,6 +123,20 @@ static void gstage_tlb_flush(struct kvm_gstage *gstage, u32 level, gpa_t addr)
gstage->vmid);
}
bool kvm_riscv_gstage_try_update_pte(struct kvm_gstage *gstage, u32 level,
gpa_t addr, pte_t *ptep,
pte_t old_pte, pte_t new_pte)
{
if (cmpxchg(&ptep->pte, pte_val(old_pte), pte_val(new_pte)) !=
pte_val(old_pte))
return false;
if (pte_val(old_pte) != pte_val(new_pte))
gstage_tlb_flush(gstage, level, addr);
return true;
}
int kvm_riscv_gstage_set_pte(struct kvm_gstage *gstage,
struct kvm_mmu_memory_cache *pcache,
const struct kvm_gstage_mapping *map)
@ -168,17 +182,22 @@ int kvm_riscv_gstage_set_pte(struct kvm_gstage *gstage,
static void kvm_riscv_gstage_update_pte_prot(struct kvm_gstage *gstage, u32 level,
gpa_t addr, pte_t *ptep, pgprot_t prot)
{
pte_t new_pte;
pte_t old_pte, new_pte;
if (pgprot_val(pte_pgprot(ptep_get(ptep))) == pgprot_val(prot))
return;
for (;;) {
old_pte = ptep_get(ptep);
if (pgprot_val(pte_pgprot(old_pte)) == pgprot_val(prot))
return;
new_pte = pfn_pte(pte_pfn(ptep_get(ptep)), prot);
new_pte = pte_mkdirty(new_pte);
new_pte = pfn_pte(pte_pfn(old_pte), prot);
new_pte = pte_mkdirty(new_pte);
set_pte(ptep, new_pte);
if (kvm_riscv_gstage_try_update_pte(gstage, level, addr, ptep,
old_pte, new_pte))
return;
gstage_tlb_flush(gstage, level, addr);
cpu_relax();
}
}
int kvm_riscv_gstage_map_page(struct kvm_gstage *gstage,
@ -337,35 +356,36 @@ int kvm_riscv_gstage_split_huge(struct kvm_gstage *gstage,
return 0;
}
void kvm_riscv_gstage_op_pte(struct kvm_gstage *gstage, gpa_t addr,
bool kvm_riscv_gstage_op_pte(struct kvm_gstage *gstage, gpa_t addr,
pte_t *ptep, u32 ptep_level, enum kvm_riscv_gstage_op op)
{
int i, ret;
pte_t old_pte, *next_ptep;
u32 next_ptep_level;
unsigned long next_page_size, page_size;
bool flush = false;
ret = gstage_level_to_page_size(gstage, ptep_level, &page_size);
if (ret)
return;
return false;
WARN_ON(addr & (page_size - 1));
if (!pte_val(ptep_get(ptep)))
return;
return false;
if (ptep_level && !gstage_pte_leaf(ptep)) {
next_ptep = (pte_t *)gstage_pte_page_vaddr(ptep_get(ptep));
next_ptep_level = ptep_level - 1;
ret = gstage_level_to_page_size(gstage, next_ptep_level, &next_page_size);
if (ret)
return;
return false;
if (op == GSTAGE_OP_CLEAR)
set_pte(ptep, __pte(0));
for (i = 0; i < PTRS_PER_PTE; i++)
kvm_riscv_gstage_op_pte(gstage, addr + i * next_page_size,
&next_ptep[i], next_ptep_level, op);
flush |= kvm_riscv_gstage_op_pte(gstage, addr + i * next_page_size,
&next_ptep[i], next_ptep_level, op);
if (op == GSTAGE_OP_CLEAR)
put_page(virt_to_page(next_ptep));
} else {
@ -375,11 +395,13 @@ void kvm_riscv_gstage_op_pte(struct kvm_gstage *gstage, gpa_t addr,
else if (op == GSTAGE_OP_WP)
set_pte(ptep, __pte(pte_val(ptep_get(ptep)) & ~_PAGE_WRITE));
if (pte_val(*ptep) != pte_val(old_pte))
gstage_tlb_flush(gstage, ptep_level, addr);
flush = true;
}
return flush;
}
void kvm_riscv_gstage_unmap_range(struct kvm_gstage *gstage,
bool kvm_riscv_gstage_unmap_range(struct kvm_gstage *gstage,
gpa_t start, gpa_t size, bool may_block)
{
int ret;
@ -388,6 +410,7 @@ void kvm_riscv_gstage_unmap_range(struct kvm_gstage *gstage,
bool found_leaf;
unsigned long page_size;
gpa_t addr = start, end = start + size;
bool flush = false;
while (addr < end) {
found_leaf = kvm_riscv_gstage_get_leaf(gstage, addr, &ptep, &ptep_level);
@ -395,26 +418,32 @@ void kvm_riscv_gstage_unmap_range(struct kvm_gstage *gstage,
if (ret)
break;
if (!found_leaf)
goto next;
if (!found_leaf) {
addr = ALIGN(addr + 1, page_size);
} else {
if (!(addr & (page_size - 1)) && ((end - addr) >= page_size))
flush |= kvm_riscv_gstage_op_pte(gstage, addr, ptep,
ptep_level, GSTAGE_OP_CLEAR);
else {
WARN_ONCE(1, "Skip unmap range addr: %#llx, end: %#llx, page_size: %#lx\n",
addr, end, page_size);
}
if (!(addr & (page_size - 1)) && ((end - addr) >= page_size))
kvm_riscv_gstage_op_pte(gstage, addr, ptep,
ptep_level, GSTAGE_OP_CLEAR);
next:
addr += page_size;
addr += page_size;
}
/*
* If the range is too large, release the kvm->mmu_lock
* to prevent starvation and lockup detector warnings.
*/
if (!(gstage->flags & KVM_GSTAGE_FLAGS_LOCAL) && may_block && addr < end)
cond_resched_lock(&gstage->kvm->mmu_lock);
cond_resched_rwlock_write(&gstage->kvm->mmu_lock);
}
return flush;
}
void kvm_riscv_gstage_wp_range(struct kvm_gstage *gstage, gpa_t start, gpa_t end)
bool kvm_riscv_gstage_wp_range(struct kvm_gstage *gstage, gpa_t start, gpa_t end)
{
int ret;
pte_t *ptep;
@ -422,6 +451,7 @@ void kvm_riscv_gstage_wp_range(struct kvm_gstage *gstage, gpa_t start, gpa_t end
bool found_leaf;
gpa_t addr = start;
unsigned long page_size;
bool flush = false;
while (addr < end) {
found_leaf = kvm_riscv_gstage_get_leaf(gstage, addr, &ptep, &ptep_level);
@ -429,15 +459,17 @@ void kvm_riscv_gstage_wp_range(struct kvm_gstage *gstage, gpa_t start, gpa_t end
if (ret)
break;
if (!found_leaf)
goto next;
addr = ALIGN_DOWN(addr, page_size);
kvm_riscv_gstage_op_pte(gstage, addr, ptep,
ptep_level, GSTAGE_OP_WP);
next:
addr += page_size;
if (!found_leaf) {
addr = ALIGN(addr + 1, page_size);
} else {
addr = ALIGN_DOWN(addr, page_size);
flush |= kvm_riscv_gstage_op_pte(gstage, addr, ptep,
ptep_level, GSTAGE_OP_WP);
addr += page_size;
}
}
return flush;
}
void __init kvm_riscv_gstage_mode_detect(void)

8
arch/riscv/kvm/main.c
View File

@ -168,10 +168,6 @@ static int __init riscv_kvm_init(void)
kvm_info("VMID %ld bits available\n", kvm_riscv_gstage_vmid_bits());
if (kvm_riscv_aia_available())
kvm_info("AIA available with %d guest external interrupts\n",
kvm_riscv_aia_nr_hgei);
kvm_riscv_setup_vendor_features();
kvm_register_perf_callbacks();
@ -182,6 +178,10 @@ static int __init riscv_kvm_init(void)
return rc;
}
if (kvm_riscv_aia_available())
kvm_info("AIA available with %d guest external interrupts\n",
atomic_read(&kvm_riscv_aia_nr_hgei));
return 0;
}
module_init(riscv_kvm_init);

211
arch/riscv/kvm/mmu.c
View File

@ -23,13 +23,15 @@ static void mmu_wp_memory_region(struct kvm *kvm, int slot)
phys_addr_t start = memslot->base_gfn << PAGE_SHIFT;
phys_addr_t end = (memslot->base_gfn + memslot->npages) << PAGE_SHIFT;
struct kvm_gstage gstage;
bool flush;
kvm_riscv_gstage_init(&gstage, kvm);
spin_lock(&kvm->mmu_lock);
kvm_riscv_gstage_wp_range(&gstage, start, end);
spin_unlock(&kvm->mmu_lock);
kvm_flush_remote_tlbs_memslot(kvm, memslot);
write_lock(&kvm->mmu_lock);
flush = kvm_riscv_gstage_wp_range(&gstage, start, end);
write_unlock(&kvm->mmu_lock);
if (flush)
kvm_flush_remote_tlbs_memslot(kvm, memslot);
}
int kvm_riscv_mmu_ioremap(struct kvm *kvm, gpa_t gpa, phys_addr_t hpa,
@ -65,9 +67,9 @@ int kvm_riscv_mmu_ioremap(struct kvm *kvm, gpa_t gpa, phys_addr_t hpa,
if (ret)
goto out;
spin_lock(&kvm->mmu_lock);
write_lock(&kvm->mmu_lock);
ret = kvm_riscv_gstage_set_pte(&gstage, &pcache, &map);
spin_unlock(&kvm->mmu_lock);
write_unlock(&kvm->mmu_lock);
if (ret)
goto out;
@ -82,12 +84,17 @@ int kvm_riscv_mmu_ioremap(struct kvm *kvm, gpa_t gpa, phys_addr_t hpa,
void kvm_riscv_mmu_iounmap(struct kvm *kvm, gpa_t gpa, unsigned long size)
{
struct kvm_gstage gstage;
bool flush;
kvm_riscv_gstage_init(&gstage, kvm);
spin_lock(&kvm->mmu_lock);
kvm_riscv_gstage_unmap_range(&gstage, gpa, size, false);
spin_unlock(&kvm->mmu_lock);
write_lock(&kvm->mmu_lock);
flush = kvm_riscv_gstage_unmap_range(&gstage, gpa, size, false);
write_unlock(&kvm->mmu_lock);
if (flush)
kvm_flush_remote_tlbs_range(kvm, gpa >> PAGE_SHIFT,
size >> PAGE_SHIFT);
}
void kvm_arch_mmu_enable_log_dirty_pt_masked(struct kvm *kvm,
@ -99,10 +106,14 @@ void kvm_arch_mmu_enable_log_dirty_pt_masked(struct kvm *kvm,
phys_addr_t start = (base_gfn + __ffs(mask)) << PAGE_SHIFT;
phys_addr_t end = (base_gfn + __fls(mask) + 1) << PAGE_SHIFT;
struct kvm_gstage gstage;
bool flush;
kvm_riscv_gstage_init(&gstage, kvm);
kvm_riscv_gstage_wp_range(&gstage, start, end);
flush = kvm_riscv_gstage_wp_range(&gstage, start, end);
if (flush)
kvm_flush_remote_tlbs_range(kvm, start >> PAGE_SHIFT,
(end - start) >> PAGE_SHIFT);
}
void kvm_arch_sync_dirty_log(struct kvm *kvm, struct kvm_memory_slot *memslot)
@ -128,12 +139,16 @@ void kvm_arch_flush_shadow_memslot(struct kvm *kvm,
gpa_t gpa = slot->base_gfn << PAGE_SHIFT;
phys_addr_t size = slot->npages << PAGE_SHIFT;
struct kvm_gstage gstage;
bool flush;
kvm_riscv_gstage_init(&gstage, kvm);
spin_lock(&kvm->mmu_lock);
kvm_riscv_gstage_unmap_range(&gstage, gpa, size, false);
spin_unlock(&kvm->mmu_lock);
write_lock(&kvm->mmu_lock);
flush = kvm_riscv_gstage_unmap_range(&gstage, gpa, size, false);
write_unlock(&kvm->mmu_lock);
if (flush)
kvm_flush_remote_tlbs_range(kvm, gpa >> PAGE_SHIFT,
size >> PAGE_SHIFT);
}
void kvm_arch_commit_memory_region(struct kvm *kvm,
@ -142,9 +157,8 @@ void kvm_arch_commit_memory_region(struct kvm *kvm,
enum kvm_mr_change change)
{
/*
* At this point memslot has been committed and there is an
* allocated dirty_bitmap[], dirty pages will be tracked while
* the memory slot is write protected.
* At this point memslot has been committed and dirty pages will be
* tracked while the memory slot is write protected.
*/
if (change != KVM_MR_DELETE && new->flags & KVM_MEM_LOG_DIRTY_PAGES) {
if (kvm_dirty_log_manual_protect_and_init_set(kvm))
@ -230,18 +244,20 @@ int kvm_arch_prepare_memory_region(struct kvm *kvm,
bool kvm_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range)
{
struct kvm_gstage gstage;
bool mmu_locked;
bool flush;
if (!kvm->arch.pgd)
return false;
lockdep_assert_held_write(&kvm->mmu_lock);
kvm_riscv_gstage_init(&gstage, kvm);
mmu_locked = spin_trylock(&kvm->mmu_lock);
kvm_riscv_gstage_unmap_range(&gstage, range->start << PAGE_SHIFT,
(range->end - range->start) << PAGE_SHIFT,
range->may_block);
if (mmu_locked)
spin_unlock(&kvm->mmu_lock);
flush = kvm_riscv_gstage_unmap_range(&gstage, range->start << PAGE_SHIFT,
(range->end - range->start) << PAGE_SHIFT,
range->may_block);
if (flush)
kvm_flush_remote_tlbs_range(kvm, range->start,
range->end - range->start);
return false;
}
@ -286,7 +302,8 @@ bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range)
}
static bool fault_supports_gstage_huge_mapping(struct kvm_memory_slot *memslot,
unsigned long hva)
unsigned long hva,
unsigned long map_size)
{
hva_t uaddr_start, uaddr_end;
gpa_t gpa_start;
@ -300,8 +317,8 @@ static bool fault_supports_gstage_huge_mapping(struct kvm_memory_slot *memslot,
/*
* Pages belonging to memslots that don't have the same alignment
* within a PMD for userspace and GPA cannot be mapped with g-stage
* PMD entries, because we'll end up mapping the wrong pages.
* within a huge page for userspace and GPA cannot be mapped with
* g-stage block entries, because we'll end up mapping the wrong pages.
*
* Consider a layout like the following:
*
@ -321,7 +338,7 @@ static bool fault_supports_gstage_huge_mapping(struct kvm_memory_slot *memslot,
* e -> g
* f -> h
*/
if ((gpa_start & (PMD_SIZE - 1)) != (uaddr_start & (PMD_SIZE - 1)))
if ((gpa_start & (map_size - 1)) != (uaddr_start & (map_size - 1)))
return false;
/*
@ -336,7 +353,8 @@ static bool fault_supports_gstage_huge_mapping(struct kvm_memory_slot *memslot,
* userspace_addr or the base_gfn, as both are equally aligned (per
* the check above) and equally sized.
*/
return (hva >= ALIGN(uaddr_start, PMD_SIZE)) && (hva < ALIGN_DOWN(uaddr_end, PMD_SIZE));
return (hva >= ALIGN(uaddr_start, map_size)) &&
(hva < ALIGN_DOWN(uaddr_end, map_size));
}
static int get_hva_mapping_size(struct kvm *kvm,
@ -404,7 +422,7 @@ static unsigned long transparent_hugepage_adjust(struct kvm *kvm,
* sure that the HVA and GPA are sufficiently aligned and that the
* block map is contained within the memslot.
*/
if (fault_supports_gstage_huge_mapping(memslot, hva)) {
if (fault_supports_gstage_huge_mapping(memslot, hva, PMD_SIZE)) {
int sz;
sz = get_hva_mapping_size(kvm, hva);
@ -421,20 +439,114 @@ static unsigned long transparent_hugepage_adjust(struct kvm *kvm,
return PAGE_SIZE;
}
static unsigned long hugetlb_mapping_size(struct kvm_memory_slot *memslot,
unsigned long hva,
unsigned long map_size)
{
switch (map_size) {
#ifndef CONFIG_32BIT
case PUD_SIZE:
if (fault_supports_gstage_huge_mapping(memslot, hva, PUD_SIZE))
return PUD_SIZE;
fallthrough;
#endif
case PMD_SIZE:
if (fault_supports_gstage_huge_mapping(memslot, hva, PMD_SIZE))
return PMD_SIZE;
fallthrough;
case PAGE_SIZE:
return PAGE_SIZE;
default:
return map_size;
}
}
static bool kvm_riscv_mmu_dirty_log_write_fault_fast(struct kvm *kvm,
struct kvm_memory_slot *memslot,
gpa_t gpa,
struct kvm_gstage_mapping *out_map)
{
struct kvm_gstage gstage;
unsigned long mmu_seq;
pte_t old_pte, new_pte;
pte_t *ptep;
gfn_t gfn = gpa >> PAGE_SHIFT;
u32 ptep_level;
bool dirty_marked = false;
bool ret;
kvm_riscv_gstage_init(&gstage, kvm);
mmu_seq = kvm->mmu_invalidate_seq;
read_lock(&kvm->mmu_lock);
if (mmu_invalidate_retry_gfn(kvm, mmu_seq, gfn)) {
ret = false;
goto out_unlock;
}
if (!kvm_riscv_gstage_get_leaf(&gstage, gpa, &ptep, &ptep_level) ||
ptep_level) {
ret = false;
goto out_unlock;
}
for (;;) {
old_pte = ptep_get(ptep);
if (!(pte_val(old_pte) & _PAGE_LEAF)) {
ret = false;
break;
}
if (!dirty_marked) {
mark_page_dirty_in_slot(kvm, memslot, gfn);
dirty_marked = true;
}
if ((pte_val(old_pte) & (_PAGE_WRITE | _PAGE_DIRTY)) ==
(_PAGE_WRITE | _PAGE_DIRTY)) {
new_pte = old_pte;
ret = true;
break;
}
new_pte = pte_mkdirty(pte_mkwrite_novma(old_pte));
if (kvm_riscv_gstage_try_update_pte(&gstage, ptep_level, gpa,
ptep, old_pte, new_pte)) {
ret = true;
break;
}
cpu_relax();
}
out_unlock:
read_unlock(&kvm->mmu_lock);
if (ret) {
out_map->addr = gpa & PAGE_MASK;
out_map->level = 0;
out_map->pte = new_pte;
}
return ret;
}
int kvm_riscv_mmu_map(struct kvm_vcpu *vcpu, struct kvm_memory_slot *memslot,
gpa_t gpa, unsigned long hva, bool is_write,
struct kvm_gstage_mapping *out_map)
{
int ret;
kvm_pfn_t hfn;
bool is_hugetlb;
bool writable;
short vma_pageshift;
gfn_t gfn = gpa >> PAGE_SHIFT;
struct vm_area_struct *vma;
struct kvm *kvm = vcpu->kvm;
struct kvm_mmu_memory_cache *pcache = &vcpu->arch.mmu_page_cache;
bool logging = (memslot->dirty_bitmap &&
!(memslot->flags & KVM_MEM_READONLY)) ? true : false;
bool logging = kvm_slot_dirty_track_enabled(memslot) &&
!(memslot->flags & KVM_MEM_READONLY);
unsigned long vma_pagesize, mmu_seq;
struct kvm_gstage gstage;
struct page *page;
@ -444,6 +556,10 @@ int kvm_riscv_mmu_map(struct kvm_vcpu *vcpu, struct kvm_memory_slot *memslot,
/* Setup initial state of output mapping */
memset(out_map, 0, sizeof(*out_map));
if (is_write && logging &&
kvm_riscv_mmu_dirty_log_write_fault_fast(kvm, memslot, gpa, out_map))
return 0;
/* We need minimum second+third level pages */
ret = kvm_mmu_topup_memory_cache(pcache, kvm->arch.pgd_levels);
if (ret) {
@ -460,16 +576,23 @@ int kvm_riscv_mmu_map(struct kvm_vcpu *vcpu, struct kvm_memory_slot *memslot,
return -EFAULT;
}
if (is_vm_hugetlb_page(vma))
is_hugetlb = is_vm_hugetlb_page(vma);
if (is_hugetlb)
vma_pageshift = huge_page_shift(hstate_vma(vma));
else
vma_pageshift = PAGE_SHIFT;
vma_pagesize = 1ULL << vma_pageshift;
if (logging || (vma->vm_flags & VM_PFNMAP))
vma_pagesize = PAGE_SIZE;
else if (is_hugetlb)
vma_pagesize = hugetlb_mapping_size(memslot, hva, vma_pagesize);
/*
* For hugetlb mappings, vma_pagesize might have been reduced from the
* VMA size to a smaller safe mapping size.
*/
if (vma_pagesize == PMD_SIZE || vma_pagesize == PUD_SIZE)
gfn = (gpa & huge_page_mask(hstate_vma(vma))) >> PAGE_SHIFT;
gfn = ALIGN_DOWN(gpa, vma_pagesize) >> PAGE_SHIFT;
/*
* Read mmu_invalidate_seq so that KVM can detect if the results of
@ -506,13 +629,17 @@ int kvm_riscv_mmu_map(struct kvm_vcpu *vcpu, struct kvm_memory_slot *memslot,
if (logging && !is_write)
writable = false;
spin_lock(&kvm->mmu_lock);
write_lock(&kvm->mmu_lock);
if (mmu_invalidate_retry(kvm, mmu_seq))
goto out_unlock;
/* Check if we are backed by a THP and thus use block mapping if possible */
if (!logging && (vma_pagesize == PAGE_SIZE))
/*
* Check if we are backed by a THP and thus use block mapping if
* possible. Hugetlb mappings already selected their target size above,
* so do not promote them through the THP helper.
*/
if (!logging && !is_hugetlb && vma_pagesize == PAGE_SIZE)
vma_pagesize = transparent_hugepage_adjust(kvm, memslot, hva, &hfn, &gpa);
if (writable) {
@ -529,7 +656,7 @@ int kvm_riscv_mmu_map(struct kvm_vcpu *vcpu, struct kvm_memory_slot *memslot,
out_unlock:
kvm_release_faultin_page(kvm, page, ret && ret != -EEXIST, writable);
spin_unlock(&kvm->mmu_lock);
write_unlock(&kvm->mmu_lock);
return ret;
}
@ -557,18 +684,22 @@ void kvm_riscv_mmu_free_pgd(struct kvm *kvm)
{
struct kvm_gstage gstage;
void *pgd = NULL;
bool flush = false;
spin_lock(&kvm->mmu_lock);
write_lock(&kvm->mmu_lock);
if (kvm->arch.pgd) {
kvm_riscv_gstage_init(&gstage, kvm);
kvm_riscv_gstage_unmap_range(&gstage, 0UL,
flush = kvm_riscv_gstage_unmap_range(&gstage, 0UL,
kvm_riscv_gstage_gpa_size(kvm->arch.pgd_levels), false);
pgd = READ_ONCE(kvm->arch.pgd);
kvm->arch.pgd = NULL;
kvm->arch.pgd_phys = 0;
kvm->arch.pgd_levels = 0;
}
spin_unlock(&kvm->mmu_lock);
write_unlock(&kvm->mmu_lock);
if (flush)
kvm_flush_remote_tlbs(kvm);
if (pgd)
free_pages((unsigned long)pgd, get_order(kvm_riscv_gstage_pgd_size));

2
arch/riscv/kvm/nacl.c
View File

@ -20,7 +20,7 @@ void __kvm_riscv_nacl_hfence(void *shmem,
unsigned long page_count)
{
int i, ent = -1, try_count = 5;
unsigned long *entp;
__lelong *entp;
again:
for (i = 0; i < SBI_NACL_SHMEM_HFENCE_ENTRY_MAX; i++) {

1
arch/riscv/kvm/vcpu.c
View File

@ -538,6 +538,7 @@ int kvm_arch_vcpu_ioctl_set_guest_debug(struct kvm_vcpu *vcpu,
else
vcpu->guest_debug = 0;
kvm_riscv_vcpu_config_guest_debug(vcpu);
return 0;
}

44
arch/riscv/kvm/vcpu_sbi_fwft.c
View File

@ -35,6 +35,16 @@ struct kvm_sbi_fwft_feature {
*/
bool (*supported)(struct kvm_vcpu *vcpu);
/**
* @init: Probe and initialize the feature on the vcpu
*
* This callback is optional. If provided, it will be called during
* vcpu initialization to probe the feature availability and perform
* any necessary initialization. Returns true if the feature is supported
* and initialized successfully, false otherwise.
*/
bool (*init)(struct kvm_vcpu *vcpu);
/**
* @reset: Reset the feature value irrespective whether feature is supported or not
*
@ -131,19 +141,30 @@ static long kvm_sbi_fwft_get_misaligned_delegation(struct kvm_vcpu *vcpu,
static bool try_to_set_pmm(unsigned long value)
{
unsigned long prev;
bool ret;
prev = csr_read_clear(CSR_HENVCFG, ENVCFG_PMM);
csr_set(CSR_HENVCFG, value);
return (csr_read_clear(CSR_HENVCFG, ENVCFG_PMM) & ENVCFG_PMM) == value;
ret = (csr_read_clear(CSR_HENVCFG, ENVCFG_PMM) & ENVCFG_PMM) == value;
csr_write(CSR_HENVCFG, prev);
return ret;
}
static bool kvm_sbi_fwft_pointer_masking_pmlen_supported(struct kvm_vcpu *vcpu)
{
return riscv_isa_extension_available(vcpu->arch.isa, SMNPM);
}
static bool kvm_sbi_fwft_pointer_masking_pmlen_init(struct kvm_vcpu *vcpu)
{
struct kvm_sbi_fwft *fwft = vcpu_to_fwft(vcpu);
if (!riscv_isa_extension_available(vcpu->arch.isa, SMNPM))
return false;
preempt_disable();
fwft->have_vs_pmlen_7 = try_to_set_pmm(ENVCFG_PMM_PMLEN_7);
fwft->have_vs_pmlen_16 = try_to_set_pmm(ENVCFG_PMM_PMLEN_16);
preempt_enable();
return fwft->have_vs_pmlen_7 || fwft->have_vs_pmlen_16;
}
@ -231,6 +252,7 @@ static const struct kvm_sbi_fwft_feature features[] = {
.first_reg_num = offsetof(struct kvm_riscv_sbi_fwft, pointer_masking.enable) /
sizeof(unsigned long),
.supported = kvm_sbi_fwft_pointer_masking_pmlen_supported,
.init = kvm_sbi_fwft_pointer_masking_pmlen_init,
.reset = kvm_sbi_fwft_reset_pointer_masking_pmlen,
.set = kvm_sbi_fwft_set_pointer_masking_pmlen,
.get = kvm_sbi_fwft_get_pointer_masking_pmlen,
@ -281,6 +303,8 @@ static int kvm_fwft_get_feature(struct kvm_vcpu *vcpu, u32 feature,
if (!tconf->supported || !tconf->enabled)
return SBI_ERR_NOT_SUPPORTED;
else if (tconf->feature->supported && !tconf->feature->supported(vcpu))
return SBI_ERR_NOT_SUPPORTED;
*conf = tconf;
@ -365,6 +389,9 @@ static int kvm_sbi_ext_fwft_init(struct kvm_vcpu *vcpu)
else
conf->supported = true;
if (conf->supported && feature->init)
conf->supported = feature->init(vcpu);
conf->enabled = conf->supported;
conf->feature = feature;
}
@ -408,6 +435,8 @@ static unsigned long kvm_sbi_ext_fwft_get_reg_count(struct kvm_vcpu *vcpu)
conf = kvm_sbi_fwft_get_config(vcpu, feature->id);
if (!conf || !conf->supported)
continue;
else if (conf->feature->supported && !conf->feature->supported(vcpu))
continue;
ret++;
}
@ -430,6 +459,8 @@ static int kvm_sbi_ext_fwft_get_reg_id(struct kvm_vcpu *vcpu, int index, u64 *re
conf = kvm_sbi_fwft_get_config(vcpu, feature->id);
if (!conf || !conf->supported)
continue;
else if (conf->feature->supported && !conf->feature->supported(vcpu))
continue;
if (index == idx) {
*reg_id = KVM_REG_RISCV |
@ -465,6 +496,8 @@ static int kvm_sbi_ext_fwft_get_reg(struct kvm_vcpu *vcpu, unsigned long reg_num
conf = kvm_sbi_fwft_get_config(vcpu, feature->id);
if (!conf || !conf->supported)
return -ENOENT;
else if (conf->feature->supported && !conf->feature->supported(vcpu))
return -ENOENT;
switch (reg_num - feature->first_reg_num) {
case 0:
@ -502,6 +535,8 @@ static int kvm_sbi_ext_fwft_set_reg(struct kvm_vcpu *vcpu, unsigned long reg_num
conf = kvm_sbi_fwft_get_config(vcpu, feature->id);
if (!conf || !conf->supported)
return -ENOENT;
else if (conf->feature->supported && !conf->feature->supported(vcpu))
return -ENOENT;
switch (reg_num - feature->first_reg_num) {
case 0:
@ -521,6 +556,7 @@ static int kvm_sbi_ext_fwft_set_reg(struct kvm_vcpu *vcpu, unsigned long reg_num
break;
case 2:
ret = conf->feature->set(vcpu, conf, true, value);
vcpu->arch.csr_dirty = true;
break;
default:
return -ENOENT;

14
arch/riscv/kvm/vcpu_sbi_system.c
View File

@ -35,6 +35,20 @@ static int kvm_sbi_ext_susp_handler(struct kvm_vcpu *vcpu, struct kvm_run *run,
return 0;
}
/*
* Check that all other vCPUs are stopped before entering
* system suspend.
*
* There is a known TOCTOU race here: a concurrent HSM
* HART_START on another vCPU can start a vCPU after it
* has already passed this check, violating the invariant.
*
* We do not fix this because:
* 1. Triggering the race requires a pathological guest.
* 2. Only guest state is at risk, not host integrity.
* 3. Userspace can double-check vCPU states before
* proceeding with suspend.
*/
kvm_for_each_vcpu(i, tmp, vcpu->kvm) {
if (tmp == vcpu)
continue;

2
arch/riscv/kvm/vcpu_timer.c
View File

@ -231,7 +231,7 @@ int kvm_riscv_vcpu_set_reg_timer(struct kvm_vcpu *vcpu,
break;
case KVM_REG_RISCV_TIMER_REG(state):
if (reg_val == KVM_RISCV_TIMER_STATE_ON)
ret = kvm_riscv_vcpu_timer_next_event(vcpu, reg_val);
ret = kvm_riscv_vcpu_timer_next_event(vcpu, t->next_cycles);
else
ret = kvm_riscv_vcpu_timer_cancel(t);
break;

23
arch/s390/include/asm/kvm_host.h
View File

@ -359,7 +359,7 @@ struct kvm_s390_float_interrupt {
struct kvm_s390_mchk_info mchk;
struct kvm_s390_ext_info srv_signal;
int last_sleep_cpu;
struct mutex ais_lock;
spinlock_t ais_lock;
u8 simm;
u8 nimm;
};
@ -448,6 +448,12 @@ struct kvm_vcpu_arch {
struct kvm_vm_stat {
struct kvm_vm_stat_generic generic;
u64 inject_io;
u64 io_390_adapter_map;
u64 io_390_adapter_unmap;
u64 io_390_inatomic;
u64 io_flic_inject_airq;
u64 io_set_adapter_int;
u64 io_390_inatomic_no_inject;
u64 inject_float_mchk;
u64 inject_pfault_done;
u64 inject_service_signal;
@ -479,6 +485,9 @@ struct s390_io_adapter {
bool masked;
bool swap;
bool suppressible;
spinlock_t maps_lock;
struct list_head maps;
unsigned int nr_maps;
};
#define MAX_S390_IO_ADAPTERS ((MAX_ISC + 1) * 8)
@ -504,6 +513,18 @@ struct kvm_s390_cpu_model {
struct kvm_s390_vm_cpu_uv_feat uv_feat_guest;
};
#define S390_ARCH_FAC_FORMAT_2 2
struct kvm_s390_flcb2 {
union {
struct {
u8 reserved0[7];
u8 length;
};
u64 header_val;
};
u64 facilities[S390_ARCH_FAC_LIST_SIZE_U64];
};
typedef int (*crypto_hook)(struct kvm_vcpu *vcpu);
struct kvm_s390_crypto {

1
arch/s390/include/asm/sclp.h
View File

@ -105,6 +105,7 @@ struct sclp_info {
unsigned char has_aisii : 1;
unsigned char has_aeni : 1;
unsigned char has_aisi : 1;
unsigned char has_astfleie2 : 1;
unsigned int ibc;
unsigned int mtid;
unsigned int mtid_cp;

1
arch/s390/include/uapi/asm/kvm.h
View File

@ -444,6 +444,7 @@ struct kvm_s390_vm_cpu_machine {
#define KVM_S390_VM_CPU_FEAT_PFMFI 11
#define KVM_S390_VM_CPU_FEAT_SIGPIF 12
#define KVM_S390_VM_CPU_FEAT_KSS 13
#define KVM_S390_VM_CPU_FEAT_ASTFLEIE2 14
struct kvm_s390_vm_cpu_feat {
__u64 feat[16];
};

1
arch/s390/kvm/Kconfig
View File

@ -30,6 +30,7 @@ config KVM
select KVM_VFIO
select VIRT_XFER_TO_GUEST_WORK
select KVM_MMU_LOCKLESS_AGING
select KVM_GENERIC_PRE_FAULT_MEMORY
help
Support hosting paravirtualized guest machines using the SIE
virtualization capability on the mainframe. This should work

2
arch/s390/kvm/dat.c
View File

@ -45,7 +45,7 @@ int kvm_s390_mmu_cache_topup(struct kvm_s390_mmu_cache *mc)
mc->pts[mc->n_pts] = o;
}
for ( ; mc->n_rmaps < KVM_S390_MMU_CACHE_N_RMAPS; mc->n_rmaps++) {
o = kzalloc_obj(*mc->rmaps[0], GFP_KERNEL_ACCOUNT);
o = kzalloc_obj(struct vsie_rmap, GFP_KERNEL_ACCOUNT);
if (!o)
return -ENOMEM;
mc->rmaps[mc->n_rmaps] = o;

1
arch/s390/kvm/dat.h
View File

@ -501,6 +501,7 @@ struct guest_fault {
bool write_attempt; /* Write access attempted */
bool attempt_pfault; /* Attempt a pfault first */
bool valid; /* This entry contains valid data */
bool crste_region3; /* Whether crstep refers to a region3 entry */
void (*callback)(struct guest_fault *f);
void *priv;
};

73
arch/s390/kvm/gaccess.c
View File

@ -1419,8 +1419,8 @@ static int walk_guest_tables(struct gmap *sg, unsigned long saddr, struct pgtwal
return kvm_s390_get_guest_page(kvm, entries + LEVEL_MEM, table.pte.pfra, wr);
}
static int _do_shadow_pte(struct gmap *sg, gpa_t raddr, union pte *ptep_h, union pte *ptep,
struct guest_fault *f, bool p)
static int _do_shadow_pte(struct kvm_s390_mmu_cache *mc, struct gmap *sg, gpa_t raddr,
union pte *ptep_h, union pte *ptep, struct guest_fault *f, bool p)
{
union pgste pgste;
union pte newpte;
@ -1430,7 +1430,7 @@ static int _do_shadow_pte(struct gmap *sg, gpa_t raddr, union pte *ptep_h, union
lockdep_assert_held(&sg->parent->children_lock);
scoped_guard(spinlock, &sg->host_to_rmap_lock)
rc = gmap_insert_rmap(sg, f->gfn, gpa_to_gfn(raddr), TABLE_TYPE_PAGE_TABLE);
rc = gmap_insert_rmap(mc, sg, f->gfn, gpa_to_gfn(raddr), TABLE_TYPE_PAGE_TABLE);
if (rc)
return rc;
@ -1462,8 +1462,8 @@ static int _do_shadow_pte(struct gmap *sg, gpa_t raddr, union pte *ptep_h, union
return 0;
}
static int _do_shadow_crste(struct gmap *sg, gpa_t raddr, union crste *host, union crste *table,
struct guest_fault *f, bool p)
static int _do_shadow_crste(struct kvm_s390_mmu_cache *mc, struct gmap *sg, gpa_t raddr,
union crste *host, union crste *table, struct guest_fault *f, bool p)
{
union crste newcrste, oldcrste;
unsigned long mask;
@ -1476,7 +1476,7 @@ static int _do_shadow_crste(struct gmap *sg, gpa_t raddr, union crste *host, uni
mask = is_pmd(*table) ? _SEGMENT_FR_MASK : _REGION3_FR_MASK;
r_gfn = gpa_to_gfn(raddr) & mask;
scoped_guard(spinlock, &sg->host_to_rmap_lock)
rc = gmap_insert_rmap(sg, f->gfn & mask, r_gfn, host->h.tt);
rc = gmap_insert_rmap(mc, sg, f->gfn & mask, r_gfn, host->h.tt);
if (rc)
return rc;
@ -1578,39 +1578,52 @@ static int _gaccess_do_shadow(struct kvm_s390_mmu_cache *mc, struct gmap *sg,
if (KVM_BUG_ON(l > TABLE_TYPE_REGION3, sg->kvm))
return -EFAULT;
if (l == TABLE_TYPE_PAGE_TABLE)
return _do_shadow_pte(sg, saddr, ptep_h, ptep, entries + LEVEL_MEM, w->p);
return _do_shadow_crste(sg, saddr, host, table, entries + LEVEL_MEM, w->p);
return _do_shadow_pte(mc, sg, saddr, ptep_h, ptep, entries + LEVEL_MEM, w->p);
return _do_shadow_crste(mc, sg, saddr, host, table, entries + LEVEL_MEM, w->p);
}
static inline int ___gaccess_shadow_fault(struct kvm_vcpu *vcpu, struct gmap *sg, gpa_t saddr,
unsigned long seq, struct pgtwalk *walk)
{
struct gmap *parent;
int rc;
if (kvm_s390_array_needs_retry_safe(vcpu->kvm, seq, walk->raw_entries))
return -EAGAIN;
parent = READ_ONCE(sg->parent);
if (!parent)
return -EAGAIN;
scoped_guard(spinlock, &parent->children_lock) {
if (READ_ONCE(sg->parent) != parent)
return -EAGAIN;
sg->invalidated = false;
rc = _gaccess_do_shadow(vcpu->arch.mc, sg, saddr, walk);
}
if (!rc)
kvm_s390_release_faultin_array(vcpu->kvm, walk->raw_entries, false);
return rc;
}
static inline int _gaccess_shadow_fault(struct kvm_vcpu *vcpu, struct gmap *sg, gpa_t saddr,
unsigned long seq, struct pgtwalk *walk)
{
struct gmap *parent;
int rc;
if (kvm_s390_array_needs_retry_unsafe(vcpu->kvm, seq, walk->raw_entries))
return -EAGAIN;
again:
rc = kvm_s390_mmu_cache_topup(vcpu->arch.mc);
if (rc)
return rc;
scoped_guard(read_lock, &vcpu->kvm->mmu_lock) {
if (kvm_s390_array_needs_retry_safe(vcpu->kvm, seq, walk->raw_entries))
return -EAGAIN;
parent = READ_ONCE(sg->parent);
if (!parent)
return -EAGAIN;
scoped_guard(spinlock, &parent->children_lock) {
if (READ_ONCE(sg->parent) != parent)
return -EAGAIN;
sg->invalidated = false;
rc = _gaccess_do_shadow(vcpu->arch.mc, sg, saddr, walk);
}
if (rc == -ENOMEM)
goto again;
if (!rc)
kvm_s390_release_faultin_array(vcpu->kvm, walk->raw_entries, false);
}
do {
rc = kvm_s390_mmu_cache_topup(vcpu->arch.mc);
if (rc)
return rc;
rc = radix_tree_preload(GFP_KERNEL);
if (rc)
return rc;
scoped_guard(read_lock, &vcpu->kvm->mmu_lock)
rc = ___gaccess_shadow_fault(vcpu, sg, saddr, seq, walk);
radix_tree_preload_end();
} while (rc == -ENOMEM);
return rc;
}

33
arch/s390/kvm/gmap.c
View File

@ -105,6 +105,11 @@ static void gmap_add_child(struct gmap *parent, struct gmap *child)
else
clear_bit(GMAP_FLAG_ALLOW_HPAGE_1M, &child->flags);
if (test_bit(GMAP_FLAG_ALLOW_HPAGE_2G, &parent->flags))
set_bit(GMAP_FLAG_ALLOW_HPAGE_2G, &child->flags);
else
clear_bit(GMAP_FLAG_ALLOW_HPAGE_2G, &child->flags);
if (kvm_is_ucontrol(parent->kvm))
clear_bit(GMAP_FLAG_OWNS_PAGETABLES, &child->flags);
list_add(&child->list, &parent->children);
@ -543,6 +548,7 @@ static int gmap_handle_minor_crste_fault(struct gmap *gmap, struct guest_fault *
f->pfn = PHYS_PFN(large_crste_to_phys(oldcrste, f->gfn));
f->writable = oldcrste.s.fc1.w;
f->crste_region3 = is_pud(oldcrste);
/* Appropriate permissions already (race with another handler), nothing to do. */
if (!oldcrste.h.i && !(f->write_attempt && oldcrste.h.p))
return 0;
@ -630,10 +636,27 @@ int gmap_try_fixup_minor(struct gmap *gmap, struct guest_fault *fault)
return rc;
}
/**
* gmap_2g_allowed() - Check whether a 2G hugepage is allowed.
* @gmap: The gmap of the guest.
* @f: Describes the fault that is being resolved.
* @slot: The memslot the faulting address belongs to.
*
* The function checks whether the GMAP_FLAG_ALLOW_HPAGE_2G flag is set for
* @gmap, whether the offset of the address in the 2G virtual frame is the
* same as the offset in the physical 2G frame, and finally whether the whole
* 2G page would fit in the given memslot.
*
* Return: true if a 2G hugepage is allowed to back the faulting address, false
* otherwise.
*/
static inline bool gmap_2g_allowed(struct gmap *gmap, struct guest_fault *f,
struct kvm_memory_slot *slot)
{
return false;
return test_bit(GMAP_FLAG_ALLOW_HPAGE_2G, &gmap->flags) &&
!((f->gfn ^ f->pfn) & ~_REGION3_FR_MASK) &&
slot->base_gfn <= ALIGN_DOWN(f->gfn, _PAGES_PER_REGION3) &&
slot->base_gfn + slot->npages >= ALIGN(f->gfn + 1, _PAGES_PER_REGION3);
}
/**
@ -702,6 +725,7 @@ static int _gmap_link(struct kvm_s390_mmu_cache *mc, struct gmap *gmap, int leve
if (oldval.val != _CRSTE_EMPTY(oldval.h.tt).val &&
crste_origin_large(oldval) != crste_origin_large(newval))
return -EAGAIN;
f->crste_region3 = is_pud(newval);
} while (!gmap_crstep_xchg_atomic(gmap, f->crstep, oldval, newval, f->gfn));
if (f->callback)
f->callback(f);
@ -1000,7 +1024,8 @@ int gmap_pv_destroy_range(struct gmap *gmap, gfn_t start, gfn_t end, bool interr
return 0;
}
int gmap_insert_rmap(struct gmap *sg, gfn_t p_gfn, gfn_t r_gfn, int level)
int gmap_insert_rmap(struct kvm_s390_mmu_cache *mc, struct gmap *sg, gfn_t p_gfn,
gfn_t r_gfn, int level)
{
struct vsie_rmap *rmap __free(kvfree) = NULL;
struct vsie_rmap *temp;
@ -1010,7 +1035,7 @@ int gmap_insert_rmap(struct gmap *sg, gfn_t p_gfn, gfn_t r_gfn, int level)
KVM_BUG_ON(!is_shadow(sg), sg->kvm);
lockdep_assert_held(&sg->host_to_rmap_lock);
rmap = kzalloc_obj(*rmap, GFP_ATOMIC);
rmap = kvm_s390_mmu_cache_alloc_rmap(mc);
if (!rmap)
return -ENOMEM;
@ -1057,7 +1082,7 @@ int gmap_protect_rmap(struct kvm_s390_mmu_cache *mc, struct gmap *sg, gfn_t p_gf
if (level <= TABLE_TYPE_REGION1) {
bitmask = -1UL << (8 + 11 * level);
scoped_guard(spinlock, &sg->host_to_rmap_lock)
rc = gmap_insert_rmap(sg, p_gfn, r_gfn & bitmask, level);
rc = gmap_insert_rmap(mc, sg, p_gfn, r_gfn & bitmask, level);
}
if (rc)
return rc;

14
arch/s390/kvm/gmap.h
View File

@ -100,7 +100,8 @@ int gmap_ucas_map(struct gmap *gmap, gfn_t p_gfn, gfn_t c_gfn, unsigned long cou
void gmap_ucas_unmap(struct gmap *gmap, gfn_t c_gfn, unsigned long count);
int gmap_enable_skeys(struct gmap *gmap);
int gmap_pv_destroy_range(struct gmap *gmap, gfn_t start, gfn_t end, bool interruptible);
int gmap_insert_rmap(struct gmap *sg, gfn_t p_gfn, gfn_t r_gfn, int level);
int gmap_insert_rmap(struct kvm_s390_mmu_cache *mc, struct gmap *sg, gfn_t p_gfn,
gfn_t r_gfn, int level);
int gmap_protect_rmap(struct kvm_s390_mmu_cache *mc, struct gmap *sg, gfn_t p_gfn, gfn_t r_gfn,
kvm_pfn_t pfn, int level, bool wr);
void gmap_set_cmma_all_dirty(struct gmap *gmap);
@ -279,7 +280,16 @@ static inline bool __must_check _gmap_crstep_xchg_atomic(struct gmap *gmap, unio
gmap_handle_vsie_unshadow_event(gmap, gfn);
else
_gmap_handle_vsie_unshadow_event(gmap, gfn);
dat_crstep_xchg_atomic(crstep, oldcrste, newcrste, gfn, gmap->asce);
if (!dat_crstep_xchg_atomic(crstep, oldcrste, newcrste, gfn, gmap->asce))
return false;
/*
* Return false even if the swap was successful, as it only
* indicates that the best effort clearing of the vsie_notif
* bit was successful. The caller will have to try again
* regardless, since the desired value has not been set.
* This pointless check is needed to silence a potential
* __must_check warning.
*/
return false;
}
if (!oldcrste.s.fc1.d && newcrste.s.fc1.d && !newcrste.s.fc1.s)

5
arch/s390/kvm/intercept.c
View File

@ -517,8 +517,9 @@ static int handle_pv_spx(struct kvm_vcpu *vcpu)
static int handle_pv_sclp(struct kvm_vcpu *vcpu)
{
struct kvm_s390_float_interrupt *fi = &vcpu->kvm->arch.float_int;
unsigned long flags;
spin_lock(&fi->lock);
spin_lock_irqsave(&fi->lock, flags);
/*
* 2 cases:
* a: an sccb answering interrupt was already pending or in flight.
@ -534,7 +535,7 @@ static int handle_pv_sclp(struct kvm_vcpu *vcpu)
fi->srv_signal.ext_params |= 0x43000;
set_bit(IRQ_PEND_EXT_SERVICE, &fi->pending_irqs);
clear_bit(IRQ_PEND_EXT_SERVICE, &fi->masked_irqs);
spin_unlock(&fi->lock);
spin_unlock_irqrestore(&fi->lock, flags);
return 0;
}

563
arch/s390/kvm/interrupt.c
View File

@ -624,8 +624,9 @@ static int __must_check __deliver_machine_check(struct kvm_vcpu *vcpu)
struct kvm_s390_mchk_info mchk = {};
int deliver = 0;
int rc = 0;
unsigned long flags;
spin_lock(&fi->lock);
spin_lock_irqsave(&fi->lock, flags);
spin_lock(&li->lock);
if (test_bit(IRQ_PEND_MCHK_EX, &li->pending_irqs) ||
test_bit(IRQ_PEND_MCHK_REP, &li->pending_irqs)) {
@ -654,7 +655,7 @@ static int __must_check __deliver_machine_check(struct kvm_vcpu *vcpu)
deliver = 1;
}
spin_unlock(&li->lock);
spin_unlock(&fi->lock);
spin_unlock_irqrestore(&fi->lock, flags);
if (deliver) {
VCPU_EVENT(vcpu, 3, "deliver: machine check mcic 0x%llx",
@ -941,11 +942,12 @@ static int __must_check __deliver_service(struct kvm_vcpu *vcpu)
{
struct kvm_s390_float_interrupt *fi = &vcpu->kvm->arch.float_int;
struct kvm_s390_ext_info ext;
unsigned long flags;
spin_lock(&fi->lock);
spin_lock_irqsave(&fi->lock, flags);
if (test_bit(IRQ_PEND_EXT_SERVICE, &fi->masked_irqs) ||
!(test_bit(IRQ_PEND_EXT_SERVICE, &fi->pending_irqs))) {
spin_unlock(&fi->lock);
spin_unlock_irqrestore(&fi->lock, flags);
return 0;
}
ext = fi->srv_signal;
@ -954,7 +956,7 @@ static int __must_check __deliver_service(struct kvm_vcpu *vcpu)
clear_bit(IRQ_PEND_EXT_SERVICE_EV, &fi->pending_irqs);
if (kvm_s390_pv_cpu_is_protected(vcpu))
set_bit(IRQ_PEND_EXT_SERVICE, &fi->masked_irqs);
spin_unlock(&fi->lock);
spin_unlock_irqrestore(&fi->lock, flags);
if (!ext.ext_params)
return 0;
@ -972,17 +974,18 @@ static int __must_check __deliver_service_ev(struct kvm_vcpu *vcpu)
{
struct kvm_s390_float_interrupt *fi = &vcpu->kvm->arch.float_int;
struct kvm_s390_ext_info ext;
unsigned long flags;
spin_lock(&fi->lock);
spin_lock_irqsave(&fi->lock, flags);
if (!(test_bit(IRQ_PEND_EXT_SERVICE_EV, &fi->pending_irqs))) {
spin_unlock(&fi->lock);
spin_unlock_irqrestore(&fi->lock, flags);
return 0;
}
ext = fi->srv_signal;
/* only clear the event bits */
fi->srv_signal.ext_params &= ~SCCB_EVENT_PENDING;
clear_bit(IRQ_PEND_EXT_SERVICE_EV, &fi->pending_irqs);
spin_unlock(&fi->lock);
spin_unlock_irqrestore(&fi->lock, flags);
VCPU_EVENT(vcpu, 4, "%s", "deliver: sclp parameter event");
vcpu->stat.deliver_service_signal++;
@ -997,8 +1000,9 @@ static int __must_check __deliver_pfault_done(struct kvm_vcpu *vcpu)
struct kvm_s390_float_interrupt *fi = &vcpu->kvm->arch.float_int;
struct kvm_s390_interrupt_info *inti;
int rc = 0;
unsigned long flags;
spin_lock(&fi->lock);
spin_lock_irqsave(&fi->lock, flags);
inti = list_first_entry_or_null(&fi->lists[FIRQ_LIST_PFAULT],
struct kvm_s390_interrupt_info,
list);
@ -1008,7 +1012,7 @@ static int __must_check __deliver_pfault_done(struct kvm_vcpu *vcpu)
}
if (list_empty(&fi->lists[FIRQ_LIST_PFAULT]))
clear_bit(IRQ_PEND_PFAULT_DONE, &fi->pending_irqs);
spin_unlock(&fi->lock);
spin_unlock_irqrestore(&fi->lock, flags);
if (inti) {
trace_kvm_s390_deliver_interrupt(vcpu->vcpu_id,
@ -1039,8 +1043,9 @@ static int __must_check __deliver_virtio(struct kvm_vcpu *vcpu)
struct kvm_s390_float_interrupt *fi = &vcpu->kvm->arch.float_int;
struct kvm_s390_interrupt_info *inti;
int rc = 0;
unsigned long flags;
spin_lock(&fi->lock);
spin_lock_irqsave(&fi->lock, flags);
inti = list_first_entry_or_null(&fi->lists[FIRQ_LIST_VIRTIO],
struct kvm_s390_interrupt_info,
list);
@ -1058,7 +1063,7 @@ static int __must_check __deliver_virtio(struct kvm_vcpu *vcpu)
}
if (list_empty(&fi->lists[FIRQ_LIST_VIRTIO]))
clear_bit(IRQ_PEND_VIRTIO, &fi->pending_irqs);
spin_unlock(&fi->lock);
spin_unlock_irqrestore(&fi->lock, flags);
if (inti) {
rc = put_guest_lc(vcpu, EXT_IRQ_CP_SERVICE,
@ -1116,10 +1121,11 @@ static int __must_check __deliver_io(struct kvm_vcpu *vcpu,
struct kvm_s390_io_info io;
u32 isc;
int rc = 0;
unsigned long flags;
fi = &vcpu->kvm->arch.float_int;
spin_lock(&fi->lock);
spin_lock_irqsave(&fi->lock, flags);
isc = irq_type_to_isc(irq_type);
isc_list = &fi->lists[isc];
inti = list_first_entry_or_null(isc_list,
@ -1146,7 +1152,7 @@ static int __must_check __deliver_io(struct kvm_vcpu *vcpu,
}
if (list_empty(isc_list))
clear_bit(irq_type, &fi->pending_irqs);
spin_unlock(&fi->lock);
spin_unlock_irqrestore(&fi->lock, flags);
if (inti) {
rc = __do_deliver_io(vcpu, &(inti->io));
@ -1662,8 +1668,9 @@ static struct kvm_s390_interrupt_info *get_io_int(struct kvm *kvm,
struct kvm_s390_interrupt_info *iter;
u16 id = (schid & 0xffff0000U) >> 16;
u16 nr = schid & 0x0000ffffU;
unsigned long flags;
spin_lock(&fi->lock);
spin_lock_irqsave(&fi->lock, flags);
list_for_each_entry(iter, isc_list, list) {
if (schid && (id != iter->io.subchannel_id ||
nr != iter->io.subchannel_nr))
@ -1673,10 +1680,10 @@ static struct kvm_s390_interrupt_info *get_io_int(struct kvm *kvm,
fi->counters[FIRQ_CNTR_IO] -= 1;
if (list_empty(isc_list))
clear_bit(isc_to_irq_type(isc), &fi->pending_irqs);
spin_unlock(&fi->lock);
spin_unlock_irqrestore(&fi->lock, flags);
return iter;
}
spin_unlock(&fi->lock);
spin_unlock_irqrestore(&fi->lock, flags);
return NULL;
}
@ -1769,9 +1776,10 @@ static int __inject_service(struct kvm *kvm,
struct kvm_s390_interrupt_info *inti)
{
struct kvm_s390_float_interrupt *fi = &kvm->arch.float_int;
unsigned long flags;
kvm->stat.inject_service_signal++;
spin_lock(&fi->lock);
spin_lock_irqsave(&fi->lock, flags);
fi->srv_signal.ext_params |= inti->ext.ext_params & SCCB_EVENT_PENDING;
/* We always allow events, track them separately from the sccb ints */
@ -1791,7 +1799,7 @@ static int __inject_service(struct kvm *kvm,
fi->srv_signal.ext_params |= inti->ext.ext_params & SCCB_MASK;
set_bit(IRQ_PEND_EXT_SERVICE, &fi->pending_irqs);
out:
spin_unlock(&fi->lock);
spin_unlock_irqrestore(&fi->lock, flags);
kfree(inti);
return 0;
}
@ -1800,17 +1808,18 @@ static int __inject_virtio(struct kvm *kvm,
struct kvm_s390_interrupt_info *inti)
{
struct kvm_s390_float_interrupt *fi = &kvm->arch.float_int;
unsigned long flags;
kvm->stat.inject_virtio++;
spin_lock(&fi->lock);
spin_lock_irqsave(&fi->lock, flags);
if (fi->counters[FIRQ_CNTR_VIRTIO] >= KVM_S390_MAX_VIRTIO_IRQS) {
spin_unlock(&fi->lock);
spin_unlock_irqrestore(&fi->lock, flags);
return -EBUSY;
}
fi->counters[FIRQ_CNTR_VIRTIO] += 1;
list_add_tail(&inti->list, &fi->lists[FIRQ_LIST_VIRTIO]);
set_bit(IRQ_PEND_VIRTIO, &fi->pending_irqs);
spin_unlock(&fi->lock);
spin_unlock_irqrestore(&fi->lock, flags);
return 0;
}
@ -1818,18 +1827,19 @@ static int __inject_pfault_done(struct kvm *kvm,
struct kvm_s390_interrupt_info *inti)
{
struct kvm_s390_float_interrupt *fi = &kvm->arch.float_int;
unsigned long flags;
kvm->stat.inject_pfault_done++;
spin_lock(&fi->lock);
spin_lock_irqsave(&fi->lock, flags);
if (fi->counters[FIRQ_CNTR_PFAULT] >=
(ASYNC_PF_PER_VCPU * KVM_MAX_VCPUS)) {
spin_unlock(&fi->lock);
spin_unlock_irqrestore(&fi->lock, flags);
return -EBUSY;
}
fi->counters[FIRQ_CNTR_PFAULT] += 1;
list_add_tail(&inti->list, &fi->lists[FIRQ_LIST_PFAULT]);
set_bit(IRQ_PEND_PFAULT_DONE, &fi->pending_irqs);
spin_unlock(&fi->lock);
spin_unlock_irqrestore(&fi->lock, flags);
return 0;
}
@ -1838,13 +1848,14 @@ static int __inject_float_mchk(struct kvm *kvm,
struct kvm_s390_interrupt_info *inti)
{
struct kvm_s390_float_interrupt *fi = &kvm->arch.float_int;
unsigned long flags;
kvm->stat.inject_float_mchk++;
spin_lock(&fi->lock);
spin_lock_irqsave(&fi->lock, flags);
fi->mchk.cr14 |= inti->mchk.cr14 & (1UL << CR_PENDING_SUBCLASS);
fi->mchk.mcic |= inti->mchk.mcic;
set_bit(IRQ_PEND_MCHK_REP, &fi->pending_irqs);
spin_unlock(&fi->lock);
spin_unlock_irqrestore(&fi->lock, flags);
kfree(inti);
return 0;
}
@ -1855,6 +1866,7 @@ static int __inject_io(struct kvm *kvm, struct kvm_s390_interrupt_info *inti)
struct kvm_s390_float_interrupt *fi;
struct list_head *list;
int isc;
unsigned long flags;
kvm->stat.inject_io++;
isc = int_word_to_isc(inti->io.io_int_word);
@ -1873,9 +1885,9 @@ static int __inject_io(struct kvm *kvm, struct kvm_s390_interrupt_info *inti)
}
fi = &kvm->arch.float_int;
spin_lock(&fi->lock);
spin_lock_irqsave(&fi->lock, flags);
if (fi->counters[FIRQ_CNTR_IO] >= KVM_S390_MAX_FLOAT_IRQS) {
spin_unlock(&fi->lock);
spin_unlock_irqrestore(&fi->lock, flags);
return -EBUSY;
}
fi->counters[FIRQ_CNTR_IO] += 1;
@ -1890,7 +1902,7 @@ static int __inject_io(struct kvm *kvm, struct kvm_s390_interrupt_info *inti)
list = &fi->lists[FIRQ_LIST_IO_ISC_0 + isc];
list_add_tail(&inti->list, list);
set_bit(isc_to_irq_type(isc), &fi->pending_irqs);
spin_unlock(&fi->lock);
spin_unlock_irqrestore(&fi->lock, flags);
return 0;
}
@ -1966,15 +1978,10 @@ static int __inject_vm(struct kvm *kvm, struct kvm_s390_interrupt_info *inti)
}
int kvm_s390_inject_vm(struct kvm *kvm,
struct kvm_s390_interrupt *s390int)
struct kvm_s390_interrupt *s390int, struct kvm_s390_interrupt_info *inti)
{
struct kvm_s390_interrupt_info *inti;
int rc;
inti = kzalloc_obj(*inti, GFP_KERNEL_ACCOUNT);
if (!inti)
return -ENOMEM;
inti->type = s390int->type;
switch (inti->type) {
case KVM_S390_INT_VIRTIO:
@ -2003,15 +2010,13 @@ int kvm_s390_inject_vm(struct kvm *kvm,
inti->io.io_int_word = s390int->parm64 & 0x00000000ffffffffull;
break;
default:
kfree(inti);
return -EINVAL;
}
trace_kvm_s390_inject_vm(s390int->type, s390int->parm, s390int->parm64,
2);
rc = __inject_vm(kvm, inti);
if (rc)
kfree(inti);
return rc;
}
@ -2176,12 +2181,13 @@ void kvm_s390_clear_float_irqs(struct kvm *kvm)
{
struct kvm_s390_float_interrupt *fi = &kvm->arch.float_int;
int i;
unsigned long flags;
mutex_lock(&kvm->lock);
if (!kvm_s390_pv_is_protected(kvm))
fi->masked_irqs = 0;
mutex_unlock(&kvm->lock);
spin_lock(&fi->lock);
spin_lock_irqsave(&fi->lock, flags);
fi->pending_irqs = 0;
memset(&fi->srv_signal, 0, sizeof(fi->srv_signal));
memset(&fi->mchk, 0, sizeof(fi->mchk));
@ -2189,7 +2195,7 @@ void kvm_s390_clear_float_irqs(struct kvm *kvm)
clear_irq_list(&fi->lists[i]);
for (i = 0; i < FIRQ_MAX_COUNT; i++)
fi->counters[i] = 0;
spin_unlock(&fi->lock);
spin_unlock_irqrestore(&fi->lock, flags);
kvm_s390_gisa_clear(kvm);
};
@ -2204,6 +2210,7 @@ static int get_all_floating_irqs(struct kvm *kvm, u8 __user *usrbuf, u64 len)
int ret = 0;
int n = 0;
int i;
unsigned long flags;
if (len > KVM_S390_FLIC_MAX_BUFFER || len == 0)
return -EINVAL;
@ -2235,7 +2242,7 @@ static int get_all_floating_irqs(struct kvm *kvm, u8 __user *usrbuf, u64 len)
}
}
fi = &kvm->arch.float_int;
spin_lock(&fi->lock);
spin_lock_irqsave(&fi->lock, flags);
for (i = 0; i < FIRQ_LIST_COUNT; i++) {
list_for_each_entry(inti, &fi->lists[i], list) {
if (n == max_irqs) {
@ -2272,7 +2279,7 @@ static int get_all_floating_irqs(struct kvm *kvm, u8 __user *usrbuf, u64 len)
}
out:
spin_unlock(&fi->lock);
spin_unlock_irqrestore(&fi->lock, flags);
out_nolock:
if (!ret && n > 0) {
if (copy_to_user(usrbuf, buf, sizeof(struct kvm_s390_irq) * n))
@ -2287,6 +2294,7 @@ static int flic_ais_mode_get_all(struct kvm *kvm, struct kvm_device_attr *attr)
{
struct kvm_s390_float_interrupt *fi = &kvm->arch.float_int;
struct kvm_s390_ais_all ais;
unsigned long flags;
if (attr->attr < sizeof(ais))
return -EINVAL;
@ -2294,10 +2302,10 @@ static int flic_ais_mode_get_all(struct kvm *kvm, struct kvm_device_attr *attr)
if (!test_kvm_facility(kvm, 72))
return -EOPNOTSUPP;
mutex_lock(&fi->ais_lock);
spin_lock_irqsave(&fi->ais_lock, flags);
ais.simm = fi->simm;
ais.nimm = fi->nimm;
mutex_unlock(&fi->ais_lock);
spin_unlock_irqrestore(&fi->ais_lock, flags);
if (copy_to_user((void __user *)attr->addr, &ais, sizeof(ais)))
return -EFAULT;
@ -2411,34 +2419,46 @@ static int register_io_adapter(struct kvm_device *dev,
{
struct s390_io_adapter *adapter;
struct kvm_s390_io_adapter adapter_info;
int rc = 0;
mutex_lock(&dev->kvm->lock);
if (copy_from_user(&adapter_info,
(void __user *)attr->addr, sizeof(adapter_info)))
return -EFAULT;
if (adapter_info.id >= MAX_S390_IO_ADAPTERS)
return -EINVAL;
(void __user *)attr->addr, sizeof(adapter_info))) {
rc = -EFAULT;
goto out;
}
if (adapter_info.id >= MAX_S390_IO_ADAPTERS) {
rc = -EINVAL;
goto out;
}
adapter_info.id = array_index_nospec(adapter_info.id,
MAX_S390_IO_ADAPTERS);
if (dev->kvm->arch.adapters[adapter_info.id] != NULL)
return -EINVAL;
if (dev->kvm->arch.adapters[adapter_info.id] != NULL) {
rc = -EINVAL;
goto out;
}
adapter = kzalloc_obj(*adapter, GFP_KERNEL_ACCOUNT);
if (!adapter)
return -ENOMEM;
if (!adapter) {
rc = -ENOMEM;
goto out;
}
INIT_LIST_HEAD(&adapter->maps);
spin_lock_init(&adapter->maps_lock);
adapter->nr_maps = 0;
adapter->id = adapter_info.id;
adapter->isc = adapter_info.isc;
adapter->maskable = adapter_info.maskable;
adapter->masked = false;
adapter->swap = adapter_info.swap;
adapter->suppressible = (adapter_info.flags) &
adapter->suppressible = adapter_info.flags &
KVM_S390_ADAPTER_SUPPRESSIBLE;
dev->kvm->arch.adapters[adapter->id] = adapter;
return 0;
out:
mutex_unlock(&dev->kvm->lock);
return rc;
}
int kvm_s390_mask_adapter(struct kvm *kvm, unsigned int id, bool masked)
@ -2453,12 +2473,151 @@ int kvm_s390_mask_adapter(struct kvm *kvm, unsigned int id, bool masked)
return ret;
}
static struct page *pin_map_page(struct kvm *kvm, u64 uaddr,
unsigned int gup_flags)
{
struct mm_struct *mm = kvm->mm;
struct page *page = NULL;
int locked = 1;
if (mmget_not_zero(mm)) {
mmap_read_lock(mm);
pin_user_pages_remote(mm, uaddr, 1, FOLL_WRITE | gup_flags,
&page, &locked);
if (locked)
mmap_read_unlock(mm);
mmput(mm);
}
return page;
}
static int kvm_s390_adapter_map(struct kvm *kvm, unsigned int id, __u64 addr)
{
struct s390_io_adapter *adapter = get_io_adapter(kvm, id);
struct s390_map_info *map;
unsigned long flags;
__u64 host_addr;
int ret, idx;
if (!adapter || !addr)
return -EINVAL;
map = kzalloc_obj(*map, GFP_KERNEL_ACCOUNT);
if (!map)
return -ENOMEM;
INIT_LIST_HEAD(&map->list);
idx = srcu_read_lock(&kvm->srcu);
host_addr = gpa_to_hva(kvm, addr);
if (kvm_is_error_hva(host_addr)) {
srcu_read_unlock(&kvm->srcu, idx);
ret = -EFAULT;
goto out;
}
srcu_read_unlock(&kvm->srcu, idx);
map->guest_addr = addr;
map->addr = host_addr;
map->page = pin_map_page(kvm, host_addr, FOLL_LONGTERM);
if (!map->page) {
ret = -EINVAL;
goto out;
}
spin_lock_irqsave(&adapter->maps_lock, flags);
if (adapter->nr_maps < MAX_S390_ADAPTER_MAPS) {
list_add_tail(&map->list, &adapter->maps);
adapter->nr_maps++;
ret = 0;
} else {
ret = -EINVAL;
}
spin_unlock_irqrestore(&adapter->maps_lock, flags);
if (ret)
unpin_user_page(map->page);
out:
if (ret)
kfree(map);
return ret;
}
static int kvm_s390_adapter_unmap(struct kvm *kvm, unsigned int id, __u64 addr)
{
struct s390_io_adapter *adapter = get_io_adapter(kvm, id);
struct s390_map_info *map, *tmp, *map_to_free;
struct page *map_page_to_put = NULL;
u64 map_addr_to_mark = 0;
unsigned long flags;
int found = 0, idx;
if (!adapter || !addr)
return -EINVAL;
spin_lock_irqsave(&adapter->maps_lock, flags);
list_for_each_entry_safe(map, tmp, &adapter->maps, list) {
if (map->guest_addr == addr) {
found = 1;
adapter->nr_maps--;
list_del(&map->list);
map_page_to_put = map->page;
map_addr_to_mark = map->guest_addr;
map_to_free = map;
break;
}
}
spin_unlock_irqrestore(&adapter->maps_lock, flags);
if (found) {
kfree(map_to_free);
idx = srcu_read_lock(&kvm->srcu);
mark_page_dirty(kvm, map_addr_to_mark >> PAGE_SHIFT);
set_page_dirty_lock(map_page_to_put);
srcu_read_unlock(&kvm->srcu, idx);
unpin_user_page(map_page_to_put);
}
return found ? 0 : -ENOENT;
}
void kvm_s390_unmap_all_adapters(struct kvm *kvm)
{
struct s390_map_info *map, *tmp;
unsigned long flags;
int i, idx;
for (i = 0; i < MAX_S390_IO_ADAPTERS; i++) {
struct s390_io_adapter *adapter = kvm->arch.adapters[i];
LIST_HEAD(local_list);
if (!adapter)
continue;
spin_lock_irqsave(&adapter->maps_lock, flags);
list_splice_init(&adapter->maps, &local_list);
adapter->nr_maps = 0;
spin_unlock_irqrestore(&adapter->maps_lock, flags);
list_for_each_entry_safe(map, tmp, &local_list, list) {
list_del(&map->list);
idx = srcu_read_lock(&kvm->srcu);
mark_page_dirty(kvm, map->guest_addr >> PAGE_SHIFT);
set_page_dirty_lock(map->page);
srcu_read_unlock(&kvm->srcu, idx);
unpin_user_page(map->page);
kfree(map);
}
}
}
void kvm_s390_destroy_adapters(struct kvm *kvm)
{
int i;
for (i = 0; i < MAX_S390_IO_ADAPTERS; i++)
kvm_s390_unmap_all_adapters(kvm);
for (i = 0; i < MAX_S390_IO_ADAPTERS; i++) {
kfree(kvm->arch.adapters[i]);
kvm->arch.adapters[i] = NULL;
}
}
static int modify_io_adapter(struct kvm_device *dev,
@ -2480,14 +2639,22 @@ static int modify_io_adapter(struct kvm_device *dev,
if (ret > 0)
ret = 0;
break;
/*
* The following operations are no longer needed and therefore no-ops.
* The gpa to hva translation is done when an IRQ route is set up. The
* set_irq code uses get_user_pages_remote() to do the actual write.
*/
case KVM_S390_IO_ADAPTER_MAP:
case KVM_S390_IO_ADAPTER_UNMAP:
ret = 0;
/* If in Secure Execution mode do not long term pin. */
mutex_lock(&dev->kvm->lock);
if (kvm_s390_pv_is_protected(dev->kvm)) {
mutex_unlock(&dev->kvm->lock);
return 0;
}
if (req.type == KVM_S390_IO_ADAPTER_MAP) {
dev->kvm->stat.io_390_adapter_map++;
ret = kvm_s390_adapter_map(dev->kvm, req.id, req.addr);
} else {
dev->kvm->stat.io_390_adapter_unmap++;
ret = kvm_s390_adapter_unmap(dev->kvm, req.id, req.addr);
}
mutex_unlock(&dev->kvm->lock);
break;
default:
ret = -EINVAL;
@ -2524,6 +2691,7 @@ static int modify_ais_mode(struct kvm *kvm, struct kvm_device_attr *attr)
struct kvm_s390_float_interrupt *fi = &kvm->arch.float_int;
struct kvm_s390_ais_req req;
int ret = 0;
unsigned long flags;
if (!test_kvm_facility(kvm, 72))
return -EOPNOTSUPP;
@ -2540,7 +2708,7 @@ static int modify_ais_mode(struct kvm *kvm, struct kvm_device_attr *attr)
2 : KVM_S390_AIS_MODE_SINGLE :
KVM_S390_AIS_MODE_ALL, req.mode);
mutex_lock(&fi->ais_lock);
spin_lock_irqsave(&fi->ais_lock, flags);
switch (req.mode) {
case KVM_S390_AIS_MODE_ALL:
fi->simm &= ~AIS_MODE_MASK(req.isc);
@ -2553,7 +2721,7 @@ static int modify_ais_mode(struct kvm *kvm, struct kvm_device_attr *attr)
default:
ret = -EINVAL;
}
mutex_unlock(&fi->ais_lock);
spin_unlock_irqrestore(&fi->ais_lock, flags);
return ret;
}
@ -2567,25 +2735,41 @@ static int kvm_s390_inject_airq(struct kvm *kvm,
.parm = 0,
.parm64 = isc_to_int_word(adapter->isc),
};
struct kvm_s390_interrupt_info *inti;
unsigned long flags;
int ret = 0;
if (!test_kvm_facility(kvm, 72) || !adapter->suppressible)
return kvm_s390_inject_vm(kvm, &s390int);
inti = kzalloc_obj(*inti, GFP_KERNEL_ACCOUNT);
if (!inti)
return -ENOMEM;
mutex_lock(&fi->ais_lock);
if (fi->nimm & AIS_MODE_MASK(adapter->isc)) {
trace_kvm_s390_airq_suppressed(adapter->id, adapter->isc);
goto out;
if (!test_kvm_facility(kvm, 72) || !adapter->suppressible) {
ret = kvm_s390_inject_vm(kvm, &s390int, inti);
if (ret)
kfree(inti);
return ret;
}
ret = kvm_s390_inject_vm(kvm, &s390int);
spin_lock_irqsave(&fi->ais_lock, flags);
if (fi->nimm & AIS_MODE_MASK(adapter->isc)) {
trace_kvm_s390_airq_suppressed(adapter->id, adapter->isc);
spin_unlock_irqrestore(&fi->ais_lock, flags);
kfree(inti);
return ret;
}
ret = kvm_s390_inject_vm(kvm, &s390int, inti);
if (!ret && (fi->simm & AIS_MODE_MASK(adapter->isc))) {
fi->nimm |= AIS_MODE_MASK(adapter->isc);
trace_kvm_s390_modify_ais_mode(adapter->isc,
KVM_S390_AIS_MODE_SINGLE, 2);
}
out:
mutex_unlock(&fi->ais_lock);
spin_unlock_irqrestore(&fi->ais_lock, flags);
if (ret)
kfree(inti);
return ret;
}
@ -2594,6 +2778,8 @@ static int flic_inject_airq(struct kvm *kvm, struct kvm_device_attr *attr)
unsigned int id = attr->attr;
struct s390_io_adapter *adapter = get_io_adapter(kvm, id);
kvm->stat.io_flic_inject_airq++;
if (!adapter)
return -EINVAL;
@ -2604,6 +2790,7 @@ static int flic_ais_mode_set_all(struct kvm *kvm, struct kvm_device_attr *attr)
{
struct kvm_s390_float_interrupt *fi = &kvm->arch.float_int;
struct kvm_s390_ais_all ais;
unsigned long flags;
if (!test_kvm_facility(kvm, 72))
return -EOPNOTSUPP;
@ -2611,10 +2798,10 @@ static int flic_ais_mode_set_all(struct kvm *kvm, struct kvm_device_attr *attr)
if (copy_from_user(&ais, (void __user *)attr->addr, sizeof(ais)))
return -EFAULT;
mutex_lock(&fi->ais_lock);
spin_lock_irqsave(&fi->ais_lock, flags);
fi->simm = ais.simm;
fi->nimm = ais.nimm;
mutex_unlock(&fi->ais_lock);
spin_unlock_irqrestore(&fi->ais_lock, flags);
return 0;
}
@ -2733,22 +2920,19 @@ static unsigned long get_ind_bit(__u64 addr, unsigned long bit_nr, bool swap)
return swap ? (bit ^ (BITS_PER_LONG - 1)) : bit;
}
static struct page *get_map_page(struct kvm *kvm, u64 uaddr)
static struct s390_map_info *get_map_info(struct s390_io_adapter *adapter,
u64 addr)
{
struct mm_struct *mm = kvm->mm;
struct page *page = NULL;
int locked = 1;
struct s390_map_info *map;
if (mmget_not_zero(mm)) {
mmap_read_lock(mm);
get_user_pages_remote(mm, uaddr, 1, FOLL_WRITE,
&page, &locked);
if (locked)
mmap_read_unlock(mm);
mmput(mm);
if (!adapter)
return NULL;
list_for_each_entry(map, &adapter->maps, list) {
if (map->addr == addr)
return map;
}
return page;
return NULL;
}
static int adapter_indicators_set(struct kvm *kvm,
@ -2757,35 +2941,98 @@ static int adapter_indicators_set(struct kvm *kvm,
{
unsigned long bit;
int summary_set, idx;
struct page *ind_page, *summary_page;
struct s390_map_info *ind_info, *summary_info;
void *map;
struct page *ind_page, *summary_page;
unsigned long flags;
ind_page = get_map_page(kvm, adapter_int->ind_addr);
if (!ind_page)
return -1;
summary_page = get_map_page(kvm, adapter_int->summary_addr);
if (!summary_page) {
put_page(ind_page);
return -1;
ind_page = NULL;
spin_lock_irqsave(&adapter->maps_lock, flags);
ind_info = get_map_info(adapter, adapter_int->ind_addr);
if (!ind_info) {
spin_unlock_irqrestore(&adapter->maps_lock, flags);
ind_page = pin_map_page(kvm, adapter_int->ind_addr, 0);
if (!ind_page)
return -1;
idx = srcu_read_lock(&kvm->srcu);
map = page_address(ind_page);
bit = get_ind_bit(adapter_int->ind_addr,
adapter_int->ind_offset, adapter->swap);
set_bit(bit, map);
mark_page_dirty(kvm, adapter_int->ind_gaddr >> PAGE_SHIFT);
set_page_dirty_lock(ind_page);
srcu_read_unlock(&kvm->srcu, idx);
unpin_user_page(ind_page);
} else {
map = page_address(ind_info->page);
bit = get_ind_bit(ind_info->addr, adapter_int->ind_offset, adapter->swap);
set_bit(bit, map);
spin_unlock_irqrestore(&adapter->maps_lock, flags);
}
idx = srcu_read_lock(&kvm->srcu);
map = page_address(ind_page);
bit = get_ind_bit(adapter_int->ind_addr,
adapter_int->ind_offset, adapter->swap);
set_bit(bit, map);
mark_page_dirty(kvm, adapter_int->ind_gaddr >> PAGE_SHIFT);
set_page_dirty_lock(ind_page);
map = page_address(summary_page);
bit = get_ind_bit(adapter_int->summary_addr,
adapter_int->summary_offset, adapter->swap);
summary_set = test_and_set_bit(bit, map);
mark_page_dirty(kvm, adapter_int->summary_gaddr >> PAGE_SHIFT);
set_page_dirty_lock(summary_page);
srcu_read_unlock(&kvm->srcu, idx);
spin_lock_irqsave(&adapter->maps_lock, flags);
summary_info = get_map_info(adapter, adapter_int->summary_addr);
if (!summary_info) {
spin_unlock_irqrestore(&adapter->maps_lock, flags);
summary_page = pin_map_page(kvm, adapter_int->summary_addr, 0);
if (WARN_ON_ONCE(!summary_page))
return -1;
idx = srcu_read_lock(&kvm->srcu);
map = page_address(summary_page);
bit = get_ind_bit(adapter_int->summary_addr,
adapter_int->summary_offset, adapter->swap);
summary_set = test_and_set_bit(bit, map);
mark_page_dirty(kvm, adapter_int->summary_gaddr >> PAGE_SHIFT);
set_page_dirty_lock(summary_page);
srcu_read_unlock(&kvm->srcu, idx);
unpin_user_page(summary_page);
} else {
map = page_address(summary_info->page);
bit = get_ind_bit(summary_info->addr, adapter_int->summary_offset,
adapter->swap);
summary_set = test_and_set_bit(bit, map);
spin_unlock_irqrestore(&adapter->maps_lock, flags);
}
put_page(ind_page);
put_page(summary_page);
return summary_set ? 0 : 1;
}
static int adapter_indicators_set_fast(struct kvm *kvm,
struct s390_io_adapter *adapter,
struct kvm_s390_adapter_int *adapter_int,
int setbit)
{
unsigned long bit;
int summary_set;
struct s390_map_info *ind_info, *summary_info;
void *map;
spin_lock(&adapter->maps_lock);
ind_info = get_map_info(adapter, adapter_int->ind_addr);
if (!ind_info) {
spin_unlock(&adapter->maps_lock);
return -EWOULDBLOCK;
}
map = page_address(ind_info->page);
bit = get_ind_bit(ind_info->addr, adapter_int->ind_offset, adapter->swap);
if (setbit)
set_bit(bit, map);
summary_info = get_map_info(adapter, adapter_int->summary_addr);
if (!summary_info) {
spin_unlock(&adapter->maps_lock);
return -EWOULDBLOCK;
}
map = page_address(summary_info->page);
bit = get_ind_bit(summary_info->addr, adapter_int->summary_offset,
adapter->swap);
/* If setbit then set summary bit. Else if falling back to the slow path */
/* with setbit==0 then clear the summary bit so the slow path re-injects */
if (setbit)
summary_set = test_and_set_bit(bit, map);
else
summary_set = test_and_clear_bit(bit, map);
spin_unlock(&adapter->maps_lock);
return summary_set ? 0 : 1;
}
@ -2801,6 +3048,8 @@ static int set_adapter_int(struct kvm_kernel_irq_routing_entry *e,
int ret;
struct s390_io_adapter *adapter;
kvm->stat.io_set_adapter_int++;
/* We're only interested in the 0->1 transition. */
if (!level)
return 0;
@ -2869,7 +3118,6 @@ int kvm_set_routing_entry(struct kvm *kvm,
int idx;
switch (ue->type) {
/* we store the userspace addresses instead of the guest addresses */
case KVM_IRQ_ROUTING_S390_ADAPTER:
if (kvm_is_ucontrol(kvm))
return -EINVAL;
@ -3459,3 +3707,86 @@ int __init kvm_s390_gib_init(u8 nisc)
out:
return rc;
}
/*
* kvm_arch_set_irq_inatomic: fast-path for irqfd injection
*/
int kvm_arch_set_irq_inatomic(struct kvm_kernel_irq_routing_entry *e,
struct kvm *kvm, int irq_source_id, int level,
bool line_status)
{
int ret, setbit;
struct s390_io_adapter *adapter;
struct kvm_s390_float_interrupt *fi = &kvm->arch.float_int;
struct kvm_s390_interrupt_info *inti;
struct kvm_s390_interrupt s390int = {
.type = KVM_S390_INT_IO(1, 0, 0, 0),
.parm = 0,
};
kvm->stat.io_390_inatomic++;
/* We're only interested in the 0->1 transition. */
if (!level)
return 0;
if (e->type != KVM_IRQ_ROUTING_S390_ADAPTER)
return -EWOULDBLOCK;
adapter = get_io_adapter(kvm, e->adapter.adapter_id);
if (!adapter)
return -EWOULDBLOCK;
s390int.parm64 = isc_to_int_word(adapter->isc);
setbit = 1;
ret = adapter_indicators_set_fast(kvm, adapter, &e->adapter, setbit);
if (ret < 0)
return -EWOULDBLOCK;
if (!ret || adapter->masked) {
kvm->stat.io_390_inatomic_no_inject++;
return 0;
}
inti = kzalloc_obj(*inti, GFP_ATOMIC);
if (!inti) {
setbit = 0;
adapter_indicators_set_fast(kvm, adapter, &e->adapter, setbit);
return -EWOULDBLOCK;
}
if (!test_kvm_facility(kvm, 72) || !adapter->suppressible) {
ret = kvm_s390_inject_vm(kvm, &s390int, inti);
if (ret == 0) {
return ret;
} else {
setbit = 0;
adapter_indicators_set_fast(kvm, adapter, &e->adapter, setbit);
kfree(inti);
return -EWOULDBLOCK;
}
}
spin_lock(&fi->ais_lock);
if (fi->nimm & AIS_MODE_MASK(adapter->isc)) {
trace_kvm_s390_airq_suppressed(adapter->id, adapter->isc);
spin_unlock(&fi->ais_lock);
kfree(inti);
kvm->stat.io_390_inatomic_no_inject++;
return 0;
}
ret = kvm_s390_inject_vm(kvm, &s390int, inti);
if (!ret && (fi->simm & AIS_MODE_MASK(adapter->isc))) {
fi->nimm |= AIS_MODE_MASK(adapter->isc);
trace_kvm_s390_modify_ais_mode(adapter->isc,
KVM_S390_AIS_MODE_SINGLE, 2);
} else if (ret) {
spin_unlock(&fi->ais_lock);
setbit = 0;
adapter_indicators_set_fast(kvm, adapter, &e->adapter, setbit);
kfree(inti);
return -EWOULDBLOCK;
}
spin_unlock(&fi->ais_lock);
return 0;
}

161
arch/s390/kvm/kvm-s390.c
View File

@ -68,6 +68,12 @@
const struct kvm_stats_desc kvm_vm_stats_desc[] = {
KVM_GENERIC_VM_STATS(),
STATS_DESC_COUNTER(VM, inject_io),
STATS_DESC_COUNTER(VM, io_390_adapter_map),
STATS_DESC_COUNTER(VM, io_390_adapter_unmap),
STATS_DESC_COUNTER(VM, io_390_inatomic),
STATS_DESC_COUNTER(VM, io_flic_inject_airq),
STATS_DESC_COUNTER(VM, io_set_adapter_int),
STATS_DESC_COUNTER(VM, io_390_inatomic_no_inject),
STATS_DESC_COUNTER(VM, inject_float_mchk),
STATS_DESC_COUNTER(VM, inject_pfault_done),
STATS_DESC_COUNTER(VM, inject_service_signal),
@ -204,11 +210,16 @@ static int nested;
module_param(nested, int, S_IRUGO);
MODULE_PARM_DESC(nested, "Nested virtualization support");
/* allow 1m huge page guest backing, if !nested */
/* allow 1m huge page guest backing */
static int hpage;
module_param(hpage, int, 0444);
MODULE_PARM_DESC(hpage, "1m huge page backing support");
/* allow 2g huge page guest backing */
static int hpage_2g;
module_param(hpage_2g, int, 0444);
MODULE_PARM_DESC(hpage_2g, "2g huge page backing support");
/* maximum percentage of steal time for polling. >100 is treated like 100 */
static u8 halt_poll_max_steal = 10;
module_param(halt_poll_max_steal, byte, 0644);
@ -232,33 +243,25 @@ static int async_destroy = 1;
module_param(async_destroy, int, 0444);
MODULE_PARM_DESC(async_destroy, "Asynchronous destroy for protected guests");
/*
* For now we handle at most 16 double words as this is what the s390 base
* kernel handles and stores in the prefix page. If we ever need to go beyond
* this, this requires changes to code, but the external uapi can stay.
*/
#define SIZE_INTERNAL 16
#define HMFAI_DWORDS 16
/*
* Base feature mask that defines default mask for facilities. Consists of the
* defines in FACILITIES_KVM and the non-hypervisor managed bits.
*/
static unsigned long kvm_s390_fac_base[SIZE_INTERNAL] = { FACILITIES_KVM };
static unsigned long kvm_s390_fac_base[HMFAI_DWORDS] = { FACILITIES_KVM };
static_assert(ARRAY_SIZE(((long[]){ FACILITIES_KVM })) <= HMFAI_DWORDS);
static_assert(ARRAY_SIZE(kvm_s390_fac_base) <= S390_ARCH_FAC_MASK_SIZE_U64);
static_assert(ARRAY_SIZE(kvm_s390_fac_base) <= S390_ARCH_FAC_LIST_SIZE_U64);
static_assert(ARRAY_SIZE(kvm_s390_fac_base) <= ARRAY_SIZE(stfle_fac_list));
/*
* Extended feature mask. Consists of the defines in FACILITIES_KVM_CPUMODEL
* and defines the facilities that can be enabled via a cpu model.
*/
static unsigned long kvm_s390_fac_ext[SIZE_INTERNAL] = { FACILITIES_KVM_CPUMODEL };
static unsigned long kvm_s390_fac_size(void)
{
BUILD_BUG_ON(SIZE_INTERNAL > S390_ARCH_FAC_MASK_SIZE_U64);
BUILD_BUG_ON(SIZE_INTERNAL > S390_ARCH_FAC_LIST_SIZE_U64);
BUILD_BUG_ON(SIZE_INTERNAL * sizeof(unsigned long) >
sizeof(stfle_fac_list));
return SIZE_INTERNAL;
}
static const unsigned long kvm_s390_fac_ext[] = { FACILITIES_KVM_CPUMODEL };
static_assert(ARRAY_SIZE(kvm_s390_fac_ext) <= S390_ARCH_FAC_MASK_SIZE_U64);
static_assert(ARRAY_SIZE(kvm_s390_fac_ext) <= S390_ARCH_FAC_LIST_SIZE_U64);
static_assert(ARRAY_SIZE(kvm_s390_fac_ext) <= ARRAY_SIZE(stfle_fac_list));
/* available cpu features supported by kvm */
static DECLARE_BITMAP(kvm_s390_available_cpu_feat, KVM_S390_VM_CPU_FEAT_NR_BITS);
@ -468,6 +471,8 @@ static void __init kvm_s390_cpu_feat_init(void)
allow_cpu_feat(KVM_S390_VM_CPU_FEAT_IBS);
if (sclp.has_kss)
allow_cpu_feat(KVM_S390_VM_CPU_FEAT_KSS);
if (sclp.has_astfleie2)
allow_cpu_feat(KVM_S390_VM_CPU_FEAT_ASTFLEIE2);
/*
* KVM_S390_VM_CPU_FEAT_SKEY: Wrong shadow of PTE.I bits will make
* all skey handling functions read/set the skey from the PGSTE
@ -630,6 +635,7 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
case KVM_CAP_S390_USER_OPEREXEC:
case KVM_CAP_S390_KEYOP:
case KVM_CAP_S390_VSIE_ESAMODE:
case KVM_CAP_PRE_FAULT_MEMORY:
r = 1;
break;
case KVM_CAP_SET_GUEST_DEBUG2:
@ -640,6 +646,11 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
if (hpage && !(kvm && kvm_is_ucontrol(kvm)))
r = 1;
break;
case KVM_CAP_S390_HPAGE_2G:
r = 0;
if (hpage_2g && !(kvm && kvm_is_ucontrol(kvm)))
r = 1;
break;
case KVM_CAP_S390_MEM_OP:
r = MEM_OP_MAX_SIZE;
break;
@ -896,6 +907,27 @@ int kvm_vm_ioctl_enable_cap(struct kvm *kvm, struct kvm_enable_cap *cap)
VM_EVENT(kvm, 3, "ENABLE: CAP_S390_HPAGE %s",
r ? "(not available)" : "(success)");
break;
case KVM_CAP_S390_HPAGE_2G:
mutex_lock(&kvm->lock);
if (kvm->created_vcpus) {
r = -EBUSY;
} else if (!hpage_2g || kvm->arch.use_cmma || kvm_is_ucontrol(kvm)) {
r = -EINVAL;
} else {
r = 0;
set_bit(GMAP_FLAG_ALLOW_HPAGE_2G, &kvm->arch.gmap->flags);
/*
* We might have to create fake 4k page
* tables. To avoid that the hardware works on
* stale PGSTEs, we emulate these instructions.
*/
kvm->arch.use_skf = 0;
kvm->arch.use_pfmfi = 0;
}
mutex_unlock(&kvm->lock);
VM_EVENT(kvm, 3, "ENABLE: CAP_S390_HPAGE_2G %s",
r ? "(not available)" : "(success)");
break;
case KVM_CAP_S390_USER_STSI:
VM_EVENT(kvm, 3, "%s", "ENABLE: CAP_S390_USER_STSI");
kvm->arch.user_stsi = 1;
@ -2252,7 +2284,7 @@ static int kvm_s390_get_cmma_bits(struct kvm *kvm,
return 0;
}
values = vmalloc(args->count);
values = vzalloc(args->count);
if (!values)
return -ENOMEM;
@ -2534,6 +2566,7 @@ static int kvm_s390_handle_pv(struct kvm *kvm, struct kvm_pv_cmd *cmd)
if (kvm_s390_pv_is_protected(kvm))
break;
kvm_s390_unmap_all_adapters(kvm);
mmap_write_lock(kvm->mm);
/*
* Disable creation of new THPs. Existing THPs can stay, they
@ -2869,6 +2902,7 @@ int kvm_arch_vm_ioctl(struct file *filp, unsigned int ioctl, unsigned long arg)
void __user *argp = (void __user *)arg;
struct kvm_device_attr attr;
int r;
struct kvm_s390_interrupt_info *inti;
switch (ioctl) {
case KVM_S390_INTERRUPT: {
@ -2877,7 +2911,12 @@ int kvm_arch_vm_ioctl(struct file *filp, unsigned int ioctl, unsigned long arg)
r = -EFAULT;
if (copy_from_user(&s390int, argp, sizeof(s390int)))
break;
r = kvm_s390_inject_vm(kvm, &s390int);
inti = kzalloc_obj(*inti, GFP_KERNEL_ACCOUNT);
if (!inti)
return -ENOMEM;
r = kvm_s390_inject_vm(kvm, &s390int, inti);
if (r)
kfree(inti);
break;
}
case KVM_CREATE_IRQCHIP: {
@ -3238,13 +3277,16 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned long type)
kvm->arch.sie_page2->kvm = kvm;
kvm->arch.model.fac_list = kvm->arch.sie_page2->fac_list;
for (i = 0; i < kvm_s390_fac_size(); i++) {
for (i = 0; i < ARRAY_SIZE(kvm_s390_fac_base); i++) {
kvm->arch.model.fac_mask[i] = stfle_fac_list[i] &
(kvm_s390_fac_base[i] |
kvm_s390_fac_ext[i]);
kvm_s390_fac_base[i];
kvm->arch.model.fac_list[i] = stfle_fac_list[i] &
kvm_s390_fac_base[i];
}
for (i = 0; i < ARRAY_SIZE(kvm_s390_fac_ext); i++) {
kvm->arch.model.fac_mask[i] |= stfle_fac_list[i] &
kvm_s390_fac_ext[i];
}
kvm->arch.model.subfuncs = kvm_s390_available_subfunc;
/* we are always in czam mode - even on pre z14 machines */
@ -3275,7 +3317,7 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned long type)
mutex_unlock(&kvm->lock);
}
mutex_init(&kvm->arch.float_int.ais_lock);
spin_lock_init(&kvm->arch.float_int.ais_lock);
spin_lock_init(&kvm->arch.float_int.lock);
for (i = 0; i < FIRQ_LIST_COUNT; i++)
INIT_LIST_HEAD(&kvm->arch.float_int.lists[i]);
@ -4397,19 +4439,28 @@ int kvm_s390_try_set_tod_clock(struct kvm *kvm, const struct kvm_s390_vm_tod_clo
}
static void __kvm_inject_pfault_token(struct kvm_vcpu *vcpu, bool start_token,
unsigned long token)
unsigned long token)
{
struct kvm_s390_interrupt inti;
struct kvm_s390_irq irq;
struct kvm_s390_interrupt_info *inti_mem = NULL;
int ret = 0;
if (start_token) {
irq.u.ext.ext_params2 = token;
irq.type = KVM_S390_INT_PFAULT_INIT;
WARN_ON_ONCE(kvm_s390_inject_vcpu(vcpu, &irq));
} else {
inti_mem = kzalloc_obj(*inti_mem, GFP_KERNEL_ACCOUNT);
if (WARN_ON_ONCE(!inti_mem))
return;
inti.type = KVM_S390_INT_PFAULT_DONE;
inti.parm64 = token;
WARN_ON_ONCE(kvm_s390_inject_vm(vcpu->kvm, &inti));
ret = kvm_s390_inject_vm(vcpu->kvm, &inti, inti_mem);
if (ret)
kfree(inti_mem);
WARN_ON_ONCE(ret);
}
}
@ -5759,6 +5810,50 @@ void kvm_arch_commit_memory_region(struct kvm *kvm,
return;
}
/**
* kvm_arch_vcpu_pre_fault_memory() -- pre-fault and link gmap dat tables
* @vcpu: the vcpu that shall appear to have generated the fault-in.
* @range: the range that needs to be faulted in.
*
* The first page of the given range is faulted in and the corresponding gmap
* page tables are created, as if the given vCPU had performed a read
* operation.
* If the range starts outside any memslots, an error is returned. An error is
* also returned for UCONTROL VMs, which should instead use the
* KVM_S390_VCPU_FAULT ioctl.
*
* Return:
* * %-ENOENT if the range lies outside of a memslot.
* * %-EINVAL in case of invalid state (for example if the VM is UCONTROL).
* * %-EIO if errors happen while faulting-in the page (will trigger a warning
* in the caller).
* * other error codes < 0 in case of other errors.
* * otherwise a number > 0 of bytes that have been faulted in successfully.
*/
long kvm_arch_vcpu_pre_fault_memory(struct kvm_vcpu *vcpu, struct kvm_pre_fault_memory *range)
{
struct guest_fault f = { .gfn = gpa_to_gfn(range->gpa), };
gpa_t end;
int rc;
if (kvm_is_ucontrol(vcpu->kvm))
return -EINVAL;
rc = kvm_s390_faultin_gfn(vcpu, NULL, &f);
if (rc == PGM_ADDRESSING)
return -ENOENT;
if (rc > 0)
return -EIO;
if (rc < 0)
return rc;
if (f.ptep)
return PAGE_SIZE;
end = ALIGN(range->gpa + PAGE_SIZE, f.crste_region3 ? _REGION3_SIZE : HPAGE_SIZE);
return min(range->size, end - range->gpa);
}
/**
* kvm_test_age_gfn() - test young
* @kvm: the kvm instance
@ -5820,9 +5915,13 @@ static int __init kvm_s390_init(void)
return -ENODEV;
}
for (i = 0; i < 16; i++)
kvm_s390_fac_base[i] |=
stfle_fac_list[i] & nonhyp_mask(i);
if (hpage_2g && !hpage) {
hpage_2g = 0;
pr_info("Disabling 2G hugepage support, since 1M hugepage support is not enabled.\n");
}
for (i = 0; i < HMFAI_DWORDS; i++)
kvm_s390_fac_base[i] |= nonhyp_mask(i);
r = __kvm_s390_init();
if (r)

5
arch/s390/kvm/kvm-s390.h
View File

@ -376,7 +376,8 @@ int __must_check kvm_s390_deliver_pending_interrupts(struct kvm_vcpu *vcpu);
void kvm_s390_clear_local_irqs(struct kvm_vcpu *vcpu);
void kvm_s390_clear_float_irqs(struct kvm *kvm);
int __must_check kvm_s390_inject_vm(struct kvm *kvm,
struct kvm_s390_interrupt *s390int);
struct kvm_s390_interrupt *s390int,
struct kvm_s390_interrupt_info *inti);
int __must_check kvm_s390_inject_vcpu(struct kvm_vcpu *vcpu,
struct kvm_s390_irq *irq);
static inline int kvm_s390_inject_prog_irq(struct kvm_vcpu *vcpu,
@ -560,6 +561,8 @@ void kvm_s390_gisa_disable(struct kvm *kvm);
void kvm_s390_gisa_enable(struct kvm *kvm);
int __init kvm_s390_gib_init(u8 nisc);
void kvm_s390_gib_destroy(void);
void kvm_s390_unmap_all_adapters(struct kvm *kvm);
/* implemented in guestdbg.c */
void kvm_s390_backup_guest_per_regs(struct kvm_vcpu *vcpu);

5
arch/s390/kvm/pv.c
View File

@ -740,7 +740,10 @@ int kvm_s390_pv_init_vm(struct kvm *kvm, u16 *rc, u16 *rrc)
uvcb.flags.ap_allow_instr = kvm->arch.model.uv_feat_guest.ap;
uvcb.flags.ap_instr_intr = kvm->arch.model.uv_feat_guest.ap_intr;
clear_bit(GMAP_FLAG_ALLOW_HPAGE_1M, &kvm->arch.gmap->flags);
scoped_guard(write_lock, &kvm->mmu_lock) {
clear_bit(GMAP_FLAG_ALLOW_HPAGE_1M, &kvm->arch.gmap->flags);
clear_bit(GMAP_FLAG_ALLOW_HPAGE_2G, &kvm->arch.gmap->flags);
}
gmap_split_huge_pages(kvm->arch.gmap);
cc = uv_call_sched(0, (u64)&uvcb);

85
arch/s390/kvm/vsie.c
View File

@ -6,12 +6,15 @@
*
* Author(s): David Hildenbrand <dahi@linux.vnet.ibm.com>
*/
#include <linux/align.h>
#include <linux/vmalloc.h>
#include <linux/kvm_host.h>
#include <linux/bug.h>
#include <linux/compiler.h>
#include <linux/list.h>
#include <linux/bitmap.h>
#include <linux/sched/signal.h>
#include <linux/stddef.h>
#include <linux/io.h>
#include <linux/mman.h>
@ -62,9 +65,9 @@ struct vsie_page {
gpa_t scb_gpa; /* 0x0258 */
/* the shadow gmap in use by the vsie_page */
struct gmap_cache gmap_cache; /* 0x0260 */
__u8 reserved[0x0700 - 0x0278]; /* 0x0278 */
struct kvm_s390_crypto_cb crycb; /* 0x0700 */
__u8 fac[S390_ARCH_FAC_LIST_SIZE_BYTE]; /* 0x0800 */
__u8 reserved[0x06f8 - 0x0278]; /* 0x0278 */
struct kvm_s390_crypto_cb crycb; /* 0x06f8 */
__u8 fac[8 + S390_ARCH_FAC_LIST_SIZE_BYTE];/* 0x07f8 */
};
static_assert(sizeof(struct vsie_page) == PAGE_SIZE);
@ -1000,6 +1003,45 @@ static void retry_vsie_icpt(struct vsie_page *vsie_page)
clear_vsie_icpt(vsie_page);
}
static int handle_stfle_0(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page,
u32 fac_list_origin)
{
struct kvm_s390_sie_block *scb_s = &vsie_page->scb_s;
/*
* format-0 -> size of nested guest's facility list == guest's size
* guest's size == host's size, since STFLE is interpretatively executed
* using a format-0 for the guest, too.
*/
if (read_guest_real(vcpu, fac_list_origin, &vsie_page->fac,
stfle_size() * sizeof(u64)))
return set_validity_icpt(scb_s, 0x1090U);
scb_s->fac = (u32)virt_to_phys(&vsie_page->fac);
return 0;
}
static int handle_stfle_2(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page, u32 fac_list_origin)
{
struct kvm_s390_flcb2 *flcb_s = (struct kvm_s390_flcb2 *)vsie_page->fac;
struct kvm_s390_sie_block *scb_s = &vsie_page->scb_s;
u64 len;
if (read_guest_real(vcpu, fac_list_origin, &len, sizeof(len)))
return set_validity_icpt(scb_s, 0x1090U);
/* discard reserved bits */
len = (len & U8_MAX);
flcb_s->header_val = len;
len += 1;
if (read_guest_real(vcpu, fac_list_origin + offsetof(struct kvm_s390_flcb2, facilities),
&flcb_s->facilities, len * sizeof(u64)))
return set_validity_icpt(scb_s, 0x1090U);
scb_s->fac = (u32)virt_to_phys(&vsie_page->fac) | S390_ARCH_FAC_FORMAT_2;
return 0;
}
/*
* Try to shadow + enable the guest 2 provided facility list.
* Retry instruction execution if enabled for and provided by guest 2.
@ -1009,29 +1051,34 @@ static void retry_vsie_icpt(struct vsie_page *vsie_page)
*/
static int handle_stfle(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page)
{
struct kvm_s390_sie_block *scb_s = &vsie_page->scb_s;
__u32 fac = READ_ONCE(vsie_page->scb_o->fac);
bool has_astfleie2 = test_kvm_cpu_feat(vcpu->kvm, KVM_S390_VM_CPU_FEAT_ASTFLEIE2);
u32 fac = READ_ONCE(vsie_page->scb_o->fac);
int format_mask, format;
u32 origin;
/* assert no overflow with maximum len */
BUILD_BUG_ON(sizeof(vsie_page->fac) < ((S390_ARCH_FAC_LIST_SIZE_U64 + 1) * sizeof(u64)));
BUILD_BUG_ON(!IS_ALIGNED(offsetof(struct vsie_page, fac), 8));
/*
* Alternate-STFLE-Interpretive-Execution facilities are not supported
* -> format-0 flcb
*/
if (fac && test_kvm_facility(vcpu->kvm, 7)) {
retry_vsie_icpt(vsie_page);
/*
* The facility list origin (FLO) is in bits 1 - 28 of the FLD
* so we need to mask here before reading.
*/
fac = fac & 0x7ffffff8U;
/*
* format-0 -> size of nested guest's facility list == guest's size
* guest's size == host's size, since STFLE is interpretatively executed
* using a format-0 for the guest, too.
*/
if (read_guest_real(vcpu, fac, &vsie_page->fac,
stfle_size() * sizeof(u64)))
return set_validity_icpt(scb_s, 0x1090U);
scb_s->fac = (u32)virt_to_phys(&vsie_page->fac);
origin = fac & 0x7ffffff8U;
format_mask = has_astfleie2 ? 3 : 0;
format = fac & format_mask;
switch (format) {
case 0:
return handle_stfle_0(vcpu, vsie_page, origin);
case 1:
return set_validity_icpt(&vsie_page->scb_s, 0x1330U);
case 2:
return handle_stfle_2(vcpu, vsie_page, origin);
case 3:
return set_validity_icpt(&vsie_page->scb_s, 0x1330U);
}
}
return 0;
}

6
arch/s390/mm/gmap_helpers.c
View File

@ -51,15 +51,15 @@ pte_t *try_get_locked_pte(struct mm_struct *mm, unsigned long vmaddr, spinlock_t
pgd = pgdp_get(pgdp);
if (pgd_none(pgd) || !pgd_present(pgd))
return NULL;
p4dp = p4d_offset(pgdp, vmaddr);
p4dp = p4d_offset_lockless(pgdp, pgd, vmaddr);
p4d = p4dp_get(p4dp);
if (p4d_none(p4d) || !p4d_present(p4d))
return NULL;
pudp = pud_offset(p4dp, vmaddr);
pudp = pud_offset_lockless(p4dp, p4d, vmaddr);
pud = pudp_get(pudp);
if (pud_none(pud) || pud_leaf(pud) || !pud_present(pud))
return NULL;
pmdp = pmd_offset(pudp, vmaddr);
pmdp = pmd_offset_lockless(pudp, pud, vmaddr);
pmd = pmdp_get_lockless(pmdp);
if (pmd_none(pmd) || pmd_leaf(pmd) || !pmd_present(pmd))
return NULL;

1
arch/x86/include/asm/cpufeatures.h
View File

@ -379,6 +379,7 @@
#define X86_FEATURE_AVIC (15*32+13) /* "avic" Virtual Interrupt Controller */
#define X86_FEATURE_V_VMSAVE_VMLOAD (15*32+15) /* "v_vmsave_vmload" Virtual VMSAVE VMLOAD */
#define X86_FEATURE_VGIF (15*32+16) /* "vgif" Virtual GIF */
#define X86_FEATURE_GMET (15*32+17) /* Guest Mode Execution Trap */
#define X86_FEATURE_X2AVIC (15*32+18) /* "x2avic" Virtual x2apic */
#define X86_FEATURE_V_SPEC_CTRL (15*32+20) /* "v_spec_ctrl" Virtual SPEC_CTRL */
#define X86_FEATURE_VNMI (15*32+25) /* "vnmi" Virtual NMI */

5
arch/x86/include/asm/kvm-x86-ops.h
View File

@ -94,11 +94,10 @@ KVM_X86_OP_OPTIONAL(sync_pir_to_irr)
KVM_X86_OP_OPTIONAL_RET0(set_tss_addr)
KVM_X86_OP_OPTIONAL_RET0(set_identity_map_addr)
KVM_X86_OP_OPTIONAL_RET0(get_mt_mask)
KVM_X86_OP_OPTIONAL_RET0(tdp_has_smep)
KVM_X86_OP(load_mmu_pgd)
KVM_X86_OP_OPTIONAL(link_external_spt)
KVM_X86_OP_OPTIONAL(set_external_spte)
KVM_X86_OP_OPTIONAL_RET0(set_external_spte)
KVM_X86_OP_OPTIONAL(free_external_spt)
KVM_X86_OP_OPTIONAL(remove_external_spte)
KVM_X86_OP(has_wbinvd_exit)
KVM_X86_OP(get_l2_tsc_offset)
KVM_X86_OP(get_l2_tsc_multiplier)

5
arch/x86/include/asm/kvm-x86-pmu-ops.h
View File

@ -1,6 +1,7 @@
/* SPDX-License-Identifier: GPL-2.0 */
#if !defined(KVM_X86_PMU_OP) || \
!defined(KVM_X86_PMU_OP_OPTIONAL)
!defined(KVM_X86_PMU_OP_OPTIONAL) || \
!defined(KVM_X86_PMU_OP_OPTIONAL_RET0)
#error Missing one or more KVM_X86_PMU_OP #defines
#else
@ -23,6 +24,7 @@ KVM_X86_PMU_OP(init)
KVM_X86_PMU_OP_OPTIONAL(reset)
KVM_X86_PMU_OP_OPTIONAL(deliver_pmi)
KVM_X86_PMU_OP_OPTIONAL(cleanup)
KVM_X86_PMU_OP_OPTIONAL_RET0(pmc_is_disabled_in_current_mode)
KVM_X86_PMU_OP_OPTIONAL(write_global_ctrl)
KVM_X86_PMU_OP(mediated_load)
@ -31,3 +33,4 @@ KVM_X86_PMU_OP(mediated_put)
#undef KVM_X86_PMU_OP
#undef KVM_X86_PMU_OP_OPTIONAL
#undef KVM_X86_PMU_OP_OPTIONAL_RET0

147
arch/x86/include/asm/kvm_host.h
View File

@ -182,20 +182,21 @@ enum kvm_reg {
VCPU_REGS_RSI = __VCPU_REGS_RSI,
VCPU_REGS_RDI = __VCPU_REGS_RDI,
#ifdef CONFIG_X86_64
VCPU_REGS_R8 = __VCPU_REGS_R8,
VCPU_REGS_R9 = __VCPU_REGS_R9,
VCPU_REGS_R10 = __VCPU_REGS_R10,
VCPU_REGS_R11 = __VCPU_REGS_R11,
VCPU_REGS_R12 = __VCPU_REGS_R12,
VCPU_REGS_R13 = __VCPU_REGS_R13,
VCPU_REGS_R14 = __VCPU_REGS_R14,
VCPU_REGS_R15 = __VCPU_REGS_R15,
VCPU_REGS_R8 = 8,
VCPU_REGS_R9,
VCPU_REGS_R10,
VCPU_REGS_R11,
VCPU_REGS_R12,
VCPU_REGS_R13,
VCPU_REGS_R14,
VCPU_REGS_R15,
#endif
VCPU_REGS_RIP,
NR_VCPU_REGS,
NR_VCPU_GENERAL_PURPOSE_REGS,
VCPU_EXREG_PDPTR = NR_VCPU_REGS,
VCPU_EXREG_CR0,
VCPU_REG_RIP = NR_VCPU_GENERAL_PURPOSE_REGS,
VCPU_REG_PDPTR,
VCPU_REG_CR0,
/*
* Alias AMD's ERAPS (not a real register) to CR3 so that common code
* can trigger emulation of the RAP (Return Address Predictor) with
@ -203,13 +204,15 @@ enum kvm_reg {
* is cleared on writes to CR3, i.e. marking CR3 dirty will naturally
* mark ERAPS dirty as well.
*/
VCPU_EXREG_CR3,
VCPU_EXREG_ERAPS = VCPU_EXREG_CR3,
VCPU_EXREG_CR4,
VCPU_EXREG_RFLAGS,
VCPU_EXREG_SEGMENTS,
VCPU_EXREG_EXIT_INFO_1,
VCPU_EXREG_EXIT_INFO_2,
VCPU_REG_CR3,
VCPU_REG_ERAPS = VCPU_REG_CR3,
VCPU_REG_CR4,
VCPU_REG_RFLAGS,
VCPU_REG_SEGMENTS,
VCPU_REG_EXIT_INFO_1,
VCPU_REG_EXIT_INFO_2,
NR_VCPU_TOTAL_REGS,
};
enum {
@ -281,6 +284,8 @@ enum x86_intercept_stage;
#define PFERR_GUEST_RMP_MASK BIT_ULL(31)
#define PFERR_GUEST_FINAL_MASK BIT_ULL(32)
#define PFERR_GUEST_PAGE_MASK BIT_ULL(33)
#define PFERR_GUEST_FAULT_STAGE_MASK \
(PFERR_GUEST_FINAL_MASK | PFERR_GUEST_PAGE_MASK)
#define PFERR_GUEST_ENC_MASK BIT_ULL(34)
#define PFERR_GUEST_SIZEM_MASK BIT_ULL(35)
#define PFERR_GUEST_VMPL_MASK BIT_ULL(36)
@ -328,11 +333,11 @@ struct kvm_kernel_irq_routing_entry;
* the number of unique SPs that can theoretically be created is 2^n, where n
* is the number of bits that are used to compute the role.
*
* But, even though there are 20 bits in the mask below, not all combinations
* But, even though there are 21 bits in the mask below, not all combinations
* of modes and flags are possible:
*
* - invalid shadow pages are not accounted, mirror pages are not shadowed,
* so the bits are effectively 18.
* so the bits are effectively 19.
*
* - quadrant will only be used if has_4_byte_gpte=1 (non-PAE paging);
* execonly and ad_disabled are only used for nested EPT which has
@ -343,11 +348,11 @@ struct kvm_kernel_irq_routing_entry;
* paging has exactly one upper level, making level completely redundant
* when has_4_byte_gpte=1.
*
* - on top of this, smep_andnot_wp and smap_andnot_wp are only set if
* cr0_wp=0, therefore these three bits only give rise to 5 possibilities.
* - on top of this, smap_andnot_wp is only set if cr0_wp=0,
* therefore these two bits only give rise to 3 possibilities.
*
* Therefore, the maximum number of possible upper-level shadow pages for a
* single gfn is a bit less than 2^13.
* single gfn is a bit less than 2^14.
*/
union kvm_mmu_page_role {
u32 word;
@ -356,17 +361,26 @@ union kvm_mmu_page_role {
unsigned has_4_byte_gpte:1;
unsigned quadrant:2;
unsigned direct:1;
unsigned access:3;
unsigned access:4;
unsigned invalid:1;
unsigned efer_nx:1;
unsigned cr0_wp:1;
unsigned smep_andnot_wp:1;
unsigned smap_andnot_wp:1;
unsigned ad_disabled:1;
unsigned guest_mode:1;
unsigned passthrough:1;
unsigned is_mirror:1;
unsigned :4;
/*
* cr4_smep is also set for EPT MBEC. Because it affects
* which pages are considered non-present (bit 10 additionally
* must be zero if MBEC is on) it has to be in the base role.
* It also has to be in the base role for AMD GMET because
* kernel-executable pages need to have U=0 with GMET enabled.
*/
unsigned cr4_smep:1;
unsigned:3;
/*
* This is left at the top of the word so that
@ -392,10 +406,10 @@ union kvm_mmu_page_role {
* tables (because KVM doesn't support Protection Keys with shadow paging), and
* CR0.PG, CR4.PAE, and CR4.PSE are indirectly reflected in role.level.
*
* Note, SMEP and SMAP are not redundant with sm*p_andnot_wp in the page role.
* If CR0.WP=1, KVM can reuse shadow pages for the guest regardless of SMEP and
* SMAP, but the MMU's permission checks for software walks need to be SMEP and
* SMAP aware regardless of CR0.WP.
* Note, SMAP is not redundant with smap_andnot_wp in the page role. If
* CR0.WP=1, KVM can reuse shadow pages for the guest regardless of SMAP,
* but the MMU's permission checks for software walks need to be SMAP
* aware regardless of CR0.WP.
*/
union kvm_mmu_extended_role {
u32 word;
@ -405,9 +419,15 @@ union kvm_mmu_extended_role {
unsigned int cr4_pse:1;
unsigned int cr4_pke:1;
unsigned int cr4_smap:1;
unsigned int cr4_smep:1;
unsigned int cr4_la57:1;
unsigned int efer_lma:1;
/*
* True if either CR4.SMEP or EFER.NXE are set. For AMD NPT
* this is the "real" host CR4.SMEP whereas cr4_smep is
* actually GMET.
*/
unsigned int has_pferr_fetch:1;
};
};
@ -466,7 +486,8 @@ struct kvm_mmu {
u64 (*get_pdptr)(struct kvm_vcpu *vcpu, int index);
int (*page_fault)(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault);
void (*inject_page_fault)(struct kvm_vcpu *vcpu,
struct x86_exception *fault);
struct x86_exception *fault,
bool from_hardware);
gpa_t (*gva_to_gpa)(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu,
gpa_t gva_or_gpa, u64 access,
struct x86_exception *exception);
@ -492,7 +513,7 @@ struct kvm_mmu {
* Byte index: page fault error code [4:1]
* Bit index: pte permissions in ACC_* format
*/
u8 permissions[16];
u16 permissions[16];
u64 *pae_root;
u64 *pml4_root;
@ -504,10 +525,7 @@ struct kvm_mmu {
* the bits spte never used.
*/
struct rsvd_bits_validate shadow_zero_check;
struct rsvd_bits_validate guest_rsvd_check;
u64 pdptrs[4]; /* pae */
};
enum pmc_type {
@ -594,6 +612,8 @@ struct kvm_pmu {
DECLARE_BITMAP(pmc_counting_instructions, X86_PMC_IDX_MAX);
DECLARE_BITMAP(pmc_counting_branches, X86_PMC_IDX_MAX);
DECLARE_BITMAP(pmc_has_mode_specific_enables, X86_PMC_IDX_MAX);
u64 ds_area;
u64 pebs_enable;
u64 pebs_enable_rsvd;
@ -799,9 +819,10 @@ struct kvm_vcpu_arch {
* rip and regs accesses must go through
* kvm_{register,rip}_{read,write} functions.
*/
unsigned long regs[NR_VCPU_REGS];
u32 regs_avail;
u32 regs_dirty;
unsigned long regs[NR_VCPU_GENERAL_PURPOSE_REGS];
unsigned long rip;
DECLARE_BITMAP(regs_avail, NR_VCPU_TOTAL_REGS);
DECLARE_BITMAP(regs_dirty, NR_VCPU_TOTAL_REGS);
unsigned long cr0;
unsigned long cr0_guest_owned_bits;
@ -864,6 +885,8 @@ struct kvm_vcpu_arch {
*/
struct kvm_mmu *walk_mmu;
u64 pdptrs[4]; /* pae */
struct kvm_mmu_memory_cache mmu_pte_list_desc_cache;
struct kvm_mmu_memory_cache mmu_shadow_page_cache;
struct kvm_mmu_memory_cache mmu_shadowed_info_cache;
@ -1038,8 +1061,6 @@ struct kvm_vcpu_arch {
u16 vec;
u32 id;
u32 host_apf_flags;
bool send_always;
bool delivery_as_pf_vmexit;
bool pageready_pending;
} apf;
@ -1422,6 +1443,7 @@ struct kvm_arch {
bool has_private_mem;
bool has_protected_state;
bool has_protected_eoi;
bool has_protected_pmu;
bool pre_fault_allowed;
struct hlist_head *mmu_page_hash;
struct list_head active_mmu_pages;
@ -1888,24 +1910,18 @@ struct kvm_x86_ops {
int (*set_tss_addr)(struct kvm *kvm, unsigned int addr);
int (*set_identity_map_addr)(struct kvm *kvm, u64 ident_addr);
u8 (*get_mt_mask)(struct kvm_vcpu *vcpu, gfn_t gfn, bool is_mmio);
bool (*tdp_has_smep)(struct kvm *kvm);
void (*load_mmu_pgd)(struct kvm_vcpu *vcpu, hpa_t root_hpa,
int root_level);
/* Update external mapping with page table link. */
int (*link_external_spt)(struct kvm *kvm, gfn_t gfn, enum pg_level level,
void *external_spt);
/* Update the external page table from spte getting set. */
int (*set_external_spte)(struct kvm *kvm, gfn_t gfn, enum pg_level level,
u64 mirror_spte);
int (*set_external_spte)(struct kvm *kvm, gfn_t gfn, u64 old_spte,
u64 new_spte, enum pg_level level);
/* Update external page tables for page table about to be freed. */
int (*free_external_spt)(struct kvm *kvm, gfn_t gfn, enum pg_level level,
void *external_spt);
void (*free_external_spt)(struct kvm *kvm, struct kvm_mmu_page *sp);
/* Update external page table from spte getting removed, and flush TLB. */
void (*remove_external_spte)(struct kvm *kvm, gfn_t gfn, enum pg_level level,
u64 mirror_spte);
bool (*has_wbinvd_exit)(void);
@ -2011,6 +2027,10 @@ struct kvm_x86_nested_ops {
struct kvm_nested_state *kvm_state);
bool (*get_nested_state_pages)(struct kvm_vcpu *vcpu);
int (*write_log_dirty)(struct kvm_vcpu *vcpu, gpa_t l2_gpa);
gpa_t (*translate_nested_gpa)(struct kvm_vcpu *vcpu, gpa_t gpa,
u64 access,
struct x86_exception *exception,
u64 pte_access);
int (*enable_evmcs)(struct kvm_vcpu *vcpu,
uint16_t *vmcs_version);
@ -2042,7 +2062,6 @@ extern bool __read_mostly enable_device_posted_irqs;
extern struct kvm_x86_ops kvm_x86_ops;
#define kvm_x86_call(func) static_call(kvm_x86_##func)
#define kvm_pmu_call(func) static_call(kvm_x86_pmu_##func)
#define KVM_X86_OP(func) \
DECLARE_STATIC_CALL(kvm_x86_##func, *(((struct kvm_x86_ops *)0)->func));
@ -2131,8 +2150,6 @@ int load_pdptrs(struct kvm_vcpu *vcpu, unsigned long cr3);
extern bool tdp_enabled;
u64 vcpu_tsc_khz(struct kvm_vcpu *vcpu);
/*
* EMULTYPE_NO_DECODE - Set when re-emulating an instruction (after completing
* userspace I/O) to indicate that the emulation context
@ -2252,7 +2269,6 @@ int kvm_emulate_wbinvd(struct kvm_vcpu *vcpu);
void kvm_get_segment(struct kvm_vcpu *vcpu, struct kvm_segment *var, int seg);
void kvm_set_segment(struct kvm_vcpu *vcpu, struct kvm_segment *var, int seg);
int kvm_load_segment_descriptor(struct kvm_vcpu *vcpu, u16 selector, int seg);
void kvm_vcpu_deliver_sipi_vector(struct kvm_vcpu *vcpu, u8 vector);
int kvm_task_switch(struct kvm_vcpu *vcpu, u16 tss_selector, int idt_index,
@ -2283,10 +2299,18 @@ void kvm_queue_exception_e(struct kvm_vcpu *vcpu, unsigned nr, u32 error_code);
void kvm_queue_exception_p(struct kvm_vcpu *vcpu, unsigned nr, unsigned long payload);
void kvm_requeue_exception(struct kvm_vcpu *vcpu, unsigned int nr,
bool has_error_code, u32 error_code);
void kvm_inject_page_fault(struct kvm_vcpu *vcpu, struct x86_exception *fault);
void kvm_inject_emulated_page_fault(struct kvm_vcpu *vcpu,
struct x86_exception *fault);
bool kvm_require_cpl(struct kvm_vcpu *vcpu, int required_cpl);
void kvm_inject_page_fault(struct kvm_vcpu *vcpu, struct x86_exception *fault,
bool from_hardware);
void __kvm_inject_emulated_page_fault(struct kvm_vcpu *vcpu,
struct x86_exception *fault,
bool from_hardware);
static inline void kvm_inject_emulated_page_fault(struct kvm_vcpu *vcpu,
struct x86_exception *fault)
{
__kvm_inject_emulated_page_fault(vcpu, fault, false);
}
bool kvm_require_dr(struct kvm_vcpu *vcpu, int dr);
static inline int __kvm_irq_line_state(unsigned long *irq_state,
@ -2527,7 +2551,8 @@ int memslot_rmap_alloc(struct kvm_memory_slot *slot, unsigned long npages);
KVM_X86_QUIRK_SLOT_ZAP_ALL | \
KVM_X86_QUIRK_STUFF_FEATURE_MSRS | \
KVM_X86_QUIRK_IGNORE_GUEST_PAT | \
KVM_X86_QUIRK_VMCS12_ALLOW_FREEZE_IN_SMM)
KVM_X86_QUIRK_VMCS12_ALLOW_FREEZE_IN_SMM | \
KVM_X86_QUIRK_NESTED_SVM_SHARED_PAT)
#define KVM_X86_CONDITIONAL_QUIRKS \
(KVM_X86_QUIRK_CD_NW_CLEARED | \

109
arch/x86/include/asm/kvm_vcpu_regs.h
View File

@ -11,15 +11,108 @@
#define __VCPU_REGS_RSI 6
#define __VCPU_REGS_RDI 7
#ifdef __ASSEMBLER__
#define REG_NUM_INVALID 100
# convert the 32-bit register operand \r32 into a register number;
# store it in the value whose name is in \opd.
# only for !CONFIG_X86_64 (does not support r8d-r15d)
.macro R32_NUM opd r32
\opd = REG_NUM_INVALID
.ifc \r32,%eax
\opd = __VCPU_REGS_RAX
.endif
.ifc \r32,%ecx
\opd = __VCPU_REGS_RCX
.endif
.ifc \r32,%edx
\opd = __VCPU_REGS_RDX
.endif
.ifc \r32,%ebx
\opd = __VCPU_REGS_RBX
.endif
.ifc \r32,%esp
\opd = __VCPU_REGS_RSP
.endif
.ifc \r32,%ebp
\opd = __VCPU_REGS_RBP
.endif
.ifc \r32,%esi
\opd = __VCPU_REGS_RSI
.endif
.ifc \r32,%edi
\opd = __VCPU_REGS_RDI
.endif
.endm
# convert the 64-bit register operand \r64 into a register number;
# store it in the value whose name is in \opd.
.macro R64_NUM opd r64
\opd = REG_NUM_INVALID
#ifdef CONFIG_X86_64
#define __VCPU_REGS_R8 8
#define __VCPU_REGS_R9 9
#define __VCPU_REGS_R10 10
#define __VCPU_REGS_R11 11
#define __VCPU_REGS_R12 12
#define __VCPU_REGS_R13 13
#define __VCPU_REGS_R14 14
#define __VCPU_REGS_R15 15
.ifc \r64,%rax
\opd = __VCPU_REGS_RAX
.endif
.ifc \r64,%rcx
\opd = __VCPU_REGS_RCX
.endif
.ifc \r64,%rdx
\opd = __VCPU_REGS_RDX
.endif
.ifc \r64,%rbx
\opd = __VCPU_REGS_RBX
.endif
.ifc \r64,%rsp
\opd = __VCPU_REGS_RSP
.endif
.ifc \r64,%rbp
\opd = __VCPU_REGS_RBP
.endif
.ifc \r64,%rsi
\opd = __VCPU_REGS_RSI
.endif
.ifc \r64,%rdi
\opd = __VCPU_REGS_RDI
.endif
.ifc \r64,%r8
\opd = 8
.endif
.ifc \r64,%r9
\opd = 9
.endif
.ifc \r64,%r10
\opd = 10
.endif
.ifc \r64,%r11
\opd = 11
.endif
.ifc \r64,%r12
\opd = 12
.endif
.ifc \r64,%r13
\opd = 13
.endif
.ifc \r64,%r14
\opd = 14
.endif
.ifc \r64,%r15
\opd = 15
.endif
#endif
.endm
.macro REG_NUM reg_num reg
#ifdef CONFIG_X86_64
R64_NUM \reg_num \reg
#else
R32_NUM \reg_num \reg
#endif
.if \reg_num == REG_NUM_INVALID
.error "invalid register"
.endif
.endm
#endif
#endif /* _ASM_X86_KVM_VCPU_REGS_H */

1
arch/x86/include/asm/msr-index.h
View File

@ -899,6 +899,7 @@
#define MSR_K7_HWCR_IRPERF_EN_BIT 30
#define MSR_K7_HWCR_IRPERF_EN BIT_ULL(MSR_K7_HWCR_IRPERF_EN_BIT)
#define MSR_K7_HWCR_CPUID_USER_DIS_BIT 35
#define MSR_K7_HWCR_CPUID_USER_DIS BIT_ULL(MSR_K7_HWCR_CPUID_USER_DIS_BIT)
#define MSR_K7_FID_VID_CTL 0xc0010041
#define MSR_K7_FID_VID_STATUS 0xc0010042
#define MSR_K7_HWCR_CPB_DIS_BIT 25

2
arch/x86/include/asm/perf_event.h
View File

@ -60,6 +60,8 @@
#define AMD64_EVENTSEL_INT_CORE_ENABLE (1ULL << 36)
#define AMD64_EVENTSEL_GUESTONLY (1ULL << 40)
#define AMD64_EVENTSEL_HOSTONLY (1ULL << 41)
#define AMD64_EVENTSEL_HOST_GUEST_MASK \
(AMD64_EVENTSEL_HOSTONLY | AMD64_EVENTSEL_GUESTONLY)
#define AMD64_EVENTSEL_INT_CORE_SEL_SHIFT 37
#define AMD64_EVENTSEL_INT_CORE_SEL_MASK \

1
arch/x86/include/asm/svm.h
View File

@ -243,6 +243,7 @@ struct __attribute__ ((__packed__)) vmcb_control_area {
#define SVM_MISC_ENABLE_NP BIT(0)
#define SVM_MISC_ENABLE_SEV BIT(1)
#define SVM_MISC_ENABLE_SEV_ES BIT(2)
#define SVM_MISC_ENABLE_GMET BIT(3)
#define SVM_MISC2_ENABLE_V_LBR BIT_ULL(0)
#define SVM_MISC2_ENABLE_V_VMLOAD_VMSAVE BIT_ULL(1)

34
arch/x86/include/asm/tdx.h
View File

@ -6,6 +6,7 @@
#include <linux/init.h>
#include <linux/bits.h>
#include <linux/mmzone.h>
#include <linux/kvm_types.h>
#include <asm/errno.h>
#include <asm/ptrace.h>
@ -121,7 +122,7 @@ int tdx_guest_keyid_alloc(void);
u32 tdx_get_nr_guest_keyids(void);
void tdx_guest_keyid_free(unsigned int keyid);
void tdx_quirk_reset_page(struct page *page);
void tdx_quirk_reset_paddr(unsigned long base, unsigned long size);
struct tdx_td {
/* TD root structure: */
@ -145,32 +146,17 @@ struct tdx_vp {
struct page **tdcx_pages;
};
static inline u64 mk_keyed_paddr(u16 hkid, struct page *page)
{
u64 ret;
ret = page_to_phys(page);
/* KeyID bits are just above the physical address bits: */
ret |= (u64)hkid << boot_cpu_data.x86_phys_bits;
return ret;
}
static inline int pg_level_to_tdx_sept_level(enum pg_level level)
{
WARN_ON_ONCE(level == PG_LEVEL_NONE);
return level - 1;
}
void tdx_sys_disable(void);
u64 tdh_vp_enter(struct tdx_vp *vp, struct tdx_module_args *args);
u64 tdh_mng_addcx(struct tdx_td *td, struct page *tdcs_page);
u64 tdh_mem_page_add(struct tdx_td *td, u64 gpa, struct page *page, struct page *source, u64 *ext_err1, u64 *ext_err2);
u64 tdh_mem_sept_add(struct tdx_td *td, u64 gpa, int level, struct page *page, u64 *ext_err1, u64 *ext_err2);
u64 tdh_mem_page_add(struct tdx_td *td, u64 gpa, kvm_pfn_t pfn, struct page *source,
u64 *ext_err1, u64 *ext_err2);
u64 tdh_mem_sept_add(struct tdx_td *td, u64 gpa, enum pg_level level, struct page *page, u64 *ext_err1, u64 *ext_err2);
u64 tdh_vp_addcx(struct tdx_vp *vp, struct page *tdcx_page);
u64 tdh_mem_page_aug(struct tdx_td *td, u64 gpa, int level, struct page *page, u64 *ext_err1, u64 *ext_err2);
u64 tdh_mem_range_block(struct tdx_td *td, u64 gpa, int level, u64 *ext_err1, u64 *ext_err2);
u64 tdh_mem_page_aug(struct tdx_td *td, u64 gpa, enum pg_level level, kvm_pfn_t pfn,
u64 *ext_err1, u64 *ext_err2);
u64 tdh_mem_range_block(struct tdx_td *td, u64 gpa, enum pg_level level, u64 *ext_err1, u64 *ext_err2);
u64 tdh_mng_key_config(struct tdx_td *td);
u64 tdh_mng_create(struct tdx_td *td, u16 hkid);
u64 tdh_vp_create(struct tdx_td *td, struct tdx_vp *vp);
@ -186,10 +172,10 @@ u64 tdh_vp_rd(struct tdx_vp *vp, u64 field, u64 *data);
u64 tdh_vp_wr(struct tdx_vp *vp, u64 field, u64 data, u64 mask);
u64 tdh_phymem_page_reclaim(struct page *page, u64 *tdx_pt, u64 *tdx_owner, u64 *tdx_size);
u64 tdh_mem_track(struct tdx_td *tdr);
u64 tdh_mem_page_remove(struct tdx_td *td, u64 gpa, u64 level, u64 *ext_err1, u64 *ext_err2);
u64 tdh_mem_page_remove(struct tdx_td *td, u64 gpa, enum pg_level level, u64 *ext_err1, u64 *ext_err2);
u64 tdh_phymem_cache_wb(bool resume);
u64 tdh_phymem_page_wbinvd_tdr(struct tdx_td *td);
u64 tdh_phymem_page_wbinvd_hkid(u64 hkid, struct page *page);
u64 tdh_phymem_page_wbinvd_hkid(u64 hkid, kvm_pfn_t pfn);
#else
static inline void tdx_init(void) { }
static inline u32 tdx_get_nr_guest_keyids(void) { return 0; }

14
arch/x86/include/asm/vmx.h
View File

@ -536,6 +536,7 @@ enum vmcs_field {
#define VMX_EPT_1GB_PAGE_BIT (1ull << 17)
#define VMX_EPT_INVEPT_BIT (1ull << 20)
#define VMX_EPT_AD_BIT (1ull << 21)
#define VMX_EPT_ADVANCED_VMEXIT_INFO_BIT (1ull << 22)
#define VMX_EPT_EXTENT_CONTEXT_BIT (1ull << 25)
#define VMX_EPT_EXTENT_GLOBAL_BIT (1ull << 26)
@ -561,10 +562,12 @@ enum vmcs_field {
#define VMX_EPT_ACCESS_BIT (1ull << 8)
#define VMX_EPT_DIRTY_BIT (1ull << 9)
#define VMX_EPT_SUPPRESS_VE_BIT (1ull << 63)
#define VMX_EPT_RWX_MASK (VMX_EPT_READABLE_MASK | \
VMX_EPT_WRITABLE_MASK | \
VMX_EPT_EXECUTABLE_MASK)
#define VMX_EPT_MT_MASK (7ull << VMX_EPT_MT_EPTE_SHIFT)
#define VMX_EPT_USER_EXECUTABLE_MASK (1ull << 10)
static inline u8 vmx_eptp_page_walk_level(u64 eptp)
{
@ -609,17 +612,24 @@ enum vm_entry_failure_code {
#define EPT_VIOLATION_PROT_READ BIT(3)
#define EPT_VIOLATION_PROT_WRITE BIT(4)
#define EPT_VIOLATION_PROT_EXEC BIT(5)
#define EPT_VIOLATION_EXEC_FOR_RING3_LIN BIT(6)
#define EPT_VIOLATION_PROT_USER_EXEC BIT(6)
#define EPT_VIOLATION_PROT_MASK (EPT_VIOLATION_PROT_READ | \
EPT_VIOLATION_PROT_WRITE | \
EPT_VIOLATION_PROT_EXEC)
EPT_VIOLATION_PROT_EXEC | \
EPT_VIOLATION_PROT_USER_EXEC)
#define EPT_VIOLATION_GVA_IS_VALID BIT(7)
#define EPT_VIOLATION_GVA_TRANSLATED BIT(8)
#define EPT_VIOLATION_GVA_USER BIT(9)
#define EPT_VIOLATION_GVA_WRITABLE BIT(10)
#define EPT_VIOLATION_GVA_NX BIT(11)
#define EPT_VIOLATION_RWX_TO_PROT(__epte) (((__epte) & VMX_EPT_RWX_MASK) << 3)
#define EPT_VIOLATION_USER_EXEC_TO_PROT(__epte) (((__epte) & VMX_EPT_USER_EXECUTABLE_MASK) >> 4)
static_assert(EPT_VIOLATION_RWX_TO_PROT(VMX_EPT_RWX_MASK) ==
(EPT_VIOLATION_PROT_READ | EPT_VIOLATION_PROT_WRITE | EPT_VIOLATION_PROT_EXEC));
static_assert(EPT_VIOLATION_USER_EXEC_TO_PROT(VMX_EPT_USER_EXECUTABLE_MASK) ==
(EPT_VIOLATION_PROT_USER_EXEC));
/*
* Exit Qualifications for NOTIFY VM EXIT

2
arch/x86/include/uapi/asm/kvm.h
View File

@ -477,6 +477,7 @@ struct kvm_sync_regs {
#define KVM_X86_QUIRK_STUFF_FEATURE_MSRS (1 << 8)
#define KVM_X86_QUIRK_IGNORE_GUEST_PAT (1 << 9)
#define KVM_X86_QUIRK_VMCS12_ALLOW_FREEZE_IN_SMM (1 << 10)
#define KVM_X86_QUIRK_NESTED_SVM_SHARED_PAT (1 << 11)
#define KVM_STATE_NESTED_FORMAT_VMX 0
#define KVM_STATE_NESTED_FORMAT_SVM 1
@ -532,6 +533,7 @@ struct kvm_svm_nested_state_data {
struct kvm_svm_nested_state_hdr {
__u64 vmcb_pa;
__u64 gpat;
};
/* for KVM_CAP_NESTED_STATE */

21
arch/x86/kvm/cpuid.c
View File

@ -11,7 +11,7 @@
#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
#include <linux/kvm_host.h>
#include "linux/lockdep.h"
#include <linux/lockdep.h>
#include <linux/export.h>
#include <linux/vmalloc.h>
#include <linux/uaccess.h>
@ -1248,7 +1248,7 @@ void kvm_initialize_cpu_caps(void)
F(AUTOIBRS),
EMULATED_F(NO_SMM_CTL_MSR),
/* PrefetchCtlMsr */
/* GpOnUserCpuid */
EMULATED_F(GP_ON_USER_CPUID),
/* EPSF */
F(PREFETCHI),
F(AVX512_BMM),
@ -2161,17 +2161,18 @@ int kvm_emulate_cpuid(struct kvm_vcpu *vcpu)
{
u32 eax, ebx, ecx, edx;
if (!is_smm(vcpu) && cpuid_fault_enabled(vcpu) &&
!kvm_require_cpl(vcpu, 0))
if (!kvm_is_cpuid_allowed(vcpu)) {
kvm_queue_exception_e(vcpu, GP_VECTOR, 0);
return 1;
}
eax = kvm_rax_read(vcpu);
ecx = kvm_rcx_read(vcpu);
eax = kvm_eax_read(vcpu);
ecx = kvm_ecx_read(vcpu);
kvm_cpuid(vcpu, &eax, &ebx, &ecx, &edx, false);
kvm_rax_write(vcpu, eax);
kvm_rbx_write(vcpu, ebx);
kvm_rcx_write(vcpu, ecx);
kvm_rdx_write(vcpu, edx);
kvm_eax_write(vcpu, eax);
kvm_ebx_write(vcpu, ebx);
kvm_ecx_write(vcpu, ecx);
kvm_edx_write(vcpu, edx);
return kvm_skip_emulated_instruction(vcpu);
}
EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_emulate_cpuid);

20
arch/x86/kvm/cpuid.h
View File

@ -7,6 +7,8 @@
#include <asm/processor.h>
#include <uapi/asm/kvm_para.h>
#include "smm.h"
extern u32 kvm_cpu_caps[NR_KVM_CPU_CAPS] __read_mostly;
extern bool kvm_is_configuring_cpu_caps __read_mostly;
@ -124,7 +126,7 @@ static __always_inline bool guest_cpuid_has(struct kvm_vcpu *vcpu,
entry = kvm_find_cpuid_entry_index(vcpu, cpuid.function, cpuid.index);
if (!entry)
return NULL;
return false;
reg = __cpuid_entry_get_reg(entry, cpuid.reg);
if (!reg)
@ -181,15 +183,17 @@ static inline int guest_cpuid_stepping(struct kvm_vcpu *vcpu)
return x86_stepping(best->eax);
}
static inline bool supports_cpuid_fault(struct kvm_vcpu *vcpu)
{
return vcpu->arch.msr_platform_info & MSR_PLATFORM_INFO_CPUID_FAULT;
}
static inline bool cpuid_fault_enabled(struct kvm_vcpu *vcpu)
{
return vcpu->arch.msr_misc_features_enables &
MSR_MISC_FEATURES_ENABLES_CPUID_FAULT;
return (vcpu->arch.msr_misc_features_enables &
MSR_MISC_FEATURES_ENABLES_CPUID_FAULT) ||
(vcpu->arch.msr_hwcr & MSR_K7_HWCR_CPUID_USER_DIS);
}
static inline bool kvm_is_cpuid_allowed(struct kvm_vcpu *vcpu)
{
return !cpuid_fault_enabled(vcpu) || is_smm(vcpu) ||
!kvm_x86_call(get_cpl)(vcpu);
}
static __always_inline void kvm_cpu_cap_clear(unsigned int x86_feature)

2
arch/x86/kvm/debugfs.c
View File

@ -121,7 +121,7 @@ static int kvm_mmu_rmaps_stat_show(struct seq_file *m, void *v)
lpage_size = kvm_mmu_slot_lpages(slot, k + 1);
cur = log[k];
for (l = 0; l < lpage_size; l++) {
index = ffs(pte_list_count(&rmap[l]));
index = fls(pte_list_count(&rmap[l]));
if (WARN_ON_ONCE(index >= RMAP_LOG_SIZE))
index = RMAP_LOG_SIZE - 1;
cur[index]++;

22
arch/x86/kvm/emulate.c
View File

@ -20,7 +20,7 @@
#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
#include <linux/kvm_host.h>
#include "kvm_cache_regs.h"
#include "regs.h"
#include "kvm_emulate.h"
#include <linux/stringify.h>
#include <asm/debugreg.h>
@ -540,8 +540,9 @@ static int emulate_exception(struct x86_emulate_ctxt *ctxt, int vec,
return X86EMUL_PROPAGATE_FAULT;
}
static int emulate_db(struct x86_emulate_ctxt *ctxt)
static int emulate_db(struct x86_emulate_ctxt *ctxt, unsigned long dr6)
{
ctxt->exception.dr6 = dr6;
return emulate_exception(ctxt, DB_VECTOR, 0, false);
}
@ -3593,12 +3594,8 @@ static int em_sti(struct x86_emulate_ctxt *ctxt)
static int em_cpuid(struct x86_emulate_ctxt *ctxt)
{
u32 eax, ebx, ecx, edx;
u64 msr = 0;
ctxt->ops->get_msr(ctxt, MSR_MISC_FEATURES_ENABLES, &msr);
if (!ctxt->ops->is_smm(ctxt) &&
(msr & MSR_MISC_FEATURES_ENABLES_CPUID_FAULT) &&
ctxt->ops->cpl(ctxt))
if (!ctxt->ops->is_cpuid_allowed(ctxt))
return emulate_gp(ctxt, 0);
eax = reg_read(ctxt, VCPU_REGS_RAX);
@ -3847,15 +3844,8 @@ static int check_dr_read(struct x86_emulate_ctxt *ctxt)
if ((cr4 & X86_CR4_DE) && (dr == 4 || dr == 5))
return emulate_ud(ctxt);
if (ctxt->ops->get_dr(ctxt, 7) & DR7_GD) {
ulong dr6;
dr6 = ctxt->ops->get_dr(ctxt, 6);
dr6 &= ~DR_TRAP_BITS;
dr6 |= DR6_BD | DR6_ACTIVE_LOW;
ctxt->ops->set_dr(ctxt, 6, dr6);
return emulate_db(ctxt);
}
if (ctxt->ops->get_effective_dr7(ctxt) & DR7_GD)
return emulate_db(ctxt, DR6_BD);
return X86EMUL_CONTINUE;
}

30
arch/x86/kvm/hyperv.c
View File

@ -1839,6 +1839,11 @@ static bool hv_is_vp_in_sparse_set(u32 vp_id, u64 valid_bank_mask, u64 sparse_ba
int valid_bit_nr = vp_id / HV_VCPUS_PER_SPARSE_BANK;
unsigned long sbank;
BUILD_BUG_ON(BITS_PER_TYPE(valid_bank_mask) != HV_MAX_SPARSE_VCPU_BANKS);
if (valid_bit_nr >= HV_MAX_SPARSE_VCPU_BANKS)
return false;
if (!test_bit(valid_bit_nr, (unsigned long *)&valid_bank_mask))
return false;
@ -2041,7 +2046,9 @@ static u64 kvm_hv_flush_tlb(struct kvm_vcpu *vcpu, struct kvm_hv_hcall *hc)
* read with kvm_read_guest().
*/
if (!hc->fast && mmu_is_nested(vcpu)) {
hc->ingpa = translate_nested_gpa(vcpu, hc->ingpa, 0, NULL);
hc->ingpa = kvm_x86_ops.nested_ops->translate_nested_gpa(
vcpu, hc->ingpa,
PFERR_GUEST_FINAL_MASK, NULL, 0);
if (unlikely(hc->ingpa == INVALID_GPA))
return HV_STATUS_INVALID_HYPERCALL_INPUT;
}
@ -2375,10 +2382,10 @@ static void kvm_hv_hypercall_set_result(struct kvm_vcpu *vcpu, u64 result)
longmode = is_64_bit_hypercall(vcpu);
if (longmode)
kvm_rax_write(vcpu, result);
kvm_rax_write_raw(vcpu, result);
else {
kvm_rdx_write(vcpu, result >> 32);
kvm_rax_write(vcpu, result & 0xffffffff);
kvm_edx_write(vcpu, result >> 32);
kvm_eax_write(vcpu, result);
}
}
@ -2542,18 +2549,15 @@ int kvm_hv_hypercall(struct kvm_vcpu *vcpu)
#ifdef CONFIG_X86_64
if (is_64_bit_hypercall(vcpu)) {
hc.param = kvm_rcx_read(vcpu);
hc.ingpa = kvm_rdx_read(vcpu);
hc.outgpa = kvm_r8_read(vcpu);
hc.param = kvm_rcx_read_raw(vcpu);
hc.ingpa = kvm_rdx_read_raw(vcpu);
hc.outgpa = kvm_r8_read_raw(vcpu);
} else
#endif
{
hc.param = ((u64)kvm_rdx_read(vcpu) << 32) |
(kvm_rax_read(vcpu) & 0xffffffff);
hc.ingpa = ((u64)kvm_rbx_read(vcpu) << 32) |
(kvm_rcx_read(vcpu) & 0xffffffff);
hc.outgpa = ((u64)kvm_rdi_read(vcpu) << 32) |
(kvm_rsi_read(vcpu) & 0xffffffff);
hc.param = ((u64)kvm_edx_read(vcpu) << 32) | kvm_eax_read(vcpu);
hc.ingpa = ((u64)kvm_ebx_read(vcpu) << 32) | kvm_ecx_read(vcpu);
hc.outgpa = ((u64)kvm_edi_read(vcpu) << 32) | kvm_esi_read(vcpu);
}
hc.code = hc.param & 0xffff;

4
arch/x86/kvm/hyperv.h
View File

@ -232,8 +232,8 @@ static inline bool kvm_hv_is_tlb_flush_hcall(struct kvm_vcpu *vcpu)
if (!hv_vcpu)
return false;
code = is_64_bit_hypercall(vcpu) ? kvm_rcx_read(vcpu) :
kvm_rax_read(vcpu);
code = is_64_bit_hypercall(vcpu) ? kvm_rcx_read_raw(vcpu) :
kvm_eax_read(vcpu);
return (code == HVCALL_FLUSH_VIRTUAL_ADDRESS_SPACE ||
code == HVCALL_FLUSH_VIRTUAL_ADDRESS_LIST ||

2
arch/x86/kvm/ioapic.c
View File

@ -441,7 +441,7 @@ static void ioapic_write_indirect(struct kvm_ioapic *ioapic, u32 val)
irq.dest_id = old_dest_id;
irq.dest_mode =
kvm_lapic_irq_dest_mode(
!!e->fields.dest_mode);
!!old_dest_mode);
kvm_bitmap_or_dest_vcpus(ioapic->kvm, &irq,
vcpu_bitmap);
}

4
arch/x86/kvm/irq.c
View File

@ -585,12 +585,16 @@ int kvm_vm_ioctl_get_irqchip(struct kvm *kvm, struct kvm_irqchip *chip)
r = 0;
switch (chip->chip_id) {
case KVM_IRQCHIP_PIC_MASTER:
spin_lock(&pic->lock);
memcpy(&chip->chip.pic, &pic->pics[0],
sizeof(struct kvm_pic_state));
spin_unlock(&pic->lock);
break;
case KVM_IRQCHIP_PIC_SLAVE:
spin_lock(&pic->lock);
memcpy(&chip->chip.pic, &pic->pics[1],
sizeof(struct kvm_pic_state));
spin_unlock(&pic->lock);
break;
case KVM_IRQCHIP_IOAPIC:
kvm_get_ioapic(kvm, &chip->chip.ioapic);

1
arch/x86/kvm/kvm-asm-offsets.c
View File

@ -24,6 +24,7 @@ static void __used common(void)
if (IS_ENABLED(CONFIG_KVM_INTEL)) {
BLANK();
OFFSET(VMX_vcpu_arch_regs, vcpu_vmx, vcpu.arch.regs);
OFFSET(VMX_spec_ctrl, vcpu_vmx, spec_ctrl);
}
}

Some files were not shown because too many files have changed in this diff Show More