Use kvm_is_vcpu_creation_in_progress() instead of an open-coded equivalent
during AIA initialization. Unlike similar vGIC code in arm64, the relevant
RISC-V code runs under kvm->lock, i.e. can use the standard API without
hitting lockdep false positive.
No functional change intended.
Signed-off-by: Sean Christopherson <seanjc@google.com>
Tested-by: Jean-Christophe Guillain <jean-christophe@guillain.net>
Tested-by: Naveen N Rao (AMD) <naveen@kernel.org>
Message-ID: <20260921174445.911676-4-seanjc@google.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
kvm_sbi_hsm_vcpu_get_status() returns SBI_ERR_INVALID_PARAM when
the requested hart does not exist. However, the HART_STATUS case
returns from the SBI handler without storing this error in
retdata->err_val.
As a result, a guest querying the status of a non-existent hart
observes SBI_SUCCESS instead of SBI_ERR_INVALID_PARAM.
Use the common SBI error handling path for HART_STATUS after
saving a valid hart state in retdata->out_val. This preserves
the returned error when kvm_sbi_hsm_vcpu_get_status() fails.
Fixes: bae0dfd74e ("RISC-V: KVM: Modify SBI extension handler to return SBI error code")
Cc: stable@vger.kernel.org
Assisted-by: LLM
Signed-off-by: Tan Chi <tanchi25@mails.ucas.ac.cn>
Reviewed-by: Anup Patel <anup@brainfault.org>
Link: https://lore.kernel.org/r/20260914031146.446157-1-tanchi25@mails.ucas.ac.cn
Signed-off-by: Anup Patel <anup@brainfault.org>
pmu_ctr_read() adds the event count returned by perf_event_read_value()
to counter_val, which can accumulate the same count repeatedly across
reads. kvm_riscv_vcpu_pmu_ctr_stop() also leaves counter_val stale by
not folding the current event count into it.
Make reads of perf-backed counters side-effect free, and use
perf_event_pause() when stopping a counter to fold the current event
count into counter_val while resetting it. This preserves the counter
value across stop/start and lets the snapshot path use counter_val
directly.
Fixes: 0cb74b65d2 ("RISC-V: KVM: Implement perf support without sampling")
Signed-off-by: SeungJu Cheon <suunj1331@gmail.com>
Reviewed-by: Anup Patel <anup@brainfault.org>
Link: https://lore.kernel.org/r/20260825083719.643970-4-suunj1331@gmail.com
Signed-off-by: Anup Patel <anup@brainfault.org>
If kvm_vcpu_write_guest() fails while updating the PMU snapshot area
on counter stop, the guest may receive SBI_SUCCESS without the
snapshot being updated, leaving stale data in shared memory.
Return SBI_ERR_FAILURE when the snapshot write fails.
Fixes: c2f41ddbcd ("RISC-V: KVM: Implement SBI PMU Snapshot feature")
Signed-off-by: SeungJu Cheon <suunj1331@gmail.com>
Reviewed-by: Anup Patel <anup@brainfault.org>
Link: https://lore.kernel.org/r/20260825083719.643970-3-suunj1331@gmail.com
Signed-off-by: Anup Patel <anup@brainfault.org>
Firmware events accumulate in kvpmu->fw_event[].value while running,
but counter stop only clears fw_event[].started without saving the
value back to pmc->counter_val. A subsequent counter start without
SBI_PMU_START_FLAG_SET_INIT_VALUE reloads the stale counter_val into
fw_event[].value, losing all events counted so far.
Save fw_event[].value into counter_val when actually stopping a
running counter, and remove the now redundant synchronization from
the snapshot path.
Fixes: badc386869 ("RISC-V: KVM: Support firmware events")
Signed-off-by: SeungJu Cheon <suunj1331@gmail.com>
Reviewed-by: Anup Patel <anup@brainfault.org>
Link: https://lore.kernel.org/r/20260825083719.643970-2-suunj1331@gmail.com
Signed-off-by: Anup Patel <anup@brainfault.org>
A guest may call SBI_PMU_SNAPSHOT_SET_SHMEM repeatedly. Each call
overwrites kvpmu->sdata without freeing the old buffer (memory leak),
and if a later kvm_vcpu_write_guest() fails, the error path frees
sdata but leaves snapshot_addr stale. A subsequent
SBI_PMU_COUNTER_START then passes the INVALID_GPA check and crashes
the host with a NULL buffer in kvm_vcpu_read_guest().
Fix this by clearing the previously installed snapshot area before
installing a new one, which keeps sdata and snapshot_addr consistent.
The SBI spec suggests a single invocation but defines no error code
for repeated calls, so KVM must tolerate them.
Fixes: c2f41ddbcd ("RISC-V: KVM: Implement SBI PMU Snapshot feature")
Cc: stable@vger.kernel.org
Signed-off-by: Zongmin Zhou <zhouzongmin@kylinos.cn>
Reviewed-by: Anup Patel <anup@brainfault.org>
Link: https://lore.kernel.org/r/20260826064117.58029-1-min_halo@163.com
Signed-off-by: Anup Patel <anup@brainfault.org>
The SBI v3.0 specification (section 15.1.2) requires a nested HFENCE
entry to be populated as follows:
1) find an unused entry with Config.Pending == 0
2) update the Page_Number and Page_Count words
3) update the Config word with Config.Pending set
__kvm_riscv_nacl_hfence() writes the Config word first, so the SBI
implementation (or NACL hardware) can observe a pending entry with
pnum/pcount values left over from the previous use of that entry,
resulting in incorrect TLB flush ranges.
Write pnum and pcount first and the Config word last. Since the
consumer is an external agent on coherent shared memory, use
WRITE_ONCE() to stop the compiler from reordering the stores and
smp_wmb() to make the parameter words globally visible before the
Pending bit is set.
Fixes: d466c19cea ("RISC-V: KVM: Add common nested acceleration support")
Signed-off-by: Zongmin Zhou <zhouzongmin@kylinos.cn>
Reviewed-by: Anup Patel <anup@brainfault.org>
Link: https://lore.kernel.org/r/20260826075009.68952-1-min_halo@163.com
Signed-off-by: Anup Patel <anup@brainfault.org>
__kvm_faultin_pfn() reports an interrupted host page fault with
KVM_PFN_ERR_SIGPENDING. RISC-V currently handles it as a generic
error PFN and returns -EFAULT.
Return -EINTR for the signal-pending sentinel so callers can distinguish
an interrupted fault from an invalid userspace mapping. Do not log the
expected interruption as a vCPU exit error.
Fixes: 9d05c1fee8 ("RISC-V: KVM: Implement stage2 page table programming")
Cc: stable@vger.kernel.org
Signed-off-by: Xie Bo <xb@ultrarisc.com>
Reviewed-by: Anup Patel <anup@brainfault.org>
Link: https://lore.kernel.org/r/20260810051544.3953925-3-xb@ultrarisc.com
Signed-off-by: Anup Patel <anup@brainfault.org>
If an MMU invalidation races with a G-stage fault, the fault handler skips
installing the page but leaves ret set to zero. As a result,
kvm_release_faultin_page() treats the page as used and can unnecessarily
mark it dirty.
Track the invalidation retry separately and release the page as unused,
while preserving the existing return value so that the vCPU retries the
fault.
Fixes: 2ed90cb093 ("KVM: RISC-V: Retry fault if vma_lookup() results become invalid")
Cc: stable@vger.kernel.org
Signed-off-by: Xie Bo <xb@ultrarisc.com>
Reviewed-by: Anup Patel <anup@brainfault.org>
Link: https://lore.kernel.org/r/20260810051544.3953925-2-xb@ultrarisc.com
Signed-off-by: Anup Patel <anup@brainfault.org>
KVM device ioctls are not serialized against KVM_RUN. As a result,
kvm_riscv_aia_imsic_rw_attr() can snapshot the physical CPU and HGEI of
an IMSIC VS-file before a concurrent vCPU migration releases it.
The HGEI can then be allocated to another vCPU before imsic_vsfile_rw()
uses the stale tuple. A GET or SET attribute may consequently access the
new owner's interrupt file.
Serialize the entire IMSIC attribute operation with the target vCPU
mutex. This prevents the VS-file from being migrated and recycled until
the attribute access completes. Acquire the mutex killably so that the
device ioctl remains interruptible while waiting for KVM_RUN to finish.
Fixes: db8b7e97d6 ("RISC-V: KVM: Add in-kernel virtualization of AIA IMSIC")
Cc: stable@vger.kernel.org
Signed-off-by: Xie Bo <xb@ultrarisc.com>
Reviewed-by: Anup Patel <anup@brainfault.org>
Link: https://lore.kernel.org/r/20260810-imsic-attr-race-v2-1-00ed95ad321e@ultrarisc.com
Signed-off-by: Anup Patel <anup@brainfault.org>
Per AIA spec 1.0 Section 6.3.2, the interrupt numbers 13-63
shares same bit position between related VS shadow CSRs and
hypervisor CSRs. So there's a shift only for SSI, STI and
SEI interrupt.
Currently the KVM always does a shift for all the interrupts
(include LCOFI with number 13) when doing the conversion
between vsip and hvip. Fix this by only doing shift the SSI,
STI and SEI. Add wrappers for doing the conversion between
vsip and hvip.
Fixes: 16b0bde9a3 ("RISC-V: KVM: Add perf sampling support for guests")
Signed-off-by: Yicong Yang <yang.yicong@picoheart.com>
Reviewed-by: Anup Patel <anup@brainfault.org>
Link: https://lore.kernel.org/r/20260804134018.85497-1-yang.yicong@picoheart.com
Signed-off-by: Anup Patel <anup@brainfault.org>
The non-Sstc hrtimer callback clears next_set before its final uses of
the enclosing vCPU. If teardown observes next_set as false while the
callback is still running, kvm_riscv_vcpu_timer_cancel() skips
hrtimer_cancel() and kvm_destroy_vcpus() can free the vCPU before the
callback enters kvm_riscv_vcpu_set_interrupt().
A guest can arm the timer with SBI TIME and request shutdown with SBI
legacy shutdown or SRST. A VMM that honors KVM_EXIT_SYSTEM_EVENT and
destroys the VM supplies the teardown side of the race; no post-launch
host ioctl is needed to arm or request teardown.
On upstream master 62cc902415, generic KASAN reported:
BUG: KASAN: slab-use-after-free in do_raw_spin_lock
Write of size 4 at addr ff60000005e58898
kvm_riscv_vcpu_set_interrupt
kvm_riscv_vcpu_hrtimer_expired
__hrtimer_run_queues
hrtimer_interrupt
The object was allocated by KVM_CREATE_VCPU and freed concurrently by:
kvm_destroy_vcpus
kvm_arch_destroy_vm
kvm_destroy_vm
__fput
For deterministic validation, I added mdelay(1000) immediately after
the existing next_set = false assignment. This only widens the
existing post-clear callback window. A no-delay trace build naturally
reached the callback-after-teardown-start/before-deinit ordering in 12
of 200 runs, but 1,500 stock-kernel stress iterations did not produce a
KASAN report, so natural reproduction is timing-sensitive.
Always invoke hrtimer_cancel() for an initialized timer. Preserve the
existing -EINVAL result when the timer is no longer set, but only after
synchronizing with a running callback.
With this patch, hrtimer_cancel() blocked for the full widened callback
window before vCPU destruction. KASAN reported no error in 100
fixed-and-widened runs or 200 fix-only timing-sweep runs.
Fixes: 3a9f66cb25 ("RISC-V: KVM: Add timer functionality")
Cc: stable@vger.kernel.org
Assisted-by: OpenAI:GPT-5.6
Signed-off-by: Myeonghun Pak <mhun512@gmail.com>
Reviewed-by: Anup Patel <anup@brainfault.org>
Link: https://lore.kernel.org/r/20260731163550.46991-1-mhun512@gmail.com
Signed-off-by: Anup Patel <anup@brainfault.org>
This is another run of the Coccinelle script for converting kmalloc()
family of allocations to kmalloc_obj() via the existing rules in
scripts/coccinelle/api/kmalloc_objs.cocci
This catches both the set of kmalloc() uses added since the first
kmalloc_obj() conversions in v7.0 and adds a large group missed in the
first pass due to Coccinelle not interacting well with the cleanup.h
scoped_...() family of macros[1]. I worked around this with spatch's
"--macro-file" argument to a file with all the scoped_...() macros mapped
to Coccinelle's YACFE_ITERATOR[2] as that was the closest viable control
flow indicator I could find.
Build tested allmodconfig on x86, arm64, arm, loongarch, mips, powerpc,
riscv, and s390 with no new warnings.
Link: https://lore.kernel.org/lkml/202609021314.8A9C0B8@keescook/ [1]
Link: https://github.com/coccinelle/coccinelle/blob/master/standard.h [2]
Signed-off-by: Kees Cook <kees+treewide@kernel.org>
* Add support for 'slot' based PMU events, paired with new UAPI that
compels the user to select a specific PMU implementation
* Lazy save/restore of vCPU state for pKVM, along with various fixes
and cleanups to the management of vCPU state between the untrusted
host and pKVM hypervisor
* Disable traps of EL1 registers for nested hypervisors when FEAT_NV2p1
is present, guaranteeing that EL2-specific register bits are stateful
in the EL1 counterpart
* Leverage FEAT_NV3 to avoid unnecessary ERET/TLBI traps when the scope
of those instructions remains 'in host' (i.e. L1 kernel/userspace)
* Pile of fixes for the management of the VNCR pseudo-TLB, such as
under-invalidations and races with concurrent TLBIs on other vCPUs
* Consolidate the non-protected and pKVM view of ICH_VTR_EL2 to a
runtime-patched constant, allowing the same data to be shared with
pKVM prior to dropping host privileges
* Considerable pile of LLM-assisted fixes around the shop but mostly in
the VGIC, our in-kernel generator of bugs (and sometimes interrupts)
LoongArch:
* Advertise already-supported capabilities.
* Some bug fixes about timer and MMIO.
* Some hardening about interrupt injection.
* Replace kvm_err() with kvm_pr_unimpl().
* Add FPU/LSX/LASX test cases for selftests.
RISC-V:
* Svadu/Zicfiss/Zicfilp FWFT support for Guest
* Use try_cmpxchg for IMSIC MRIF RMW
* More arch-specific tracepoints in KVM RISC-V
* Eager page splitting when enabling dirty logging
* Optimize hfence request handling for SMP Guests
* Improve dirty log clearing by skipping zero bits in mask
* Guard HFENCE range loops against overflow
* CPU PM notifiers in KVM RISC-V for non-retentive idle states
* Fix kernel-mode vector context save/restore for Guest
s390:
* Fixes for vfio-ap
* Fixes for the gmap rework
* Fixes for vsie
* AI triggered fixes all over
* diag9c tracing
* code move preparation for the additional arm64 support
* enable CONTEXT_ANALYSIS
x86:
* Perform spring cleaning on x86.{c,h} and asm/kvm_host.h, by adding regs.c
(the kvm_cache_regs.h => regs.h is already applied) and msrs.{c,h}, and moving
relevant code out of x86.c.
* Split kvm_mmu in three parts, respectively to describe the format of page
tables, walking the guest page tables and building the page tables. Always
use the same page table walker kvm->arch.gva_walk as the entry point to
convert a guest's virtual address, where the previous code used two
different kvm_mmu structs depending on whether the walk included nested
EPT/NPT or not. Make page fault vmexits reuse the permission checking
machinery that is used for guest page faults. This is both a cleanup
and a baby step towards supporting XS/XU memory permissions.
* Document some of the "fun" gotchas with the APIC base when creating IRQCHIPs
on x86.
* Remove a defunct masterclock update from kvm_xen_shared_info_init(). It
could result in incorrect kvmclock due to triggering an unnecessary
switch to/from masterclock mode.
* Skip Xen runstate time updates if time has effectively gone backwards, so
that the guest doesn't report 100% steal time for a very, very long time.
* Drop KVM's runtime updates of the Xen PV timing CPUID leaf, as KVM was
updating the wrong sub-leaf, and upstream KVM will soon provide all the
information needed by userspace to populate the CPUID field itself.
* Fix a bug where KVM would walk a newly created rmap without holding the rmap
lock (or mmu_lock) during aging.
* Fix a bug where aging TDP MMU SPTEs could clobber FROZEN SPTEs.
* Fix a variety of #DB priority bugs.
* Fix a class of races related to enabling Hyper-V emulation on a vCPU after
the vCPU is visible to the rest of KVM.
* Use static calls for nested virtualization ops.
* Move more KVM-internal code out of x86's kvm_host.h.
* Enumerate support for a variety of Zhaoxin instructions that don't require
explicit virtualization.
* Fix missing EFER validation bugs, including in the KVM_SET_SREGS* path.
* Harden kvm_vcpu_map() against double-mapping and thus leaking references.
* Misc fixes and cleanups, e.g. for largely benign syzkaller splats.
x86 (Intel):
* Zero a vCPU's entry in VMX's Posted Interrupt Descriptor table used for IPI
virtualization when the vCPU is freed, to fix a use-after-free where hardware
will write to a freed vCPU's PID.
* Service local TLB flushes on a failed nested VM-Enter to fix a bug where KVM
could miss a TLB on a future, successful VM-Enter with the same L2 VPID.
* Cap the maximum value shoved into the VMX Preemption Timer to workaround an
erratum that affects all existing Intel CPUs that support CPUID 0x15.
* Fix VPID virtualization bugs where KVM would fail to flush hardware TLBs.
* Harden the TDX "populate" ioctls against bad input, and to prepare
for supporting in-place private<=>shared conversion.
x86 (AMD):
* Forcefully invalidate SNP VMSA pages if their backing guest_memfd page is
zapped/invalidated, e.g. due to a PUNCH_HOLE in response to a Page-State
Change request.
* Remove a dying VM from the GA Log notifier list before the VM is actually
destroyed, to fix a potential use-after-free.
* While FOLL_WRITE was needed in the past to trigger CoW unsharing, nowadays
FOLL_LONGTERM does that already even without FOLL_WRITE, and in fact,
get_user_pages() actually disallows FOLL_WRITE together with FOLL_LONGTERM.
So don't pass FOLL_WRITE when registering encrypted memory regions, i.e. when
pinning SEV/SEV-ES guest memory, to fix a regression with file-backed memory
introduced by KVM's (correct) usage of long-term pins.
(This was reviewed by mm maintainers; for more information, see commit
ee1a586dd1).
* Allocate full pages for SEV/SEV-ES {DE,EN}CRYPT ops on SNP-enabled hosts to
fix a data corruption issue due to the PSP driver assigning to-be-written
pages to firmware (as required by the SNP specs).
* Unconditionally intercept ICBEP so that KVM generates the correct guest RIP
when handling an ICEBP-induced TASK_SWITCH #VMEXIT.
* Harden the SNP "populate" ioctls against bad input, and to prepare
for supporting in-place private<=>shared conversion.
Generic:
* Remove kvm_debugfs_dir if kvm_init() fails after creating KVM's debugfs.
* Add a per-VM bitmap to track which vCPU IDs have been "claimed" but for
which the vCPU isn't yet online, and use the bitmap to reject duplicate IDs
before calling into arch code. This allows arch code to consume vcpu_id
without having to worry about cross-vCPU clobbering (at least s390 and x86
have had related bugs).
* Rework the so called "prepare" and "invalidate" guest_memfd hooks to prepare
for in-place private<=>shared conversion, and clean up a few warts along the
way.
Selftests:
* Automatically allocate a full page for L2 guest stacks on x86 instead of
requiring test-specific L1 guest code to carve out a portion of the L1
stack for L2 usage, and to ensure the L2 stack also adheres to the x86-64
calling convention ABI.
* Add a selftest to verify {Guest,Host}-Only behavior in x86's mediated PMU.
* Clean up nested SVM's handling of GPRs on L2<=>L1 transitions, reuse the
functionality for nested VMX, and drop the ucall hack that was fudging
around the lack of GPR switching on nVMX.
* Add a stress test to verify KVM doesn't clobber/drop #PF state, e.g. CR2,
across save/restore, including when L2 is active.
* Add a test to verify KVM_CREATE_VM accepts exactly what is reported by
KVM_CAP_VM_TYPES.
* Misc selftests fixes and cleanups
* Fix several issues with seeding the pRNG, and rework the pRNG APIs to that
the pRNG can be sanely used in host code, not just guest code.
* Add an IRQ test to validate virtual IRQ deliverty for IRQs wired up via
KVM_IRQFD + KVM_SET_GSI_ROUTING, with optional support for triggering IRQs
via writes to an assigned VFIO device.
* Add syscall wrappers to assert success on a variety of pthreads and CPU
affinity APIs.
* Set vCPU pthread affinity as early as possible to reduce contention issues
that were surfaced by PREEMPT_LAZY, which result in runtimes of over a
minute on large hosts, versus the expected ~5 seconds.
* Rework the PMU counters test to run each testcase using a single VM with
many vCPUs for each sub-testcase, instead of using a unique VM for each
sub-testcase. This cuts the runtime by ~20x.
Miscellaneous:
* MAINTAINERS updates for vfio-ap, guest_memfd, kvm-x86. Mostly representing
the status quo more accurately, but also... welcome David Hildenbrand
as guest_memfd reviewer!
-----BEGIN PGP SIGNATURE-----
iQFIBAABCAAyFiEE8TM4V0tmI4mGbHaCv/vSX3jHroMFAmqMgosUHHBib256aW5p
QHJlZGhhdC5jb20ACgkQv/vSX3jHroP8bwf+ORImBMDM3QEmybZM3I+N2+xqSuHP
QHttbmqGbsFK/RUeH96/X/+P9waqaz3uVeUQ6Qp2r0ryqwKtLt8YvIxKDp+M0vVJ
n+iukk1xulBEc28aGdKHn9G4wayAwDA/9f7CvJ23hojaJfLScbF3OlFkDd7y5DpO
x15Rtg9folYUjjop3LDML4N9/9Qmk4KRvVZ4ZVv6IB4uGJJ72fLd5dbBMyDk+BLl
Lz9N1xVTXcnJXJmrjMB4/QNt/HiQJdun8LcokJZyykta7Xx6aY7OGZv+VmCeq47K
e9Hk8mD4AyAdVbVvIntROSeBJOrlsgWJXAPHea6wEdRQnS3XVEBuZC0yDA==
=rfLh
-----END PGP SIGNATURE-----
Merge tag 'for-linus' of git://git.kernel.org/pub/scm/virt/kvm/kvm
Pull kvm updates from Paolo Bonzini:
"ARM64:
- Add support for 'slot' based PMU events, paired with new UAPI that
compels the user to select a specific PMU implementation
- Lazy save/restore of vCPU state for pKVM, along with various fixes
and cleanups to the management of vCPU state between the untrusted
host and pKVM hypervisor
- Disable traps of EL1 registers for nested hypervisors when
FEAT_NV2p1 is present, guaranteeing that EL2-specific register bits
are stateful in the EL1 counterpart
- Leverage FEAT_NV3 to avoid unnecessary ERET/TLBI traps when the
scope of those instructions remains 'in host' (i.e. L1
kernel/userspace)
- Pile of fixes for the management of the VNCR pseudo-TLB, such as
under-invalidations and races with concurrent TLBIs on other vCPUs
- Consolidate the non-protected and pKVM view of ICH_VTR_EL2 to a
runtime-patched constant, allowing the same data to be shared with
pKVM prior to dropping host privileges
- Considerable pile of LLM-assisted fixes around the shop but mostly
in the VGIC, our in-kernel generator of bugs (and sometimes
interrupts)
LoongArch:
- Advertise already-supported capabilities
- Some bug fixes about timer and MMIO
- Some hardening about interrupt injection
- Replace kvm_err() with kvm_pr_unimpl()
- Add FPU/LSX/LASX test cases for selftests
RISC-V:
- Svadu/Zicfiss/Zicfilp FWFT support for Guest
- Use try_cmpxchg for IMSIC MRIF RMW
- More arch-specific tracepoints in KVM RISC-V
- Eager page splitting when enabling dirty logging
- Optimize hfence request handling for SMP Guests
- Improve dirty log clearing by skipping zero bits in mask
- Guard HFENCE range loops against overflow
- CPU PM notifiers in KVM RISC-V for non-retentive idle states
- Fix kernel-mode vector context save/restore for Guest
s390:
- Fixes for vfio-ap
- Fixes for the gmap rework
- Fixes for vsie
- AI triggered fixes all over
- diag9c tracing
- code move preparation for the additional arm64 support
- enable CONTEXT_ANALYSIS
x86:
- Perform spring cleaning on x86.{c,h} and asm/kvm_host.h, by adding
regs.c (the kvm_cache_regs.h => regs.h is already applied) and
msrs.{c,h}, and moving relevant code out of x86.c
- Split kvm_mmu in three parts, respectively to describe the format
of page tables, walking the guest page tables and building the page
tables. Always use the same page table walker kvm->arch.gva_walk as
the entry point to convert a guest's virtual address, where the
previous code used two different kvm_mmu structs depending on
whether the walk included nested EPT/NPT or not. Make page fault
vmexits reuse the permission checking machinery that is used for
guest page faults. This is both a cleanup and a baby step towards
supporting XS/XU memory permissions
- Document some of the "fun" gotchas with the APIC base when creating
IRQCHIPs on x86
- Remove a defunct masterclock update from kvm_xen_shared_info_init().
It could result in incorrect kvmclock due to triggering an
unnecessary switch to/from masterclock mode
- Skip Xen runstate time updates if time has effectively gone
backwards, so that the guest doesn't report 100% steal time for
a very, very long time
- Drop KVM's runtime updates of the Xen PV timing CPUID leaf, as KVM
was updating the wrong sub-leaf, and upstream KVM will soon provide
all the information needed by userspace to populate the CPUID field
itself
- Fix a bug where KVM would walk a newly created rmap without holding
the rmap lock (or mmu_lock) during aging
- Fix a bug where aging TDP MMU SPTEs could clobber FROZEN SPTEs
- Fix a variety of #DB priority bugs
- Fix a class of races related to enabling Hyper-V emulation on a
vCPU after the vCPU is visible to the rest of KVM
- Use static calls for nested virtualization ops
- Move more KVM-internal code out of x86's kvm_host.h
- Enumerate support for a variety of Zhaoxin instructions that don't
require explicit virtualization
- Fix missing EFER validation bugs, including in the KVM_SET_SREGS*
path
- Harden kvm_vcpu_map() against double-mapping and thus leaking
references
- Misc fixes and cleanups, e.g. for largely benign syzkaller splats
x86 (Intel):
- Zero a vCPU's entry in VMX's Posted Interrupt Descriptor table used
for IPI virtualization when the vCPU is freed, to fix a
use-after-free where hardware will write to a freed vCPU's PID
- Service local TLB flushes on a failed nested VM-Enter to fix a bug
where KVM could miss a TLB on a future, successful VM-Enter with
the same L2 VPID
- Cap the maximum value shoved into the VMX Preemption Timer to
workaround an erratum that affects all existing Intel CPUs that
support CPUID 0x15
- Fix VPID virtualization bugs where KVM would fail to flush hardware
TLBs
- Harden the TDX "populate" ioctls against bad input, and to prepare
for supporting in-place private<=>shared conversion
x86 (AMD):
- Forcefully invalidate SNP VMSA pages if their backing guest_memfd
page is zapped/invalidated, e.g. due to a PUNCH_HOLE in response to
a Page-State Change request
- Remove a dying VM from the GA Log notifier list before the VM is
actually destroyed, to fix a potential use-after-free
- While FOLL_WRITE was needed in the past to trigger CoW unsharing,
nowadays FOLL_LONGTERM does that already even without FOLL_WRITE,
and in fact, get_user_pages() actually disallows FOLL_WRITE
together with FOLL_LONGTERM. So don't pass FOLL_WRITE when
registering encrypted memory regions, i.e. when pinning SEV/SEV-ES
guest memory, to fix a regression with file-backed memory
introduced by KVM's (correct) usage of long-term pins
(This was reviewed by mm maintainers; for more information, see
commit ee1a586dd1 "KVM: SEV: Drop FOLL_WRITE for encrypted region
registration")
- Allocate full pages for SEV/SEV-ES {DE,EN}CRYPT ops on SNP-enabled
hosts to fix a data corruption issue due to the PSP driver
assigning to-be-written pages to firmware (as required by the SNP
specs)
- Unconditionally intercept ICBEP so that KVM generates the correct
guest RIP when handling an ICEBP-induced TASK_SWITCH #VMEXIT
- Harden the SNP "populate" ioctls against bad input, and to prepare
for supporting in-place private<=>shared conversion
Generic:
- Remove kvm_debugfs_dir if kvm_init() fails after creating KVM's
debugfs
- Add a per-VM bitmap to track which vCPU IDs have been "claimed" but
for which the vCPU isn't yet online, and use the bitmap to reject
duplicate IDs before calling into arch code. This allows arch code
to consume vcpu_id without having to worry about cross-vCPU
clobbering (at least s390 and x86 have had related bugs)
- Rework the so called "prepare" and "invalidate" guest_memfd hooks
to prepare for in-place private<=>shared conversion, and clean up a
few warts along the way
Selftests:
- Automatically allocate a full page for L2 guest stacks on x86
instead of requiring test-specific L1 guest code to carve out a
portion of the L1 stack for L2 usage, and to ensure the L2 stack
also adheres to the x86-64 calling convention ABI
- Add a selftest to verify {Guest,Host}-Only behavior in x86's
mediated PMU
- Clean up nested SVM's handling of GPRs on L2<=>L1 transitions,
reuse the functionality for nested VMX, and drop the ucall hack
that was fudging around the lack of GPR switching on nVMX
- Add a stress test to verify KVM doesn't clobber/drop #PF state,
e.g. CR2, across save/restore, including when L2 is active
- Add a test to verify KVM_CREATE_VM accepts exactly what is reported
by KVM_CAP_VM_TYPES
- Misc selftests fixes and cleanups
- Fix several issues with seeding the pRNG, and rework the pRNG APIs
to that the pRNG can be sanely used in host code, not just guest
code
- Add an IRQ test to validate virtual IRQ deliverty for IRQs wired up
via KVM_IRQFD + KVM_SET_GSI_ROUTING, with optional support for
triggering IRQs via writes to an assigned VFIO device
- Add syscall wrappers to assert success on a variety of pthreads and
CPU affinity APIs
- Set vCPU pthread affinity as early as possible to reduce contention
issues that were surfaced by PREEMPT_LAZY, which result in runtimes
of over a minute on large hosts, versus the expected ~5 seconds
- Rework the PMU counters test to run each testcase using a single VM
with many vCPUs for each sub-testcase, instead of using a unique VM
for each sub-testcase. This cuts the runtime by ~20x
Miscellaneous:
- MAINTAINERS updates for vfio-ap, guest_memfd, kvm-x86. Mostly
representing the status quo more accurately, but also... welcome
David Hildenbrand as guest_memfd reviewer!"
* tag 'for-linus' of git://git.kernel.org/pub/scm/virt/kvm/kvm: (413 commits)
KVM: arm64: Validate GICv5 timer PPIs before claiming ownership
KVM: arm64: vgic: Reject out-of-range GICv5 PPI IDs
KVM: arm64: vgic: Prevent speculative SPI array underflow
KVM: arm64: vgic: Free gic_kvm_info on initialization failure
KVM: arm64: Avoid mismatched accesses to 'struct kvm_nvhe_init_params'
s390/vfio-ap: Fix NULL deref in status_show() during queue probe
s390/vfio-ap: Fix hot-unplug skipped when last AP adapter or domain removed
s390/vfio-ap: fix potential use of uninitialized apm_filtered bitmap
s390/vfio-ap: Fix control domain removal in vfio_ap_mdev_cfg_remove
s390/vfio-ap: Fix required lock not held during update of ap_matrix_mdev object
s390/vfio-ap: Fix missing lock required to access list of ap_matrix_mdev objects
s390/vfio-ap: Fix dereference matrix_mdev->kvm without checking for NULL
s390/vfio-ap: Fix stale do_remove flag across iterations in vfio_ap_mdev_cfg_remove
RISC-V: KVM: fix vcpu vector context handling for kernel-mode vector
riscv: vector: allow non-preemptible kernel-mode vector with IRQs off
riscv: vector: refactor riscv_v_start_kernel_context
KVM: s390: gmap: Make prefix handling optional
KVM: s390: gmap: Make CMMA optional
KVM: s390: gmap: Make storage keys optional
KVM: s390: Prepare gmap for a second KVM implementation
...
Running vector workloads like perf + mcf on KVM can result in an
unexpected termination due to a vtype corruption. This happens because
the kernel-mode vector (KMV) misattributes the guest's vcpu context as
the user's context and source from a wrong status.VS.
The simplified call chain that results in this problem is shown as
follow:
__riscv_sys_ioctl()
kvm_arch_vcpu_ioctl_run()
kvm_riscv_vcpu_exit()
kvm_riscv_vcpu_sbi_ecall()
kvm_riscv_vcpu_pmu_ctr_stop()
kvm_vcpu_write_guest()
__copy_to_user()
enter_vector_usercopy()
kernel_vector_begin()
kernel_vector_begin() should use the sstatus.VS from guest's vcpu
context instead of task_pt_reg(current). Also, it should not save
guest's v-reg into the user's context memory.
To resolve this, the vcpu context must be correctly saved when KMV is
serving a guest. However, invoking KVM functions directly from generic
RISC-V architecture code introduces a reverse dependency, breaking
builds when KVM is configured as N or M.
Address this by registering an RCU-protected callback for context
flushing. KVM registers this callback at module initialization and
unregisters it on exit. When KMV starts a kernel context, it can now
safely flush the vector context via the callback.
Fixes: ecd2ada8a5 ("riscv: Add support for kernel mode vector")
Signed-off-by: Andy Chiu <tchiu@tenstorrent.com>
Reviewed-by: Yong-Xuan Wang <yongxuan.wang@sifive.com>
Reviewed-by: Anup Patel <anup@brainfault.org>
Link: https://lore.kernel.org/r/20260803215250.824417-4-tchiu@tenstorrent.com
Signed-off-by: Anup Patel <anup@brainfault.org>
Register a CPU_PM notifier to restore hypervisor CSR content during CPU
non-retention idle states. When a CPU enters a deep idle state that
powers off the CPU domain, hypervisor CSRs and VS CSRs lose their state
and must be saved before entry and restored after exit.
This completes KVM's power management coverage for RISC-V:
- CPU hotplug: handled by kvm_online_cpu/kvm_offline_cpu (cpuhp callbacks)
- System suspend: handled by kvm_suspend/kvm_resume (syscore ops)
- CPU idle (retention): no action needed, CSRs are retained
- CPU idle (non-retention): handled by this CPU_PM notifier
Signed-off-by: Yong-Xuan Wang <yongxuan.wang@sifive.com>
Reviewed-by: Anup Patel <anup@brainfault.org>
Link: https://lore.kernel.org/r/20260810-kvm-cpu-pm-v5-1-6c9e4c95caf5@sifive.com
Signed-off-by: Anup Patel <anup@brainfault.org>
When a CPU is taken offline or enters deep idle states, hypervisor CSR
state is lost. The kvm_former_vcpu fast-path optimization caches the
last VCPU that ran on each CPU to avoid expensive CSR restoration when
the same VCPU is re-scheduled on the same CPU. However, if this cache
is not cleared when CSR state is lost, the next VCPU entry will
incorrectly skip CSR restoration, leading to corrupt hypervisor state.
Add kvm_riscv_clear_former_vcpu() to invalidate the per-CPU cache and
call it from kvm_arch_disable_virtualization_cpu() to ensure proper CSR
restoration after CPU offline or system suspend events.
Fixes: 1323a5cfe5 ("KVM: riscv: Skip CSR restore if VCPU is reloaded on the same core")
Signed-off-by: Yong-Xuan Wang <yongxuan.wang@sifive.com>
Reviewed-by: Nutty Liu <nutty.liu@hotmail.com>
Reviewed-by: Anup Patel <anup@brainfault.org>
Link: https://lore.kernel.org/r/20260721-kvm-cpu-pm-v4-1-146bf942547d@sifive.com
Signed-off-by: Anup Patel <anup@brainfault.org>
SBI PMU EVENT_GET_INFO stores guest-controlled num_events * sizeof(*einfo)
in a 32-bit integer. On RV64, num_events = 0x10000001 makes 0x100000010
truncate to 16. KVM then allocates one entry but loops over the original
num_events, causing out-of-bounds reads and writes. A nested guest
triggered:
BUG: KASAN: slab-out-of-bounds in kvm_riscv_vcpu_pmu_event_info+0xa4/0x142
Read of size 4 at addr ff600000074d46b0 by task init/1
Call Trace:
[<ffffffff8006471c>] kvm_riscv_vcpu_pmu_event_info+0xa4/0x142
[<ffffffff800690c0>] kvm_sbi_ext_pmu_handler+0xca/0x268
[<ffffffff8006779e>] kvm_riscv_vcpu_sbi_ecall+0xec/0x1e6
[<ffffffff8006008c>] kvm_riscv_vcpu_exit+0x48c/0x540
[<ffffffff8005ea0a>] kvm_arch_vcpu_ioctl_run+0x37e/0xc80
Allocated by task 1:
__kmalloc_noprof+0x19e/0x4b0
kvm_riscv_vcpu_pmu_event_info+0x72/0x142
kvm_sbi_ext_pmu_handler+0xca/0x268
kvm_riscv_vcpu_sbi_ecall+0xec/0x1e6
kvm_riscv_vcpu_exit+0x48c/0x540
kvm_arch_vcpu_ioctl_run+0x37e/0xc80
The buggy address is located 0 bytes to the right of
allocated 16-byte region [ff600000074d46a0, ff600000074d46b0)
Store the shared-memory size in size_t and reject multiplication overflow.
Allocate the guest-driven array with GFP_KERNEL_ACCOUNT so it is charged
to kmemcg, and use __GFP_NOWARN to suppress allocation failure warnings.
Use kvcalloc() to allow vmalloc fallback and an unsigned long loop index
to match num_events.
Reported-by: Naveed Khan <naveed@digiscrypt.com>
Closes: https://lore.kernel.org/kvm/178345245327.72065.13249716450708539854@digiscrypt.com/
Fixes: e309fd113b ("RISC-V: KVM: Implement get event info function")
Cc: stable@vger.kernel.org
Signed-off-by: Guidong Han <2045gemini@gmail.com>
Reviewed-by: Anup Patel <anup@brainfault.org>
Link: https://lore.kernel.org/r/20260730092533.1369531-1-2045gemini@gmail.com
Signed-off-by: Anup Patel <anup@brainfault.org>
KVM already treats a NULL hart-mask pointer in SBI v0.1 calls as all
harts, matching OpenSBI's legacy handling. The current handler builds that
target set as (1UL << online_vcpus) - 1 before iterating the mask.
That expression shifts by the type width once the VM has BITS_PER_LONG
online vCPUs, and a single word cannot represent larger VMs. A count-based
mask also selects the wrong IDs when vCPU IDs are sparse.
Handle SEND_IPI with a NULL hart-mask by iterating all online vCPUs
directly. For remote fence calls, use the existing hbase == -1UL
convention so the common fence request helper targets the whole VM
without consulting hmask.
Non-NULL guest hart masks keep the existing v0.1 behavior.
Fixes: a046c2d857 ("RISC-V: KVM: Reorganize SBI code by moving SBI v0.1 to its own file")
Signed-off-by: Pengpeng Hou <pengpeng@iscas.ac.cn>
Reviewed-by: Anup Patel <anup@brainfault.org>
Link: https://lore.kernel.org/r/20260718040000.010.826fe94e-caplitmus-rv@iscas.ac.cn
Signed-off-by: Anup Patel <anup@brainfault.org>
The local HFENCE range helpers build the loop end with addr + size and
then step through the range by the requested page order. If addr + size
wraps, a small range near the top of the address space can be treated as
empty and skip the intended fence. The final loop step can also wrap
before the loop condition is evaluated again.
Treat wrapped ranges conservatively as full-range HFENCE requests and stop
each bounded loop before the next step can wrap past the computed end. This
keeps the existing large-range fallback and only changes local GVMA/VVMA
range construction.
Fixes: 2415e46e3a ("RISC-V: KVM: Introduce range based local HFENCE functions")
Signed-off-by: Pengpeng Hou <pengpeng@iscas.ac.cn>
Reviewed-by: Anup Patel <anup@brainfault.org>
Link: https://lore.kernel.org/r/20260718040000.008.17bfd3bd-caplitmus-rv@iscas.ac.cn
Signed-off-by: Anup Patel <anup@brainfault.org>
The base extensions are often lowercase and were written as lowercase in
hwcap, but other references to these extensions in the kernel are
uppercase. Standardize the case to make it easier to handle macro
expansion.
Acked-by: Anup Patel <anup@brainfault.org>
Reviewed-by: Anup Patel <anup@brainfault.org>
Signed-off-by: Charlie Jenkins <thecharlesjenkins@gmail.com>
[andrew.jones: Apply KVM_ISA_EXT_ARR(), fixup all KVM use.]
Signed-off-by: Andrew Jones <andrew.jones@oss.qualcomm.com>
Signed-off-by: Guodong Xu <docular.xu@gmail.com>
Link: https://patch.msgid.link/20260701-rva23u64-hwprobe-v2-v5-4-2c61f94a695a@gmail.com
[pjw@kernel.org: fixed a checkpatch warning; added Andrew's username to his tag comments]
Signed-off-by: Paul Walmsley <pjw@kernel.org>
KVM lets userspace configure the IMSIC guest, HART, group-width, and
group-shift fields independently. The individual bounds allow a layout
such as guest_bits=7, hart_bits=6, group_bits=1, and group_shift=24.
That layout places the group index bit at bit 24 while the lower
guest+HART field also extends through bit 24. The IMSIC address helpers
later consume these values as one address layout: aia_imsic_ppn() masks
the lower guest+HART field and the group field, while
aia_imsic_hart_index() combines the group and HART index. Allowing an
overlapping configuration therefore gives one address bit two different
meanings.
Reject group-index layouts that start before the end of the guest+HART
field before initializing the in-kernel irqchip.
Fixes: 89d01306e3 ("RISC-V: KVM: Implement device interface for AIA irqchip")
Signed-off-by: Pengpeng Hou <pengpeng@iscas.ac.cn>
Reviewed-by: Anup Patel <anup@brainfault.org>
Link: https://lore.kernel.org/r/20260718040000.005.6ede2881-caplitmus-rv@iscas.ac.cn
Signed-off-by: Anup Patel <anup@brainfault.org>
KVM memslots describe half-open ranges [base_gfn, base_gfn + npages). A
slot ending exactly at the guest physical-address limit has its final page
below that limit and is valid.
The current greater-than-or-equal comparison rejects a valid boundary case.
Use a strict greater-than comparison, matching equivalent KVM architecture
checks.
Fixes: 9d05c1fee8 ("RISC-V: KVM: Implement stage2 page table programming")
Signed-off-by: Pengpeng Hou <pengpeng@iscas.ac.cn>
Reviewed-by: Anup Patel <anup@brainfault.org>
Link: https://lore.kernel.org/r/20260718033646.002.31dc89f2-caplitmus-rv@iscas.ac.cn
Signed-off-by: Anup Patel <anup@brainfault.org>
The existing kvm_riscv_gstage_wp_range() walks the entire [start, end)
range to apply write protection for dirty log clearing, even when the
provided mask has zero bits (i.e., many GFNs do not need protection).
This leads to unnecessary page table walks when the mask is sparse.
Replace the range-based approach with a new function
kvm_riscv_gstage_wp_pt_masked() that iterates only over the set bits in
the mask. For each set bit, it looks up the leaf PTE and applies write
protection. Once a huge page is encountered, the entire huge-page range
is processed in one go, and the corresponding bits in the mask are cleared
using bitmap_clear().
Performance was measured with KVM selftests dirty_log_perf_test on a
Spacemit k3 host with the following configuration:
- vCPUs: 2 (-v 2)
- Memory: 1GB (-b 1G)
- Iterations: 3 (-i 3)
- Write percentage varied via -w parameter to simulate different
dirty mask densities.
The following data shows the time spent in the clear-dirty-log phase
(i.e., the KVM_CLEAR_DIRTY_LOG ioctl) under each configuration.
+------------------+------------------+------------------+-------------+
| Write Percentage | Original (s) | Patched (s) | Improvement |
+------------------+------------------+------------------+-------------+
| 10% | 0.012905 | 0.009213 | +28.6% |
| 30% | 0.014217 | 0.010287 | +27.6% |
| 50% | 0.014606 | 0.011772 | +19.4% |
| 70% | 0.014735 | 0.013199 | +10.4% |
| 100% | 0.014759 | 0.015294 | -3.6% |
+------------------+------------------+------------------+-------------+
The performance improvement is most significant when the dirty mask is
sparse (low write percentage), which is common in real-world scenarios
with low to moderate memory write intensity. In the worst-case scenario
where the mask is fully set (100% write), the optimization introduces
a slight 3.6% overhead due to the additional bit operations, which is
acceptable given the substantial gains in common cases.
This change significantly reduces the number of page-table walks when
the dirty mask has many zero bits, improving the efficiency of
KVM_CLEAR_DIRTY_LOG and related ioctls.
Signed-off-by: Wang Yechao <wang.yechao255@zte.com.cn>
Reviewed-by: Anup Patel <anup@brainfault.org>
Link: https://lore.kernel.org/r/20260717080654.877151-1-wang.yechao255@zte.com.cn
Signed-off-by: Anup Patel <anup@brainfault.org>
try_count is initialized to 5 but never decremented in the retry path,
making the `if (try_count)` check always true. If all NACL shared memory
hfence entries remain in the pending state after sync, the function loops
forever, causing a soft lockup. Decrement try_count on each retry so the
fallback warning and return become reachable.
Fixes: d466c19cea ("RISC-V: KVM: Add common nested acceleration support")
Signed-off-by: Zongmin Zhou <zhouzongmin@kylinos.cn>
Reviewed-by: Anup Patel <anup@brainfault.org>
Link: https://lore.kernel.org/r/20260716073756.44153-1-min_halo@163.com
Signed-off-by: Anup Patel <anup@brainfault.org>
FENCE.I does not need hfence data, but it currently goes through the
generic make_xfence_request() path with NULL data, incurring unnecessary
per-VCPU checks.
Split out a separate make_xfence_request_nodata() function to handle
FENCE.I directly, and move the data validity check to the top of the
generic function to avoid redundant checks.
Signed-off-by: Wang Yechao <wang.yechao255@zte.com.cn>
Reviewed-by: Anup Patel <anup@brainfault.org>
Link: https://lore.kernel.org/r/20260731093833.1551341-3-wang.yechao255@zte.com.cn
Signed-off-by: Anup Patel <anup@brainfault.org>
When handling hfence requests in make_xfence_request(), the current code
uses a single 'actual_req' variable and a single vcpu_mask. If any VCPU
fails to enqueue the hfence data (because its queue is full), the request
falls back to 'fallback_req' for all VCPUs, even if other VCPUs still
have available queue space.
This can cause unnecessary fallback for healthy VCPUs, and more seriously,
those healthy VCPUs will not process their already-enqueued hfence
requests because no 'req' is set for them. As a result, their queues will
quickly become full as well, degrading performance for SMP guests.
Fix this by maintaining two separate bitmaps: one for VCPUs that
successfully enqueued the hfence data (req_vcpu_mask) and another for
those that failed (fallback_req_vcpu_mask). Then send the appropriate
requests to each group. This ensures that fallback is only applied to
VCPUs that actually need it, preserving the efficiency of the normal
path for others.
Fixes: 13acfec2db ("RISC-V: KVM: Add remote HFENCE functions based on VCPU requests")
Signed-off-by: Wang Yechao <wang.yechao255@zte.com.cn>
Reviewed-by: Anup Patel <anup@brainfault.org>
Link: https://lore.kernel.org/r/20260731093833.1551341-2-wang.yechao255@zte.com.cn
Signed-off-by: Anup Patel <anup@brainfault.org>
Add an eager_page_split module parameter for RISC-V KVM, following
the same approach as on x86. This parameter controls whether eager
page splitting is enabled. The default value is on.
When eager page splitting is enabled, KVM proactively splits large
pages (huge pages) into smaller pages when needed for dirty logging
or other operations. Disabling it can be beneficial for VM workloads
that rarely perform writes, or that only write to a small region of
memory, as it allows huge pages to remain intact for read accesses.
Signed-off-by: Wang Yechao <wang.yechao255@zte.com.cn>
Reviewed-by: Anup Patel <anup@brainfault.org>
Link: https://lore.kernel.org/r/20260731091215.1549430-6-wang.yechao255@zte.com.cn
Signed-off-by: Anup Patel <anup@brainfault.org>
Split huge pages on the range specified using KVM_CLEAR_DIRTY_LOG.
And do not split when enabling dirty logging if
KVM_DIRTY_LOG_INITIALLY_SET is set.
Signed-off-by: Wang Yechao <wang.yechao255@zte.com.cn>
Reviewed-by: Anup Patel <anup@brainfault.org>
Link: https://lore.kernel.org/r/20260731091215.1549430-5-wang.yechao255@zte.com.cn
Signed-off-by: Anup Patel <anup@brainfault.org>
The function kvm_arch_mmu_enable_log_dirty_pt_masked() is invoked from
two distinct call paths:
kvm_clear_dirty_log_protect()
kvm_arch_mmu_enable_log_dirty_pt_masked()
kvm_vm_ioctl_reset_dirty_pages()
kvm_dirty_ring_reset()
kvm_reset_dirty_gfn()
kvm_arch_mmu_enable_log_dirty_pt_masked()
In both scenarios, the caller already performs a remote TLB flush after
dirty logging is enabled, so the TLB flush inside
kvm_arch_mmu_enable_log_dirty_pt_masked() is unnecessary. Remove it.
Signed-off-by: Wang Yechao <wang.yechao255@zte.com.cn>
Reviewed-by: Anup Patel <anup@brainfault.org>
Link: https://lore.kernel.org/r/20260731091215.1549430-4-wang.yechao255@zte.com.cn
Signed-off-by: Anup Patel <anup@brainfault.org>
Split huge pages eagerly when enabling dirty logging. The goal is to
avoid doing it while faulting on write-protected pages, which
negatively impacts guest performance.
The benefits of eager page splitting are the same as in x86 and arm64,
added with commit a3fe5dbda0 ("KVM: x86/mmu: Split huge pages mapped
by the TDP MMU when dirty logging is enabled") and commit e7bf7a490c
("KVM: arm64: Split huge pages when dirty logging is enabled")
Signed-off-by: Wang Yechao <wang.yechao255@zte.com.cn>
Reviewed-by: Anup Patel <anup@brainfault.org>
Link: https://lore.kernel.org/r/20260731091215.1549430-3-wang.yechao255@zte.com.cn
Signed-off-by: Anup Patel <anup@brainfault.org>
Add RISC-V KVM tracepoints for events that are useful when debugging
guest exits and in-kernel emulation paths. The existing kvm_entry and
kvm_exit tracepoints are kept, with the kvm_entry PC format fixed to use
zero-padded hexadecimal output.
The newly added tracepoints cover:
- kvm_vcpu_exit for synchronous guest traps
- kvm_vcpu_irq for VS-mode interrupt set/clear
- kvm_mmio_emulate for MMIO load/store emulation
kvm_vcpu_exit reports the full trap context (sepc/scause/stval/htval/
htinst) so userspace can filter by scause, and is placed after the
interrupt early-return so it fires only for synchronous traps.
kvm_vcpu_irq covers every set/unset_interrupt() caller from a single
place.
Example trace output:
kvm_vcpu_exit:
VCPU: 0, SEPC: 0x80200000, SCAUSE: 0x17,
STVAL: 0x10000000, HTVAL: 0x4000000, HTINST: 0x2023
kvm_mmio_emulate:
VCPU: 0, Store MMIO at 0x10000000, len 4, insn 0x2023, sepc 0x80200000
kvm_vcpu_irq:
VCPU: 0, IRQ: 10, level: 1
Testing:
A QEMU-based test program was used to exercise:
- Guest page faults
- MMIO emulation
- IRQ injection
Verified trace output generation for:
- kvm_vcpu_exit
- kvm_vcpu_irq
- kvm_mmio_emulate
Assisted-by: YuanSheng:deepseek-v4-pro
Co-developed-by: Quan Zhou <zhouquan@iscas.ac.cn>
Signed-off-by: Quan Zhou <zhouquan@iscas.ac.cn>
Signed-off-by: Yuhang.Chen <yhchen312@gmail.com>
Reviewed-by: Anup Patel <anup@brainfault.org>
Link: https://lore.kernel.org/r/20260730065711.3721489-1-yhchen312@gmail.com
Signed-off-by: Anup Patel <anup@brainfault.org>
As the CSR is only flushed when the csr_dirty is set, always set the
csr_dirty field in kvm_sbi_ext_fwft_reset() so the CSR change can take
effect immediately. This unconditional set should be safe as all
supported FWFT features change the CSR state currently.
Signed-off-by: Inochi Amaoto <inochiama@gmail.com>
Reviewed-by: Anup Patel <anup@brainfault.org>
Link: https://lore.kernel.org/r/20260729232527.139183-5-inochiama@gmail.com
Signed-off-by: Anup Patel <anup@brainfault.org>
The SBI extension validation callback can only fix the parameter if
the extension is enabled. However, if the extension is disabled after
modifty some parameters, the state of the extension is still broken.
Reset the extension when the extension is disabled so it can have
a clear context.
Signed-off-by: Inochi Amaoto <inochiama@gmail.com>
Reviewed-by: Anup Patel <anup@brainfault.org>
Link: https://lore.kernel.org/r/20260729232527.139183-4-inochiama@gmail.com
Signed-off-by: Anup Patel <anup@brainfault.org>
Since the KVM SBI has extension parameters validation support,
implement it for the SBI FWFT support.
Signed-off-by: Inochi Amaoto <inochiama@gmail.com>
Reviewed-by: Anup Patel <anup@brainfault.org>
Link: https://lore.kernel.org/r/20260729232527.139183-3-inochiama@gmail.com
Signed-off-by: Anup Patel <anup@brainfault.org>
When user disable FWFT extension after setting any value of the
FWFT feature, the state of vCPU will be broken since the value
of disable FWFT feature is still functional.
Add the generic SBI extension validate callback so the FWFT
extension can fix its parameters before the first run.
Signed-off-by: Inochi Amaoto <inochiama@gmail.com>
Reviewed-by: Anup Patel <anup@brainfault.org>
Link: https://lore.kernel.org/r/20260729232527.139183-2-inochiama@gmail.com
Signed-off-by: Anup Patel <anup@brainfault.org>
Currently the RMW for IMSIC MRIF is implemented with word LRSC loop.
This will only cover the lower 32bit on a 64bit system. Instead of
guard the implementation with CONFIG_64BIT here, use try_cmpxchg()
wrapper which has already take care this to fix this issue.
It can also use AMO instructions on supported system.
Fixes: db8b7e97d6 ("RISC-V: KVM: Add in-kernel virtualization of AIA IMSIC")
Signed-off-by: Yicong Yang <yang.yicong@picoheart.com>
Reviewed-by: Anup Patel <anup@brainfault.org>
Link: https://lore.kernel.org/r/20260728141938.95845-1-yang.yicong@picoheart.com
Signed-off-by: Anup Patel <anup@brainfault.org>
fault_addr in gstage_page_fault() and the fault_addr parameter of
kvm_riscv_vcpu_mmio_load/store() are unsigned long. On RV32 with
Sv32x4, guest physical addresses are 34 bits, so a 32-bit unsigned
long truncates bits 32/33 at two points:
- reconstruction: fault_addr = (trap->htval << 2) | ... is evaluated
in 32-bit arithmetic, dropping bits 32/33 before widening;
- the MMIO handler call: even with the local widened, the handler's
unsigned long parameter narrows it back to 32 bits, aliasing
accesses above 4 GB into the low 4 GB.
Widen fault_addr to gpa_t end to end: the local in gstage_page_fault(),
the (gpa_t) cast before the <<2 shift, and the fault_addr parameters of
kvm_riscv_vcpu_mmio_load/store(). The handlers' internal uses
(run->mmio.phys_addr is __u64, kvm_io_bus_read/write() take gpa_t) are
already 64-bit, so no further changes are needed.
Also in preparation for sharing a common struct kvm_page_fault across
architectures, where fault_addr is gpa_t. No functional change on RV64.
Fixes: 9d05c1fee8 ("RISC-V: KVM: Implement stage2 page table programming")
Assisted-by: YuanSheng: deepseek-v4-pro
Co-developed-by: Quan Zhou <zhouquan@iscas.ac.cn>
Signed-off-by: Quan Zhou <zhouquan@iscas.ac.cn>
Signed-off-by: Bingyu Xian <shanbeeyoo@gmail.com>
Reviewed-by: Anup Patel <anup@brainfault.org>
Link: https://lore.kernel.org/r/20260729120733.829457-4-shanbeeyoo@gmail.com
Signed-off-by: Anup Patel <anup@brainfault.org>
vma_pageshift in kvm_riscv_mmu_map() was declared short. Use unsigned
int, the conventional kernel type for bit widths and shift counts, in
preparation for sharing a common struct kvm_page_fault across
architectures. No functional change.
Assisted-by: YuanSheng: deepseek-v4-pro
Co-developed-by: Quan Zhou <zhouquan@iscas.ac.cn>
Signed-off-by: Quan Zhou <zhouquan@iscas.ac.cn>
Signed-off-by: Bingyu Xian <shanbeeyoo@gmail.com>
Reviewed-by: Anup Patel <anup@brainfault.org>
Link: https://lore.kernel.org/r/20260729120733.829457-3-shanbeeyoo@gmail.com
Signed-off-by: Anup Patel <anup@brainfault.org>
Control-flow integrity is controlled through a WARL field in henvcfg.
Expose the feature only if the Zicfilp/Zicfiss is supported for VS-mode.
Allow the VMM to block access to the feature by disabling the ISA
extension in the guest.
Assisted-by: YuanSheng:claude-4.7-opus
Co-developed-by: Quan Zhou <zhouquan@iscas.ac.cn>
Signed-off-by: Quan Zhou <zhouquan@iscas.ac.cn>
Signed-off-by: Inochi Amaoto <inochiama@gmail.com>
Link: https://lore.kernel.org/r/20260713010302.303278-8-inochiama@gmail.com
Signed-off-by: Anup Patel <anup@brainfault.org>
The extension Zicfiss/Zicfilp introduces a software-check exception
to the privilege architecture. Redirect this exception to VS mode
when VCPU enable Zicfiss/Zicfilp so the CFI violations in VU/VS could
be reported.
Also, update kvm_riscv_vcpu_trap_redirect() to forward the SPELP
state to the VS mode.
Assisted-by: YuanSheng:claude-4.7-opus
Co-developed-by: Quan Zhou <zhouquan@iscas.ac.cn>
Signed-off-by: Quan Zhou <zhouquan@iscas.ac.cn>
Signed-off-by: Inochi Amaoto <inochiama@gmail.com>
Reviewed-by: Anup Patel <anup@brainfault.org>
Link: https://lore.kernel.org/r/20260713010302.303278-6-inochiama@gmail.com
Signed-off-by: Anup Patel <anup@brainfault.org>
Add ssp context save/restore for guest VCPUs and also add it to the
ONE_REG interface to allow its access from user space.
Signed-off-by: Inochi Amaoto <inochiama@gmail.com>
Reviewed-by: Anup Patel <anup@brainfault.org>
Link: https://lore.kernel.org/r/20260713010302.303278-5-inochiama@gmail.com
Signed-off-by: Anup Patel <anup@brainfault.org>
Hardware updating of PTE A/D bits is controlled through ADUE bit in
henvcfg Expose the feature only if both Svadu and Svade are supported
for VS-mode.
Allow the VMM to enable/disable this feature by change the ISA
extension state in the guest.
Assisted-by: YuanSheng:claude-4.7-opus
Co-developed-by: Quan Zhou <zhouquan@iscas.ac.cn>
Signed-off-by: Quan Zhou <zhouquan@iscas.ac.cn>
Signed-off-by: Inochi Amaoto <inochiama@gmail.com>
Reviewed-by: Anup Patel <anup@brainfault.org>
Link: https://lore.kernel.org/r/20260713010302.303278-2-inochiama@gmail.com
Signed-off-by: Anup Patel <anup@brainfault.org>
User-controlled register indices from the ONE_REG ioctl are used to
index into the vector register buffer (v0..v31). Sanitize the calculated
offset with array_index_nospec() to prevent speculative out-of-bounds
access.
Signed-off-by: Zongmin Zhou <zhouzongmin@kylinos.cn>
Reviewed-by: Anup Patel <anup@brainfault.org>
Link: https://lore.kernel.org/r/20260715030818.75657-1-min_halo@163.com
Signed-off-by: Anup Patel <anup@brainfault.org>
KVM RISC-V tracks guest local interrupt state with two bitmaps:
- irqs_pending: interrupts that should be visible to the guest
- irqs_pending_mask: interrupts whose pending state changed
The current code updates those bitmaps with independent atomic bitops
and assumes a multiple-producer, single-consumer protocol. That model
does not actually hold.
kvm_riscv_vcpu_sync_interrupts() is not a pure consumer. When the guest
changes guest-visible HVIP state, sync_interrupts() writes both
irqs_pending and irqs_pending_mask to reflect the new guest state back
into KVM state. As a result, irqs_pending and irqs_pending_mask form a
single logical state transition, but they are not updated atomically as
a pair.
This allows a race where a newly injected interrupt is lost. For
example:
CPU0 CPU1
---- ----
kvm_riscv_vcpu_set_interrupt(VS_SOFT)
set_bit(VS_SOFT, irqs_pending)
kvm_riscv_vcpu_sync_interrupts()
sees guest-cleared HVIP.VSSIP
sets irqs_pending_mask
clear_bit(IRQ_VS_SOFT, irqs_pending)
set_bit(VS_SOFT, irqs_pending_mask)
kvm_vcpu_kick()
After that interleaving, a later flush can update HVIP without VSSIP
even though a new virtual interrupt was injected. In practice, the
guest can remain blocked in WFI with work pending.
The same pending/mask protocol is shared by VS soft interrupts, PMU
overflow delivery, and AIA high interrupt synchronization, so the race
is not limited to one interrupt source.
Fix this by serializing all updates to irqs_pending and irqs_pending_mask
with a per-vCPU raw spinlock. This keeps the pending bit and the dirty
mask as one state transition across:
- set/unset interrupt
- guest HVIP sync
- interrupt flush to guest CSR state
- vCPU reset
- AIA CSR writes that clear dirty state
Use non-atomic bitmap operations while holding the lock. Hold the lock
across the AIA sync, flush, and pending checks as well, so both bitmap
words share the same serialization domain.
This intentionally replaces the existing lockless protocol instead of
trying to repair it with additional barriers. The problem is not memory
ordering on a single field; it is that two separate bitmaps encode one
shared state machine while both producers and sync paths can modify
them. A per-vCPU raw spinlock keeps the fix small, local, and suitable
for backporting.
Fixes: cce69aff68 ("RISC-V: KVM: Implement VCPU interrupts and requests handling")
Cc: stable@vger.kernel.org
Signed-off-by: Xie Bo <xb@ultrarisc.com>
Reviewed-by: Anup Patel <anup@brainfault.org>
Link: https://lore.kernel.org/r/20260715020359.1521354-2-xb@ultrarisc.com
Signed-off-by: Anup Patel <anup@brainfault.org>
When an instruction guest-page-fault targets a GPA that is not backed
by any memslot, KVM has no MMIO emulation path for the fetch. Load and
store guest-page faults can be routed through MMIO emulation, but an
instruction fetch has no data payload or access size for userspace to
complete in the same way.
Treat this case as an architectural access fault in the guest. On bare
metal, fetching from an inaccessible physical address raises an
instruction access fault for the supervisor to handle through its trap
vector. Reflect EXC_INST_ACCESS back to the guest so the guest observes
the same class of exception rather than leaving the fetch as a
host-handled condition.
stval contains the virtual address of the portion of the instruction that
caused the fault, while sepc points to the beginning of the instruction.
Signed-off-by: Qingwei Hu <qingwei.hu@bytedance.com>
Reviewed-by: Anup Patel <anup@brainfault.org>
Link: https://lore.kernel.org/r/20260707122548.281685-1-qingwei.hu@bytedance.com
Signed-off-by: Anup Patel <anup@brainfault.org>
All config and core registers of the KVM RISC-V ONE_REG interface are
expected to be always available to the KVM user-space and the KVM
get-reg-list selftest assumes these registers to be as base registers.
Currently, the Zicbo[m|z|p] block size config registers are only
available when corresponding ISA extension is present on the host
which breaks the above expectation. In fact, KVM get-reg-list selftest
fails when any of the Zicbo[m|z|p] ISA extension is not present on
host. To address this issue, drop the ISA extension checks from
kvm_riscv_vcpu_get/set_reg_config() and copy_config_reg_indices()
functions.
Fixes: 031f9efafc ("KVM: riscv: Add KVM_GET_REG_LIST API support")
Fixes: a044ef7104 ("RISC-V: KVM: use ENOENT in *_one_reg() when extension is unavailable")
Fixes: 48e2febcda ("RISC-V: KVM: Provide UAPI for Zicbop block size")
Fixes: cf05b059d5 ("RISC-V: KVM: Introduce common kvm_riscv_isa_check_host()")
Signed-off-by: Anup Patel <anup.patel@oss.qualcomm.com>
Link: https://lore.kernel.org/r/20260706181522.2003922-1-anup.patel@oss.qualcomm.com
Signed-off-by: Anup Patel <anup@brainfault.org>
