When creating a vCPU, don't drop kvm->lock to when doing the bulk of actual
vCPU creation, as allowing multiple vCPUs to be created in parallel adds
significant complexity in KVM (as evidenced by the many related bugs), and
all known VMMs fully serialize vCPU creation. Remove all manually locking
of kvm->lock from kvm_arch_vcpu_{post,}create() for obvious reasons.
For many years, "everyone" has assumed that dropping kvm->lock was done for
performance reasons optimization, e.g. to allow userspace to create all
vCPUs concurrently for latency purposes. But as above, no known VMM does
that. Looking at the history of this code, before commit 11ec280471
("KVM: Convert vm lock to a mutex"), kvm->lock was a spinlock. I.e. KVM
*had* to drop kvm->lock when doing the bulk of vCPU creation, otherwise KVM
couldn't do normal memory allocations. When kvm->lock got turned into a
mutex for unrelated reasons, no one took advantage updated of the change to
simplify vCPU creation. And 19 years later, everyone just assumed that KVM
continued to deal with the complexity for performance reasons.
Furthermore, naively parallelizing vCPU creation in userspace is likely a
net negative due to the overheads of task creation. Unless a VMM carefully
avoids the extra overhead related to parallelization, e.g. spawns each
vCPU's thread before creating the vCPU, creating vCPUs concurrently is a
net *negative* up until about ~64 vCPUs, after which the times are a wash.
The absolute speed of light _is_ faster if KVM doesn't hold kvm-lock, but
at vCPU counts of ~16 or less, it's probably in the noise when considering
total VM creation time, as the added latency is less than 1ms up until 16
or so vCPUs.
On top of all that, KVM has had a *lot* of fatal bugs (most often found by
syzkaller) related to vCPUs being created while trying to do per-VM
operations (basically, see every flow that locks all vCPUs). I.e. the
parallel vCPU creation "support" is actively harmful as the only "use case"
is for misbehaving userspace to exploit KVM bugs.
Serializing vCPU creation will allow reverting commit 97d65b544f ("KVM:
Check for duplicate vcpu_id as early as possible"), which had "minor" math
error: the worst case scenario isn't "256 bytes per VM", it's "256 unsigned
longs per VM", i.e. 2048 bytes per VM, which doubles the size of each VM
and pushes several architectures into order-1 allocations.
Signed-off-by: Sean Christopherson <seanjc@google.com>
Tested-by: Jean-Christophe Guillain <jean-christophe@guillain.net>
Tested-by: Naveen N Rao (AMD) <naveen@kernel.org>
Message-ID: <20260921174445.911676-5-seanjc@google.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
- Invalidate the ITS translation cache when the guest changes the
base address of the ITS tables (Fuad Tabba)
- Skip saving ITS devices with device IDs that are out-of-bounds
rather than failing the entire ITS save ioctl (Fuad Tabba)
- Close race between VM teardown and invalidations of nested MMUs
when handling MMU operations that are allowed to block
(Lorenzo Stoakes)
- Various fixes for the handling of the host's untrusted SVE
configuration in pKVM (Fuad Tabba)
- Make sure that empty SMCCC ranges based at 0 are rejected by the
kvm_smccc_set_filter() (Karl Mehltretter)
- Revoke the host mapping for pKVM's private stack pages, along
with a new sanity check that all mappings in the hyp's private
VA range have been correctly marked as hyp-owned (Fuad Tabba)
- Lifetime fixes for the array of shadow stage-2 MMUs, ensuring that
concurrent vCPU initialization cannot relocate in-use MMUs. Defer
the freeing of shadow stage-2 MMUs to the point that no other users
(e.g. MMU notifier) could reference them (Marc Zyngier)
- Drop useless WARN when rejecting an unsupported ioctl for pKVM
(Fuad Tabba)
- Fix the steal_time selftest to install correctly-sized mappings for
non-4K hosts (Sebastian Ott)
- Correct mapping of fine-grained trap for GCSPOPX instruction
(Mark Brown)
- Fix KVM_BUG_ON() due to missing handling of DBGBXVR<n> from 32-bit
guests (Karl Mehltretter)
-----BEGIN PGP SIGNATURE-----
iI0EABYKADUWIQSNXHjWXuzMZutrKNKivnWIJHzdFgUCaq7S2hccb2xpdmVyLnVw
dG9uQGxpbnV4LmRldgAKCRCivnWIJHzdFuyFAPwMTw52duiQXiJd2MLUe7qndEWW
s5vcAPDZIdDOWY4HewD/Xh7j5EwanNDM2DDXvdrgjnsTrwsJBW2ULIBKS6lvOA4=
=fF21
-----END PGP SIGNATURE-----
Merge tag 'kvmarm-fixes-7.3-1' of https://git.kernel.org/pub/scm/linux/kernel/git/kvmarm/kvmarm into HEAD
KVM/arm64 changes for 7.3, take #2
- Invalidate the ITS translation cache when the guest changes the
base address of the ITS tables (Fuad Tabba)
- Skip saving ITS devices with device IDs that are out-of-bounds
rather than failing the entire ITS save ioctl (Fuad Tabba)
- Close race between VM teardown and invalidations of nested MMUs
when handling MMU operations that are allowed to block
(Lorenzo Stoakes)
- Various fixes for the handling of the host's untrusted SVE
configuration in pKVM (Fuad Tabba)
- Make sure that empty SMCCC ranges based at 0 are rejected by the
kvm_smccc_set_filter() (Karl Mehltretter)
- Revoke the host mapping for pKVM's private stack pages, along
with a new sanity check that all mappings in the hyp's private
VA range have been correctly marked as hyp-owned (Fuad Tabba)
- Lifetime fixes for the array of shadow stage-2 MMUs, ensuring that
concurrent vCPU initialization cannot relocate in-use MMUs. Defer
the freeing of shadow stage-2 MMUs to the point that no other users
(e.g. MMU notifier) could reference them (Marc Zyngier)
- Drop useless WARN when rejecting an unsupported ioctl for pKVM
(Fuad Tabba)
- Fix the steal_time selftest to install correctly-sized mappings for
non-4K hosts (Sebastian Ott)
- Correct mapping of fine-grained trap for GCSPOPX instruction
(Mark Brown)
- Fix KVM_BUG_ON() due to missing handling of DBGBXVR<n> from 32-bit
guests (Karl Mehltretter)
This is another run of the Coccinelle script for converting kmalloc()
family of allocations to kmalloc_obj() via the existing rules in
scripts/coccinelle/api/kmalloc_objs.cocci
This catches both the set of kmalloc() uses added since the first
kmalloc_obj() conversions in v7.0 and adds a large group missed in the
first pass due to Coccinelle not interacting well with the cleanup.h
scoped_...() family of macros[1]. I worked around this with spatch's
"--macro-file" argument to a file with all the scoped_...() macros mapped
to Coccinelle's YACFE_ITERATOR[2] as that was the closest viable control
flow indicator I could find.
Build tested allmodconfig on x86, arm64, arm, loongarch, mips, powerpc,
riscv, and s390 with no new warnings.
Link: https://lore.kernel.org/lkml/202609021314.8A9C0B8@keescook/ [1]
Link: https://github.com/coccinelle/coccinelle/blob/master/standard.h [2]
Signed-off-by: Kees Cook <kees+treewide@kernel.org>
Use READ_ONCE() in _destroy_pages_crste() to read the crste, avoid
dereferencing the pointer multiple times.
Fixes: a2c17f9270 ("KVM: s390: New gmap code")
Signed-off-by: Claudio Imbrenda <imbrenda@linux.ibm.com>
Message-ID: <20260828115439.145885-9-imbrenda@linux.ibm.com>
When dat_cond_set_storage_key() finds a large page, it will
conditionally set the storage key in absolute memory using
large_crste_to_phys() to get the absolute address.
There is a race window between dat_entry_walk() and
large_crste_to_phys(): the large page could have been split
concurrently, and large_crste_to_phys() might be called with a crste
that does not designate a large page, leading to crashes.
Similar issues were also present in dat_set_storage_key().
dat_get_storage_key() and dat_reset_reference_bit() did instead check
for a potential concurrent splitting of the large page, but then
handled it incorrectly.
Fix by performing a READ_ONCE on the crste pointer, checking and using
the result, instead of dereferencing the pointer again. In case a race
is detacted, try dat_entry_walk() again.
Fixes: 8e03e8316e ("KVM: s390: KVM page table management functions: storage keys")
Signed-off-by: Claudio Imbrenda <imbrenda@linux.ibm.com>
Message-ID: <20260828115439.145885-8-imbrenda@linux.ibm.com>
Like kvm_s390_inject_vcpu(), kvm_s390_set_irq_state() also needs the
kvm->srcu or the slots lock when performing the Store status operation.
Fix by taking kvm->srcu in kvm_s390_set_irq_state().
Fixes: ba5c1e9b6c ("KVM: s390: interrupt subsystem, cpu timer, waitpsw")
Fixes: 062e44a931 ("KVM: s390: Use srcu in kvm_arch_vcpu_unlocked_ioctl()")
Signed-off-by: Claudio Imbrenda <imbrenda@linux.ibm.com>
Message-ID: <20260828115439.145885-7-imbrenda@linux.ibm.com>
Move all code from s390_kvm_mmu_commit_memory_region() into
s390_kvm_mmu_prepare_memory_region(). This allows the function to fail
gracefully if needed. The previous behaviour was to print a warning and
continue execution with page tables inconsistent with the memslots.
Fixes: e38c884df9 ("KVM: s390: Switch to new gmap")
Signed-off-by: Claudio Imbrenda <imbrenda@linux.ibm.com>
Message-ID: <20260828115439.145885-6-imbrenda@linux.ibm.com>
Refactor dat_set_slot(), _dat_slot_pte(), _dat_slot_crste(). Now they
only take a struct kvm_s390_mmu_cache as priv. For dat_delete_slot(),
mc is NULL, as no allocations should take place.
This is needed as a prerequisite to move gmap DAT table setup from
kvm_arch_commit_memory_region() to kvm_arch_prepare_memory_region().
Signed-off-by: Claudio Imbrenda <imbrenda@linux.ibm.com>
Message-ID: <20260828115439.145885-5-imbrenda@linux.ibm.com>
In some circumstances, it is possible that the page of nested guest
memory that is being shadowed is not present at all in the parent guest
gmap. dat_entry_walk() will not find any leaf entry and return with
-ENOENT, which will erroneously be propagated all the way to userspace.
Fix by manually calling gmap_link() on the memory of the nested guest
that is being shadowed if the mapping was not already present.
Fixes: e38c884df9 ("KVM: s390: Switch to new gmap")
Signed-off-by: Claudio Imbrenda <imbrenda@linux.ibm.com>
Message-ID: <20260828115439.145885-4-imbrenda@linux.ibm.com>
The parameter "old" should be marked as const, to prevent compile-time
warnings.
Fixes: d487a24041 ("KVM: s390: Prepare gmap for a second KVM implementation")
Signed-off-by: Claudio Imbrenda <imbrenda@linux.ibm.com>
Message-ID: <20260828115439.145885-3-imbrenda@linux.ibm.com>
When the indicator and/or summary bits are set in the guest, the
accessed page was only marked dirty in KVM if the access was performed
using the slow path; accesses through the new kvm_arch_set_irq_inatomic
fast inject path would not mark the page as dirty.
Fix by adding/moving the missing calls to mark_page_dirty(). Note that
for the inatomic path set_page_dirty{,_lock}() is not needed as the
page stays pinned; the unpin path correctly marks it as dirty.
Opportunistically reorder the local variables to be in reverse
Christmas tree order and refactor to use guard().
Fixes: 1e95e3bc6b ("KVM: s390: Enable adapter_indicators_set to use mapped pages")
Signed-off-by: Claudio Imbrenda <imbrenda@linux.ibm.com>
Message-ID: <20260828115439.145885-2-imbrenda@linux.ibm.com>
* Add support for 'slot' based PMU events, paired with new UAPI that
compels the user to select a specific PMU implementation
* Lazy save/restore of vCPU state for pKVM, along with various fixes
and cleanups to the management of vCPU state between the untrusted
host and pKVM hypervisor
* Disable traps of EL1 registers for nested hypervisors when FEAT_NV2p1
is present, guaranteeing that EL2-specific register bits are stateful
in the EL1 counterpart
* Leverage FEAT_NV3 to avoid unnecessary ERET/TLBI traps when the scope
of those instructions remains 'in host' (i.e. L1 kernel/userspace)
* Pile of fixes for the management of the VNCR pseudo-TLB, such as
under-invalidations and races with concurrent TLBIs on other vCPUs
* Consolidate the non-protected and pKVM view of ICH_VTR_EL2 to a
runtime-patched constant, allowing the same data to be shared with
pKVM prior to dropping host privileges
* Considerable pile of LLM-assisted fixes around the shop but mostly in
the VGIC, our in-kernel generator of bugs (and sometimes interrupts)
LoongArch:
* Advertise already-supported capabilities.
* Some bug fixes about timer and MMIO.
* Some hardening about interrupt injection.
* Replace kvm_err() with kvm_pr_unimpl().
* Add FPU/LSX/LASX test cases for selftests.
RISC-V:
* Svadu/Zicfiss/Zicfilp FWFT support for Guest
* Use try_cmpxchg for IMSIC MRIF RMW
* More arch-specific tracepoints in KVM RISC-V
* Eager page splitting when enabling dirty logging
* Optimize hfence request handling for SMP Guests
* Improve dirty log clearing by skipping zero bits in mask
* Guard HFENCE range loops against overflow
* CPU PM notifiers in KVM RISC-V for non-retentive idle states
* Fix kernel-mode vector context save/restore for Guest
s390:
* Fixes for vfio-ap
* Fixes for the gmap rework
* Fixes for vsie
* AI triggered fixes all over
* diag9c tracing
* code move preparation for the additional arm64 support
* enable CONTEXT_ANALYSIS
x86:
* Perform spring cleaning on x86.{c,h} and asm/kvm_host.h, by adding regs.c
(the kvm_cache_regs.h => regs.h is already applied) and msrs.{c,h}, and moving
relevant code out of x86.c.
* Split kvm_mmu in three parts, respectively to describe the format of page
tables, walking the guest page tables and building the page tables. Always
use the same page table walker kvm->arch.gva_walk as the entry point to
convert a guest's virtual address, where the previous code used two
different kvm_mmu structs depending on whether the walk included nested
EPT/NPT or not. Make page fault vmexits reuse the permission checking
machinery that is used for guest page faults. This is both a cleanup
and a baby step towards supporting XS/XU memory permissions.
* Document some of the "fun" gotchas with the APIC base when creating IRQCHIPs
on x86.
* Remove a defunct masterclock update from kvm_xen_shared_info_init(). It
could result in incorrect kvmclock due to triggering an unnecessary
switch to/from masterclock mode.
* Skip Xen runstate time updates if time has effectively gone backwards, so
that the guest doesn't report 100% steal time for a very, very long time.
* Drop KVM's runtime updates of the Xen PV timing CPUID leaf, as KVM was
updating the wrong sub-leaf, and upstream KVM will soon provide all the
information needed by userspace to populate the CPUID field itself.
* Fix a bug where KVM would walk a newly created rmap without holding the rmap
lock (or mmu_lock) during aging.
* Fix a bug where aging TDP MMU SPTEs could clobber FROZEN SPTEs.
* Fix a variety of #DB priority bugs.
* Fix a class of races related to enabling Hyper-V emulation on a vCPU after
the vCPU is visible to the rest of KVM.
* Use static calls for nested virtualization ops.
* Move more KVM-internal code out of x86's kvm_host.h.
* Enumerate support for a variety of Zhaoxin instructions that don't require
explicit virtualization.
* Fix missing EFER validation bugs, including in the KVM_SET_SREGS* path.
* Harden kvm_vcpu_map() against double-mapping and thus leaking references.
* Misc fixes and cleanups, e.g. for largely benign syzkaller splats.
x86 (Intel):
* Zero a vCPU's entry in VMX's Posted Interrupt Descriptor table used for IPI
virtualization when the vCPU is freed, to fix a use-after-free where hardware
will write to a freed vCPU's PID.
* Service local TLB flushes on a failed nested VM-Enter to fix a bug where KVM
could miss a TLB on a future, successful VM-Enter with the same L2 VPID.
* Cap the maximum value shoved into the VMX Preemption Timer to workaround an
erratum that affects all existing Intel CPUs that support CPUID 0x15.
* Fix VPID virtualization bugs where KVM would fail to flush hardware TLBs.
* Harden the TDX "populate" ioctls against bad input, and to prepare
for supporting in-place private<=>shared conversion.
x86 (AMD):
* Forcefully invalidate SNP VMSA pages if their backing guest_memfd page is
zapped/invalidated, e.g. due to a PUNCH_HOLE in response to a Page-State
Change request.
* Remove a dying VM from the GA Log notifier list before the VM is actually
destroyed, to fix a potential use-after-free.
* While FOLL_WRITE was needed in the past to trigger CoW unsharing, nowadays
FOLL_LONGTERM does that already even without FOLL_WRITE, and in fact,
get_user_pages() actually disallows FOLL_WRITE together with FOLL_LONGTERM.
So don't pass FOLL_WRITE when registering encrypted memory regions, i.e. when
pinning SEV/SEV-ES guest memory, to fix a regression with file-backed memory
introduced by KVM's (correct) usage of long-term pins.
(This was reviewed by mm maintainers; for more information, see commit
ee1a586dd1).
* Allocate full pages for SEV/SEV-ES {DE,EN}CRYPT ops on SNP-enabled hosts to
fix a data corruption issue due to the PSP driver assigning to-be-written
pages to firmware (as required by the SNP specs).
* Unconditionally intercept ICBEP so that KVM generates the correct guest RIP
when handling an ICEBP-induced TASK_SWITCH #VMEXIT.
* Harden the SNP "populate" ioctls against bad input, and to prepare
for supporting in-place private<=>shared conversion.
Generic:
* Remove kvm_debugfs_dir if kvm_init() fails after creating KVM's debugfs.
* Add a per-VM bitmap to track which vCPU IDs have been "claimed" but for
which the vCPU isn't yet online, and use the bitmap to reject duplicate IDs
before calling into arch code. This allows arch code to consume vcpu_id
without having to worry about cross-vCPU clobbering (at least s390 and x86
have had related bugs).
* Rework the so called "prepare" and "invalidate" guest_memfd hooks to prepare
for in-place private<=>shared conversion, and clean up a few warts along the
way.
Selftests:
* Automatically allocate a full page for L2 guest stacks on x86 instead of
requiring test-specific L1 guest code to carve out a portion of the L1
stack for L2 usage, and to ensure the L2 stack also adheres to the x86-64
calling convention ABI.
* Add a selftest to verify {Guest,Host}-Only behavior in x86's mediated PMU.
* Clean up nested SVM's handling of GPRs on L2<=>L1 transitions, reuse the
functionality for nested VMX, and drop the ucall hack that was fudging
around the lack of GPR switching on nVMX.
* Add a stress test to verify KVM doesn't clobber/drop #PF state, e.g. CR2,
across save/restore, including when L2 is active.
* Add a test to verify KVM_CREATE_VM accepts exactly what is reported by
KVM_CAP_VM_TYPES.
* Misc selftests fixes and cleanups
* Fix several issues with seeding the pRNG, and rework the pRNG APIs to that
the pRNG can be sanely used in host code, not just guest code.
* Add an IRQ test to validate virtual IRQ deliverty for IRQs wired up via
KVM_IRQFD + KVM_SET_GSI_ROUTING, with optional support for triggering IRQs
via writes to an assigned VFIO device.
* Add syscall wrappers to assert success on a variety of pthreads and CPU
affinity APIs.
* Set vCPU pthread affinity as early as possible to reduce contention issues
that were surfaced by PREEMPT_LAZY, which result in runtimes of over a
minute on large hosts, versus the expected ~5 seconds.
* Rework the PMU counters test to run each testcase using a single VM with
many vCPUs for each sub-testcase, instead of using a unique VM for each
sub-testcase. This cuts the runtime by ~20x.
Miscellaneous:
* MAINTAINERS updates for vfio-ap, guest_memfd, kvm-x86. Mostly representing
the status quo more accurately, but also... welcome David Hildenbrand
as guest_memfd reviewer!
-----BEGIN PGP SIGNATURE-----
iQFIBAABCAAyFiEE8TM4V0tmI4mGbHaCv/vSX3jHroMFAmqMgosUHHBib256aW5p
QHJlZGhhdC5jb20ACgkQv/vSX3jHroP8bwf+ORImBMDM3QEmybZM3I+N2+xqSuHP
QHttbmqGbsFK/RUeH96/X/+P9waqaz3uVeUQ6Qp2r0ryqwKtLt8YvIxKDp+M0vVJ
n+iukk1xulBEc28aGdKHn9G4wayAwDA/9f7CvJ23hojaJfLScbF3OlFkDd7y5DpO
x15Rtg9folYUjjop3LDML4N9/9Qmk4KRvVZ4ZVv6IB4uGJJ72fLd5dbBMyDk+BLl
Lz9N1xVTXcnJXJmrjMB4/QNt/HiQJdun8LcokJZyykta7Xx6aY7OGZv+VmCeq47K
e9Hk8mD4AyAdVbVvIntROSeBJOrlsgWJXAPHea6wEdRQnS3XVEBuZC0yDA==
=rfLh
-----END PGP SIGNATURE-----
Merge tag 'for-linus' of git://git.kernel.org/pub/scm/virt/kvm/kvm
Pull kvm updates from Paolo Bonzini:
"ARM64:
- Add support for 'slot' based PMU events, paired with new UAPI that
compels the user to select a specific PMU implementation
- Lazy save/restore of vCPU state for pKVM, along with various fixes
and cleanups to the management of vCPU state between the untrusted
host and pKVM hypervisor
- Disable traps of EL1 registers for nested hypervisors when
FEAT_NV2p1 is present, guaranteeing that EL2-specific register bits
are stateful in the EL1 counterpart
- Leverage FEAT_NV3 to avoid unnecessary ERET/TLBI traps when the
scope of those instructions remains 'in host' (i.e. L1
kernel/userspace)
- Pile of fixes for the management of the VNCR pseudo-TLB, such as
under-invalidations and races with concurrent TLBIs on other vCPUs
- Consolidate the non-protected and pKVM view of ICH_VTR_EL2 to a
runtime-patched constant, allowing the same data to be shared with
pKVM prior to dropping host privileges
- Considerable pile of LLM-assisted fixes around the shop but mostly
in the VGIC, our in-kernel generator of bugs (and sometimes
interrupts)
LoongArch:
- Advertise already-supported capabilities
- Some bug fixes about timer and MMIO
- Some hardening about interrupt injection
- Replace kvm_err() with kvm_pr_unimpl()
- Add FPU/LSX/LASX test cases for selftests
RISC-V:
- Svadu/Zicfiss/Zicfilp FWFT support for Guest
- Use try_cmpxchg for IMSIC MRIF RMW
- More arch-specific tracepoints in KVM RISC-V
- Eager page splitting when enabling dirty logging
- Optimize hfence request handling for SMP Guests
- Improve dirty log clearing by skipping zero bits in mask
- Guard HFENCE range loops against overflow
- CPU PM notifiers in KVM RISC-V for non-retentive idle states
- Fix kernel-mode vector context save/restore for Guest
s390:
- Fixes for vfio-ap
- Fixes for the gmap rework
- Fixes for vsie
- AI triggered fixes all over
- diag9c tracing
- code move preparation for the additional arm64 support
- enable CONTEXT_ANALYSIS
x86:
- Perform spring cleaning on x86.{c,h} and asm/kvm_host.h, by adding
regs.c (the kvm_cache_regs.h => regs.h is already applied) and
msrs.{c,h}, and moving relevant code out of x86.c
- Split kvm_mmu in three parts, respectively to describe the format
of page tables, walking the guest page tables and building the page
tables. Always use the same page table walker kvm->arch.gva_walk as
the entry point to convert a guest's virtual address, where the
previous code used two different kvm_mmu structs depending on
whether the walk included nested EPT/NPT or not. Make page fault
vmexits reuse the permission checking machinery that is used for
guest page faults. This is both a cleanup and a baby step towards
supporting XS/XU memory permissions
- Document some of the "fun" gotchas with the APIC base when creating
IRQCHIPs on x86
- Remove a defunct masterclock update from kvm_xen_shared_info_init().
It could result in incorrect kvmclock due to triggering an
unnecessary switch to/from masterclock mode
- Skip Xen runstate time updates if time has effectively gone
backwards, so that the guest doesn't report 100% steal time for
a very, very long time
- Drop KVM's runtime updates of the Xen PV timing CPUID leaf, as KVM
was updating the wrong sub-leaf, and upstream KVM will soon provide
all the information needed by userspace to populate the CPUID field
itself
- Fix a bug where KVM would walk a newly created rmap without holding
the rmap lock (or mmu_lock) during aging
- Fix a bug where aging TDP MMU SPTEs could clobber FROZEN SPTEs
- Fix a variety of #DB priority bugs
- Fix a class of races related to enabling Hyper-V emulation on a
vCPU after the vCPU is visible to the rest of KVM
- Use static calls for nested virtualization ops
- Move more KVM-internal code out of x86's kvm_host.h
- Enumerate support for a variety of Zhaoxin instructions that don't
require explicit virtualization
- Fix missing EFER validation bugs, including in the KVM_SET_SREGS*
path
- Harden kvm_vcpu_map() against double-mapping and thus leaking
references
- Misc fixes and cleanups, e.g. for largely benign syzkaller splats
x86 (Intel):
- Zero a vCPU's entry in VMX's Posted Interrupt Descriptor table used
for IPI virtualization when the vCPU is freed, to fix a
use-after-free where hardware will write to a freed vCPU's PID
- Service local TLB flushes on a failed nested VM-Enter to fix a bug
where KVM could miss a TLB on a future, successful VM-Enter with
the same L2 VPID
- Cap the maximum value shoved into the VMX Preemption Timer to
workaround an erratum that affects all existing Intel CPUs that
support CPUID 0x15
- Fix VPID virtualization bugs where KVM would fail to flush hardware
TLBs
- Harden the TDX "populate" ioctls against bad input, and to prepare
for supporting in-place private<=>shared conversion
x86 (AMD):
- Forcefully invalidate SNP VMSA pages if their backing guest_memfd
page is zapped/invalidated, e.g. due to a PUNCH_HOLE in response to
a Page-State Change request
- Remove a dying VM from the GA Log notifier list before the VM is
actually destroyed, to fix a potential use-after-free
- While FOLL_WRITE was needed in the past to trigger CoW unsharing,
nowadays FOLL_LONGTERM does that already even without FOLL_WRITE,
and in fact, get_user_pages() actually disallows FOLL_WRITE
together with FOLL_LONGTERM. So don't pass FOLL_WRITE when
registering encrypted memory regions, i.e. when pinning SEV/SEV-ES
guest memory, to fix a regression with file-backed memory
introduced by KVM's (correct) usage of long-term pins
(This was reviewed by mm maintainers; for more information, see
commit ee1a586dd1 "KVM: SEV: Drop FOLL_WRITE for encrypted region
registration")
- Allocate full pages for SEV/SEV-ES {DE,EN}CRYPT ops on SNP-enabled
hosts to fix a data corruption issue due to the PSP driver
assigning to-be-written pages to firmware (as required by the SNP
specs)
- Unconditionally intercept ICBEP so that KVM generates the correct
guest RIP when handling an ICEBP-induced TASK_SWITCH #VMEXIT
- Harden the SNP "populate" ioctls against bad input, and to prepare
for supporting in-place private<=>shared conversion
Generic:
- Remove kvm_debugfs_dir if kvm_init() fails after creating KVM's
debugfs
- Add a per-VM bitmap to track which vCPU IDs have been "claimed" but
for which the vCPU isn't yet online, and use the bitmap to reject
duplicate IDs before calling into arch code. This allows arch code
to consume vcpu_id without having to worry about cross-vCPU
clobbering (at least s390 and x86 have had related bugs)
- Rework the so called "prepare" and "invalidate" guest_memfd hooks
to prepare for in-place private<=>shared conversion, and clean up a
few warts along the way
Selftests:
- Automatically allocate a full page for L2 guest stacks on x86
instead of requiring test-specific L1 guest code to carve out a
portion of the L1 stack for L2 usage, and to ensure the L2 stack
also adheres to the x86-64 calling convention ABI
- Add a selftest to verify {Guest,Host}-Only behavior in x86's
mediated PMU
- Clean up nested SVM's handling of GPRs on L2<=>L1 transitions,
reuse the functionality for nested VMX, and drop the ucall hack
that was fudging around the lack of GPR switching on nVMX
- Add a stress test to verify KVM doesn't clobber/drop #PF state,
e.g. CR2, across save/restore, including when L2 is active
- Add a test to verify KVM_CREATE_VM accepts exactly what is reported
by KVM_CAP_VM_TYPES
- Misc selftests fixes and cleanups
- Fix several issues with seeding the pRNG, and rework the pRNG APIs
to that the pRNG can be sanely used in host code, not just guest
code
- Add an IRQ test to validate virtual IRQ deliverty for IRQs wired up
via KVM_IRQFD + KVM_SET_GSI_ROUTING, with optional support for
triggering IRQs via writes to an assigned VFIO device
- Add syscall wrappers to assert success on a variety of pthreads and
CPU affinity APIs
- Set vCPU pthread affinity as early as possible to reduce contention
issues that were surfaced by PREEMPT_LAZY, which result in runtimes
of over a minute on large hosts, versus the expected ~5 seconds
- Rework the PMU counters test to run each testcase using a single VM
with many vCPUs for each sub-testcase, instead of using a unique VM
for each sub-testcase. This cuts the runtime by ~20x
Miscellaneous:
- MAINTAINERS updates for vfio-ap, guest_memfd, kvm-x86. Mostly
representing the status quo more accurately, but also... welcome
David Hildenbrand as guest_memfd reviewer!"
* tag 'for-linus' of git://git.kernel.org/pub/scm/virt/kvm/kvm: (413 commits)
KVM: arm64: Validate GICv5 timer PPIs before claiming ownership
KVM: arm64: vgic: Reject out-of-range GICv5 PPI IDs
KVM: arm64: vgic: Prevent speculative SPI array underflow
KVM: arm64: vgic: Free gic_kvm_info on initialization failure
KVM: arm64: Avoid mismatched accesses to 'struct kvm_nvhe_init_params'
s390/vfio-ap: Fix NULL deref in status_show() during queue probe
s390/vfio-ap: Fix hot-unplug skipped when last AP adapter or domain removed
s390/vfio-ap: fix potential use of uninitialized apm_filtered bitmap
s390/vfio-ap: Fix control domain removal in vfio_ap_mdev_cfg_remove
s390/vfio-ap: Fix required lock not held during update of ap_matrix_mdev object
s390/vfio-ap: Fix missing lock required to access list of ap_matrix_mdev objects
s390/vfio-ap: Fix dereference matrix_mdev->kvm without checking for NULL
s390/vfio-ap: Fix stale do_remove flag across iterations in vfio_ap_mdev_cfg_remove
RISC-V: KVM: fix vcpu vector context handling for kernel-mode vector
riscv: vector: allow non-preemptible kernel-mode vector with IRQs off
riscv: vector: refactor riscv_v_start_kernel_context
KVM: s390: gmap: Make prefix handling optional
KVM: s390: gmap: Make CMMA optional
KVM: s390: gmap: Make storage keys optional
KVM: s390: Prepare gmap for a second KVM implementation
...
- Add a cpuidle driver with polling and enabled wait states using the
existing CPU idle infrastructure and idle governor to improve latency
for frequent sleep/wakeup cycles. Remove the obsolete tick delay
heuristic and generic arch_needs_cpu() hook. Add the corresponding
driver entry to MAINTAINERS
- Add kCFI support using the generic support provided by Clang
- Enable Clang CONTEXT_ANALYSIS for various architecture code and for
char, PCI, CIO and virtio drivers. Add required lock annotations,
exclude unsupported mm helpers and remove conditional PCI locking
- Fix secure storage access exception handling and reintroduce
DCACHE_WORD_ACCESS previously removed as a workaround
- Fix cpum_cf perf crashes when CPUs are brought online while per-task
events are active. Allocate and remove per-CPU counter data from CPU
hotplug callbacks
- Fix a deadlock when an s390dbf debug area is unregistered while one
of its debugfs files is being written to
- Fix MVIY_PERCPU() with binutils older than 2.39, where an assembler
macro silently omitted an instruction needed to repair interrupted
operations after CPU migration
- Remove/replace cond_resched() calls which are no-ops with the supported
s390 preemption models
- Fix AP queue depth and maximum message length decoding according to
the architecture. Current hardware is not affected, but future hardware
could report values which were handled incorrectly
- Reflect the configured CPU state in cpu_enabled_mask so deconfigured
CPUs are not presented as available for onlining
- Restore the vDSO GNU_EH_FRAME program header which was lost when the
build switched to direct linker invocation, and mark it read-only
- Add SCLP action qualifiers used by Spyre for card initialization,
recoverable error and telemetry reporting
- Move KMSAN interrupt flag helpers out of line to fix
-Wstatic-in-inline build warnings
- Use level-specific page table entry accessors for hugetlb entries and
ptep_get() when accessing crashed kernel memory in kdump
- Make forced AP bus rescans killable so that a user process blocked
behind an ongoing scan can still be terminated with SIGKILL
- Rework pkey ioctl error paths to remove duplicated cleanup code and
avoid freeing error pointers
- Allow the protected guest SWIOTLB buffer to be allocated outside the
first 2GB. Also enable dynamic SWIOTLB growth and the coherent atomic
pool fallback to improve I/O behavior when the initial pool is exhausted
- Add program check statistics and spinlock contention tracepoints.
Increase the lockdep chain capacity to keep lockdep enabled for complex
code paths such as btrfs
- Simplify IPL, trap and syscall code and remove the obsolete unistd_32.h
generation entry
-----BEGIN PGP SIGNATURE-----
iQEzBAABCgAdFiEE3QHqV+H2a8xAv27vjYWKoQLXFBgFAmqLHuoACgkQjYWKoQLX
FBhf2Qf+JlV+jQM1Lvn/Dj16vuQ77a4aP5C/OnLGMaTrrzbX420qU04yvC96v2Xu
ux01aDU9VakonE74IT0NmrNo1VDUk8nSvIWUTB6GH7KvK76VEZN5Kkyn8TmeRmE0
bZ0Fg7MgnhwdYijFDiX9w4rLyirwxs7vkScdJdJd0iKEdoZHXojGSjPDvmSpXght
FgCszt+YOqu9MMf9B5oGAl+P40mgPTlm6M+ygoe2dX7qPQBUHLbDPTgZiWnKdXi2
LPx0QPEha921ePDWrWz2HEqNetMfwGl12iertXddf1uzuK6LLObi0M5QrGw/ZbOy
UJFM+AjFekTQyZPSunD4NWyCjglqrA==
=XF7Y
-----END PGP SIGNATURE-----
Merge tag 's390-7.3-1' of git://git.kernel.org/pub/scm/linux/kernel/git/s390/linux
Pull s390 updates from Vasily Gorbik:
- Add a cpuidle driver with polling and enabled wait states using the
existing CPU idle infrastructure and idle governor to improve latency
for frequent sleep/wakeup cycles. Remove the obsolete tick delay
heuristic and generic arch_needs_cpu() hook. Add the corresponding
driver entry to MAINTAINERS
- Add kCFI support using the generic support provided by Clang
- Enable Clang CONTEXT_ANALYSIS for various architecture code and for
char, PCI, CIO and virtio drivers. Add required lock annotations,
exclude unsupported mm helpers and remove conditional PCI locking
- Fix secure storage access exception handling and reintroduce
DCACHE_WORD_ACCESS previously removed as a workaround
- Fix cpum_cf perf crashes when CPUs are brought online while per-task
events are active. Allocate and remove per-CPU counter data from CPU
hotplug callbacks
- Fix a deadlock when an s390dbf debug area is unregistered while one
of its debugfs files is being written to
- Fix MVIY_PERCPU() with binutils older than 2.39, where an assembler
macro silently omitted an instruction needed to repair interrupted
operations after CPU migration
- Remove/replace cond_resched() calls which are no-ops with the
supported s390 preemption models
- Fix AP queue depth and maximum message length decoding according to
the architecture. Current hardware is not affected, but future
hardware could report values which were handled incorrectly
- Reflect the configured CPU state in cpu_enabled_mask so deconfigured
CPUs are not presented as available for onlining
- Restore the vDSO GNU_EH_FRAME program header which was lost when the
build switched to direct linker invocation, and mark it read-only
- Add SCLP action qualifiers used by Spyre for card initialization,
recoverable error and telemetry reporting
- Move KMSAN interrupt flag helpers out of line to fix
-Wstatic-in-inline build warnings
- Use level-specific page table entry accessors for hugetlb entries and
ptep_get() when accessing crashed kernel memory in kdump
- Make forced AP bus rescans killable so that a user process blocked
behind an ongoing scan can still be terminated with SIGKILL
- Rework pkey ioctl error paths to remove duplicated cleanup code and
avoid freeing error pointers
- Allow the protected guest SWIOTLB buffer to be allocated outside the
first 2GB. Also enable dynamic SWIOTLB growth and the coherent atomic
pool fallback to improve I/O behavior when the initial pool is
exhausted
- Add program check statistics and spinlock contention tracepoints.
Increase the lockdep chain capacity to keep lockdep enabled for
complex code paths such as btrfs
- Simplify IPL, trap and syscall code and remove the obsolete
unistd_32.h generation entry
* tag 's390-7.3-1' of git://git.kernel.org/pub/scm/linux/kernel/git/s390/linux: (59 commits)
s390/percpu: Fix MVIY_PERCPU() with older binutils
s390/debug: Fix deadlock during unregister
s390/cpum_cf: Handle CPU hotplug via prepare/dead callbacks
s390: Enable CONTEXT_ANALYSIS for various directories
s390/mm: Add __context_unsafe() attribute to gmap helper functions
s390/mm: Add __context_unsafe() attribute to do_secure_storage_access()
s390/sysinfo: Add context analysis attributes
s390/irqflags: Add out-of-line definitions of arch_local_irq_*() for KMSAN
s390/virtio: Enable CONTEXT_ANALYSIS
s390/cio: Enable CONTEXT_ANALYSIS
s390/vfio_ccw: Add __must_hold() attribute to vfio_ccw_sch_quiesce()
s390/pci: Enable CONTEXT_ANALYSIS
s390/pci: Rework __zpci_event_availability() to remove conditional locking
s390/pci: Rework __zpci_event_error() to remove conditional locking
s390/char: Enable CONTEXT_ANALYSIS
s390/con3215: Add __must_hold() attribute to raw3215_make_room()
s390/ap: Fix MAPML computation
s390/cio: Remove cond_resched() calls
s390: Remove cond_resched() calls
KVM: s390: Remove cond_resched() calls
...
Guard guest prefix handling behind `KVM_S390_MANAGES_S390_GUEST`. This
enables other KVM implementations to use gmap without implementing
prefix handling. The prefix handling is integrated deeply in the gmap
implementation. Therefore, provide safe default implementations for the
guarded functions.
No functional changes.
Signed-off-by: Steffen Eiden <seiden@linux.ibm.com>
Reviewed-by: Christian Borntraeger <borntraeger@linux.ibm.com>
Signed-off-by: Christian Borntraeger <borntraeger@linux.ibm.com>
Guard guest CMMA behind `KVM_S390_MANAGES_S390_GUEST`. This enables
other KVM implementations to use gmap without implementing CMMA.
No functional changes.
Signed-off-by: Steffen Eiden <seiden@linux.ibm.com>
Reviewed-by: Christian Borntraeger <borntraeger@linux.ibm.com>
Signed-off-by: Christian Borntraeger <borntraeger@linux.ibm.com>
Guard guest storage key handling behind `KVM_S390_MANAGES_S390_GUEST`.
This enables other KVM implementations to use gmap without implementing
storage key infrastructure. Define KVM_S390_MANAGES_S390_GUEST to 1 for
KVM hosts managing s390 guests (in kvm_host_s390.h). A KVM
implementation not implementing those features must define
KVM_S390_MANAGES_S390_GUEST to 0 in its kvm_host_<guest_arch>.h.
No functional changes besides the new guard.
Signed-off-by: Steffen Eiden <seiden@linux.ibm.com>
Reviewed-by: Christian Borntraeger <borntraeger@linux.ibm.com>
Signed-off-by: Christian Borntraeger <borntraeger@linux.ibm.com>
Refactor gmap code such that a second s390 (host) KVM implementation can
use the gmap code as well. Move mmu code from s390 to gmap so the other
KVM implementation can use it as well.
No functional change.
Signed-off-by: Steffen Eiden <seiden@linux.ibm.com>
Reviewed-by: Christian Borntraeger <borntraeger@linux.ibm.com>
Signed-off-by: Christian Borntraeger <borntraeger@linux.ibm.com>
Move all the code required to run s390 KVM guests on s390 to a s390
subdirectory. Move gmap related code into a gmap directory to later
share gmap code between KVM implementations.
Update S390 VFIO-PCI MAINTAINERS filepath.
No functional change.
Signed-off-by: Steffen Eiden <seiden@linux.ibm.com>
Reviewed-by: Christian Borntraeger <borntraeger@linux.ibm.com>
Signed-off-by: Christian Borntraeger <borntraeger@linux.ibm.com>
Rename kvm-s390.c to s390.c and kvm-s390.h to s390.h for consistency
with the new directory structure. Update all include statements and
simplify Makefile ccflags from explicit paths to -I$(src).
No functional change.
Signed-off-by: Steffen Eiden <seiden@linux.ibm.com>
Reviewed-by: Christian Borntraeger <borntraeger@linux.ibm.com>
Signed-off-by: Christian Borntraeger <borntraeger@linux.ibm.com>
Update include guard names in dat.h, faultin.h, and gmap.h to
ARCH_KVM_GMAP_* to reflect their upcoming relocation to a shared gmap
directory.
No functional change.
Signed-off-by: Steffen Eiden <seiden@linux.ibm.com>
Reviewed-by: Christian Borntraeger <borntraeger@linux.ibm.com>
Signed-off-by: Christian Borntraeger <borntraeger@linux.ibm.com>
Move the kvm_s390_major_guest_pfault trace event from trace.h to a new
trace_gmap.h header. This separates gmap-specific tracing from general
KVM/s390 tracing, preparing for code sharing between multiple KVM
implementations.
The trace event definition is updated to use local defines for
parameters so that they can be replaced later with ease for when another
KVM implementation uses these traces.
No functional change.
Signed-off-by: Steffen Eiden <seiden@linux.ibm.com>
Reviewed-by: Christian Borntraeger <borntraeger@linux.ibm.com>
Signed-off-by: Christian Borntraeger <borntraeger@linux.ibm.com>
In some circumstances, one bit of kernel stack could have been leaked
from dat_cond_set_storage_key().
Fix by clearing prev before use.
Fixes: 8e03e8316e ("KVM: s390: KVM page table management functions: storage keys")
Reviewed-by: Christian Borntraeger <borntraeger@linux.ibm.com>
Signed-off-by: Claudio Imbrenda <imbrenda@linux.ibm.com>
Message-ID: <20260812104436.109741-9-imbrenda@linux.ibm.com>
kvm_s390_clear_pv_state() needs to also clear the dumping flag, to
allow the protected VM to be started again (as non-protected, with all
protected state safely destroyed) after a forced reboot while a
protected dump was ongoing and not completed.
Fixes: e40df9efd6 ("KVM: s390: pv: clear the state without memset")
Reviewed-by: Christian Borntraeger <borntraeger@linux.ibm.com>
Signed-off-by: Claudio Imbrenda <imbrenda@linux.ibm.com>
Message-ID: <20260812104436.109741-8-imbrenda@linux.ibm.com>
When __inject_sigp_stop() is called for a Stop and Store Status
operation, if the vCPU is running, the interrupt is marked as pending
and the status is stored by the thread performing the KVM_RUN IOCTL.
If the vCPU is already stopped, the status is stored immediately.
Storing the status means writing into userspace, which might fault, and
__inject_sigp_stop() is called from do_inject_vcpu() which in turn is
always called holding a spinlock, which is obviously an issue.
Fix this by returning -EWOULDBLOCK from __inject_sigp_stop(), and
adding a bool flag to indicate whether a store status is needed. The
callers of do_inject_vcpu() are modified to pass the pointer to the
bool flag; whenever a Store Status operation is needed, the callers can
now perform it outside the spinlock.
Opportunistically refactor kvm_s390_set_irq_state() to use
scoped_guard() and __free().
Fixes: 6cddd432e3 ("KVM: s390: handle stop irqs without action_bits")
Signed-off-by: Claudio Imbrenda <imbrenda@linux.ibm.com>
[ Added Fixes tag while picking -- Claudio ]
Message-ID: <20260812104436.109741-7-imbrenda@linux.ibm.com>
In case of failure, pgste_get_trylock_multiple() will attempt to unlock
the locked PGSTEs based on whether the PCL is set. In some
circumstances this can lead to unlocking PGSTEs that were locked by
other threads.
Fix by unlocking the amount of PGSTEs that were actually locked,
ignoring the PCL bit in the array.
Fixes: 94fd9b16cc ("KVM: s390: KVM page table management functions: lifecycle management")
Reviewed-by: Christian Borntraeger <borntraeger@linux.ibm.com>
Reviewed-by: Christoph Schlameuss <schlameuss@linux.ibm.com>
Signed-off-by: Claudio Imbrenda <imbrenda@linux.ibm.com>
Message-ID: <20260812104436.109741-6-imbrenda@linux.ibm.com>
When attempting to report all pending floating interrupt to userspace,
the GISA IPM bits are atomically tested and cleared, and the
corresponding interrupt description is written in the output buffer. If
the output buffer is too small, an error is returned to userspace, but
the GISA IPM bits are now lost.
Moreover, the contract of KVM_DEV_FLIC_GET_ALL_IRQS, which is the only
path to get_all_floating_irqs(), states that:
> All interrupts remain pending, i.e. are not deleted from the list of
> currently pending interrupts.
Fix by non-destructively testing for the GISA IPM bits.
Fixes: 24160af6cb ("KVM: s390: add GISA interrupts to FLIC ioctl interface")
Reviewed-by: Christian Borntraeger <borntraeger@linux.ibm.com>
Signed-off-by: Claudio Imbrenda <imbrenda@linux.ibm.com>
Message-ID: <20260812104436.109741-4-imbrenda@linux.ibm.com>
kvm_arch_vcpu_unlocked_ioctl() is called without further locks held, but
kvm_s390_inject_vcpu(), which is called from there, needs either the
kvm->srcu or the slots lock.
Fix by taking the kvm->srcu in kvm_arch_vcpu_unlocked_ioctl().
Fixes: ba5c1e9b6c ("KVM: s390: interrupt subsystem, cpu timer, waitpsw")
Reviewed-by: Christian Borntraeger <borntraeger@linux.ibm.com>
Reviewed-by: Christoph Schlameuss <schlameuss@linux.ibm.com>
Signed-off-by: Claudio Imbrenda <imbrenda@linux.ibm.com>
Message-ID: <20260812104436.109741-3-imbrenda@linux.ibm.com>
Some callers pass NULL as oldkey. Calling page_cond_set_storage_key()
will cause that NULL pointer to get dereferenced.
Fix by checking for NULL and assigning the pointer to a dummy local
variable to avoid crashes.
Fixes: 8e03e8316e ("KVM: s390: KVM page table management functions: storage keys")
Reviewed-by: Christian Borntraeger <borntraeger@linux.ibm.com>
Reviewed-by: Christoph Schlameuss <schlameuss@linux.ibm.com>
Signed-off-by: Claudio Imbrenda <imbrenda@linux.ibm.com>
Message-ID: <20260812104436.109741-2-imbrenda@linux.ibm.com>
kvm_sigset_activate() installs vcpu->sigset via sigprocmask() and
stashes the caller's mask in current->real_blocked; only
kvm_sigset_deactivate() restores it.
For KVM_RUN on a STOPPED vcpu the error path will not restore the
userspace mask. Re-arrange the error handling to also restore the
signal mask.
Fixes: 6352e4d2dd ("KVM: s390: implement KVM_(S|G)ET_MP_STATE for user space state control")
Cc: stable@vger.kernel.org
Signed-off-by: Christian Borntraeger <borntraeger@linux.ibm.com>
Reviewed-by: Claudio Imbrenda <imbrenda@linux.ibm.com>
Signed-off-by: Claudio Imbrenda <imbrenda@linux.ibm.com>
Message-ID: <20260811153738.206885-5-borntraeger@linux.ibm.com>
The rc/rrc value is copied to the cmd location of the cmd in the kvm_pv_cmd
structure. Fix the offset.
Fixes: 8aba09588d ("KVM: s390: Add CPU dump functionality")
Cc: stable@vger.kernel.org
Signed-off-by: Christian Borntraeger <borntraeger@linux.ibm.com>
Reviewed-by: Claudio Imbrenda <imbrenda@linux.ibm.com>
Signed-off-by: Claudio Imbrenda <imbrenda@linux.ibm.com>
Message-ID: <20260811153738.206885-4-borntraeger@linux.ibm.com>
When shadowing crypto access bits from a format0 apcb (crycb 0 or 1),
the bits 64..255 are unchanged from whatever is in the vsie page in the
crycb and thus in the apcb. This gives a nested guest potential access
to a device no longer available. Zero out the remaining bits.
Fixes: 6b79de4b05 ("KVM: s390: vsie: allow guest FORMAT-1 CRYCB on host FORMAT-2")
Cc: stable@vger.kernel.org
Signed-off-by: Christian Borntraeger <borntraeger@linux.ibm.com>
Reviewed-by: Claudio Imbrenda <imbrenda@linux.ibm.com>
Signed-off-by: Claudio Imbrenda <imbrenda@linux.ibm.com>
Message-ID: <20260811153738.206885-3-borntraeger@linux.ibm.com>
Every other dat_* consumer in this file (kvm_s390_get_skeys,
set_skeys, get_cmma_bits, set_cmma_bits, MEM_CLR_CMMA,
kvm_s390_fixup_prefix, kvm_test_age_gfn, kvm_age_gfn) reads
kvm->arch.gmap->asce *inside* the mmu_lock read-side. keyop is the only
outlier.
gmap->asce is mutated under write_lock(mmu_lock) by gmap_set_limit()
and keyop might use a stale asce value for walking as KVM_S390_KEYOP
and KVM_S390_VM_MEM_LIMIT_SIZE can run concurrently. This can result
in memory corruption.
Fixes: 0ee4ddc164 ("KVM: s390: Storage key manipulation IOCTL")
Cc: stable@vger.kernel.org
Signed-off-by: Christian Borntraeger <borntraeger@linux.ibm.com>
Reviewed-by: Claudio Imbrenda <imbrenda@linux.ibm.com>
Signed-off-by: Claudio Imbrenda <imbrenda@linux.ibm.com>
Message-ID: <20260811153738.206885-2-borntraeger@linux.ibm.com>
Channel-subsystem damage machine checks are for the host channel
subsystem. The guest channel subsystem is emulated in the userspace VMM.
There is no point in forwarding such machine checks into the guest.
This also simplifies the machine check reinjection and avoids kfree of a
stack variable as reported by sashiko. There might be still machine
checks that have the ck bit set with another bit (like instruction
damage), mask out the CK bit in s390_backup_mcck_info(), like the CP and
ED bits already are.
Fixes: 4d62fcc0b6 ("KVM: s390: Inject machine check into the guest")
Cc: stable@vger.kernel.org
Signed-off-by: Christian Borntraeger <borntraeger@linux.ibm.com>
Acked-by: Heiko Carstens <hca@linux.ibm.com>
Acked-by: Claudio Imbrenda <imbrenda@linux.ibm.com>
Signed-off-by: Claudio Imbrenda <imbrenda@linux.ibm.com>
Message-ID: <20260806145835.31818-1-borntraeger@linux.ibm.com>
struct kvm_hw_breakpoint::len is a __u64 that is fully controlled by user
space. This is then assigned to wp_info->len, which is an int. The bounds
check is done on the truncated value while the allocation uses the
untruncated one:
wp_info->len = bp_data->len;
[...]
if (wp_info->len < 0 || wp_info->len > MAX_WP_SIZE)
return -EINVAL;
wp_info->old_data = kmalloc(bp_data->len, GFP_KERNEL_ACCOUNT);
Use the validated value for the allocation as intended. Without this
fix userspace can trigger >4GB allocations which will fail and result
in a WARN due to MAX_PAGE_ORDER.
Fixes: 27291e2165 ("KVM: s390: hardware support for guest debugging")
Cc: stable@vger.kernel.org
Signed-off-by: Christian Borntraeger <borntraeger@linux.ibm.com>
Reviewed-by: Claudio Imbrenda <imbrenda@linux.ibm.com>
Signed-off-by: Claudio Imbrenda <imbrenda@linux.ibm.com>
Message-ID: <20260805110455.7200-9-borntraeger@linux.ibm.com>
kvm_s390_clear_bp_data() is only called from
kvm_arch_vcpu_ioctl_set_guest_debug(), i.e. when user space changes or
disables debugging. A vCPU that is destroyed while hardware breakpoints
are still armed - the normal case when the VMM just exits or crashes -
leaks hw_bp_info, hw_wp_info and all old_data buffers, since generic KVM
frees the vCPU right after kvm_arch_vcpu_destroy().
That is bounded by MAX_BP_COUNT entries, so roughly 8 KiB per vCPU, but
it is unbounded over VM lifetimes. The allocations are
GFP_KERNEL_ACCOUNT, so the charge also outlives the exiting process and
pins dying memcgs.
Fix by clearing the debug data on vCPU destruction. Calling it
unconditionally is fine: struct kvm_vcpu is zero allocated, so for a vCPU
that never enabled debugging the counters are 0 and the pointers NULL.
Fixes: 27291e2165 ("KVM: s390: hardware support for guest debugging")
Cc: stable@vger.kernel.org
Signed-off-by: Christian Borntraeger <borntraeger@linux.ibm.com>
Reviewed-by: Matthew Rosato <mjrosato@linux.ibm.com>
Reviewed-by: Claudio Imbrenda <imbrenda@linux.ibm.com>
Signed-off-by: Claudio Imbrenda <imbrenda@linux.ibm.com>
Message-ID: <20260805110455.7200-8-borntraeger@linux.ibm.com>
__import_wp_info() backs up the original guest memory contents of a
watchpoint with read_guest_abs(), which is kvm_read_guest() and therefore
resolves the memslot via __kvm_memslots(). That requires kvm->srcu (or
kvm->slots_lock) to be held, otherwise a concurrent memslot update can
free the memslots array under us once its SRCU grace period has elapsed.
As this is not fast path, following lock ordering (mutex first, then
srcu) take the big hammer and hold the srcu for the full import.
Fixes: 27291e2165 ("KVM: s390: hardware support for guest debugging")
Cc: stable@vger.kernel.org
Signed-off-by: Christian Borntraeger <borntraeger@linux.ibm.com>
Reviewed-by: Claudio Imbrenda <imbrenda@linux.ibm.com>
Signed-off-by: Claudio Imbrenda <imbrenda@linux.ibm.com>
Message-ID: <20260805110455.7200-7-borntraeger@linux.ibm.com>
__import_wp_info() allocates a per-watchpoint old_data buffer to back up
the original guest memory contents. If a later watchpoint of the same
KVM_SET_GUEST_DEBUG request fails to import, kvm_s390_import_bp_data()
jumps to the error label, which frees the wp_info array but not the
old_data buffers of the entries that were imported successfully. Up to
MAX_BP_COUNT - 1 buffers of up to MAX_WP_SIZE bytes are leaked per failed
request, and the request can be repeated.
Create error handling for cleaning up all created old_data memory
areas.
Fixes: 27291e2165 ("KVM: s390: hardware support for guest debugging")
Cc: stable@vger.kernel.org
Signed-off-by: Christian Borntraeger <borntraeger@linux.ibm.com>
Reviewed-by: Matthew Rosato <mjrosato@linux.ibm.com>
Reviewed-by: Claudio Imbrenda <imbrenda@linux.ibm.com>
Signed-off-by: Claudio Imbrenda <imbrenda@linux.ibm.com>
Message-ID: <20260805110455.7200-6-borntraeger@linux.ibm.com>
bp_data is freed only for the error case by kfree(bp_data).
Every successful KVM_SET_GUEST_DEBUG will leak bp_data.
Fixes: 27291e2165 ("KVM: s390: hardware support for guest debugging")
Cc: stable@vger.kernel.org
Signed-off-by: Christian Borntraeger <borntraeger@linux.ibm.com>
Reviewed-by: Matthew Rosato <mjrosato@linux.ibm.com>
Reviewed-by: Claudio Imbrenda <imbrenda@linux.ibm.com>
Signed-off-by: Claudio Imbrenda <imbrenda@linux.ibm.com>
Message-ID: <20260805110455.7200-5-borntraeger@linux.ibm.com>
kvm_s390_reinject_machine_check() fills cr14, mcic, ext_damage_code and
failing_storage_address of the on-stack struct kvm_s390_irq, but struct
kvm_s390_mchk_info also has a pad word and a 16 byte fixed_logout array.
struct mcck_volatile_info carries no logout data, so there is nothing to
copy there and both stay whatever was on the stack.
__inject_mchk() then memcpy()s fixed_logout into the vcpu local
interrupt state unconditionally. This will reach the guest during
deliver and userspace during migration.
Reflecting zeroes is the correct behaviour here, as KVM has no logout
data for a reinjected machine check.
This needs a host machine check while the cpu is in SIE so not trivial
to trigger.
Fixes: 4d62fcc0b6 ("KVM: s390: Inject machine check into the guest")
Cc: stable@vger.kernel.org
Signed-off-by: Christian Borntraeger <borntraeger@linux.ibm.com>
Reviewed-by: Matthew Rosato <mjrosato@linux.ibm.com>
Reviewed-by: Claudio Imbrenda <imbrenda@linux.ibm.com>
Signed-off-by: Claudio Imbrenda <imbrenda@linux.ibm.com>
Message-ID: <20260805110455.7200-4-borntraeger@linux.ibm.com>
__kvm_inject_pfault_token() only sets .type and .u.ext.ext_params2 of
the on-stack struct kvm_s390_irq but the full ext substructure is copied
into the cpu local variable on inject. ext_params and pad contain stale
stack values.
Interrupt delivery only uses ext_params2, so nothing leaks to the guest,
but a host user can use the migration ioctls to get to the data.
Fix by zero-initializing the irq struct.
Do the same for the inti data structure.
Fixes: 383d0b0501 ("KVM: s390: handle pending local interrupts via bitmap")
Cc: stable@vger.kernel.org
Signed-off-by: Christian Borntraeger <borntraeger@linux.ibm.com>
Reviewed-by: Matthew Rosato <mjrosato@linux.ibm.com>
Reviewed-by: Claudio Imbrenda <imbrenda@linux.ibm.com>
Signed-off-by: Claudio Imbrenda <imbrenda@linux.ibm.com>
Message-ID: <20260805110455.7200-3-borntraeger@linux.ibm.com>
pin_map_page() fails legitimately whenever the userspace mapping behind
the adapter route has gone away, e.g. when the VMM unmaps that memory.
As this can happen without a kernel programming error, remove the
WARN_ON.
Fixes: 1e95e3bc6b ("KVM: s390: Enable adapter_indicators_set to use mapped pages")
Cc: Douglas Freimuth <freimuth@linux.ibm.com>
Cc: Matthew Rosato <mjrosato@linux.ibm.com>
Signed-off-by: Christian Borntraeger <borntraeger@linux.ibm.com>
Reviewed-by: Matthew Rosato <mjrosato@linux.ibm.com>
Reviewed-by: Douglas Freimuth <freimuth@linux.ibm.com>
Reviewed-by: Claudio Imbrenda <imbrenda@linux.ibm.com>
Signed-off-by: Claudio Imbrenda <imbrenda@linux.ibm.com>
Message-ID: <20260805110455.7200-2-borntraeger@linux.ibm.com>
Create a simple constant for the SCB alignment shift.
Signed-off-by: Christoph Schlameuss <schlameuss@linux.ibm.com>
Signed-off-by: Christian Borntraeger <borntraeger@linux.ibm.com>
The crypto control block address is required to have double word
alignment. Add a static_assert to enforce correct alignment.
Signed-off-by: Christoph Schlameuss <schlameuss@linux.ibm.com>
Reviewed-by: Christian Borntraeger <borntraeger@linux.ibm.com>
[borntraeger@linux.ibm.com: improve commit message]
Signed-off-by: Christian Borntraeger <borntraeger@linux.ibm.com>
Ensure that the backup info for machine check is the same offset as that
in struct sie_page!
With the assertion in place we do not need the comment anymore.
Signed-off-by: Christoph Schlameuss <schlameuss@linux.ibm.com>
Reviewed-by: Christian Borntraeger <borntraeger@linux.ibm.com>
Signed-off-by: Christian Borntraeger <borntraeger@linux.ibm.com>
Make the code slightly more readable by using phys_to_pfn instead of an
open coded shift.
Signed-off-by: Christoph Schlameuss <schlameuss@linux.ibm.com>
Reviewed-by: Christian Borntraeger <borntraeger@linux.ibm.com>
[borntraeger@linux.ibm.com: improve commit message]
Signed-off-by: Christian Borntraeger <borntraeger@linux.ibm.com>
Remove useless BUILD_BUG_ON() checking the size of struct vsie_page.
This is already covered by a static_assert at the struct vsie_page
definition.
Fixes: e38c884df9 ("KVM: s390: Switch to new gmap")
Signed-off-by: Christoph Schlameuss <schlameuss@linux.ibm.com>
[borntraeger@linux.ibm.com: improve commit message]
Signed-off-by: Christian Borntraeger <borntraeger@linux.ibm.com>
sclp.has_64bscao is already verified in the guard clause a few lines
above this. So we cannot reach this code if it is not true.
Reviewed-by: Hendrik Brueckner <brueckner@linux.ibm.com>
Reviewed-by: Eric Farman <farman@linux.ibm.com>
Reviewed-by: Janosch Frank <frankja@linux.ibm.com>
Signed-off-by: Christoph Schlameuss <schlameuss@linux.ibm.com>
Reviewed-by: Claudio Imbrenda <imbrenda@linux.ibm.com>
Reviewed-by: Janosch Frank <frankja@de.ibm.com>
Signed-off-by: Christian Borntraeger <borntraeger@linux.ibm.com>
Since [1] cond_resched() is a no-op on s390. Remove all calls.
This also entirely removes uv_call_sched() and replaces all call sites
with uv_call(), since both functions are identical after the removal
of cond_resched().
[1] commit 7dadeaa6e8 ("sched: Further restrict the preemption modes")
Reviewed-by: Claudio Imbrenda <imbrenda@linux.ibm.com>
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
If creating a protected vCPU in kvm_s390_pv_create_cpu() fails,
kvm_s390_pv_destroy_cpu() was called, which checks whether the vCPU has
a PV handle and exits doing nothing otherwise. At that point, due to
not having created the protected vCPU, the PV handle will not be set,
and kvm_s390_pv_destroy_cpu() will do nothing, thus leaking the
allocated memory.
Fix by factoring out the code to free and reset a PV vCPU; call it from
kvm_s390_pv_destroy_cpu() and kvm_s390_pv_create_cpu().
Opportunistically fix the return value of kvm_s390_pv_destroy_cpu() in
case of errors: return -EIO instead if EIO.
Fixes: d4074324b0 ("KVM: s390: pv: avoid double free of sida page")
Reviewed-by: Steffen Eiden <seiden@linux.ibm.com>
Reviewed-by: Janosch Frank <frankja@linux.ibm.com>
Signed-off-by: Claudio Imbrenda <imbrenda@linux.ibm.com>
Message-ID: <20260803124040.126471-14-imbrenda@linux.ibm.com>
