I hit the following on one of my machines:
mce: CPU0 BANK15 CMCI inherited storm
------------[ cut here ]------------
ODEBUG: assert_init not available (active state 0) object: (____ptrval____) object type: timer_list hint: 0x0
WARNING: lib/debugobjects.c:632 at debug_object_assert_init+0x178/0x230, CPU#0: swapper/0/0
CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Not tainted 7.2.0-rc5 #3 PREEMPTLAZY
RIP: 0010:debug_object_assert_init+0x18f/0x230
Call Trace:
<TASK>
__mod_timer
mce_timer_kick
cmci_discover
intel_init_cmci
mce_intel_feature_init
mcheck_cpu_init
identify_cpu
identify_boot_cpu
arch_cpu_finalize_init
start_kernel
A second splat follows right after, from timer_setup() finding that same
timer already queued:
ODEBUG: init active (active state 0) object: (____ptrval____) object type: timer_list hint: stub_timer+0x0/0x10
This is happening because CMCI storm detection is trying to modify the timer
before latter was properly set up.
Set up the timer first. __mcheck_cpu_setup_timer() only calls timer_setup(),
and depends on neither the generic nor the vendor init.
[ bp: Massage commit message. ]
Fixes: 1f68ce2a02 ("x86/mce: Handle Intel threshold interrupt storms")
Signed-off-by: Breno Leitao <leitao@debian.org>
Signed-off-by: Borislav Petkov (AMD) <bp@alien8.de>
Cc: stable@vger.kernel.org
Link: https://patch.msgid.link/20260803-mce_timer_init-v1-1-9539db424330@debian.org
The KVM clock is an interesting thing. It is defined as "nanoseconds
since the guest was created", but in practice it runs at two *different*
rates — or three different rates, if you count implementation bugs.
Definition A is that it runs synchronously with the CLOCK_MONOTONIC_RAW
of the host, with a delta of kvm->arch.kvmclock_offset.
But that version doesn't actually get used in the common case, where the
host has a reliable TSC and the guest TSCs are all running at the same
rate and in sync with each other, and kvm->arch.use_master_clock is set.
In that common case, definition B is used: There is a reference point in
time at kvm->arch.master_kernel_ns (again a CLOCK_MONOTONIC_RAW time),
and a corresponding host TSC value kvm->arch.master_cycle_now. This
fixed point in time is converted to guest units (the time offset by
kvmclock_offset and the TSC Value scaled and offset to be a guest TSC
value) and advertised to the guest in the pvclock structure. While in
this 'use_master_clock' mode, the fixed point in time never needs to be
changed, and the clock runs precisely in time with the guest TSC, at the
rate advertised in the pvclock structure.
The third definition C is implemented in kvm_get_wall_clock_epoch() and
__get_kvmclock(), using the master_cycle_now and master_kernel_ns fields
but converting the *host* TSC cycles directly to a value in nanoseconds
instead of scaling via the guest TSC.
One might naïvely think that all three definitions are identical, since
CLOCK_MONOTONIC_RAW is not skewed by NTP frequency corrections; all
three are just the result of counting the host TSC at a known frequency,
or the scaled guest TSC at a known precise fraction of the host's
frequency. The problem is with arithmetic precision, and the way that
frequency scaling is done in a division-free way by multiplying by a
scale factor, then shifting right. In practice, all three ways of
calculating the KVM clock will suffer a systemic drift from each other.
Eventually, definition C should just be eliminated. Commit 451a707813
("KVM: x86/xen: improve accuracy of Xen timers") worked around it for
the specific case of Xen timers, which are defined in terms of the KVM
clock and suffered from a continually increasing error in timer expiry
times. That commit notes that get_kvmclock_ns() is non-trivial to fix
and says "I'll come back to that", which remains true.
Definitions A and B do need to coexist, the former to handle the case
where the host or guest TSC is suboptimally configured. But KVM should
be more careful about switching between them, and the discontinuity in
guest time which could result.
In particular, KVM_REQ_MASTERCLOCK_UPDATE will take a new snapshot of
time as the reference in master_kernel_ns and master_cycle_now, yanking
the guest's clock back to match definition A at that moment.
When invoked from in 'use_master_clock' mode, kvm_update_masterclock()
should probably *adjust* kvm->arch.kvmclock_offset to account for the
drift, instead of yanking the clock back to definition A. But in the
meantime there are a bunch of places where it just doesn't need to be
invoked at all.
To start with: there is no need to do such an update when a Xen guest
populates the shared_info page. This seems to have been a hangover from
the very first implementation of shared_info which automatically
populated the vcpu_info structures at their default locations, but even
then it should just have raised KVM_REQ_CLOCK_UPDATE on each vCPU
instead of using KVM_REQ_MASTERCLOCK_UPDATE. And now that userspace is
expected to explicitly set the vcpu_info even in its default locations,
there's not even any need for that either.
Fixes: 629b534884 ("KVM: x86/xen: update wallclock region")
Reviewed-by: Paul Durrant <paul@xen.org>
Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
Signed-off-by: Sean Christopherson <seanjc@google.com>
In query_edd(), use bool for do_edd and initialize it directly from
CONFIG_EDD_OFF via IS_ENABLED(). Also use bool for do_mbr and be_quiet.
No change in functionality intended.
Signed-off-by: Thorsten Blum <thorsten.blum@linux.dev>
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Link: https://patch.msgid.link/20260804115227.62158-5-thorsten.blum@linux.dev
- Fix the boot-time memcmp() asm implementation's constraints
and optimization properties (Mauricio Faria de Oliveira)
- Move the 0xd0...0xd7 AMD Zen5 model range from the Zen6
range where it mistakenly ended up (Pratik Vishwakarma)
Signed-off-by: Ingo Molnar <mingo@kernel.org>
-----BEGIN PGP SIGNATURE-----
iQJFBAABCgAvFiEEBpT5eoXrXCwVQwEKEnMQ0APhK1gFAmpu9+0RHG1pbmdvQGtl
cm5lbC5vcmcACgkQEnMQ0APhK1j8Ig//WBoOGw9bbFkDJFUdIS59EdJS+4l7KTJh
eRxiExnwQluOpo1x4/mKAMmgqz7IoFCc0PjLXIBC3+rN+AvM5+L/qkBGtHmcaHEu
uXdU5zAYk6a7QfvK9C6k1C/v4HQKb3FLvUliJShKfLAv7ekLLvVHWxExmJYoBsrn
VxPIW3+rpnBJn4v5bLUVs/Z6h99aB4nicA4i6p7WKiy+Vr1qOSIu5T4piff27FJE
btT61yOfTKFoKgum5dI4ZYJTX29edXcJUMPidN/egJAxMQnUdI38fO/joYMreYsx
rJ3zuv3mdV+LiusPO7ZHcNVZtc09PPMM2krLrOyjSEHzhE/WGqLXNn6FKK5H+j86
xKyawqhD+2ANktj+09bFfu3cpZEz5bdifyKx6BtvgYkSEVaX055E1zzqYXZCIaVs
DqdAs3aSKNWAbJkfMONJMZ4P28ZGikppeefihG6hcAnBjAyQC9SaTOA37fHT6WQ/
bDf3NU5c+TX0TrWd696E9jy5cLjXL60jbVPL7SF7C3d8/QPPWCbOU14Dw3s4bWh+
WfptSuB9Iuk+aalRnAVWpUBI9DDUcomC3oDLLHttztXojdZPvP1zdUhDzjaJ0Im8
N8xC30PBMGIrFGJpXN4XRM3BlSxXBVGrmS/WAqE/KE51RcA5WzQBFMGBe4V3gRbO
OV3slKkCD9k=
=EHql
-----END PGP SIGNATURE-----
Merge tag 'x86-urgent-2026-08-02' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
Pull misc x86 fixes from Ingo Molnar:
- Fix the boot-time memcmp() asm implementation's constraints
and optimization properties (Mauricio Faria de Oliveira)
- Move the 0xd0...0xd7 AMD Zen5 model range from the Zen6
range where it mistakenly ended up (Pratik Vishwakarma)
* tag 'x86-urgent-2026-08-02' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip:
x86/CPU/AMD: Carve out a Zen5 models range
x86/boot: Add volatile, clobbers and zero-length test in memcmp()
Sort it according to the following precedence of struct cpuid_bit fields:
(level) -> (sub_leaf) -> (reg) -> (bit).
No functional changes.
Signed-off-by: Borislav Petkov (AMD) <bp@alien8.de>
btf_distill_func_proto() accepts a function argument up to 16 bytes, so a
128-bit scalar such as __int128 reaches the x86 trampoline with
arg_size == 16. But the current implementation assumes an __int128
argument only needs one register, so the register save area is
under-allocated and save_args() overwrites adjacent stack slots.
Compute the register count from arg_size for all arguments to fix it.
Fixes: a9c5ad31fb ("bpf: x86: Support in-register struct arguments in trampoline programs")
Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
Acked-by: Leon Hwang <leon.hwang@linux.dev>
Link: https://lore.kernel.org/bpf/20260729050204.2586457-1-yonghong.song@linux.dev
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
vCPU creation in kvm_vm_ioctl_create_vcpu() may fail after
kvm_arch_vcpu_create() -> vmx_vcpu_create() already succeeded. In such
case kvm_vm_ioctl_create_vcpu() destroys the newly created vCPU in the
failure path. However, that leaves a side effect: the IPIv pid_table
entry remains configured with this vCPU's pi_desc address. As a result,
when another vCPU sends an IPI to the APIC ID of this failed-to-create
vCPU, it will cause HW to write to this (freed!) pi_desc memory. [*]
Fix this by clearing the pid_table entry when destroying the vCPU.
Note that the same issue exists for SVM AVIC as well [1], to be fixed.
[*] Although, since this memory is freed into the kvm_vcpu_cache kmem
cache which is only used for allocating kvm_vcpus, _maybe_ this
memory will only be reused for pi_desc of another vCPU, not for
anything else. So _maybe_ this will only result in delivering the
IPI to a wrong vCPU (possibly of another VM) in the worst case, not
in a random corruption of kernel memory.
Link: https://lore.kernel.org/kvm/al4rNqpBYy8FGKPw@blrnaveerao1 [1]
Signed-off-by: Dmytro Maluka <dmaluka@chromium.org>
Reviewed-by: Kai Huang <kai.huang@intel.com>
Link: https://patch.msgid.link/20260729170621.308809-3-dmaluka@chromium.org
Signed-off-by: Sean Christopherson <seanjc@google.com>
On early boot CPU detection, use parsed CPUID(0x1) instead of a direct CPUID
query.
Beside the parser's centralization benefits, use the auto generated CPUID data
types, and their C99 bitfields, instead of doing ugly bitwise operations.
[ bp: - Use a common code pattern for l1 of fetching and checking it
right after that
- flip the check to save an indentation level
- align assignments vertically.
]
Signed-off-by: Ahmed S. Darwish <darwi@linutronix.de>
Signed-off-by: Borislav Petkov (AMD) <bp@alien8.de>
Link: https://patch.msgid.link/20260528153923.403473-12-darwi@linutronix.de
Rework .vcpu_pre_run() into a more specific .vcpu_needs_initialization() to
consolidate the SNP and TDX control flows, and to eliminate the potentially
confusing almost-collision between svm_vcpu_pre_run() and pre_sev_run(): the
former is SEV specific, but is pre-KVM_RUN, whereas the latter is pre-VMRUN.
Link: https://patch.msgid.link/20260731173340.2644656-4-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
Synthesize SHUTDOWN (for L1) if L2 requires unhandleable emulation after
loading guest state from SMRAM during RSM to prevent a misbehaving L1 (or
userspace via L1) from tripping the sanity check that KVM doesn't try to
cancel a pending nested VM-Enter. If SMRAM is modified such that RSM will
load what should be impossible state for L2, then KVM will detect that it
needs to emulate the current code stream and will abort VM-Entry to L2.
And because KVM (rightly) expects such a scenario to be impossible, KVM
WARNs and bugs the VM.
__ret && !(vcpu->kvm)->vm_bugged
WARNING: arch/x86/kvm/vmx/vmx.c:6741 at vmx_handle_exit+0x65/0x790 [kvm_intel], CPU#13: vmx_invalid_nes/2902
Modules linked in: kvm_intel kvm irqbypass [last unloaded: kvm]
CPU: 13 UID: 1000 PID: 2902 Comm: vmx_invalid_nes Tainted: G W 7.2.0-rc2 #124 PREEMPT
Tainted: [W]=WARN
Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
RIP: 0010:vmx_handle_exit+0x65/0x790 [kvm_intel]
Call Trace:
<TASK>
kvm_arch_vcpu_ioctl_run+0xdf8/0x1d00 [kvm]
kvm_vcpu_ioctl+0x2d5/0x960 [kvm]
__x64_sys_ioctl+0x8a/0xd0
do_syscall_64+0xb7/0x570
entry_SYSCALL_64_after_hwframe+0x4b/0x53
</TASK>
Alternatively, KVM could suppress the WARN for the RSM case, but that would
still leave the vCPU in a "bad" state that KVM doesn't know how to handle
(which is also why KVM rejects attempts to do KVM_RUN when the vCPU is
loaded with invalid state). And architecturally, the Intel SDM explicitly
states that RSM leads to shutdown if the CPU detects invalid state.
Fixes: 2bb8cafea8 ("KVM: vVMX: signal failure for nested VMEntry if emulation_required")
Reported-by: Hao Zhang <zhanghao1@kylinos.cn>
Link: https://patch.msgid.link/20260731173340.2644656-3-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
Expose VMX's check for unhandleable emulation as its own kvm_x86_ops hook,
and move the actual pre-KVM_RUN check into common x86. This will allow
sharing the core logic with KVM's RSM emulation without needed to add a
post-RSM hook, and is a step towards removing the .vcpu_pre_run() hook
entirely.
Alternatively, KVM could provide a post-RSM hook as mentioned, but pre/post
hooks tend to be unwieldy as the exact "timing" of the call often matters
greatly. E.g. in this case, the call must slot in exactly between loading
guest state from SMRAM and the hack to force the vCPU out of L2 on SHUTDOWN.
Link: https://patch.msgid.link/20260731173340.2644656-2-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
The x86 library provides x86_family() and x86_model(). They take raw
CPUID register output and calculate the CPU family and model from it.
In follow-up work, the x86 subsystem will use APIs which access
previously-parsed CPUID leafs structure instead of doing direct CPUID queries.
These new APIs force using the auto generated leaf data types at
<asm/cpuid/leaf_types.h>.
Introduce x86 family and model calculation functions that take these
auto-generated data types. Refactor the original code so that no logic is
duplicated.
[ bp: Massage commit message, unbreak too long line. ]
Signed-off-by: Ahmed S. Darwish <darwi@linutronix.de>
Signed-off-by: Borislav Petkov (AMD) <bp@alien8.de>
Link: https://patch.msgid.link/20260528153923.403473-11-darwi@linutronix.de
Due to a widespread Intel erratum (e.g. EMR158), programming the
VMX-preemption timer with certain large values may cause the timer to
expire earlier than expected. The recommended workaround is to cap the
VMX-preemption timer value to strictly less than:
2^25 * CPUID.15H:EBX[31:0] / CPUID.15H:EAX[31:0].
Calculate the maximum "safe" preemption timer value during hardware setup
based on CPUID 15H when available, and use the adjusted max value in all
locations where KVM currently hardcodes the max architectural value,
including in the subtle case where KVM soft-disables the timer.
Don't apply the workaround when running as a VM, because absent explicit
enumeration to state the bug is present (or not), it's L0's responsibility
to faithfully emulate/virtualize the VMX preemption timer.
WARN if the above logic would result in a max value of zero and fall back
to the maximum architectural value, as the expectation is that real
hardware will never provide problematic EAX/EBX values (which is another
reason to ignore the erratum when running as a VM; there's less chance of
a false positive on the WARN due to L0 providing an unanticipated ratio).
Reported-by: Sean Christopherson <seanjc@google.com>
Closes: https://lore.kernel.org/all/Zn9X0yFxZi_Mrlnt@google.com/
Suggested-by: Chao Gao <chao.gao@intel.com>
Assisted-by: Gemini:Gemini-Next
Reviewed-by: Chao Gao <chao.gao@intel.com>
Signed-off-by: Jim Mattson <jmattson@google.com>
Reviewed-by: Binbin Wu <binbin.wu@linux.intel.com>
[sean: track inclusive max instead of exclusive limit, massage changelog]
Link: https://patch.msgid.link/20260731171926.2629627-3-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
Double down on using the VMX preemption timer only for 64-bit kernels, and
bury the setup and runtime adjustment code, and all global variables, under
CONFIG_X86_64=y. This will allow addressing a widespread Intel erratum
without running afoul of unused-but-set-variable and __udivdi3() errors on
32-bit kernels.
No functional change intended.
Reviewed-by: Binbin Wu <binbin.wu@linux.intel.com>
Reviewed-by: Chao Gao <chao.gao@intel.com>
Link: https://patch.msgid.link/20260731171926.2629627-2-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
Use parsed CPUID(0x0) instead of a direct CPUID query.
[ bp: Drop the unnecessary NULL check. ]
Signed-off-by: Ahmed S. Darwish <darwi@linutronix.de>
Signed-off-by: Borislav Petkov (AMD) <bp@alien8.de>
Link: https://patch.msgid.link/20260528153923.403473-10-darwi@linutronix.de
@lost and @topa_index are no longer part of struct pt_buffer so
delete their comment lines.
Warning: ../arch/x86/events/intel/pt.h:86 Excess struct member 'lost' description in 'pt_buffer'
Warning: ../arch/x86/events/intel/pt.h:86 Excess struct member 'topa_index' description in 'pt_buffer'
Signed-off-by: Randy Dunlap <rdunlap@infradead.org>
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Link: https://patch.msgid.link/20260730233429.285788-2-rdunlap@infradead.org
Add a short description for imr_self_test() and drop the comments for
the non-existent imr_self_test_exit() to avoid kernel-doc warnings:
Warning: arch/x86/platform/intel-quark/imr_selftest.c:52 missing initial short description on line:
* imr_self_test
Warning: ../arch/x86/platform/intel-quark/imr_selftest.c:128 function parameter 'imr_self_test_init' not described in 'device_initcall'
Warning: ../arch/x86/platform/intel-quark/imr_selftest.c:128 expecting prototype for imr_self_test_exit(). Prototype was for device_initcall() instead
Signed-off-by: Randy Dunlap <rdunlap@infradead.org>
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Link: https://patch.msgid.link/20260730233429.285788-6-rdunlap@infradead.org
These function parameters don't exist so drop them to avoid kernel-doc
warnings:
Warning: ../arch/x86/include/asm/spec-ctrl.h:28 Excess function parameter 'guest_spec_ctrl' description in 'x86_spec_ctrl_set_guest'
Warning: ../arch/x86/include/asm/spec-ctrl.h:42 Excess function parameter 'guest_spec_ctrl' description in 'x86_spec_ctrl_restore_host'
Signed-off-by: Randy Dunlap <rdunlap@infradead.org>
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Link: https://patch.msgid.link/20260730233429.285788-5-rdunlap@infradead.org
- correct missing function parameter names
- add missing function return value sections
- use the correct function name in comments
to avoid these warnings (samples):
Warning: arch/x86/include/asm/checksum_64.h:23 No description found for return value of 'csum_fold'
Warning: arch/x86/include/asm/checksum_64.h:46 function parameter 'iph' not described in 'ip_fast_csum'
Warning: arch/x86/include/asm/checksum_64.h:46 function parameter 'ihl' not described in 'ip_fast_csum'
Warning: arch/x86/include/asm/checksum_64.h:46 No description found for return value of 'ip_fast_csum'
Warning: arch/x86/include/asm/checksum_64.h:89 expecting prototype for csum_tcpup_nofold(). Prototype was for csum_tcpudp_nofold() instead
Warning: arch/x86/include/asm/checksum_64.h:115 expecting prototype for csum_tcpup_magic(). Prototype was for csum_tcpudp_magic() instead
Warning: arch/x86/include/asm/checksum_64.h:119 No description found for return value of 'csum_tcpudp_magic'
Warning: arch/x86/include/asm/checksum_64.h:129 No description found for return value of 'csum_partial'
Warning: ./arch/x86/include/asm/checksum_64.h:175 function parameter '_saddr' not described in 'csum_ipv6_magic'
Warning: ./arch/x86/include/asm/checksum_64.h:175 function parameter '_daddr' not described in 'csum_ipv6_magic'
Warning: ./arch/x86/include/asm/checksum_64.h:175 Excess function parameter 'saddr' description in 'csum_ipv6_magic'
Warning: ./arch/x86/include/asm/checksum_64.h:175 Excess function parameter 'daddr' description in 'csum_ipv6_magic'
Signed-off-by: Randy Dunlap <rdunlap@infradead.org>
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Link: https://patch.msgid.link/20260730233429.285788-4-rdunlap@infradead.org
Use the correct macro parameter names in the kernel-doc comments
to avoid kernel-doc warnings:
Warning: arch/x86/include/asm/barrier.h:35 function parameter 'idx' not described in 'array_index_mask_nospec'
Warning: arch/x86/include/asm/barrier.h:35 function parameter 'sz' not described in 'array_index_mask_nospec'
Warning: arch/x86/include/asm/barrier.h:35 Excess function parameter 'index' description in 'array_index_mask_nospec'
Warning: arch/x86/include/asm/barrier.h:35 Excess function parameter 'size' description in 'array_index_mask_nospec'
Signed-off-by: Randy Dunlap <rdunlap@infradead.org>
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Link: https://patch.msgid.link/20260730233429.285788-3-rdunlap@infradead.org
Teach dma_direct_map_phys() to select the DMA address encoding based on
DMA_ATTR_CC_SHARED.
Use phys_to_dma_unencrypted() for decrypted mappings and
phys_to_dma_encrypted() otherwise. If a device requires unencrypted DMA
but the source physical address is still encrypted, force the mapping
through swiotlb so the DMA address and backing memory attributes remain
consistent.
Update the arm64, x86, s390 and powerpc secure-guest setup to not use
swiotlb force option
Tested-by: Jiri Pirko <jiri@nvidia.com>
Tested-by: Michael Kelley <mhklinux@outlook.com>
Tested-by: Mostafa Saleh <smostafa@google.com>
Signed-off-by: Aneesh Kumar K.V (Arm) <aneesh.kumar@kernel.org>
Link: https://lore.kernel.org/r/20260717180442.110954-18-aneesh.kumar@kernel.org
[mszyprow: rebased onto latest changes in arch/arm64/mm/init.c]
Signed-off-by: Marek Szyprowski <m.szyprowski@samsung.com>
Teach dma_capable() about DMA_ATTR_CC_SHARED so the capability
check can reject encrypted DMA addresses for devices that require
unencrypted/shared DMA.
Also propagate DMA_ATTR_CC_SHARED in swiotlb_map() when the selected
SWIOTLB pool is decrypted so the capability check sees the correct DMA
address attribute.
Reviewed-by: Jason Gunthorpe <jgg@nvidia.com>
Tested-by: Jiri Pirko <jiri@nvidia.com>
Tested-by: Michael Kelley <mhklinux@outlook.com>
Tested-by: Mostafa Saleh <smostafa@google.com>
Reviewed-by: Petr Tesarik <ptesarik@suse.com>
Signed-off-by: Aneesh Kumar K.V (Arm) <aneesh.kumar@kernel.org>
Link: https://lore.kernel.org/r/20260717180442.110954-16-aneesh.kumar@kernel.org
Signed-off-by: Marek Szyprowski <m.szyprowski@samsung.com>
The difference between __alloc_pages_node() and alloc_pages_node() is
that the latter allows you to pass NUMA_NO_NODE.
The former is going away and the latter works fine here so switch over.
No functional change intended.
Link: https://lore.kernel.org/20260703-alloc-trylock-v5-10-c87b714e19d3@google.com
Signed-off-by: Brendan Jackman <jackmanb@google.com>
Reviewed-by: Suren Baghdasaryan <surenb@google.com>
Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Reviewed-by: Zi Yan <ziy@nvidia.com>
Cc: Thomas Gleixner <tglx@kernel.org>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Dave Hansen <dave.hansen@linux.intel.com>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Assisted-by: Gemini:unknown-version
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
The difference between __alloc_pages_node() and alloc_pages_node() is
that the latter allows you to pass NUMA_NO_NODE.
The former is going away and the latter works fine here so switch over.
No functional change intended.
Link: https://lore.kernel.org/20260703-alloc-trylock-v5-9-c87b714e19d3@google.com
Signed-off-by: Brendan Jackman <jackmanb@google.com>
Reviewed-by: Suren Baghdasaryan <surenb@google.com>
Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Reviewed-by: Zi Yan <ziy@nvidia.com>
Cc: Sean Christopherson <seanjc@google.com>
Cc: Paolo Bonzini <pbonzini@redhat.com>
Assisted-by: Gemini:unknown-version
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
The difference between __alloc_pages_node() and alloc_pages_node() is that
the latter allows you to pass NUMA_NO_NODE.
The former is going away and the latter works fine here so switch over.
No functional change intended.
Link: https://lore.kernel.org/20260703-alloc-trylock-v5-8-c87b714e19d3@google.com
Signed-off-by: Brendan Jackman <jackmanb@google.com>
Reviewed-by: Suren Baghdasaryan <surenb@google.com>
Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Reviewed-by: Zi Yan <ziy@nvidia.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Arnaldo Carvalho de Melo <acme@kernel.org>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Ian Rogers <irogers@google.com>
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: James Clark <james.clark@linaro.org>
Assisted-by: Gemini:unknown-version
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
CONFIG_ARCH_ENABLE_THP_MIGRATION gates PMD-level migration entries.
PMD-level device-private entries use the same migration mechanism and
therefore require the same architecture support.
Upcoming PMD-level swap entries can use the same PMD softleaf helpers
without depending on page migration, so rename the architecture gate to
CONFIG_ARCH_HAS_PMD_SOFTLEAVES. This describes the PMD entry capability
rather than one current user of it.
This is a pure rename: the set of selecting architectures (x86, arm64,
s390, riscv, loongarch, and powerpc on PPC_BOOK3S_64) and the gating
semantics are unchanged.
No functional change intended.
Link: https://lore.kernel.org/20260706114320.1643046-7-usama.arif@linux.dev
Signed-off-by: Usama Arif <usama.arif@linux.dev>
Reviewed-by: Zi Yan <ziy@nvidia.com>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Cc: Alexandre Ghiti <alex@ghiti.fr>
Cc: Baolin Wang <baolin.wang@linux.alibaba.com>
Cc: Baoquan He <baoquan.he@linux.dev>
Cc: Barry Song <baohua@kernel.org>
Cc: Chris Li <chrisl@kernel.org>
Cc: Dev Jain <dev.jain@arm.com>
Cc: "Huang, Ying" <ying.huang@linux.alibaba.com>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Kairui Song <kasong@tencent.com>
Cc: Kemeng Shi <shikemeng@huaweicloud.com>
Cc: Kiryl Shutsemau <kas@kernel.org>
Cc: Lance Yang <lance.yang@linux.dev>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Matthew Wilcox (Oracle) <willy@infradead.org>
Cc: Nhat Pham <nphamcs@gmail.com>
Cc: Nico Pache <npache@redhat.com>
Cc: Rik van Riel <riel@surriel.com>
Cc: Ryan Roberts <ryan.roberts@arm.com>
Cc: Shakeel Butt <shakeel.butt@linux.dev>
Cc: Vlastimil Babka <vbabka@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
As noted in commit 85a2d4a890 ("x86,ibt: Use UDB instead of 0xEA") Jcc should
be assumed not-taken, however the normal kCFI (ABI) emits the following sequence:
movl $(-hash), %r10d
addl -15(%r11), %r10d
je 1f
ud2
1: cs call __x86_indirect_thunk_r11
(when used in conjunction with -mretpoline-external-thunk).
Notably, the Jcc here is always taken, resulting in lower throughput than would
be ideal. Replace it with the following sequence on boot:
movl $(-hash), %r10d
addl -15(%r11), %r10d
jne . + 3
test $0xd6, %al
cs call __x86_indirect_thunk_r11
This jumps to the UDB instruction used as an immediate byte in the test
instruction. The test instruction will clobber eflags, but that is immaterial,
eflags is already changed by the preceding addl.
Intel recommends the FineIBT sequence on platforms that support IBT; older
platforms are still widely used and would benefit from this.
An earlier PoC was benchmarked by Scott:
Indirect branch miss rate (br_misp_retired.indirect:k / br_inst_retired.indirect:k)
BHI_DIS_S=1
Benchmark Baseline IBT kCFI kCFI-opt
-----------------------------------------------------------------------------
iperf3 UDP 0.103764 0.103180 0.104311 0.102945
hackbench 0.000885 0.000876 0.001996 0.000826
lmbench syscall 0.005089 0.004486 0.016990 0.005852
lmbench fork+exit 0.018454 0.019176 0.031085 0.015153
lmbench fork+exec 0.017147 0.021613 0.029129 0.016337
redis 0.032220 0.032655 0.045540 0.027946
nginx+wrk 0.109033 0.112765 0.132557 0.102417
fio randread 0.009704 0.009620 0.008548 0.000962
fio seqwrite 0.006927 0.006707 0.019372 0.004590
kbuild 0.056748 0.057324 0.064640 0.048136
BHI_DIS_S=0
Benchmark Baseline IBT kCFI kCFI-opt
-----------------------------------------------------------------------------
iperf3 UDP 0.000077 0.000106 0.000186 0.000073
hackbench 0.000123 0.000132 0.000367 0.000097
lmbench syscall 0.023259 0.018319 0.040903 0.012772
lmbench fork+exit 0.011494 0.011887 0.029079 0.016415
lmbench fork+exec 0.037782 0.038994 0.055378 0.026381
redis 0.002481 0.003152 0.017073 0.000184
nginx+wrk 0.015478 0.016266 0.033637 0.000268
fio randread 0.009836 0.007949 0.007096 0.000143
fio seqwrite 0.014587 0.014165 0.041792 0.002157
kbuild 0.055774 0.055249 0.062590 0.046546
Suggested-by: Scott D Constable <scott.d.constable@intel.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Acked-by: Borislav Petkov (AMD) <bp@alien8.de>
Tested-by: Nathan Chancellor <nathan@kernel.org>
Link: https://patch.msgid.link/20260612071506.GQ187714%40noisy.programming.kicks-ass.net
Always update x2APIC MSR intercepts for L1 when AVIC is deactivated, even
if L2 is active and KVM is using a separate MSR bitmap to run L2. If AVIC
is fully enabled prior to running L2, and is then inhibited while L2 is
active (for a VM-scoped inhibit), then KVM will run L1 with AVIC disabled,
but with x2APIC MSR intercepts disabled, i.e. will allow L1 to read most of
the host's APIC state, send arbitrary interrupts, change task priority, and
ultimately trivially DoS the host.
E.g. sending a self-IPI in L1 on HYPERV_REENLIGHTENMENT_VECTOR, 0xee, with
CONFIG_HYPERV=n in the host kernel as a "safe" PoC, yields:
Spurious interrupt (vector 0xee) on CPU#425. Acked
And hacking KVM to abuse kvm_set_posted_intr_wakeup_handler() to register a
handler and WARN on POSTED_INTR_WAKEUP_VECTOR yields:
------------[ cut here ]------------
WARNING: arch/x86/kvm/svm/svm.c:5594 at pi_wakeup_handler+0x9/0x10 [kvm_amd], CPU#156: nested_x2apic_t/316940
CPU: 156 UID: 0 PID: 316940 Comm: nested_x2apic_t Tainted: G S U
Tainted: [S]=CPU_OUT_OF_SPEC, [U]=USER
Hardware name: Google Astoria-Turin/astoria, BIOS 0.20260209.0-0 02/09/2026
RIP: 0010:pi_wakeup_handler+0x9/0x10 [kvm_amd]
Call Trace:
<IRQ>
sysvec_kvm_posted_intr_wakeup_ipi+0x64/0x80
</IRQ>
<TASK>
asm_sysvec_kvm_posted_intr_wakeup_ipi+0x1a/0x20
RIP: 0010:vcpu_run+0x1430/0x1e40 [kvm]
kvm_arch_vcpu_ioctl_run+0x2c1/0x600 [kvm]
kvm_vcpu_ioctl+0x580/0x6b0 [kvm]
__se_sys_ioctl+0x6d/0xb0
do_syscall_64+0x10a/0x480
entry_SYSCALL_64_after_hwframe+0x4b/0x53
RIP: 0033:0x46ff4b
</TASK>
---[ end trace 0000000000000000 ]---
Fixes: 091abbf578 ("KVM: x86: nSVM: optimize svm_set_x2apic_msr_interception")
Cc: stable@vger.kernel.org
Cc: Yosry Ahmed <yosry@kernel.org>
Signed-off-by: Sean Christopherson <seanjc@google.com>
Link: https://patch.msgid.link/20260729213558.639074-1-pbonzini@redhat.com/
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Family 0x1a, model 0xd0..0xd7 belongs to the Zen5 generation. Carve it
out from the larger, Zen6 range where former doesn't belong.
[ bp: Rewrite commit message, add tags. ]
Fixes: b5f53e6d3d ("x86/CPU/AMD: Add more Zen6 models")
Signed-off-by: Pratik Vishwakarma <Pratik.Vishwakarma@amd.com>
Signed-off-by: Borislav Petkov (AMD) <bp@alien8.de>
Cc: <stable@kernel.org>
Link: https://patch.msgid.link/20260729055459.15904-1-Pratik.Vishwakarma@amd.com
Use CMPXCHG instead of clear_bit(), which currently emits a LOCK BTR since
the to-be-cleared bit isn't a compile-time constant, when aging SPTEs in
the shadow MMU to align with the approach taken by the TDP MMU, and because
using CMPXCHG is far more robust against bugs in KVM. E.g. if the SPTE is
somehow no longer an SPTE due to a KVM bug, CMPXCHG will fail gracefully,
whereas clear_bit() would potentially corrupt/clobber memory.
Clearing the Accessed bit without atomically ensuring the SPTE is still the
old SPTE is "fine", as holding the rmap's lock ensures zapping the old SPTE
can't fully complete, which in turn ensures a new, different SPTE can't be
installed. But that chain of logic isn't exactly obvious, and there's zero
reason to avoid CMPXCHG as its cost on modern hardware is within ~1-2 uops
of LOCK BTR (and may even be cheaper on some microarchitectures). Doing a
64-bit CMPXCHG on 32-bit kernels does require a more expensive CMPXCHG8B,
but 32-bit KVM is all but dead at this point.
Cc: James Houghton <jthoughton@google.com>
Reviewed-by: Kai Huang <kai.huang@intel.com>
Reviewed-by: James Houghton <jthoughton@google.com>
Link: https://patch.msgid.link/20260728002236.869865-3-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
Use LOCK CMPXCHG instead of LOCK AND to clear the Accessed bit when aging
SPTEs in the TDP MMU, as doing a LOCK AND can corrupt a FROZEN SPTE and
allow a third CPU to effectively overwrite the FROZEN SPTE. As pointed
out by AI of some kind, because the magic FROZEN_SPTE value is a "full"
SPTE, not a single bit, and includes the Accessed bit, clearing the
Accessed bit in a FROZEN SPTE will result in is_frozen_spte() getting a
false negative.
E.g. if CPU0 freezes an SPTE, and CPU1 clears the Accessed bit in the
frozen SPTE, then CPU2 could come along and overwrite the frozen SPTE with
a shadow-present SPTE.
Thankfully, the false negative is largely benign, because outside of TDX,
which doesn't support aging, KVM only freezes leaf SPTEs when removing an
upper level shadow page. So while KVM could clobber a frozen SPTE back to
a shadow-present SPTE, and could even use the new SPTE, the subsequent TLB
flush will make the orphaned, shadow-present SPTE unreachable. Failure to
ever zap the orphaned leaf SPTE would show up in KVM's stats, but otherwise
is benign (because KVM no longer keeps an elevated refcount for leaf SPTEs).
Opportunistically add a comment to warn future developers away from using
kvm_tdp_mmu_write_spte_atomic() and tdp_mmu_clear_spte_bits_atomic(), as
they are generally unsafe. Keep the helpers, e.g. instead of open-coding
the atomic64_fetch_and() in tdp_mmu_clear_spte_bits(), as scary warnings
usually are more effective deterrent against recidivism than removal of the
dangerous code.
Alternatively, KVM could use different bits for the magic FROZEN_SPTE value,
e.g. setting the Dirty bits (with effective IPAT and Global aliases) would
likely be "ok", as IPAT/Global are extremely unlikely to be cleared without
doing a full SPTE write, and KVM's clearing of Dirty bits shares logic with
Write-Protection, which must do a full SPTE write (via cmpxchg64() in the
TDP MMU) to ensure KVM isn't clobbering state. But there is zero reason to
carry that risk (beyond stubbornness in wanting to preserve a "cute" idea),
as the cost of LOCK CMPXCHG and LOCK AND are within 1-2 uops of each other
on modern hardware.
Fixes: b146a9b34a ("KVM: x86/mmu: Age TDP MMU SPTEs without holding mmu_lock")
Cc: stable@vger.kernel.org
Reviewed-by: Kai Huang <kai.huang@intel.com>
Reviewed-by: James Houghton <jthoughton@google.com>
Link: https://patch.msgid.link/20260728002236.869865-2-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
Add the volatile qualifier and clobbers parameter to prevent bugs with
instruction reordering and optimization.
Also add TEST for the zero-length case to set ZF, as, if the count register
is zero, the REPE prefix does not run the CMPSB instruction, leaving the ZF
flag undetermined.
[ bp: Add a comment about the len==0 case. ]
Fixes: 62bd0337d0 ("Top header file for new x86 setup code")
Closes: https://sashiko.dev/#/patchset/20260701-pvh-kasan-inline-v6-0-ba99045dfa9f%40igalia.com
Suggested-by: Borislav Petkov <bp@alien8.de>
Signed-off-by: Mauricio Faria de Oliveira <mfo@igalia.com>
Signed-off-by: Borislav Petkov (AMD) <bp@alien8.de>
Link: https://lore.kernel.org/all/20260721-pvh-kasan-inline-v7-2-38979a50cef0@igalia.com
Convert the kvm_vcpu_map_readonly() usage in nVMX's MSR bitmap merging to
the new CLASS(kvm_vcpu_map_local_readonly) implementation, to eliminate the
last of the open-coded on-stack "struct kvm_host_map" declarations (in x86,
PPC still has one more to convert).
No functional change intended.
Link: https://patch.msgid.link/20260724004757.131420-5-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
Convert the kvm_vcpu_map() usage in the enter/leave SMM flows to the new
CLASS(kvm_vcpu_map_local) implementations, to eliminate the need to
manually do unmap() in error paths, and more importantly to eliminate more
of the open-coded on-stack "struct kvm_host_map" declarations.
No functional change intended.
Link: https://patch.msgid.link/20260724004757.131420-4-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
Add CLASS() definitions for locally mapping a PFN using kvm_vcpu_map()
given a vCPU+gfn pair. In addition to eliminating the need to manually
do unmap(), e.g. in error paths, this will allow hardening KVM against
double-mapping without having to manually ensure every on-stack declaration
is zero-initialized.
Use "map local" as the primary terminology as the basic concept is more or
less the same as kmap_local(): ensure the current context has a kernel
mapping to the underlying memory.
Immediately convert the relatively straightforward nested SVM flows, and
defer converting the more involved SMM flows to a separate change.
No functional change intended.
Cc: Yosry Ahmed <yosry@kernel.org>
Link: https://patch.msgid.link/20260724004757.131420-3-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
Always check and clear KVM_REQ_GET_NESTED_STATE_PAGES when emulating a
nested VM-Exit to ensure the request is cleared, even when KVM was built
with CONFIG_KVM_HYPERV=n, as KVM subtly relies on the "check" to clear
the flag and thus avoid double-mapping the vmcs12 pages, e.g. if KVM
manages to bail from VM-Enter without processing the request, and then
emulates VMLAUNCH or VMRESUME.
Fixes: b4f69df0f6 ("KVM: x86: Make Hyper-V emulation optional")
Cc: stable@vger.kernel.org
Reported-by: Yosry Ahmed <yosry@kernel.org>
Reviewed-by: Yosry Ahmed <yosry@kernel.org>
Link: https://patch.msgid.link/20260724004757.131420-2-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
The metric table needs to be delivered to userspace as a single
atomic snapshot, but the current sysfs metrics_bin path is a file
read: userspace can read it in chunks and observe a torn snapshot
if an SMU refresh happens between read() calls. The same path is
also bounded by PAGE_SIZE, so the ~13 KB table used by HSMP protocol
version 7 on Family 1Ah Model 50h-5Fh cannot be returned at all,
regardless of how userspace reads it. Rather than extend sysfs to
lift both restrictions, expose the metric table through the
existing HSMP character device using a new ioctl that always copies
the table in one shot.
Add struct hsmp_telemetry_data and HSMP_IOCTL_GET_TELEMETRY_DATA
to the UAPI header. Under the surrounding #pragma pack(4), placing
the __u64 user pointer first gives a tight 16-byte layout that is
identical for 32- and 64-bit callers, and the trailing __u16
reserved field is rejected with -EINVAL if non-zero so future
kernels can repurpose it without breaking already-deployed
userspace. The command is encoded with _IOW because the kernel only
reads the request struct; the snapshot travels through the user
pointer it carries.
The requested size may be anything from one byte up to the size
firmware reported for that socket's table. A short request returns
the leading bytes of the snapshot, so userspace built against an
older table layout keeps working on firmware that grew the table,
mirroring the relaxed response_sz rule applied to HSMP messages
earlier in this series. A request larger than the firmware table is
rejected with -EINVAL rather than short-written, so a caller can
never mistake a partial copy for a full one.
Dispatch hsmp_ioctl() on the ioctl command: the existing message
handler is factored out as hsmp_ioctl_msg() for HSMP_IOCTL_CMD, and
HSMP_IOCTL_GET_TELEMETRY_DATA goes to a new
hsmp_ioctl_get_telemetry() helper.
/dev/hsmp is a singleton character device that outlives an
individual socket unbind, so an ioctl issued on an already-open fd
can run concurrently with socket teardown. hsmp_sock_rwsem is the
driver's contract for that: the data plane takes it for read, and
probe and remove take it for write to drain the data plane before
freeing the socket array, unmapping the metric tables and
destroying the per-socket mutexes. hsmp_ioctl_get_telemetry() takes
it for read across the socket lookup, the checks on that socket's
metric-table state and the table read itself, so none of that state
can be torn down underneath it. Without this the handler would
sleep in its kvmalloc() holding no lock at all, and could resume
with a freed socket, locking a destroyed mutex and reading from an
unmapped iomem region.
The lock is dropped before the copy_to_user(), because faulting in
the destination can block indefinitely on a userfaultfd-backed
buffer and would otherwise leave a socket unbind waiting for the
write lock.
Since hsmp_metric_tbl_read() reached the mailbox through
hsmp_send_message(), which takes hsmp_sock_rwsem itself, calling it
with the lock already held would recursively take the read side and
can deadlock against a queued writer. Split out
hsmp_metric_tbl_read_locked(), which asserts the lock and uses
hsmp_send_message_locked(), and leave hsmp_metric_tbl_read() as a
wrapper that takes the read lock for the sysfs callers. This also
brings the whole fill-and-copy under the rwsem for those callers,
where the memcpy_fromio() previously ran outside it, and makes the
lock order uniformly hsmp_sock_rwsem -> metric_read_lock ->
hsmp_sem.
The user-controlled socket index in HSMP_IOCTL_GET_TELEMETRY_DATA is
clamped with array_index_nospec() before indexing hsmp_pdev.sock[],
mitigating Spectre v1 (CVE-2017-5753). Include linux/nospec.h, which
the file relied on getting transitively.
Co-developed-by: Muthusamy Ramalingam <muthusamy.ramalingam@amd.com>
Signed-off-by: Muthusamy Ramalingam <muthusamy.ramalingam@amd.com>
Signed-off-by: Muralidhara M K <muralidhara.mk@amd.com>
Link: https://patch.msgid.link/20260727141542.3370108-5-muralidhara.mk@amd.com
Reviewed-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
The driver hard-codes the metric-table region size to
sizeof(struct hsmp_metric_table). That is correct for HSMP protocol
version 6 but mis-sizes the ioremap of the SMU DRAM region on newer
platforms: Family 1Ah Model 50h-5Fh exposes a ~13 KB table under
protocol version 7, and the table is expected to keep growing on
future firmware. The same hard-coded value also forces
hsmp_metric_tbl_read() to reject any read that follows the actual
firmware layout.
Pick up the table size from firmware instead. SMU on Family 1Ah
Model 50h and later populates HSMP_GET_METRIC_TABLE_DRAM_ADDR's
args[2] with the DRAM region size in bytes; older firmware leaves it
0. Bump the descriptor's response_sz to 3 so the field is read, and
store the result in the new per-socket hsmp_socket.metric_tbl_size,
which is then used both for the ioremap() of the region and as the
expected size in hsmp_metric_tbl_read().
The size is stored per socket rather than per platform because
hsmp_get_tbl_dram_base() runs once per socket and each socket maps
its own region. A single platform-wide field would let the last
socket's size be used to copy out of an earlier socket's smaller
mapping.
Bump DRIVER_VERSION to 2.6.
Behaviour on existing protocol-version-6 hardware is unchanged.
Reading a third response word is safe there: for this command SMU
leaves args[2] as 0 rather than a stale value from an earlier
mailbox transaction, so the fallback always applies, yielding the
same value as the previous hard-coded one, and both the ioremap and
the size check produce the same result as before.
Co-developed-by: Muthusamy Ramalingam <muthusamy.ramalingam@amd.com>
Signed-off-by: Muthusamy Ramalingam <muthusamy.ramalingam@amd.com>
Signed-off-by: Muralidhara M K <muralidhara.mk@amd.com>
Link: https://patch.msgid.link/20260727141542.3370108-4-muralidhara.mk@amd.com
Reviewed-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
Family 1Ah Model 50h-5Fh firmware exposes new HSMP messages (0x29-0x2A,
0x33-0x3A) for PC6/CC6 control, CCD power/thermal monitoring, DIMM
sideband access, floor- and SDPS-limit control, and command-enable
discovery. The same firmware extends three existing SET-only messages
(HSMP_SET_XGMI_LINK_WIDTH 0x0C, HSMP_SET_DF_PSTATE 0x0D,
HSMP_SET_PSTATE_MAX_MIN 0x22) with a read-back path selected by bit[31]
of args[0] (0 = set, 1 = get). Add the new IDs and convert the three
messages to HSMP_SET_GET.
Also add PQoS-related HSMP messages HSMP_PQOS_TRAFFIC_PRIORITY (0x3B)
and HSMP_PQOS_FLOATING_BW (0x3C) with matching hsmp_msg_desc_table[]
descriptors so userspace can reach the new functionality.
Backward compatibility is preserved on prior platforms: new IDs
previously occupied HSMP_RSVD slots, and existing userspace that leaves
bit[31] = 0 continues to take a pure SET path. Converting the three
SET messages to HSMP_SET_GET also keeps them accepted by
validate_message(), which already applies a relaxed upper-bound check
on response_sz for that type.
Co-developed-by: Muthusamy Ramalingam <muthusamy.ramalingam@amd.com>
Signed-off-by: Muthusamy Ramalingam <muthusamy.ramalingam@amd.com>
Signed-off-by: Muralidhara M K <muralidhara.mk@amd.com>
Link: https://patch.msgid.link/20260727141542.3370108-2-muralidhara.mk@amd.com
Reviewed-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
mmu_try_to_unsync_pages() skips the shadow-page lookup when the
supplied memslot allows a hugepage, because a shadow page would disallow
hugepages. But hugepage metadata is per-address-space while shadow pages
are shared across all address spaces. With SMM, the other address space
can therefore have a shadow page even when the supplied memslot allows a
hugepage.
Check the corresponding memslot in the other address space before
taking the fast path. Skip the shadow-page lookup only when all address
spaces allow a hugepage.
Fixes: b3ae3ceb55 ("KVM: x86/mmu: KVM: x86/mmu: Skip unsync when large pages are allowed")
Assisted-by: Codex:GPT-5
Signed-off-by: Jinu Kim <kimjw04271234@gmail.com>
[invert direction of the conditional. - Paolo]
Message-ID: <20260721103512.2136240-3-kimjw04271234@gmail.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
kvm_gfn_is_write_tracked() checks only the supplied memslot, but page
tracking is per-address-space and shadow pages are shared across all
address spaces. With SMM, a GFN can therefore be write-tracked in one
address space and appear untracked through the other.
Check the supplied slot first, then the slot for the other address space.
This ensures all callers honor write tracking regardless of the active
address space. In particular, it prevents mmu_try_to_unsync_pages() from
marking an upper-level shadow page unsync and eventually triggering the
BUG in pte_list_remove().
Fixes: 699023e239 ("KVM: x86: add SMM to the MMU role, support SMRAM address space")
Assisted-by: Codex:GPT-5
Signed-off-by: Jinu Kim <kimjw04271234@gmail.com>
Message-ID: <20260721103512.2136240-2-kimjw04271234@gmail.com>
[invert direction of the conditional. - Paolo]
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Cancel (and flush) the I/O APIC's delayed EOI handling work during the
"pre VM destroy" phase, before vCPUs are destroyed, as processing the EOI
broadcast will inject another IRQ if the line is asserted, i.e. will try
to deliver an IRQ to the target vCPU(s). Canceling the work after vCPUs
are destroyed leads to UAF if the delayed work is processed after vCPUs are
destroyed.
BUG: KASAN: slab-use-after-free in __kvm_irq_delivery_to_apic_fast+0x9bf/0xa20 arch/x86/kvm/lapic.c:1250
Read of size 8 at addr ffff8880499abea0 by task kworker/1:2/1218
CPU: 1 UID: 0 PID: 1218 Comm: kworker/1:2 Not tainted 7.1.0-rc7 #5 PREEMPT(lazy)
Hardware name: QEMU Ubuntu 25.10 PC v2 (i440FX + PIIX, + 10.1 machine, 1996), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
Workqueue: events kvm_ioapic_eoi_inject_work
Call Trace:
<TASK>
__dump_stack lib/dump_stack.c:94
dump_stack_lvl+0x100/0x190 lib/dump_stack.c:120
print_address_description mm/kasan/report.c:378
print_report+0x139/0x4ad mm/kasan/report.c:482
kasan_report+0xe4/0x1d0 mm/kasan/report.c:595
__kvm_irq_delivery_to_apic_fast+0x9bf/0xa20 arch/x86/kvm/lapic.c:1250
__kvm_irq_delivery_to_apic+0xd8/0xbf0 arch/x86/kvm/lapic.c:1345
kvm_irq_delivery_to_apic arch/x86/kvm/lapic.h:129
ioapic_service+0x308/0x590 arch/x86/kvm/ioapic.c:492
kvm_ioapic_eoi_inject_work+0x13c/0x190 arch/x86/kvm/ioapic.c:532
process_one_work+0xa59/0x19a0 kernel/workqueue.c:3314
process_scheduled_works kernel/workqueue.c:3397
worker_thread+0x5eb/0xe50 kernel/workqueue.c:3478
kthread+0x370/0x450 kernel/kthread.c:436
ret_from_fork+0x72b/0xd30 arch/x86/kernel/process.c:158
ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:245
</TASK>
Note, the VM is unreachable once kvm_destroy_vm() starts, and scheduling
new work via kvm_ioapic_send_eoi() can only be done via KVM_RUN, i.e.
requires a live vCPU.
Alternatively, KVM could simply destroy the I/O APIC during the "pre" phase
of VM destruction, but that gets more than a bit sketchy as KVM expects the
I/O APIC to exist if ioapic_in_kernel() is true, and nested virtualization
in particular has a bad habit of touching VM-scope state during vCPU
destruction. E.g. attempting to free the PIC during the pre phase would
lead to a NULL pointer dereference in kvm_cpu_has_extint(), and it's not
hard to imagine the I/O APIC having a similar flaw.
Fixes: 17bcd71442 ("KVM: x86: Free vCPUs before freeing VM state")
Reported-by: <zdi-disclosures@trendmicro.com>
Reported-by: Zhong Wang <wangzhong.c0ss4ck@bytedance.com>
Reported-by: Xuanqing Shi <shixuanqing.11@bytedance.com>
Cc: stable@vger.kernel.org
Signed-off-by: Weiming Shi <bestswngs@gmail.com>
Co-developed-by: Sean Christopherson <seanjc@google.com>
Signed-off-by: Sean Christopherson <seanjc@google.com>
Message-ID: <20260727171718.543491-1-seanjc@google.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
VMCLEAR/VMREAD/VMWRITE/VMPTRLD access the internal VMCS cache, which
is not visible to the compiler; without a memory clobber, the compiler
can reorder them in troublesome ways because "asm volatile" and "asm goto"
only protect against removal of the asm. For example, placing a VMWRITE
before the corresponding VMCS pointer is loaded can lead to corruption.
While none of this has been observed, it is better to prevent than cure.
Likewise, INVEPT and INVVPID access the TLB and, even though in their
case the effect is only visible to the next VMLAUNCH/VMRESUME, it is
technically correct to add the clobber there too. So avoid any urge to
special case them, and simply hardcode "memory" into the clobber list
of vmx_asm1() and vmx_asm2(). __vmcs_readl() open-codes its own asm,
so add the clobber there as well.
Link: https://lore.kernel.org/kvm/CABgObfbL3t21yVeSwiLSjjOUER+rTYDPHYAH9YU4TWGRjx6XHg@mail.gmail.com/
Cc: Sean Christopherson <seanjc@google.com>
Cc: stable@vger.kernel.org
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Futex hash computation requires a mask operation with read-only after
init data that will be converted to a runtime constant in the subsequent
commit.
Introduce runtime_const_mask_32 to further optimize the mask operation
in the futex hash computation hot path.
[ prateek: Broke off the x86 chunk, commit message. ]
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Signed-off-by: K Prateek Nayak <kprateek.nayak@amd.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/20260227161841.GH606826@noisy.programming.kicks-ass.net
Link: https://patch.msgid.link/20260728052540.4728-2-kprateek.nayak@amd.com
Transmeta CPUs allow masking CPUID(0x1).EDX feature flags via MSR writes. If
a bit is cleared in the 0x80860004 MSR, its corresponding feature flag is not
reported by CPUID.
Refresh the CPUID parser's CPUID(0x1) cache after all MSR bits are unmasked.
Note, the MSR 0x80860004 semantics are documented in the "BIOS Programmer's
Guide: Transmeta Crusoe Processor" document, dated June 14, 2002.
Signed-off-by: Ahmed S. Darwish <darwi@linutronix.de>
Signed-off-by: Borislav Petkov (AMD) <bp@alien8.de>
Link: https://patch.msgid.link/20260528153923.403473-9-darwi@linutronix.de
When getting a to-be-injected IRQ, don't WARN if the IRQ disappeared and
Xen emulation is supported, as a guest could concurrently toggle its
evtchn_upcall_pending flag in shared memory and deassert the IRQ. Even
more annoyingly, userspace could disable Xen emulation for the entire VM
KVM_XEN_HVM_CONFIG. So, suppress WARNs on lost IRQs if Xen emulation is
supported to prevent false positives.
Alternatively, KVM could track if the VM has ever used Xen emulation, but
the added complexity isn't worth carrying given that the vast majority of
deployments can and should disable Xen emulation.
Fixes: bf672720e8 ("KVM: x86: check the kvm_cpu_get_interrupt result before using it")
Reported-by: Sashiko Bot <sashiko-bot@kernel.org>
Closes: https://lore.kernel.org/all/20260625212001.3B6561F000E9@smtp.kernel.org
Link: https://patch.msgid.link/20260724173425.278753-3-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
When getting a to-be-injected IRQ, don't WARN if the IRQ disappeared and
the VM has an in-kernel PIC, as the ExtINT handling that's routed through
KVM's virtual PIC is tracked per-VM, not per-vCPU. If another vCPU grabs
the IRQ, or deasserts the interrupt (which is level-triggered), then it's
both expected and "fine" for a
Keep the assert for split IRQCHIP VMs to help detect KVM bugs, as userspace
is responsible for routing ExtINT to the intended vCPU, i.e. once an ExtINT
is pending, it can't be cleared without holding the vCPU's mutex, and thus
false positives are impossible.
Fixes: bf672720e8 ("KVM: x86: check the kvm_cpu_get_interrupt result before using it")
Debugged-by: Alexander Potapenko <glider@google.com>
Reported-by: syzbot+dd769db18693736eee89@syzkaller.appspotmail.com
Closes: https://syzkaller.appspot.com/bug?extid=dd769db18693736eee89
Closes: https://lore.kernel.org/all/6a360fdf.871e809a.2d6dda.0000.GAE@google.com
Link: https://patch.msgid.link/20260724173425.278753-2-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
The default case in __apic_accept_irq() has carried a printk("TODO:
unsupported delivery mode") since the original LAPIC emulation was
introduced in commit 97222cc831 ("KVM: Emulate local APIC in
kernel").
The switch now handles all eight delivery modes defined by the x86
architecture and is always constrained to the three-bit field
defined by the architecture: Either by masking with APIC_MODE_MASK,
by three-bit bitfield widths in the IOAPIC and MSI structs, or by
using APIC_DM_* constants directly.
Replace the unreachable printk with WARN_ON_ONCE(1) to match the
existing pattern for impossible defaults elsewhere in the same file.
Signed-off-by: Tim Wiederhake <twiederh@redhat.com>
Link: https://patch.msgid.link/20260715120341.2661873-1-twiederh@redhat.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
Tweak the guest_memfd make_private() hook to explicitly pass the number of
pages to align with the signature of the make_shared() hook, and because
the existing code is outright broken if a guest_memfd folio is comprised of
more than one page (which can't happen, yet). The SNP code *tries* to
create a corresponding huge entry, but if the RMP must use 4KiB entries for
whatever reason, KVM will only convert the first pfn, and not the entire
range of pfns that will be mapped into the guest.
Alternatively, @max_order could simply be repurposed as _the_ @order, but
that will fall apart when in-place conversion comes along, at which point
KVM will need to deal with conversions that aren't bound 1:1 to a folio.
I.e. the number of pages to convert may not be exactly a power-of-2 (and
folios don't strictly guarantee power-of-2 pages anyways).
WARN in the SNP code if the number of pages to prepare is anything other
than '1', i.e. if guest_memfd is trying to prepare/convert more than a
single 4KiB page, as sev_gmem_prepare() doesn't actually handle conversion
greater than order-0 folios.
Opportunistically swap the ordering of @pfn and @gfn params for
kvm_x86_ops.gmem_make_private() to match kvm_arch_gmem_make_private().
Fixes: b85524314a ("KVM: guest_memfd: delay kvm_gmem_prepare_folio() until the memory is passed to the guest")
Reviewed-by: Xiaoyao Li <xiaoyao.li@intel.com>
Reviewed-by: Ackerley Tng <ackerleytng@google.com>
Link: https://patch.msgid.link/20260723210811.72720-9-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
Rework guest_memfd's prepare() hook into a more accurate make_private(),
and rework its Kconfig from PREPARE to a more generic CONVERT. This will
allow x86 to share (pun intended) a kvm_x86_ops.gmem_make_shared() hook
between the "convert to shared" and "reclaim" flows, which are one and the
same for SNP.
No functional change intended.
Reviewed-by: Ackerley Tng <ackerleytng@google.com>
Reviewed-by: Fuad Tabba <fuad.tabba@linux.dev>
Link: https://patch.msgid.link/20260723210811.72720-8-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
Rename kvm_x86_ops's gmem_invalidate() hook to gmem_make_shared(), as the
hook doesn't invalidate anything, and so that KVM doesn't need to add yet
another vendor callback to support "convert to shared" once in-place
conversion comes along.
Opportunistically wrap the ops declarations with a GMEM_RECLAIM guard so
that attempting to wire up a .gmem_make_shared() hook without selecting
CONFIG_HAVE_KVM_ARCH_GMEM_RECLAIM will result in a build failure.
No functional change intended.
Reviewed-by: Xiaoyao Li <xiaoyao.li@intel.com>
Reviewed-by: Ackerley Tng <ackerleytng@google.com>
Reviewed-by: Fuad Tabba <fuad.tabba@linux.dev>
Link: https://patch.msgid.link/20260723210811.72720-4-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
Rename guest_memfd's invalidate() hook to reclaim() and isolate it via its
own RECLAIM Kconfig, as the hook is called when a folio is freed, which is
far too late and lacks sufficient information for KVM to actually
invalidate its usage of the memory. E.g. SNP uses the hook to convert
memory back to SHARED so that it can be safely accessed by the host, there
is no invalidation of guest mappings anywhere. Isolating the hook will
also allow pKVM on arm64 to opt-in to reclaim() without also having to
differentiate between reclaim and conversions to shared for active VMs.
Keep guest_memfd's trampoline, even though it would be trivial to wire up
.free_folio() directly to an arch callback, to avoid bleeding guest_memfd
internals into arch code (specifically, avoid referencing folios in arch
code).
Leave the kvm_x86_ops hook as-is for the moment, as "reclaim" on SNP is
the same as convert-to-shared, i.e. using a different name for the x86 hook
will allow reusing it for in-place conversion.
Reviewed-by: Xiaoyao Li <xiaoyao.li@intel.com>
Reviewed-by: Fuad Tabba <fuad.tabba@linux.dev>
Reviewed-by: Ackerley Tng <ackerleytng@google.com>
Link: https://patch.msgid.link/20260723210811.72720-3-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
Pass the number of pages to "invalidate", i.e. reclaim, instead of the end
pfn, as a first step towards aligning the function prototypes between the
de facto "to private" and "to shared" arch hooks. Eventually, the goal
is to end up with kvm_gmem_arch_make_{private,shared}(), and in both cases,
providing the number of pages makes the call sites slightly nicer, and also
avoids any confusion over whether the end pfn is inclusive or exclusive.
Opportunistically rename "start" to "pfn", again to align with the expected
signature of make_private() (which needs to pass a starting gfn as well, at
which point the "start" becomes noise).
No functional change intended.
Cc: Fuad Tabba <fuad.tabba@linux.dev>
Cc: Ackerley Tng <ackerleytng@google.com>
Reviewed-by: Xiaoyao Li <xiaoyao.li@intel.com>
Reviewed-by: Ackerley Tng <ackerleytng@google.com>
Reviewed-by: Fuad Tabba <fuad.tabba@linux.dev>
Link: https://patch.msgid.link/20260723210811.72720-2-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
Wire up the fchroot() system call as number 472 on (nearly) all
architectures and sync the mirrored copies of the syscall tables and
the asm-generic unistd.h under tools/.
Link: https://patch.msgid.link/20260724-work-failfs-v2-5-485dabbae185@kernel.org
Signed-off-by: Christian Brauner (Amutable) <brauner@kernel.org>
These are the remaining patches for legacy gpio interface
cleanup, ensuring that files referencing them are sure to
have CONFIG_GPIOLIB_LEGACY set, and use linux/gpio/legacy.h
instead of the old linux/gpio.h header.
There are a few more patches already merged in other
branches for 7.3, including soc, led, net and media. Once
everything is in, we can apply the final patches to turn
off CONFIG_GPIOLIB_LEGACY by default and remove the obsolete
header file.
This branch is based on the immutable ib-mfd-legacy-gpio-7.3
branch from mfd.git to avoid a conflict in the rohm drivers.
-----BEGIN PGP SIGNATURE-----
iQIzBAABCgAdFiEEo6/YBQwIrVS28WGKmmx57+YAGNkFAmpjrCwACgkQmmx57+YA
GNlhuA//eRNxgyDFRUtuRchUUnIjpCzPdZOXTvnTmDTVYIsHaKS6J/KTBZcxRG7g
tA749mgpQ2tT0hZFhT1XHCaxHwwBfmV0lomIO65pNcmqmEAUODxTwEy3t4ON4loi
Ee0SqEXhI+NhMbEoQkrpcCk5y+89Xo+iABbEv12UL/V/K0PKfRDCRs2ZbARnE0yu
L9wiqt2TJn3baHMabxBZJSWXrYpHCaPqm+vQgIhELQZ4TGswfi6OvCEPhUkk1QUa
mVvWH7LtUi1A0Ya1HvAhgncvboY6LZYjDHMw7meQmUOSTIu7XoQ6+XjEJ6LJSgtH
rLAdrFUHFezoHd0UaklnB3taEa7KOZIljNIY8IGRMSqv7jYF8fYXTPuxnS9fuj2o
u+GxDriWYLp1DfH7HlkrVbmVeX6zunH8qYx91z5aRl7YHNKfwg0wRv8mxAgeBTJy
1HO/su+ZT0X21hvVibiQNy08VWiQ5+9F5DV+SPTKtaT8chGjjx6uw7oyZkgvzWyu
xdHuwOzH9WAhL7WnGcJ7PkJzWDoLVkl3KzRCCaYMXBCyLRwCIJTfC5CH7hJNh41i
fW8b5Nfh7DRAhEuapQk87/NE/sRIFKZW2kfyzbCzsJHm+TW/5X+O6OwoJ+KUSdWQ
SqVJOVUKDWjg0S8+jxu0mr1ikzwKgrFv2gzWFxaffb6Oi0rqBc4=
=Ybrr
-----END PGP SIGNATURE-----
Merge tag 'config-gpio-legacy-for-7.3' of https://git.kernel.org/pub/scm/linux/kernel/git/arnd/playground into gpio/for-next
gpio: legacy interface cleanups for 7.3
These are the remaining patches for legacy gpio interface
cleanup, ensuring that files referencing them are sure to
have CONFIG_GPIOLIB_LEGACY set, and use linux/gpio/legacy.h
instead of the old linux/gpio.h header.
There are a few more patches already merged in other
branches for 7.3, including soc, led, net and media. Once
everything is in, we can apply the final patches to turn
off CONFIG_GPIOLIB_LEGACY by default and remove the obsolete
header file.
This branch is based on the immutable ib-mfd-legacy-gpio-7.3
branch from mfd.git to avoid a conflict in the rohm drivers.
a bit more widely, because newer versions of LLVM started
optimizing it a bit better and introduced run-time relocations
in PIE code. (Nathan Chancellor)
Signed-off-by: Ingo Molnar <mingo@kernel.org>
-----BEGIN PGP SIGNATURE-----
iQJFBAABCgAvFiEEBpT5eoXrXCwVQwEKEnMQ0APhK1gFAmpl0uARHG1pbmdvQGtl
cm5lbC5vcmcACgkQEnMQ0APhK1hmpxAAltRv2TwPA4MeZt10iYTG4BdwMp4OleR4
hDxqYL9T1OJev4wLujlri3/Vb1MVKG1S3YQjeG6hWAB0L+//hbx2e/nEU24uW5xb
j+0kAWSSZPUIP3KRc5Cy4mb1EXiumjk3GHnwXZpRV5XA4RRoTmpexd09J5v5KNvt
/q/c8ZzVMB8p8S1K98BdL/wMAyEHLg+iSywTdNsa8Zyw8qB8qAlRgLHTlhYjkcnO
E1+g3MY7QDEjMYDgHAMP1GHvtXTktXfp2Neo4s0WLmY/C6LEyWE/+HIDrMVuzjat
AU5IWbE0tDxGYdeyVvVoxRafFWmVg7mfnazkORp6iojbfJeVBB+IkNAVX0fbKEBT
Yx6Gcm/x0Xb2pf8zaUWE2q02Sz4WYpWrkcLZrLIVnCphyAK7K7V+I7ZkFP1reQR9
fr9RIsKC+6K5mWivFpOjkEzYp+D9Yp2OJyAgK0oE7F7Zd6u0+qpCWaM+VPHsYp41
wsVGTPgD5JP871o6ln38YskBct70E3BnzO5p10a5PbyRaaBCSBZC1XGCglIs7PS9
TmsZOI4Wc8dHzobUTXNdgOymfqXV/bQEITL8qRt9Ge2GUiUROFLp8IVzmJIADWKv
oqtoNo/vjthpFTXVZ3c4beACEaNu71bDbLjwjOdIb2W8NPY+Z+WeSkHDcg5Gt7qN
2cYWBUyhc3o=
=3e+9
-----END PGP SIGNATURE-----
Merge tag 'x86-urgent-2026-07-26' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
Pull x86 fix from Ingo Molnar:
- Disable jump/lookup tables in the x86 boot decompressor code
a bit more widely, because newer versions of LLVM started
optimizing it a bit better and introduced run-time relocations
in PIE code (Nathan Chancellor)
* tag 'x86-urgent-2026-07-26' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip:
x86/boot/compressed: Disable jump tables
An attacker injecting interrupts while the Safe-RET mitigation executes
on machines affected by SRSO can neutralize the safe return sequence,
potentially leading to data leakage through speculative execution.
Fixup register state as if the Safe-RET sequence executed successfully
by "emulating" it, in a manner of speaking, and avoid executing a RET
instruction after returning from the interrupt.
Co-developed-by: David Kaplan <David.Kaplan@amd.com>
Signed-off-by: David Kaplan <David.Kaplan@amd.com>
Signed-off-by: Borislav Petkov (AMD) <bp@alien8.de>
Unconditionally intercept and emulate ICEBP, i.e. INT1 #DBs, on SVM to
workaround a bug/misfeature in current AMD CPUs. AMD CPUs don't provide
information to allow software to distinguish instruction-induced #DBs (i.e.
ICEBP) from exception-induced #DBs (all others), and also don't report an
instruction length for an ICEBP-induced TASK_SWITCH.
On an intercepted TASK_SWITCH (which always has fault semantics), KVM
(any hypervisor, really) looks at the vectoring event type to decide
whether it was logically caused by a trap, and therefore whether to
advance guest RIP before entering the new task. If the guest IDT is
configured to deliver #DBs via a task gate, then the guest will see broken
behavior for ICEBP #DB because KVM doesn't have enough information to
detect and skip the ICEBP.
The typical workaround is to intercept ICEBP unconditionally and handle the
FAULT=>TRAP conversion in the hypervisor, at which point the #DB-induced
TASK_SWITCH occurs with RIP on the correct instruction boundary regardless
of whether it was instruction-induced or exception-induced.
As a bonus, intercepting ICEBP more or less aligns SVM with VMX (KVM always
intercepts #DBs on VMX, and ICEBP #DB VM-Exits on Intel have fault-like
behavior).
Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
Link: https://patch.msgid.link/e03f092dfbb7d391a6bf2797ba01e122ba080bcd.camel@infradead.org
[sean: drop selftest, reword changelog to provide more details]
Signed-off-by: Sean Christopherson <seanjc@google.com>
The OLPC GPIO controller sets up a fixed number space that is used
by at least two drivers:
arch/x86/platform/olpc/olpc-xo1-sci.c: In function 'setup_ec_sci':
arch/x86/platform/olpc/olpc-xo1-sci.c:358:13: error: implicit declaration of function 'gpio_request' [-Wimplicit-function-declaration]
358 | r = gpio_request(OLPC_GPIO_ECSCI, "OLPC-ECSCI");
| ^~~~~~~~~~~~
sound/pci/cs5535audio/cs5535audio_olpc.c: In function 'olpc_analog_input':
sound/pci/cs5535audio/cs5535audio_olpc.c:41:9: error: implicit declaration of function 'gpio_set_value'; did you mean 'gpiod_set_value'? [-Wimplicit-function-declaration]
41 | gpio_set_value(OLPC_GPIO_MIC_AC, on);
The AMD Geode platform that this is based on is now marked as
'Orphaned' in Kconfig, and it is likely that there are no XO1
users on modern kernels, but so far there is no consensus on
removing it entirely.
Select CONFIG_GPIOLIB_LEGACY for this platform and make sure the
sound driver portion cannot be compiled without this.
Acked-by: Borislav Petkov (AMD) <bp@alien8.de>
Acked-by: Takashi Iwai <tiwai@suse.de>
Reviewed-by: Bartosz Golaszewski <bartosz.golaszewski@oss.qualcomm.com>
Reviewed-by: Linus Walleij <linusw@kernel.org>
Acked-by: Dmitry Torokhov <dmitry.torokhov@gmail.com> # for input
Signed-off-by: Arnd Bergmann <arnd@arndb.de>
KVM services local TLB flushes on "full" nested VM-Exits (through
__nested_vmx_vmexit()), but not if a nested VM-Enter fails (e.g. due to
failed VMCS checks in nested_vmx_enter_non_root_mode()).
However, it is possible that KVM had queued TLB flushes that need to be
performed, even if the nested VM-Enter was not successful. For example,
if VPID is disabled for L2 (via nested_vmx_transition_tlb_flush(), or if
via the MSR load lists, as the SDM says:
If any MSR is being loaded in such a way that would architecturally
require a TLB flush, the TLBs are updated so that, after VM entry, the
logical processor will not use any translations that were cached before
the transition.
The SDM is unclear about when the TLB flush should occur, and whether or
not a failed VM entry would flush the TLB, so it is safer to always
do the TLB flush in this case.
More concretely, KVM also updates the last VPID L1 used for L2 in
nested_vmx_transition_tlb_flush() (i.e. last_vpid), even if the VM entry
ultimately fails. With the current code, KVM could miss a TLB flush if
L1 changes L2's VPID, then does a failed VM entry followed by a
successful one, as the failed VM entry would update last_vpid but not
actually flush the TLB. Servicing local TLB flushes on failed VM entries
makes sure that the TLB is always flushed when last_vpid is updated.
Fixes: 5c614b3583 ("KVM: nVMX: nested VPID emulation")
Cc: stable@vger.kernel.org
Reported-by: Sashiko <sashiko-bot@kernel.org> # Internal review
Suggested-by: Sean Christopherson <seanjc@google.com>
Signed-off-by: Yosry Ahmed <yosry@kernel.org>
Link: https://patch.msgid.link/20260722230128.1587363-1-yosry@kernel.org
Signed-off-by: Sean Christopherson <seanjc@google.com>
When {de,en}crypting memory of an SEV or SEV-ES guest on an SNP-enabled
host via a temporary buffer, allocate a full 4KiB page for the buffer to
ensure the page containing the buffer is wholly owned by KVM, i.e. won't
be concurrently allocated and accessed by other kernel code while KVM is
using the buffer to {de,en}crypt memory. On SNP-enabled platforms, when
sending SEV/SEV-ES commands that trigger firmware writes to memory, the
to-be-written page(s) must be (temporarily) assigned to Firmware (as
required by the SNP architecture, to guard against using such commands as
gadgets to attack SNP guests). See snp_map_cmd_buf_desc() and friends.
Unfortunately, transferring ownership of a page to Firmware makes the page
inaccessible to software, and thus writes generate RMP #PF violations. If
KVM uses a sub-page allocation for its temporary buffer, some other actor
in the kernel can allocate and use the other portions of the page, and thus
trigger unexpected (and seemingly spurious) RMP #PF violations due to
software attempting to access a Firmware-owned page.
BUG: unable to handle page fault for address: ffff906ae30f0300
#PF: supervisor write access in kernel mode
#PF: error_code(0x80000003) - RMP violation
PGD 6b1b80d067 P4D 6b1b80d067 PUD 100231e2063 PMD 10055a88063 PTE 80000100630f0163
SEV-SNP: PFN 0x100630f0 unassigned, dumping non-zero entries in 2M PFN region: [0x10063000 - 0x10063200]
Oops: Oops: 0003 [#1] SMP
CPU: 70 UID: 0 PID: 10658 Comm: svw_WaiterThrea Tainted: G U W O 7.1.0-smp--c22293789940-seanjc-next #1 PREEMPTLAZY
Tainted: [U]=USER, [W]=WARN, [O]=OOT_MODULE
Hardware name: Google, Inc. Arcadia_IT_80/Arcadia_IT_80, BIOS 34.86.0-102 01/25/2026
RIP: 0010:memset+0xf/0x20
Call Trace:
<TASK>
__kvmalloc_node_noprof+0x2a4/0x710
do_getxattr+0x4e/0x130
path_getxattrat+0x125/0x1b0
do_syscall_64+0x10a/0x480
entry_SYSCALL_64_after_hwframe+0x4b/0x53
RIP: 0033:0x7f3a22cb6daa
</TASK>
Modules linked in: kvm_amd kvm irqbypass vfat fat ccp k10temp sha3 libsha3 i2c_piix4 gq(O) cdc_acm xhci_pci xhci_hcd
gsmi: Log Shutdown Reason 0x03
CR2: ffff906ae30f0300
---[ end trace 0000000000000000 ]---
RIP: 0010:memset+0xf/0x20
Kernel panic - not syncing: Fatal exception
Kernel Offset: 0x39e00000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
gsmi: Log Shutdown Reason 0x02
Fixes: 4c735bf1bc ("KVM: SEV: Allocate only as many bytes as needed for temp crypt buffers")
Cc: stable@vger.kernel.org
Cc: Michael Roth <michael.roth@amd.com>
Debugged-by: Michael Roth <michael.roth@amd.com>
Link: https://patch.msgid.link/20260723211306.75397-1-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
AMD heterogeneous parts report a third core type via CPUID Fn0x80000026
EBX[31:28] (Extended CPU Topology, Core Type):
0 - Performance
1 - Efficiency
2 - Low Power
get_topology_cpu_type() only translates the first two, so on parts that ship
low-power cores the third type falls through to TOPO_CPU_TYPE_UNKNOWN.
Add TOPO_CPU_TYPE_LOW_POWER which takes care of this new type.
[ bp: Massage commit message, zap the "what" explanation. ]
Signed-off-by: Vishal Badole <Vishal.Badole@amd.com>
Signed-off-by: Borislav Petkov (AMD) <bp@alien8.de>
Link: https://patch.msgid.link/20260721154514.2506972-4-Vishal.Badole@amd.com
Replace the bare 0/1 in get_topology_cpu_type() with named constants that
mirror what the AMD APM publishes for CPUID Fn0x80000026 EBX[31:28] (Extended
CPU Topology, Core Type):
0 - Performance core
1 - Efficient core
2 - Low-power core (used by a follow-up)
No functional change.
Suggested-by: Borislav Petkov (AMD) <bp@alien8.de>
Signed-off-by: Vishal Badole <Vishal.Badole@amd.com>
Signed-off-by: Borislav Petkov (AMD) <bp@alien8.de>
Link: https://patch.msgid.link/20260721154514.2506972-3-Vishal.Badole@amd.com
Sashiko reported¹ that a confusion could ensue if a vendor-specific CPU
type 0 (performance) gets attempted to be used in a x86_cpu_id match
table. The current logic treats 0 as the wildcard X86_CPU_TYPE_ANY and
such a thing would end up matching the wrong CPUs.
This is all backwards because we started using the vendor-specific CPU
type number instead of using a Linux-defined, generic CPU type which
is agnostic.
Convert the current logic to it before users start appearing.
¹https://sashiko.dev/#/patchset/20260629094349.533301-1-Vishal.Badole%40amd.com
Signed-off-by: Borislav Petkov (AMD) <bp@alien8.de>
Signed-off-by: Vishal Badole <Vishal.Badole@amd.com>
Link: https://patch.msgid.link/20260721154514.2506972-2-Vishal.Badole@amd.com
Virtual addresses used with the AES-GCM library don't have to be in the
linear mapping. Remove the virt_addr_valid() checks from
snp_send_guest_request(), which were a workaround left over from when
this code used crypto_aead, which required scatterlists.
Link: https://lore.kernel.org/all/20251010151036.42241-2-ardb+git@google.com/
Reviewed-by: Alexey Kardashevskiy <aik@amd.com>
Tested-by: Alexey Kardashevskiy <aik@amd.com>
Tested-by: Nikunj A Dadhania <nikunj@amd.com>
Reviewed-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://patch.msgid.link/20260722025338.33354-3-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
The old AES-GCM library code is being replaced as part of an overhaul
that is adding support for all the common AES encryption modes with
consistent conventions. The SEV code is the only user of the old
AES-GCM library. Update it to use the new API instead.
Besides adjustments to the key struct name, function names, return
value, and parameter order, the only notable changes are:
- Replace the direct accesses to the auth tag length field (which should
be considered private until someone truly needs it) with the
AUTHTAG_LEN constant which is already defined in <asm/sev.h>.
- Replace kfree() with kfree_sensitive() on lines being changed anyway.
Tested-by: Nikunj A Dadhania <nikunj@amd.com>
Reviewed-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://patch.msgid.link/20260722025338.33354-2-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
Change the function's return type from 'int' to 'bool' and return the result
of memcmp() directly to simplify the code. While at it, cast ->start to
const u8 * to better match the expected type.
Also inline the verify_sha256_digest() check and remove the redundant
local 'ret' variable in purgatory().
No change in functionality expected.
Signed-off-by: Thorsten Blum <thorsten.blum@linux.dev>
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Link: https://patch.msgid.link/20260723133634.161598-3-thorsten.blum@linux.dev
flush_tlb_mm_range() and arch_tlbbatch_flush() pin the current CPU while
they decide whether the flush can be handled locally or must be sent to
remote CPUs. The CPU pinning is needed for the current CPU number and for
the local TLB flush path, which reads per-CPU TLB state.
The caller does not need to remain pinned while waiting for a remote TLB
flush to complete. After the remote-flush path has been selected,
flush_tlb_info is caller-private stack storage, so the caller no longer
has to stay on the same CPU to protect a shared per-CPU flush_tlb_info
object.
flush_tlb_multi() may also route through x86 PV backends. Those backends
must protect their own CPU-local scratch state instead of relying on the
caller to stay pinned. Hyper-V already does this by disabling interrupts
while using hyperv_pcpu_input_arg, and Xen's multicall path brackets its
per-CPU multicall buffer with xen_mc_batch() and xen_mc_issue().
kvm_flush_tlb_multi() also disables preemption while using __pv_cpu_mask.
Remote TLB flushes may synchronously wait for many CPUs, and the wait can
take tens of milliseconds when remote CPUs have interrupts disabled or
when many CPUs are involved. Keeping preemption disabled for that whole
wait unnecessarily increases scheduling latency on the initiating CPU.
Drop the CPU pinning before calling flush_tlb_multi() in the remote paths
of flush_tlb_mm_range() and arch_tlbbatch_flush(). Keep the local paths
inside the pinned section because they still access this CPU's TLB state.
Signed-off-by: Chuyi Zhou <zhouchuyi@bytedance.com>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Tested-by: Paul E. McKenney <paulmck@kernel.org>
Reviewed-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Link: https://patch.msgid.link/20260709122933.4021501-15-zhouchuyi@bytedance.com
kvm_flush_tlb_multi() is installed as an x86 PV TLB flush backend, so
flush_tlb_multi() can reach it through pv_ops when running as a KVM
guest.
kvm_flush_tlb_multi() uses the per-CPU scratch cpumask __pv_cpu_mask.
That buffer must remain tied to the current CPU until the mask has been
copied, filtered, and consumed by native_flush_tlb_multi().
The x86/mm callers currently enter flush_tlb_multi() while pinned to a
CPU. To let those callers drop CPU pinning before issuing the remote TLB
flush, each PV backend must protect its own CPU-local scratch state.
Make the KVM backend protect its per-CPU scratch cpumask by disabling
preemption locally. This is harmless with the current callers, where the
preemption disable is nested, and makes the KVM pv_ops dependency
explicit before changing the x86/mm call sites.
Signed-off-by: Chuyi Zhou <zhouchuyi@bytedance.com>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Reviewed-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Link: https://patch.msgid.link/20260709122933.4021501-14-zhouchuyi@bytedance.com
flush_tlb_info benefits from cacheline alignment, but using
cacheline-aligned stack storage directly can grow stack usage too much on
configurations with large SMP_CACHE_BYTES values. Commit 515ab7c413
("x86/mm: Align TLB invalidation info") attempted to align stack storage,
and commit 780e0106d4 ("x86/mm/tlb: Revert "x86/mm: Align TLB
invalidation info"") reverted it because using SMP_CACHE_BYTES led to 320
bytes of stack consumption. Commit 3db6d5a5ec ("x86/mm/tlb: Remove
'struct flush_tlb_info' from the stack") moved flush_tlb_info to per-CPU
storage, which avoided the stack growth problem while preserving
cacheline alignment. That was a good fit while the callers kept
preemption disabled for the whole flush operation.
However, a single per-CPU flush_tlb_info also requires all flush_tlb*()
operations to keep preemption disabled while the object is in use, so
that it cannot be overwritten by another flush on the same CPU.
flush_tlb*() may send IPIs to remote CPUs and synchronously wait for all
remote CPUs to complete their local TLB flushes. That wait can take tens
of milliseconds when interrupts are disabled on a remote CPU or when a
large number of remote CPUs are involved.
To shorten the CPU-pinned and preemption-disabled section around those
remote TLB flush waits, move flush_tlb_info back to caller-private stack
storage. The caller then does not have to stay on the same CPU until the
remote flush completes.
The type alignment is capped at 64 bytes. This keeps the alignment
benefit for stack objects without reintroducing the old large-cacheline
stack usage problem.
To evaluate the performance impact, use the following script to reproduce
the microbenchmark mentioned in commit 3db6d5a5ec ("x86/mm/tlb: Remove
'struct flush_tlb_info' from the stack"). The test environment is an Ice
Lake system (Intel(R) Xeon(R) Platinum 8336C) with 128 CPUs and 2 NUMA
nodes. During the test, the threads were bound to specific CPUs, and both
pti and mitigations were disabled:
#include <stdio.h>
#include <stdlib.h>
#include <pthread.h>
#include <sys/mman.h>
#include <sys/time.h>
#include <unistd.h>
#define NUM_OPS 1000000
#define NUM_THREADS 3
#define NUM_RUNS 5
#define PAGE_SIZE 4096
volatile int stop_threads = 0;
void *busy_wait_thread(void *arg) {
while (!stop_threads) {
__asm__ volatile ("nop");
}
return NULL;
}
long long get_usec() {
struct timeval tv;
gettimeofday(&tv, NULL);
return tv.tv_sec * 1000000LL + tv.tv_usec;
}
int main() {
pthread_t threads[NUM_THREADS];
char *addr;
int i, r;
addr = mmap(NULL, PAGE_SIZE, PROT_READ | PROT_WRITE, MAP_PRIVATE
| MAP_ANONYMOUS, -1, 0);
if (addr == MAP_FAILED) {
perror("mmap");
exit(1);
}
for (i = 0; i < NUM_THREADS; i++) {
if (pthread_create(&threads[i], NULL, busy_wait_thread, NULL))
exit(1);
}
printf("Running benchmark: %d runs, %d ops each, %d background\n"
"threads\n", NUM_RUNS, NUM_OPS, NUM_THREADS);
for (r = 0; r < NUM_RUNS; r++) {
long long start, end;
start = get_usec();
for (i = 0; i < NUM_OPS; i++) {
addr[0] = 1;
if (madvise(addr, PAGE_SIZE, MADV_DONTNEED)) {
perror("madvise");
exit(1);
}
}
end = get_usec();
double duration = (double)(end - start);
double avg_lat = duration / NUM_OPS;
printf("Run %d: Total time %.2f us, Avg latency %.4f us/op\n",
r + 1, duration, avg_lat);
}
stop_threads = 1;
for (i = 0; i < NUM_THREADS; i++)
pthread_join(threads[i], NULL);
munmap(addr, PAGE_SIZE);
return 0;
}
base on-stack-aligned on-stack-not-aligned
---- --------- -----------
avg (usec/op) 2.5278 2.5261 2.5508
stddev 0.0007 0.0027 0.0023
The benchmark results show that the average latency difference between
the baseline (base) and the properly aligned stack variable
(on-stack-aligned) is within the standard deviation (stddev). This
indicates that the variations are caused by testing noise, and reverting
to a stack variable with proper alignment causes no performance
regression compared to the per-CPU implementation. The unaligned version
(on-stack-not-aligned) shows a minor performance drop. The
CPU-pinned/preemption-disabled section can therefore be shortened without
sacrificing performance.
With caller-private storage there is no shared per-CPU object to protect,
so remove the DEBUG_VM reentrancy counter as well.
Signed-off-by: Chuyi Zhou <zhouchuyi@bytedance.com>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Tested-by: Paul E. McKenney <paulmck@kernel.org>
Reviewed-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Acked-by: Nadav Amit <nadav.amit@gmail.com>
Link: https://patch.msgid.link/20260709122933.4021501-13-zhouchuyi@bytedance.com
A stack allocated flush_tlb_info should keep cacheline alignment to avoid
the regression that motivated the per-CPU storage, but using
SMP_CACHE_BYTES directly can make the stack frame grow excessively on
configurations with large cache lines. This was addressed by
commit 780e0106d4 ("x86/mm/tlb: Revert "x86/mm: Align TLB
invalidation info""), where the stack consumption reached 320 bytes.
Add FLUSH_TLB_INFO_ALIGN and cap the type alignment at 64 bytes. The
existing per-CPU flush_tlb_info instance remains
DEFINE_PER_CPU_SHARED_ALIGNED(), so its per-CPU shared-cacheline
alignment is unchanged.
This prepares for moving flush_tlb_info back to stack storage without
reintroducing the old large-cacheline stack usage problem.
Signed-off-by: Chuyi Zhou <zhouchuyi@bytedance.com>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Reviewed-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Link: https://patch.msgid.link/20260709122933.4021501-12-zhouchuyi@bytedance.com
get_flush_tlb_info() has two responsibilities: it reserves the per-CPU
flush_tlb_info storage and it initializes the fields that describe the
flush operation. The per-CPU storage also carries the DEBUG_VM
reentrancy check and the matching put_flush_tlb_info() lifetime rules.
Moving flush_tlb_info back to caller-provided storage requires the same
field initialization without tying the caller to the per-CPU object.
Leaving the field setup embedded in get_flush_tlb_info() would either
keep those callers tied to the per-CPU object or duplicate the
initialization logic.
Split the field setup into init_flush_tlb_info(). Keep the per-CPU
storage selection, DEBUG_VM reentrancy check and put_flush_tlb_info()
lifetime rules in get_flush_tlb_info().
No functional change intended.
Signed-off-by: Chuyi Zhou <zhouchuyi@bytedance.com>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Reviewed-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Link: https://patch.msgid.link/20260709122933.4021501-11-zhouchuyi@bytedance.com
After a recent upstream LLVM change to start generating jump and lookup
tables in switch statements in more instances [1], linking the
compressed x86 boot image when CONFIG_KERNEL_ZSTD is enabled fails with:
ld.lld: error: Unexpected run-time relocations (.rela) detected!
Dumping the relocations in misc.o, which is the only file influenced by
CONFIG_KERNEL_ZSTD in the decompressor, shows dynamic relocations to
some string constants, which correspond to the string literals in the
switch statement in handle_zstd_error():
Relocation section '.rela.data.rel.ro' at offset 0x277b0 contains 31 entries:
Offset Info Type Symbol's Value Symbol's Name + Addend
0000000000000000 0000006600000001 R_X86_64_64 0000000000000000 .rodata.str1.1 + 73a
0000000000000008 0000006600000001 R_X86_64_64 0000000000000000 .rodata.str1.1 + 78e
0000000000000010 0000006600000001 R_X86_64_64 0000000000000000 .rodata.str1.1 + 78e
0000000000000018 0000006600000001 R_X86_64_64 0000000000000000 .rodata.str1.1 + 78e
...
This optimization is problematic for the decompressor environment, as it
is built as -fPIE without any explicit absolute references (as described
at the top of misc.c) while not applying any dynamic relocations, hence
the linker assertion. To opt out of this optimization, which is of
little value in this special early boot code, and to mirror the other
x86 startup code in arch/x86/boot/startup, disable jump tables in the
decompressor.
Signed-off-by: Nathan Chancellor <nathan@kernel.org>
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Acked-by: Ard Biesheuvel <ardb@kernel.org>
Cc: Bill Wendling <morbo@google.com>
Cc: Justin Stitt <justinstitt@google.com>
Cc: Nick Desaulniers <ndesaulniers@google.com>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: stable@vger.kernel.org
Link: fa02a6ed66 [1]
Link: https://patch.msgid.link/20260722-x86-boot-compressed-disable-jt-clang-v2-1-7373d38482fb@kernel.org
Closes: https://github.com/ClangBuiltLinux/linux/issues/2165
Commit 913bd90601 ("[PATCH] x86_64: Increase the variability of the
process stack on 64bit architectures"), merged in 2006, increased stack
randomisation from 8 MiB to an ostensible 1 GiB for 64-bit systems.
However the commit actually introduced a mask of 0x3fffff * 4 KiB pages
which provides 16 GiB of randomisation, not 1 GiB.
It appears Andi perhaps typo'd an additional 'f', as 0x3ffff would indeed
provide the stated 1 GiB.
Since this value has been in use for 20 years it seems sensible to retain
it, so correct the comment instead.
Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Acked-by: SJ Park <sj@kernel.org>
Link: https://patch.msgid.link/20260722-fix-x86-comment-v1-1-605080af988d@kernel.org
Signed-off-by: Kees Cook <kees@kernel.org>
__kvm_rmap_lock() deliberately elides the rmap lock when it observes an
empty rmap. In that case kvm_rmap_lock_readonly() also re-enables
preemption and returns zero, so the caller holds neither the rmap lock
nor a preemption reference. The elision documents the invariant it
relies on:
* Elide the lock if the rmap is empty, as lockless walkers (read-only
* mode) don't need to (and can't) walk an empty rmap, nor can they add
* entries to the rmap. I.e. the only paths that process empty rmaps
* do so while holding mmu_lock for write, and are mutually exclusive.
kvm_rmap_age_gfn_range() ignores the returned value and unconditionally
enters for_each_rmap_spte_lockless(). The iterator started with
rmap_get_first(), which re-reads rmap_head->val rather than using the
value returned by the lock. If a writer populates the rmap between the
lock's read and the iterator's re-read, the aging path walks the newly
installed rmap without holding its lock.
For a KVM_RMAP_MANY rmap this leaves the walker following a
pte_list_desc chain that it never locked. A writer holding mmu_lock for
write may free that chain (e.g. kvm_zap_all_rmap_sptes() on the recycle
path, or any rmap zap) via kmem_cache_free() while the walk is in
progress, giving a slab use-after-free. Nothing serialises the two: the
aging path runs without mmu_lock when CONFIG_KVM_MMU_LOCKLESS_AGING=y,
and the rmap lock that would otherwise exclude the writer was elided.
Because the empty path re-enables preemption, the interval between the
two reads can span an arbitrary scheduling delay.
Fix the class of bug by having the lockless walk consume the value
returned by the lock instead of re-reading the rmap. Split
rmap_get_first() into __rmap_get_first(), which starts an iterator from
an already-read rmap value, and make for_each_rmap_spte_lockless() take
that value and call __rmap_get_first() directly.
kvm_rmap_age_gfn_range() passes the value returned by
kvm_rmap_lock_readonly(): when the lock was elided the value is zero,
__rmap_get_first() returns NULL, and the walk is skipped. No lockless
walker re-reads the rmap, so the lock-elision invariant cannot be
violated, and no lock()-without-paired-unlock() path is added to the
aging code.
Fixes: af3b6a9eba ("KVM: x86/mmu: Walk rmaps (shadow MMU) without holding mmu_lock when aging gfns")
Suggested-by: Sean Christopherson <seanjc@google.com>
Cc: stable@vger.kernel.org
Signed-off-by: Phil Rosenthal <phil@phil.gs>
Link: https://patch.msgid.link/20260720-rmap-age-elided-submit-v2-1-668973030d47@phil.gs
Signed-off-by: Sean Christopherson <seanjc@google.com>
The size of the bandwidth field is enumerated from AMD hardware. resctrl uses
this field width to determine the maximum bandwidth supported that is stored
in resctrl_membw::max_bw. User space allocation requests ("control values")
are compared against this maximum for validity before being programmed to
hardware.
resctrl filesystem and resctrl x86 architecture code only support u32 control
values: resctrl_membw::max_bw is a u32, the control value provided by user
space is parsed into u32 local variables, and after validity checks, the
control value is staged into the u32 resctrl_staged_config::new_ctrl for
architecture consumption. The resctrl x86 architecture code in turn caches the
new control value into the u32 array rdt_hw_ctrl_domain::ctrl_val[].
The AMD bandwidth field to which control values are written can be up to
64 bits wide. While not an issue with current hardware (bandwidths that
require more than a u32, more than 536870911.875 GB/s, seem unreasonable
today), it is theoretically possible that enumeration of maximum bandwidth
field width will return values that are according to specification but
cannot be supported by resctrl.
Static checkers complain about this size mismatch.
Fix the static checker complaint by explicitly encoding the fact that resctrl
is unable to support all values that the hardware specification allows. Switch
to BIT() instead of open-coding the bitshift to avoid signed integer overflow
if the number of bits is a valid 31.
Signed-off-by: Reinette Chatre <reinette.chatre@intel.com>
Signed-off-by: Borislav Petkov (AMD) <bp@alien8.de>
Reviewed-by: Tony Luck <tony.luck@intel.com>
Link: https://patch.msgid.link/dd9fc3505c0ed250c5f14898b0ed6d7460a3cb82.1784753375.git.reinette.chatre@intel.com
When pinning SEV guest memory, drop FOLL_WRITE and rely on FOLL_LONGTERM
to break CoW, as *KVM* doesn't actually to the memory using the GUP'd
pages. Omitting FOLL_WRITE fixes a regression when using file-backed guest
memory that was introduced when KVM (correctly) added FOLL_LONG (e.g. to
ensure anonymous memory is migrated out of MIGRATE_CMA/ZONE_MOVABLE before
a long term pin).
Unfortunately, as of commits:
8ac268436e ("mm/gup: disallow FOLL_LONGTERM GUP-nonfast writing to file-backed mappings")
a6e79df92e ("mm/gup: disallow FOLL_LONGTERM GUP-fast writing to file-backed mappings")
GUP uses FOLL_LONGTERM as a canary of sorts to detect pins that are likely
to be problematic, and disallows WRITE+LONGTERM pins for file-backed memory.
As a result, backing SEV+ guests with file-backed memory, e.g. virtio-pmem,
fails due to the disallowed FOLL_LONGTERM+FOLL_WRITE combination.
Note, in the past, FOLL_WRITE was required to trigger CoW unsharing, to
prevent replacing the page in the (primary MMU's) page tables during a
later write fault after already having pinned a (shared) page in
MAP_PRIVATE mappings. FOLL_LONGTERM does that nowadays, even without
FOLL_WRITE (see gup_must_unshare()).
Fixes: 7e066cb9b7 ("KVM: SEV: Use long-term pin when registering encrypted memory regions")
Cc: stable@vger.kernel.org
Suggested-by: "David Hildenbrand (Arm)" <david@kernel.org>
Link: https://lore.kernel.org/all/ad784f05-b36c-4e91-9f17-4c5b826735d0@kernel.org/
Signed-off-by: Pankaj Gupta <pankaj.gupta@amd.com>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Acked-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Link: https://patch.msgid.link/20260715063626.65899-1-pankaj.gupta@amd.com
[sean: massage changelog, add comment about CoW unsharing]
Signed-off-by: Sean Christopherson <seanjc@google.com>
When handling userspace SREGS writes, check the validity of EFER (i.e.
allowed bits) before writing the new value of EFER through the
per-vendor set_efer callbacks. This prevents userspace from writing
bogus values (e.g. EFER.SVME=1 with nested=0).
Note: on KVM_SET_MSRS, KVM only checks EFER validity in terms of KVM
caps, not guest caps, so it is possible to set EFER bits that are
supported by KVM but not by the guest CPUID. Potentially allowing
userspace to set msrs before CPUID.
However, for KVM_SET_SREGS*, check the validity of the set bits against
both KVM and guest caps. This is consistent with other validity checks
(e.g. for CR4) that check validity against guest caps, which already
imposes the need to set CPUID before SREGS.
Cc: stable@vger.kernel.org
Signed-off-by: Yosry Ahmed <yosry@kernel.org>
Link: https://patch.msgid.link/20260713180153.2728382-2-yosry@kernel.org
Signed-off-by: Sean Christopherson <seanjc@google.com>
Supported EFER bits naturally fits into kvm_caps because it gets
recomputed during vendor initialization (e.g. to account for EFER.SVME
being allowed/disallowed based on nested being enabled/disabled). Move
efer_supported_bits into kvm_caps as supported_efer_bits (for naming
consistency).
As the bitmask is now globally visible as part of kvm_caps, there's
little use for helpers to enable/disable specific bits, so drop them and
open-code updates to kvm_caps.supported_efer_bits.
No functional change intended.
Suggested-by: Sean Christopherson <seanjc@google.com>
Reviewed-by: Nikolay Borisov <nik.borisov@suse.com>
Signed-off-by: Yosry Ahmed <yosry@kernel.org>
Link: https://patch.msgid.link/20260713181020.2735367-6-yosry@kernel.org
Signed-off-by: Sean Christopherson <seanjc@google.com>
In preparation for moving efer_reserved_bits into kvm_caps, reverse its
polarity and make it efer_supported_bits, to be more consistent with
other fields in kvm_caps.
No functional change intended.
Reviewed-by: Nikolay Borisov <nik.borisov@suse.com>
Signed-off-by: Yosry Ahmed <yosry@kernel.org>
Link: https://patch.msgid.link/20260713181020.2735367-5-yosry@kernel.org
Signed-off-by: Sean Christopherson <seanjc@google.com>
EFER reserved bits are statically initialized, and do not reset if a
vendor module is re-loaded. For example, loading kvm_amd with nested=1
removes EFER.SVME (and potentially EFER.LMSLE) from the reserved bits.
Reloading kvm_amd with nested=0 does not add them back, allowing
userspace to set EFER.SVME with nested=0.
Re-initializing EFER reserved bits before configuring them on vendor
initialization.
Cc: stable@vger.kernel.org
Signed-off-by: Yosry Ahmed <yosry@kernel.org>
Link: https://patch.msgid.link/20260713181020.2735367-4-yosry@kernel.org
Signed-off-by: Sean Christopherson <seanjc@google.com>
Remove EFER.LME and EFER.LMA from EFER reserved bits only if long mode
is actually supported. KVM does check long-mode support before allowing
the bits for guest writes and userspace writes through KVM_SET_SREGS*
(in __kvm_valid_efer()), but userspace writes through KVM_SET_MSRS only
check reserved bits.
In practice, this doesn't really matter. The true motiviation is getting
rid of the #ifdeffery when initializing efer_reserved_bits.
Cc: stable@vger.kernel.org
Signed-off-by: Yosry Ahmed <yosry@kernel.org>
Link: https://patch.msgid.link/20260713181020.2735367-3-yosry@kernel.org
Signed-off-by: Sean Christopherson <seanjc@google.com>
Move SVM-specific EFER bit enablement to generic x86 code, with the rest
of EFER bit enablement. Unifying the code for EFER bit enablement allows
for a later change to re-initialize EFER bits on module init.
No functional change intended.
Cc: stable@vger.kernel.org
Suggested-by: Sean Christopherson <seanjc@google.com>
Signed-off-by: Yosry Ahmed <yosry@kernel.org>
Link: https://patch.msgid.link/20260713181020.2735367-2-yosry@kernel.org
Signed-off-by: Sean Christopherson <seanjc@google.com>
Fix an issue where userspace or the guest can program an Hyper-V
synthetic timer to have a deadline in the past via integer overflow,
preventing the CPU from making progress and triggering an RCU stall.
Hyper-V's SynIC exposes 4 per-vCPU synthetic timers to the
guest, which are emulated by KVM. Each is programmed through the
HV_X64_MSR_STIMERi_CONFIG and HV_X64_MSR_STIMERi_COUNT MSRs. Depending
on CONFIG, COUNT represents either the absolute expiration time or the
period of a periodic timer, both expressed in 100ns ticks. These timers
may be set both by the guest (WRMSR) and the host (KVM_SET_MSRS).
When the timer is enabled, stimer_start() translates COUNT to an
absolute monotonic deadline and arms an hrtimer. If COUNT is set to a
value close to U64_MAX, the deadline calculation can overflow.
ktime_add_ns(ktime_now, 100 * (stimer->exp_time - time_now))
This can result in a CPU livelock. stimer_start() arms the timer
via hrtimer_start() with a deadline in the past, which causes it to
immediately fire. The stimer callback then raises KVM_RQ_HV_STIMER, with
the intention of causing KVM to deliver a synthetic interrupt on the
next vCPU guest enter.
Then, once userspace issues KVM_RUN, vcpu_enter_guest() consumes the
request, calling kvm_hv_process_stimers(). This would normally disable
the timer via stimer_expiration() once the deadline is in the past.
However, the deadline comparison is done between the KVM reference
counter and stime->exp_time, which is a big value close to U64_MAX, so
this never happens for a few thousand years.
kvm_hv_process_timers() then re-arms the timer via stimer_start(), since
it was not disabled, which again fires immediately. Before entering
the guest, kvm_vcpu_exit_request() checks kvm_request_pending(),
which returns true due to the newly raised KVM_REQ_HV_STIMER. Then
vcpu_enter_guest() aborts the guest entry, returning early into
vcpu_run(), which loops back again into vcpu_enter_guest(), restarting
the cycle.
Since there are no manual yields in this loop, a task with SCHED_FIFO
may starve RCU grace-period kthreads, which exposes the stalls found
by syzcaller:
rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
rcu: (detected by 1, t=10502 jiffies, g=14269, q=1142 ncpus=2)
rcu: All QSes seen, last rcu_preempt kthread activity 10500 (4294965239-4294954739), jiffies_till_next_fqs=1, root ->qsmask 0x0
rcu: rcu_preempt kthread starved for 10500 jiffies! g14269 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0
rcu: Unless rcu_preempt kthread gets sufficient CPU time, OOM is now expected behavior.
( ... )
Call Trace:
<IRQ>
__run_hrtimer kernel/time/hrtimer.c:1773 [inline]
__hrtimer_run_queues+0x408/0xc30 kernel/time/hrtimer.c:1841
hrtimer_interrupt+0x45b/0xaa0 kernel/time/hrtimer.c:1903
local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1045 [inline]
__sysvec_apic_timer_interrupt+0x102/0x3e0 arch/x86/kernel/apic/apic.c:1062
instr_sysvec_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1056 [inline]
sysvec_apic_timer_interrupt+0xa1/0xc0 arch/x86/kernel/apic/apic.c:1056
</IRQ>
<TASK>
asm_sysvec_apic_timer_interrupt+0x1a/0x20 arch/x86/include/asm/idtentry.h:697
RIP: 0010:__raw_spin_unlock_irqrestore include/linux/spinlock_api_smp.h:152 [inline]
RIP: 0010:_raw_spin_unlock_irqrestore+0xa8/0x110 kernel/locking/spinlock.c:194
Code: 74 05 e8 0b f4 5f f6 48 c7 44 24 20 00 00 00 00 9c 8f 44 24 20 f6 44 24 21 02 75 4f f7 c3 00 02 00 00 74 01 fb bf 01 00 00 00 <e8> 23 6b 27 f6 65 8b 05 7c 60 5a 07 85 c0 74 40 48 c7 04 24 0e 36
RSP: 0018:ffffc900040a7320 EFLAGS: 00000206
RAX: 5de15cb931505900 RBX: 0000000000000a06 RCX: 5de15cb931505900
RDX: 0000000000000007 RSI: ffffffff8daa9dc3 RDI: 0000000000000001
RBP: ffffc900040a73b0 R08: ffffffff8fc3d077 R09: 1ffffffff1f87a0e
R10: dffffc0000000000 R11: fffffbfff1f87a0f R12: dffffc0000000000
R13: 0000000000000000 R14: ffff8880b8628240 R15: 1ffff92000814e64
hrtimer_start include/linux/hrtimer.h:259 [inline]
stimer_start arch/x86/kvm/hyperv.c:682 [inline]
kvm_hv_process_stimers+0xd0a/0x16a0 arch/x86/kvm/hyperv.c:893
vcpu_enter_guest arch/x86/kvm/x86.c:11193 [inline]
vcpu_run+0x2240/0x76b0 arch/x86/kvm/x86.c:11639
kvm_arch_vcpu_ioctl_run+0x1148/0x1c90 arch/x86/kvm/x86.c:11984
kvm_vcpu_ioctl+0x99a/0xed0 virt/kvm/kvm_main.c:4492
vfs_ioctl fs/ioctl.c:51 [inline]
__do_sys_ioctl fs/ioctl.c:597 [inline]
__se_sys_ioctl+0xfc/0x170 fs/ioctl.c:583
do_syscall_x64 arch/x86/entry/syscall_64.c:63 [inline]
do_syscall_64+0xfa/0xf80 arch/x86/entry/syscall_64.c:94
entry_SYSCALL_64_after_hwframe+0x77/0x7f
RIP: 0033:0x7f635278f749
Code: ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 a8 ff ff ff f7 d8 64 89 01 48
RSP: 002b:00007f635365c038 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
RAX: ffffffffffffffda RBX: 00007f63529e5fa0 RCX: 00007f635278f749
RDX: 0000000000000000 RSI: 000000000000ae80 RDI: 0000000000000005
RBP: 00007f6352813f91 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
R13: 00007f63529e6038 R14: 00007f63529e5fa0 R15: 00007ffd5b219358
</TASK>
Fix this by clamping the deadline computation to KTIME_MAX, which
preserves the intent of arming a timer very far in the future.
ktime_add_safe() already does this type of clamping, so use it after
checking that that multiplying by the 100ns time tick also does not
overflow.
Reviewed-by: Vitaly Kuznetsov <vkuznets@redhat.com>
Reported-by: syzbot+3d5461510f8dc4adfe30@syzkaller.appspotmail.com
Closes: https://syzkaller.appspot.com/bug?extid=3d5461510f8dc4adfe30
Fixes: 1f4b34f825 ("kvm/x86: Hyper-V SynIC timers")
Cc: stable@vger.kernel.org
Signed-off-by: Carlos López <clopez@suse.de>
Link: https://patch.msgid.link/20260714133212.3916611-3-clopez@suse.de
[sean: tag for stable]
Signed-off-by: Sean Christopherson <seanjc@google.com>
Change to using KVM_X86_OP() instead of KVM_X86_OP_OPTIONAL() for the
.pi_update_irte() hook in kvm-x86-ops.h since now both VMX and SVM have
implemented it.
For the Fixes tag:
This hook was introduced for VMX posted-interrupt support. SVM later
added its implementation, but at this point KVM_X86_OP* had not been
introduced yet.
Initially KVM introduced KVM_X86_OP_NULL (and KVM_X86_OP) and used it
for this hook. But this was correct, because the use of KVM_X86_OP_NULL
was "to mark calls that do not follow the [svm|vmx]_func_name naming
convention" and the VMX one was named pi_update_irte(), i.e., did not
follow the convention. See commit 9af5471bdb ("KVM: x86: introduce
definitions to support static calls for kvm_x86_ops").
KVM later removed KVM_X86_OP_NULL (due to "the naming convention is not
in use anymore"), and added KVM_X86_OP_OPTIONAL for the hooks that can
be NULL pointer. It used KVM_X86_OP_OPTIONAL for this hook, but should
use KVM_X86_OP instead. See commit e4fc23bad8 ("KVM: x86: remove
KVM_X86_OP_NULL and mark optional kvm_x86_ops").
Note this hook was named .update_pi_irte() when it was introduced, but
got renamed to .pi_update_irte() at some point between the above two
commits.
Fixes: e4fc23bad8 ("KVM: x86: remove KVM_X86_OP_NULL and mark optional kvm_x86_ops")
Signed-off-by: Kai Huang <kai.huang@intel.com>
Link: https://patch.msgid.link/20260715090505.601174-1-kai.huang@intel.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
These callbacks are sysfs show paths.
Use sysfs_emit() and cpumask_pr_args() to emit the masks.
This prepares for removing cpumap_print_to_pagebuf().
Signed-off-by: Yury Norov <ynorov@nvidia.com>
BPF core and x86 JIT use text poking and temporary writable buffers and
thus can handle ROX memory.
Enable ROX cache for EXECMEM_BPF when configuration and CPU features allow
that.
Signed-off-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Acked-by: Song Liu <song@kernel.org>
Link: https://lore.kernel.org/bpf/20260716-execmem-x86-rox-bpf-v0-v3-5-4e76158c01c5@kernel.org
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
arch_bpf_trampoline_size() allocates a buffer to get actual size required
for a trampoline.
This buffer must be in the module address space because
__arch_prepare_bpf_trampoline() calculates rel32 offsets relatively to
that buffer.
In preparation for enabling ROX mode for EXECMEM_BPF make sure that the
allocated memory is writable.
Add bpf_jit_alloc_exec_rw() wrapper for execmem_alloc_rw() and use it for
buffer allocation in arch_bpf_trampoline_size().
Signed-off-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Acked-by: Song Liu <song@kernel.org>
Link: https://lore.kernel.org/bpf/20260716-execmem-x86-rox-bpf-v0-v3-4-4e76158c01c5@kernel.org
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
- Avoid redundant allocations when allocating IMSIC page tables
- Apply SBI FWFT LOCK flag only on successful set
- Bound SBI PMU counter mask scan to BITS_PER_LONG, since on RV32
the PMU SBI start/stop helper can only access 32 PMU counters.
- Skip TLB flush when G-stage PTE becomes valid if the Svvptc
extension is available.
- Always show Zicbo[m|z|p] block sizes in ONE_REG
- Inject instruction access fault on unmapped guest fetch
- Use raw spinlock for irqs_pending and irqs_pending_mask
- Fix Spectre-v1 in vector register access via ONE_REG
x86:
- Fixes to SEV selftests.
- Once free_nested() did a VMCLEAR of shadow VMCS, there's no need to VMCLEAR it
again if the kernel is preempted and thread migration happens
- Preserve nested TDP shadow page tables if they are used as roots, instead of
clearing them unnecessarily
- Fix use of stale data if out-of-memory happens after vendor module reload
- Check for invalid/obsolete root *after* making MMU pages available, because
the latter can make a page invalid
- Only reset TSC Deadline Timer in apic_timer_expired on KVM_RUN
-----BEGIN PGP SIGNATURE-----
iQFIBAABCgAyFiEE8TM4V0tmI4mGbHaCv/vSX3jHroMFAmpfSUwUHHBib256aW5p
QHJlZGhhdC5jb20ACgkQv/vSX3jHroOrWQf/S8Za6t7bZbQbJHDV0NXCKufpyKyz
l6WMsLGdn4LFLNqCtKhvQf5lEZ5PKRHFTGoWrO1wQSbREHRUCnfMD5lsSKQA4Grf
0QpQ6YBjXFv6zg0M456/Q/S1oyusyqkXXBS8BxlBQyfqR+ByKx8f4PXTGazfUua3
4SPHr5TUxr6CXy8715OQUGa1Xdf0/5d3OseiAgBwqM31yZPhK0tsar77Cd9P93bs
DC+FzfphHg0gb3s/xSfuVR0BPnqHvQMDF9rS/YKgJgJdZkzHXCo7muWRQmHWK+FR
Q2zOTWSlJATQucC0+2j1yj2Wdhg1itgZTfj/8RaIDRKbmjk0S1bLc3bamg==
=Svds
-----END PGP SIGNATURE-----
Merge tag 'for-linus' of git://git.kernel.org/pub/scm/virt/kvm/kvm
Pull kvm fixes from Paolo Bonzini:
"RISC-V:
- Avoid redundant allocations when allocating IMSIC page tables
- Apply SBI FWFT LOCK flag only on successful set
- Bound SBI PMU counter mask scan to BITS_PER_LONG, since on RV32 the
PMU SBI start/stop helper can only access 32 PMU counters.
- Skip TLB flush when G-stage PTE becomes valid if the Svvptc
extension is available.
- Always show Zicbo[m|z|p] block sizes in ONE_REG
- Inject instruction access fault on unmapped guest fetch
- Use raw spinlock for irqs_pending and irqs_pending_mask
- Fix Spectre-v1 in vector register access via ONE_REG
x86:
- Fixes to SEV selftests
- Once free_nested() did a VMCLEAR of shadow VMCS, there's no need to
VMCLEAR it again if the kernel is preempted and thread migration
happens
- Preserve nested TDP shadow page tables if they are used as roots,
instead of clearing them unnecessarily
- Fix use of stale data if out-of-memory happens after vendor module
reload
- Check for invalid/obsolete root *after* making MMU pages available,
because the latter can make a page invalid
- Only reset TSC Deadline Timer in apic_timer_expired on KVM_RUN"
* tag 'for-linus' of git://git.kernel.org/pub/scm/virt/kvm/kvm:
KVM: x86: Only reset TSC Deadline Timer in apic_timer_expired on KVM_RUN
KVM: selftests: sev_init2_tests: Derive SEV availability from KVM
KVM: selftests: sev_smoke_test: Only run VM types the host offers
KVM: x86/mmu: Fix use-after-free on vendor module reload
KVM: x86/mmu: Preserve nested TDP shadow page tables if they are used as roots
KVM: x86: Check for invalid/obsolete root *after* making MMU pages available
KVM: nVMX: Hide shadow VMCS right after VMCLEAR
KVM: riscv: Fix Spectre-v1 in vector register access
RISC-V: KVM: Serialize virtual interrupt pending state updates
RISC-V: KVM: Inject instruction access fault on unmapped guest fetch
RISC-V: KVM: Zicbo[m|z|p] block sizes should be always present in ONE_REG
riscv: kvm: Skip TLB flush when G-stage PTE becomes valid with Svvptc
KVM: riscv: PMU: Bound counter mask scan to BITS_PER_LONG
KVM: riscv: SBI FWFT: Apply LOCK flag only on successful set
RISC-V: KVM: Avoid redundant page-table allocations in ioremap topup
On Intel platforms with a VMX preemption timer and APICv, if a VMM
calls KVM_GET_LAPIC before KVM_GET_MSRS to save the vCPU state, it is
possible to lose a pending timer interrupt.
If the thread running these ioctls is migrated to another core after
calling KVM_GET_LAPIC but before KVM_GET_MSRS and the guest is using
their LAPIC timer in TSC-deadline mode, not only does the save LAPIC
state not carry the pending interrupt, the TSCDEADLINE MSR will be
zeroed.
After migration across CPUs, KVM_GET_MSRS calls vcpu_load, posting the
interrupt and clearing the MSR:
vcpu_load() ->
kvm_arch_vcpu_load() ->
kvm_lapic_restart_hv_timer() ->
start_hv_timer() ->
apic_timer_expired() ->
kvm_apic_inject_pending_timer_irqs()
. post interrupt into the LAPIC state
. clear IA32_TSCDEADLINE
The saved LAPIC state will be missing the pending interrupt and the saved
MSR will be zero. Oops.
Fix by only posting an interrupt when we're attempting to enter the guest
(vcpu->wants_to_run == true), not for vcpu_load from other paths.
Assisted-by: gemini:gemini-3.1-pro-preview
Debugged-by: David Matlack <dmatlack@google.com>
Debugged-by: Sean Christopherson <seanjc@google.com>
Debugged-by: Jim Mattson <jmattson@google.com>
Debugged-by: James Houghton <jthoughton@google.com>
Signed-off-by: Venkatesh Srinivas <venkateshs@chromium.org>
Message-ID: <20260715234234.15382-2-venkateshs@chromium.org>
Reviewed-by: James Houghton <jthoughton@google.com>
Reviewed-by: Chao Gao <chao.gao@intel.com>
Cc: stable@vger.kernel.org
Fixes: ae95f566b3 ("KVM: X86: TSCDEADLINE MSR emulation fastpath", 2020-05-15)
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
mmu_destroy_caches() destroys pte_list_desc_cache and
mmu_page_header_cache, but leaves both pointers unchanged. The pointers
live in kvm.ko, and therefore survive when a vendor module is unloaded
while kvm.ko remains loaded.
If creation of pte_list_desc_cache fails during a subsequent vendor
module load, its assignment sets pte_list_desc_cache to NULL and the
error path calls mmu_destroy_caches(). mmu_page_header_cache still
points to the cache destroyed during the preceding vendor module
unload. Passing that stale pointer to kmem_cache_destroy() causes a
slab use-after-free.
Reproduce the issue on a v7.1.3 kernel with CONFIG_KASAN=y,
CONFIG_KASAN_GENERIC=y, CONFIG_KVM=m, and CONFIG_KVM_INTEL=m. A
one-shot test hook forces pte_list_desc_cache to NULL on the second
invocation of kvm_mmu_vendor_module_init():
1. Load kvm.ko and kvm-intel.ko, creating both caches.
2. Unload only kvm_intel, leaving kvm.ko loaded.
3. Reload kvm_intel and force initialization through the -ENOMEM path.
KASAN reports:
BUG: KASAN: slab-use-after-free in
kvm_mmu_vendor_module_init+0x5b/0x170 [kvm]
...
kmem_cache_destroy+0x21/0x1d0
kvm_mmu_vendor_module_init+0x5b/0x170 [kvm]
...
Allocated by task 16817:
__kmem_cache_create_args+0x12c/0x3b0
__kmem_cache_create.constprop.0+0xb6/0xf0 [kvm]
kvm_mmu_vendor_module_init+0x13b/0x170 [kvm]
...
Freed by task 16820:
kmem_cache_destroy+0x117/0x1d0
kvm_mmu_vendor_module_exit+0x21/0x30 [kvm]
Clear both pointers immediately after destroying their caches so that
the stored state reflects the caches' lifetime and repeated cleanup is
safe.
With the fix applied, the same injected vendor module reload fails with
-ENOMEM as expected and produces no KASAN report.
Fixes: cb498ea2ce ("KVM: Portability: Combine kvm_init and kvm_init_x86")
Cc: stable@vger.kernel.org
Signed-off-by: Phil Rosenthal <phil@phil.gs>
Message-ID: <20260718-kvm-mmu-cache-uaf-v3-1-e103b93c74e1@phil.gs>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
kvm_mmu_zap_oldest_mmu_pages() excludes a shadow page whose root_count
is non-zero from top-level reclaim, because such a page cannot be
freed. The path in mmu_page_zap_pte() that recursively zaps a parentless
nested TDP child has no such check. As a result, a shadow page can
be zapped even if the page itself can't be freed; as the comment in
kvm_mmu_zap_oldest_mmu_pages() notes, zapping it will just force vCPUs
to rebuild the page.
As in top-level reclaim, do not recursively prepare zapping of a
nested TDP child whose root_count is non-zero.
Fixes: 2de4085ccc ("KVM: x86/MMU: Recursively zap nested TDP SPs when zapping last/only parent")
Signed-off-by: Hyunwoo Kim <imv4bel@gmail.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Check for a "stale" page fault, i.e. for an invalid and/or obsolete root,
after making MMU pages available for the shadow MMU. If reclaiming shadow
pages zaps an in-use root, i.e. marks it invalid, then KVM will attempt to
map memory into an invalid root. On its own, populating an invalid root is
"fine", but because child shadow pages inherit their parent's role, any
children created during the map/fetch will be created as invalid pages,
thus violating KVM's invariant that invalid pages are never on the list of
active MMU pages.
Note, the underlying flaw has existed since KVM first started tracking
invalid roots in 2008 (commit 2e53d63acb, "KVM: MMU: ignore zapped root
pagetables"), but the true badness only came along in 2020 (Linux 5.9)
with the invariant that invalid shadow pages can't be on the list of
active pages.
Note #2, inheriting role.invalid when creating child shadow pages is also
far from ideal; that flaw will be addressed separately.
Reported-by: Hyunwoo Kim <imv4bel@gmail.com>
Fixes: f95eec9bed ("KVM: x86/mmu: Don't put invalid SPs back on the list of active pages")
Cc: stable@vger.kernel.org
Signed-off-by: Sean Christopherson <seanjc@google.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
free_nested() frees the shadow VMCS while vmcs01 still points to it. But
because it is asynchronous with respect to loaded_vmcs_clear(), the vCPU
might migrate before the pointer is cleared and __loaded_vmcs_clear()
may then execute VMCLEAR.
The VMCS needs to stay attached until its explicit VMCLEAR completes, but
then it can be hidden and the page safely freed.
Fixes: 355f4fb140 ("kvm: nVMX: VMCLEAR an active shadow VMCS after last use")
Cc: stable@vger.kernel.org
Signed-off-by: Hyunwoo Kim <imv4bel@gmail.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
The low 32-bit ICR data is prepared by __prepare_ICR(), which returns a 32-bit
value. However, when this value is assigned to a new variable, it's easy to
mistakenly declare that variable with a different width.
To avoid this class of mistakes, use __prepare_ICR() directly as the function
argument instead of storing its result in an intermediate variable. This also
shaves off a bunch of lines in the code.
There should be no functionality change resulting from this.
Signed-off-by: Melody Wang <huibo.wang@amd.com>
Signed-off-by: Borislav Petkov (AMD) <bp@alien8.de>
Acked-by: Thomas Gleixner <tglx@kernel.org>
Link: https://patch.msgid.link/20260708012117.177959-1-huibo.wang@amd.com
The return values of syscall_enter_from_user_mode[_work]() are
non-intuitive. Both functions return the syscall number which should be
invoked by the architecture specific syscall entry code. The returned
number can be:
- the unmodified syscall number which was handed in by the caller
- a modified syscall number (ptrace, seccomp, trace/probe/bpf)
That has an additional twist. If the return value is -1L then the caller is
not allowed to modify the return value as that indicates that the modifying
entity requests to abort the syscall and set the return value already. That
can obviously not be differentiated from a syscall which handed in -1 as
syscall number.
The most trivial way to deal with that is:
set_return_value(regs, -ENOSYS);
nr = syscall_enter_from_user_mode(regs, nr);
if (valid(nr))
handle_syscall(regs, nr);
That's what LOONGARCH, RISCV, and X86 do. But PowerPC and S390 do not
preset the return value, so when user space hands in -1 and there is
nothing setting the return value in the entry work code, then the syscall
is skipped but the return value is whatever random data has been in the
return value register.
Change the return values of syscall_enter_from_user_mode[_work]() to
boolean and return false, when either ptrace or seccomp request to skip the
syscall. If they return true, update the syscall number as it might have
been changed.
That results in slightly different behaviour of the architectures versus
tracing.
If the syscall tracepoint has probe/BPF attached, those might set the
syscall number to -1 and also set the return value. PowerPC and S390 will
then overwrite that value with -ENOSYS. The other architectures will just
ignore it like any other invalid syscall and use the modified one.
Originally-by: Michal Suchánek <msuchanek@suse.de>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Tested-by: Michal Suchánek <msuchanek@suse.de>
Link: https://patch.msgid.link/20260712141346.772209074@kernel.org
Thomas Gleixner pointed out that cpu-type is a per-CPU property while hybrid
is a system property; conflating the two in the CPU matching infrastructure is
wrong. Currently, on a hybrid system x86_match_cpu() matches any cpu-type.
This works if the intent is to find the possibility of a cpu-type in a system.
But fails if matching for the cpu-type of a given CPU.
Borislav posted a cleanup here:
https://lore.kernel.org/all/20260703193222.GFakgORjvxwnZTPRnI@fat_crate.local
To make way for the cleanup stop matching cpu-type in cpu_vuln_blacklist.
RFDS is the only user, so drop the VULNBL_INTEL_TYPE entries and fold their
RFDS bit into the base Alder Lake (0x97) and Raptor Lake (0xB7) blacklist
entries. For now open-code cpu-type check in vulnerable_to_rfds(). In the
future, if more vulnerabilities need cpu-type matching a helper can be added.
No functional change intended.
Fixes: 722fa0dba7 ("x86/rfds: Exclude P-only parts from the RFDS affected list")
Signed-off-by: Pawan Gupta <pawan.kumar.gupta@linux.intel.com>
Signed-off-by: Borislav Petkov (AMD) <bp@alien8.de>
Link: https://patch.msgid.link/20260708-cpu-type-vuln-v1-1-85c1d3c704db@linux.intel.com
- Reject too long acpi_rsdp= boot parameter values
(Thorsten Blum)
- Validate console=uart8250 baud rate to fix early boot hang
(Thorsten Blum)
- Remove dead Makefile rule (Ethan Nelson-Moore)
Signed-off-by: Ingo Molnar <mingo@kernel.org>
-----BEGIN PGP SIGNATURE-----
iQJFBAABCgAvFiEEBpT5eoXrXCwVQwEKEnMQ0APhK1gFAmpcjhMRHG1pbmdvQGtl
cm5lbC5vcmcACgkQEnMQ0APhK1gVUw//UmbQ3jby1pvjDfAeJeqI8x2iCY2F9N5l
AqPD3371RPNzyx5uqyz4dLY94M31JW4jqzxntbIhGJrm9aXPLRTVzfMjF6sd9Lmj
lkMyHW+a4DVtA7oVSk2INebGUX2t+cpGLcF+9/yH3QavpLIXL9BNz5vDYqS0POMc
EitNNNqmO4+/f6oLf7Xxfl8Z2z2cVhcPCj1EG8b/rOWKeFmw1teuVu+dSagnV7Cn
/Hi4S8Q5HYUvvFBIFxgtuPW5S8HlTINhaE8hiat+Z0BENlAnbNThGEpz7hdogXAq
VDFwGPsjg3AF2agTWnF3e7Vs2CdDLHgcdTxUGII2lhZ9wCXzQ72+qXajf9PPynzH
YV9xtPw0a4sTDcR4QtdgBneCLf320WmbV6aH+tm9aZ1LIcPUgBO0qxYI/Ff1Frm8
NI1LxZJMiVfNNpFBk4JW/V1QKo3Padjhzfc23Rb4/8Xpu+S+Cu31pligIH3Rtnhm
pNt6Bv8eV61QJMZvjy4+5MouFfK2rL3YyEiE/WMP5cRhOCaK10hgDDEqCbuL5IRV
AgYMqGQPLTbZbdZZ1oSub+OPIRidMDfITlLoR9ggcJgNrV5J4tGysgAJWU2fCDoR
5dvRlzeQ0cjmCVMC+FVjQAahjBX5KqAQolDuDGB/vAAvxiir711ZO7g9UTCCRWQg
Bq1slSZ0WEc=
=upW0
-----END PGP SIGNATURE-----
Merge tag 'x86-urgent-2026-07-19' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
Pull x86 fixes from Ingo Molnar:
- Reject too long acpi_rsdp= boot parameter values (Thorsten Blum)
- Validate console=uart8250 baud rate to fix early boot hang (Thorsten
Blum)
- Remove dead Makefile rule (Ethan Nelson-Moore)
* tag 'x86-urgent-2026-07-19' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip:
x86/boot: Validate console=uart8250 baud rate to fix early boot hang
x86/boot: Reject too long acpi_rsdp= values
x86/cpu: Remove Makefile rule for removed UMC CPU support
The recent switch to 64-bit MSR interfaces introduced a build warning.
Work it around with a __maybe_unused annotation. After the full conversion
of the APIs we'll be able to address this more robustly by making sure
the APIs consume all the values as far the compiler is concerned, so that
the compiler doesn't emit such nuisance warnings.
[ mingo: Updated the changelog. ]
Fixes: cff219368b ("x86/mce: Stop using 32-bit MSR interfaces")
Closes: https://lore.kernel.org/oe-kbuild-all/202607031726.ZOwu4snu-lkp@intel.com/
Reported-by: kernel test robot <lkp@intel.com>
Signed-off-by: Juergen Gross <jgross@suse.com>
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Link: https://patch.msgid.link/20260703105555.1758819-1-jgross@suse.com
The 32-bit MSR interfaces rdmsr() and rdmsr_safe() are planned to be
removed. Use the related 64-bit variants instead.
Signed-off-by: Juergen Gross <jgross@suse.com>
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Cc: H. Peter Anvin <hpa@zytor.com>
Link: https://patch.msgid.link/20260703112426.1763049-1-jgross@suse.com
The 32-bit MSR interfaces rdmsr(), wrmsr() and rdmsr_safe() are
planned to be removed. Use the related 64-bit variants instead.
No change in functionality intended.
Signed-off-by: Juergen Gross <jgross@suse.com>
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Cc: H. Peter Anvin <hpa@zytor.com>
Cc: Kiryl Shutsemau <kas@kernel.org>
Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: Rick Edgecombe <rick.p.edgecombe@intel.com>
Cc: Sean Christopherson <seanjc@google.com>
Cc: kvm@vger.kernel.org
Link: https://patch.msgid.link/20260629060526.3638272-10-jgross@suse.com
The 32-bit MSR interfaces rdmsr(), wrmsr(), rdmsr_safe() and
wrmsr_safe() are planned to be removed. Use the related 64-bit variants
instead.
Signed-off-by: Juergen Gross <jgross@suse.com>
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Cc: H. Peter Anvin <hpa@zytor.com>
Link: https://patch.msgid.link/20260703112344.1762985-1-jgross@suse.com
The 32-bit MSR interfaces rdmsr(), wrmsr(), rdmsr_safe() and
wrmsr_safe() are planned to be removed. Use the related 64-bit variants
instead.
In processor_throttling.c drop needless initializers.
Signed-off-by: Juergen Gross <jgross@suse.com>
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Acked-by: Rafael J. Wysocki (Intel) <rafael@kernel.org>
Link: https://patch.msgid.link/20260629060526.3638272-5-jgross@suse.com
If a vCPU stays scheduled out (or blocked) while the last pCPU it ran
on goes through a hotplug cycle (online->offline->online), and the vCPU
then resumes execution on the same pCPU, then it is possible for it to
run with an ASID that has now been assigned to a different vCPU,
resulting in stale TLB translations being used.
svm_enable_virtualization_cpu() resets asid_generation to 1 and sets
next_asid to max_asid + 1 on every CPU online event, including hotplug
cycles. Because next_asid starts beyond the pool boundary, the first
call to new_asid() after an online event always wraps the pool,
incrementing asid_generation to 2 and assigning ASIDs starting from
min_asid.
Consider two vCPUs from different VMs, vCPU-A pinned to CPU-X holding
asid_generation=2 and ASID=N from before the hotplug event:
1. CPU-X goes offline and back online: asid_generation resets to 1,
next_asid = max_asid + 1.
2. One or more vCPUs migrate to CPU-X and call new_asid(), wrapping
the pool and consuming ASIDs starting from min_asid. Eventually
vCPU-B from a different VM is assigned asid_generation=2, ASID=N
— the same ASID that vCPU-A held before the hotplug.
3. vCPU-A enters pre_svm_run() on CPU-X: current_vmcb->cpu is
unchanged so the migration branch is skipped. Its saved
asid_generation=2 matches sd->asid_generation=2, so the generation
check silently passes and vCPU-A continues running with ASID=N —
the same ASID just freshly assigned to vCPU-B.
Both vCPUs from different VMs now run on CPU-X with the same ASID,
causing them to share NPT TLB entries and producing stale translations.
The collision manifests as a KVM internal error (Suberror: 1, emulation
failure). The NPT page fault reports a faulting GPA far outside the
VM's physical memory range — a sign of stale TLB translations being
used. KVM falls back to instruction emulation, which fails on
FPU/XSave instructions (XRSTOR, STMXCSR) that the emulator does not
implement.
Fix this by incrementing asid_generation instead of resetting it to 1
in svm_enable_virtualization_cpu(). On module load, asid_generation
starts at 0 (memset) and the increment produces 1, identical to the
old behaviour. On subsequent hotplug cycles the generation advances
beyond any value a vCPU previously observed on this CPU, so the
generation check in pre_svm_run() reliably forces new_asid() on every
vCPU after every hotplug cycle.
Fixes: 774c47f1d7 ("[PATCH] KVM: cpu hotplug support")
Reported-by: Chandrakanth Silveru <Chandrakanth.Silveru@amd.com>
Tested-by: Srikanth Aithal <Srikanth.Aithal@amd.com>
Reviewed-by: K Prateek Nayak <kprateek.nayak@amd.com>
Reviewed-by: Tom Lendacky <thomas.lendacky@amd.com>
Signed-off-by: Nikunj A Dadhania <nikunj@amd.com>
Message-ID: <20260715063506.672432-1-nikunj@amd.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Loading and unloading modules concurrently on several CPUs on a KASAN
build, with a short delay injected at the CPA page-table lookup to
widen the window, faults within minutes:
BUG: KASAN: use-after-free in __change_page_attr+0x7cc/0x7e0
Write of size 8 at addr ffff888181139718 by task modprobe
...
The buggy address belongs to the physical page:
pfn:0x181139 ... page_type: f2(table)
cpa_collapse_large_pages() rebuilds a leaf PMD from its 4K PTEs and
frees the old PTE-table pages, while __change_page_attr() fetches a
PTE pointer from a lockless lookup_address_in_pgd_attr() and writes
it with set_pte_atomic() only later. When module text is served from
a shared large ROX mapping the two run on the same PMD:
CPU A (module load) CPU B (module finalize)
------------------- -----------------------
execmem_make_temp_rw
set_memory_nx
__change_page_attr
split 2M -> 4K table P
kpte = &P[i] (lockless)
execmem_restore_rox
set_memory_rox (CPA_COLLAPSE)
cpa_collapse_large_pages
rebuild leaf PMD
flush_tlb_all
pagetable_free(P)
set_pte_atomic(kpte, ...)
-> writes into freed P
P is a page-table page (page_type: table), reused at once, so the
write corrupts whatever got the page next: a bad-pte or bad-page
splat, or a fatal fault once P has been turned into read-only text.
The flush_tlb_all() before the free does not close this: its IPI only
serializes against page-table walkers that run with interrupts off
(e.g. GUP-fast); the walk in __change_page_attr() runs with interrupts
on, so nothing stops it from holding a stale pointer into P.
Serialize the collapse - the PMD rebuild, TLB flush and PTE-table
free - under cpa_lock, the same lock __change_page_attr() now takes
unconditionally since commit ("x86/mm/pat: stop gating cpa_lock on
debug_pagealloc_enabled()"), so a concurrent walker can no longer
hold a pointer into a table the collapse is about to free.
Fixes: 41d88484c7 ("x86/mm/pat: restore large ROX pages after fragmentation")
Signed-off-by: Denis V. Lunev <den@openvz.org>
Signed-off-by: Dave Hansen <dave.hansen@linux.intel.com>
Acked-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
Link: https://patch.msgid.link/20260715183453.2381141-1-den@openvz.org
The splitting and merging of kernel page table mappings between small and
large is protected by cpa_lock. The merging is relatively new but the
splitting is ancient.
The splitting has a locking optimization: since DEBUG_PAGEALLOC forces all
mappings to 4k, there are no large pages to split. So the code that *might*
cause a split can just skip the locking (and a few other things).
This is entertaining, but it adds complexity and makes for weird locking
rules. Plus it's all for a debugging feature which makes the kernel super
slow in the first place. Optimizing something which is already super slow
and not used in production is not the best way to spend our complexity
budget.
Stop gating cpa_lock on debug_pagealloc_enabled() to simplify the code
and the locking rules.
[ dhansen: flesh out changelog ]
Suggested-by: Dave Hansen <dave.hansen@intel.com>
Signed-off-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Signed-off-by: Dave Hansen <dave.hansen@linux.intel.com>
Link: https://patch.msgid.link/20260715144519.934289-1-rppt@kernel.org
Link: https://lore.kernel.org/all/aab44f08-89f8-47fe-bee4-0ab6b25968c6@intel.com/
The header define in asm/timex,h and the naming of the function to read the
delay timer are confusing at best.
Convert it to a config switch selected by the archictures which provide the
functionality and rename the function to delay_read_timer(), which makes
the purpose clear. Move the declaration to linux/delay.h where it belongs.
Remove the resulting empty asm/timex.h files as well.
No functional change.
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Link: https://patch.msgid.link/20260410120317.978403520@kernel.org
This has been scheduled for removal more than a decade ago and the comments
related to it have been dutifully ignored. The last dependencies are gone.
Remove it along with various now empty asm/timex.h files.
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Reviewed-by: Christophe Leroy (CS GROUP) <chleroy@kernel.org>
Link: https://patch.msgid.link/20260410120317.910770161@kernel.org
CLOCK_TICK_RATE is only used in x86 but defined all over the tree for no
reason with comments that it's scheduled for removal for more than a
decade.
Use PIT_TICK_RATE for registering refined jiffies to get rid of the last
dependency.
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Link: https://patch.msgid.link/20260410120317.843419270@kernel.org
Put all vmcs12 pages if KVM synthesizes a nested VM-Exit due to invalid
guest while emulating VMLAUNCH or VMRESUME. The invalid guest state path
doesn't use nested_vmx_vmexit() as that API is intended to be used if and
only if L2 is active, and the open coded equivalent neglects to put the
vmcs12 pages. Failure to put the vmcs12 pages leaks any pinned pages
(and/or mappings) if L1 retries VMLAUNCH/VMRESUME.
Note, the !from_vmenter scenario doesn't suffer the same problem, as
vmx_get_nested_state_pages() only gets/pins/maps the vmcs12 pages if L2 is
active, i.e. if a "full" VM-Exit is guaranteed before KVM will retry
getting vmcs12 pages.
Fixes: 96c66e87de ("KVM/nVMX: Use kvm_vcpu_map when mapping the virtual APIC page")
Fixes: 3278e04925 ("KVM/nVMX: Use kvm_vcpu_map when mapping the posted interrupt descriptor table")
Fixes: fe1911aa44 ("KVM: nVMX: Use kvm_vcpu_map() to get/pin vmcs12's APIC-access page")
Reported-by: Minh Nguyen <minhnguyen.080505@gmail.com>
Cc: stable@vger.kernel.org
Signed-off-by: Sean Christopherson <seanjc@google.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
- Fix a bug where KVM will trigger a UAF if updating IOMMU IRTEs fails when
registering an IRQ-bypass producer.
- Ignore pending PV EOI instead of BUG()ing the host if the feature was
disabled by the guest.
- Fix nVMX bugs where KVM would run L1 with an L1-controlled CR3 after a
failed "late" consistency check when KVM is NOT using EPT.
- Disallow intra-host migration/mirroring of SNP VMs as KVM doesn't yet
support moving/mirroring SNP state.
- Fix a TOCTOU bug in KVM's handling of the "trusted" CPUID for TDX guests.
- Fix a NULL pointer deref in trace_kvm_inj_exception() where a change to the
core infrastructure missed KVM's unique (ab)use of __print_symbolic().
-----BEGIN PGP SIGNATURE-----
iQIzBAABCgAdFiEEKTobbabEP7vbhhN9OlYIJqCjN/0FAmpWb/kACgkQOlYIJqCj
N/1eAA//RDKkj5grTOKN7n9aY+uBrEFSH0VsEoyfg0GgiFZNOFWq4J6yiPTYb9kk
5Jxnl0LObI1sQ1Ini7LvB2sNa7YR5M3za3WNzUvynjw2gBvQGxU/7JMUqwcFlcXt
85RH/Wg7eNPCUvRUSsKEQwtdfi2NiGCxdg/AVnHs8/IPl8Ce7n3tQeKq4zm44mN9
eJ5hV5cCp6ytV7iM3N89yXo2LgLg2FWTg4vPA6MwxmTjNaouft3XH0428sNyI4zw
4NhLAnPLL8zmUs2zCxMSBYNkKiJ8Ou7jwD+1hny8MdHFbnrkNeNbk4Pbpol9AQ+Z
PhYeSsL0rk4bw+l5OEH5mxvtqjPV0qsH/ntN99W6kDh/gYDgJr7d70DsWxHWvluV
1Ewu26fjaUhKKwBoi4pDsXJVSHK2rTfeufeKegVW5IqEKLXYDzG2Kaj41UdPcFbo
HDv2FdUiqQo8PG2PUZzTduMOpcWggDjVgiqhAr4dL/Mf+IH+pDtPdCHM4XpvYHkQ
a+lFO76ZA/7fBVl8QgeZy8qRkbqSZLh00isUXXSRBamSL/rD27ODGOFrw+R9ct4s
hfUlC5GQDgrMmkqdweILfVip3/4B3Im2UI/5/QJB1bouf4r6w5A5fseTnMek445j
9zcFWquD17H/hg2DaS7+J0eZMKxhOZXegVLcE3UVqZLYw4VUBEw=
=+cUz
-----END PGP SIGNATURE-----
Merge tag 'kvm-x86-fixes-7.2-rc4' of https://github.com/kvm-x86/linux into HEAD
KVM x86 fixes for 7.2-rcN
- Fix a bug where KVM will trigger a UAF if updating IOMMU IRTEs fails when
registering an IRQ-bypass producer.
- Ignore pending PV EOI instead of BUG()ing the host if the feature was
disabled by the guest.
- Fix nVMX bugs where KVM would run L1 with an L1-controlled CR3 after a
failed "late" consistency check when KVM is NOT using EPT.
- Disallow intra-host migration/mirroring of SNP VMs as KVM doesn't yet
support moving/mirroring SNP state.
- Fix a TOCTOU bug in KVM's handling of the "trusted" CPUID for TDX guests.
- Fix a NULL pointer deref in trace_kvm_inj_exception() where a change to the
core infrastructure missed KVM's unique (ab)use of __print_symbolic().
Now that GENERIC_GETTIMEOFDAY and VDSO_GETRANDOM enable
HAVE_GENERIC_VDSO automatically, the architecture specific kconfig files
do not need to do so anymore.
Remove the corresponding lines.
Keep them for riscv, loongarch and powerpc as those have their own
explicit usage of the vDSO datastore provided by HAVE_GENERIC_VDSO.
Signed-off-by: Thomas Weißschuh <thomas.weissschuh@linutronix.de>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Reviewed-by: Vincenzo Frascino <vincenzo.frascino@arm.com>
Link: https://patch.msgid.link/20260709-vdso-have_generic_vdso-v1-5-d2e1061f268b@linutronix.de
Add missing ENCODE_FRAME_POINTER macro invocation into FRED_ENTER macro,
to prevent the unwinder from encountering a NULL stack frame pointer
when CONFIG_UNWINDER_FRAME_POINTER is enabled
Fixes: 14619d912b ("x86/fred: FRED entry/exit and dispatch code")
Signed-off-by: David Stevens <stevensd@google.com>
Signed-off-by: Dave Hansen <dave.hansen@linux.intel.com>
Acked-by: H. Peter Anvin (Intel) <hpa@zytor.com>
Link: https://patch.msgid.link/20260424191456.2679717-12-stevensd@google.com
Currently, only x86 genuinely implements and supports Syscall User Dispatch
(SUD).
Multiple architectures provide a stub arch_syscall_is_vdso_sigreturn()
returning 'false' simply to satisfy GENERIC_ENTRY compilation, which
creates a false impression of feature support.
Introduce ARCH_SUPPORTS_SYSCALL_USER_DISPATCH to decouple this mechanism
from GENERIC_ENTRY. Select it exclusively on x86 and remove the redundant
stub functions from other architectures.
Suggested-by: Mark Rutland <mark.rutland@arm.com>
Signed-off-by: Jinjie Ruan <ruanjinjie@huawei.com>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Reviewed-by: Thomas Weißschuh <thomas.weissschuh@linutronix.de>
Link: https://patch.msgid.link/20260713035422.582771-1-ruanjinjie@huawei.com
When the baud rate is empty, 0, invalid, or overflows to 0 when stored
as an int, the system will hang during early boot because of a division
by zero in early_serial_init().
Fall back to DEFAULT_BAUD when the resulting baud rate is 0 to prevent
an early system hang.
Fixes: ce0aa5dd20 ("x86, setup: Make the setup code also accept console=uart8250")
Signed-off-by: Thorsten Blum <thorsten.blum@linux.dev>
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Cc: stable@vger.kernel.org
Link: https://patch.msgid.link/20260713194924.126472-3-thorsten.blum@linux.dev
Convert xen_mm_unpin_all() to ptdescs in preparation for the eventual
splitting of ptdescs from struct page. Continue checking PagePinned
through the underlying page as we do not have a per-memdesc api yet.
Signed-off-by: Vishal Moola <vishal.moola@gmail.com>
Signed-off-by: Dave Hansen <dave.hansen@linux.intel.com>
Reviewed-by: William Kucharski <william.kucharski@linux.dev>
Reviewed-by: Juergen Gross <jgross@suse.com>
Acked-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Link: https://patch.msgid.link/20260629185742.126987-10-vishal.moola@gmail.com
Convert xen_mm_pin_all() to ptdescs in preparation for the eventual
splitting of ptdescs from struct page. Continue checking PagePinned
through the underlying page as we do not have a per-memdesc api for
page flags yet.
Signed-off-by: Vishal Moola <vishal.moola@gmail.com>
Signed-off-by: Dave Hansen <dave.hansen@linux.intel.com>
Reviewed-by: William Kucharski <william.kucharski@linux.dev>
Reviewed-by: Juergen Gross <jgross@suse.com>
Acked-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Link: https://patch.msgid.link/20260629185742.126987-9-vishal.moola@gmail.com
Convert pgd_page_get_mm() to ptdescs. Define struct ptdesc in our
pgtable_types so that our declarations recognize ptdesc as an
appropriate page table type.
Now that all callers are using ptdescs, we can pass in that
ptdesc to get the underlying mm_struct.
Signed-off-by: Vishal Moola <vishal.moola@gmail.com>
Signed-off-by: Dave Hansen <dave.hansen@linux.intel.com>
Reviewed-by: William Kucharski <william.kucharski@linux.dev>
Acked-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Link: https://patch.msgid.link/20260629185742.126987-8-vishal.moola@gmail.com
Convert sync_global_pgds_l4() to ptdescs in preparation for the
eventual splitting of ptdescs from struct page.
Signed-off-by: Vishal Moola <vishal.moola@gmail.com>
Signed-off-by: Dave Hansen <dave.hansen@linux.intel.com>
Reviewed-by: William Kucharski <william.kucharski@linux.dev>
Acked-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Link: https://patch.msgid.link/20260629185742.126987-7-vishal.moola@gmail.com
Convert sync_global_pgds_l5() to ptdescs in preparation for the
eventual splitting of ptdescs from struct page.
Signed-off-by: Vishal Moola <vishal.moola@gmail.com>
Signed-off-by: Dave Hansen <dave.hansen@linux.intel.com>
Reviewed-by: William Kucharski <william.kucharski@linux.dev>
Acked-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Link: https://patch.msgid.link/20260629185742.126987-6-vishal.moola@gmail.com
Convert arch_sync_kernel_mappings() to ptdescs in preparation for
the eventual splitting of ptdescs from struct page.
Following this patch, we can successfully boot a 32-bit x86 kernel with
separately allocated ptdescs.
Signed-off-by: Vishal Moola <vishal.moola@gmail.com>
Signed-off-by: Dave Hansen <dave.hansen@linux.intel.com>
Reviewed-by: William Kucharski <william.kucharski@linux.dev>
Acked-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Link: https://patch.msgid.link/20260629185742.126987-5-vishal.moola@gmail.com
Convert collapse_pmd_page() to ptdescs in preparation for the
eventual splitting of ptdescs from struct page.
Signed-off-by: Vishal Moola <vishal.moola@gmail.com>
Signed-off-by: Dave Hansen <dave.hansen@linux.intel.com>
Reviewed-by: William Kucharski <william.kucharski@linux.dev>
Acked-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Link: https://patch.msgid.link/20260629185742.126987-4-vishal.moola@gmail.com
Convert __set_pmd_pte() to ptdescs in preparation for the eventual
splitting of ptdescs from struct page.
Signed-off-by: Vishal Moola <vishal.moola@gmail.com>
Signed-off-by: Dave Hansen <dave.hansen@linux.intel.com>
Reviewed-by: William Kucharski <william.kucharski@linux.dev>
Acked-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Link: https://patch.msgid.link/20260629185742.126987-3-vishal.moola@gmail.com
Use IS_ENABLED() to check if we are on 32 bit. This standardizes this
check with the other 32 bit check in the file. No functional changes.
Signed-off-by: Vishal Moola <vishal.moola@gmail.com>
Signed-off-by: Dave Hansen <dave.hansen@linux.intel.com>
Reviewed-by: William Kucharski <william.kucharski@linux.dev>
Acked-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Link: https://patch.msgid.link/20260629185742.126987-2-vishal.moola@gmail.com
According to x86 architecture rules, 32-bit operations zero-extend the
result to 64 bits. The current implementation of handle_in() only masks
the lower 32 bits, which preserves the upper 32 bits of RAX when a
32-bit port IN instruction is emulated.
Use insn_assign_reg() to write the result back into RAX with proper
partial-register-write semantics: 1- and 2-byte forms leave the upper
bits untouched, the 4-byte form zero-extends to the full register.
Fixes: 0314994883 ("x86/tdx: Port I/O: Add runtime hypercalls")
Reported-by: Borys Tsyrulnikov <tsyrulnikov.borys@gmail.com>
Signed-off-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
Signed-off-by: Dave Hansen <dave.hansen@linux.intel.com>
Reviewed-by: Binbin Wu <binbin.wu@linux.intel.com>
Link: https://lore.kernel.org/all/CAKw_Dz96rfSQc6Rn+9QBcUFHhmkK+9zu+P=bxowfZwxrATCBRg@mail.gmail.com/
Cc:stable@vger.kernel.org
Link: https://patch.msgid.link/20260713133753.223947-4-kirill@shutemov.name
KVM's instruction emulator has a small helper, assign_register(), that
writes a value into a register following the x86 rules for writes to
general-purpose registers: an 8- or 16-bit write leaves the rest of the
register untouched, a 32-bit write zero-extends the result to 64 bits,
and a 64-bit write replaces the whole register.
The TDX guest #VE handler needs the same logic for port I/O emulation
to get 32-bit zero-extension right. Rather than add a third copy of
the same switch, move the helper verbatim to <asm/insn-eval.h>, rename
it to insn_assign_reg(), and route KVM's callers through it.
Add <asm/insn.h> to the header's includes so it builds standalone in
callers that have not pulled it in transitively.
No functional change.
Signed-off-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
Signed-off-by: Dave Hansen <dave.hansen@linux.intel.com>
Acked-by: Sean Christopherson <seanjc@google.com>
Cc:stable@vger.kernel.org
Link: https://patch.msgid.link/20260713133753.223947-3-kirill@shutemov.name
handle_in() and handle_out() in arch/x86/coco/tdx/tdx.c use:
u64 mask = GENMASK(BITS_PER_BYTE * size, 0);
GENMASK(h, l) includes bit h. For size=1 (INB), this produces
GENMASK(8, 0) = 0x1FF (9 bits) instead of GENMASK(7, 0) = 0xFF (8
bits). The mask is one bit too wide for all I/O sizes.
Fix the mask calculation.
Fixes: 0314994883 ("x86/tdx: Port I/O: Add runtime hypercalls")
Reported-by: Borys Tsyrulnikov <tsyrulnikov.borys@gmail.com>
Signed-off-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
Signed-off-by: Dave Hansen <dave.hansen@linux.intel.com>
Reviewed-by: Kai Huang <kai.huang@intel.com>
Reviewed-by: Kuppuswamy Sathyanarayanan <sathyanarayanan.kuppuswamy@linux.intel.com>
Reviewed-by: Binbin Wu <binbin.wu@linux.intel.com>
Reviewed-by: Rick Edgecombe <rick.p.edgecombe@intel.com>
Link: https://lore.kernel.org/all/CAKw_Dz96rfSQc6Rn+9QBcUFHhmkK+9zu+P=bxowfZwxrATCBRg@mail.gmail.com/
Cc:stable@vger.kernel.org
Link: https://patch.msgid.link/20260713133753.223947-2-kirill@shutemov.name
A resctrl domain consists of the domain structure self that includes
pointers to dynamically allocated filesystem as well as architecture
specific data. For example, the L3 monitoring domain structure consists
of the architecture specific struct rdt_hw_l3_mon_domain that contains
the dynamically allocated rdt_hw_l3_mon_domain::arch_mbm_states
architectural state and the embedded struct rdt_l3_mon_domain contains
the dynamically allocated rdt_l3_mon_domain::mbm_states resctrl fs state.
The domains are added to and removed from an RCU protected list while
cpus_write_lock() is held so that readers could access domains via
cpus_read_lock() or from an RCU read-side critical section. A reader
accessing a domain via the RCU list expects that the domain and all its
dynamically allocated data is accessible.
Only place the domain on the RCU list when all its dynamically allocated
data is ready, similarly unlink it from RCU list (again with cpus_write_lock()
held) before removing any of its dynamically allocated data.
Calling resctrl_online_mon_domain() before adding the domain to the RCU
list creates the kernfs files that expose the domain's monitoring data to
user space before adding the domain to the RCU list. This is safe because
rdtgroup_mondata_show() acquires cpus_read_lock() before it traverses the
RCU list and will thus block until the domain is added to the RCU list.
There are no readers accessing a domain via RCU list. Ensure safety of
access when such a reader arrives.
Signed-off-by: Reinette Chatre <reinette.chatre@intel.com>
Signed-off-by: Borislav Petkov (AMD) <bp@alien8.de>
Reviewed-by: Tony Luck <tony.luck@intel.com>
Reviewed-by: Chen Yu <yu.c.chen@intel.com>
Link: https://patch.msgid.link/31ae67084c983e8cb8c5ef2c65e1096de5e8f9b0.1783963505.git.reinette.chatre@intel.com
rdt_resource::ctrl_domains and rdt_resource::mon_domains are RCU lists with
entries added and removed by architecture from CPU hotplug callbacks that are
run with cpus_write_lock() held. These lists can be traversed safely from
resctrl fs by either holding cpus_read_lock() or relying on an RCU read-side
critical section.
resctrl fs traversals of rdt_resource::ctrl_domains and
rdt_resource::mon_domains are done using list_for_each_entry() with
cpus_read_lock() held. Similarly, x86 architecture callbacks use
list_for_each_entry() expecting that resctrl fs makes the call with
cpus_read_lock() held. Inconsistently, a lockdep_assert_cpus_held() precedes
the list_for_each_entry() call with varying distance to document this safe RCU
list traversal.
In preparation for an upcoming traversal of rdt_resource::ctrl_domains that
needs to be done from RCU read-side critical section there is a requirement
for developers to always know exactly in which context the list is being
traversed.
Replace the list_for_each_entry() traversals of RCU list with
list_for_each_entry_rcu() to document that an RCU list is being traversed
while making use of the built-in lockdep expression that additionally
documents that it is cpus_read_lock() that enables the list to be
traversed from non-RCU protection. Only revert to documenting the
safety of traversal using a comment when lockdep does not have needed
visibility in functions called via smp_call*().
The lockdep expression within list_for_each_entry_rcu() depends on
RCU_EXPERT that is not set in a typical debug kernel so keep the existing
lockdep_assert_cpus_held() that is active with CONFIG_LOCKDEP=y found in
typical debug kernel.
Signed-off-by: Reinette Chatre <reinette.chatre@intel.com>
Signed-off-by: Borislav Petkov (AMD) <bp@alien8.de>
Link: https://patch.msgid.link/d9373f8da8ffde667740e186ffc96ab69628ac9a.1783963505.git.reinette.chatre@intel.com
This was there to be able to patch out locking instructions when running
a SMP kernel build on a UP CPU. The times are long gone when single-CPU
x86 machines were relevant so drop that machinery and simplify the code
considerably.
LOCK_PREFIX needs to stay for when one wants to do a UP build for
whatever reason. That'll go away when CONFIG_SMP becomes unconditional.
Kill a bunch of leftover, unused prototypes while at it.
Signed-off-by: Borislav Petkov (AMD) <bp@alien8.de>
Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Acked-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Link: https://lore.kernel.org/r/20260328081634.797552-1-rppt@kernel.org
The trace_kvm_inj_exception tracepoint takes as arguments the exception
vector, whether the exception has an error code (and subsequently, the
error code), and whether it is being reinjected. Because '0' is a valid
error code, KVM uses __print_symbolic() to format the error code as a
string to avoid printing the error code entirely if the exception doesn't
have an error code (see commit 21d4c575eb ("KVM: x86: Print error code
in exception injection tracepoint iff valid").
KVM's abuse of __print_symbolic() was all fine and dandy, until commit
754e38d2d1 ("tracing: Use explicit array size instead of sentinel
elements in symbol printing") reworked the printing to avoid terminating
the arrays with NULL/0 values, and missed KVM's clever use of not-quite
empty array of symbols.
BUG: kernel NULL pointer dereference, address: 0000000000000000
#PF: supervisor read access in kernel mode
#PF: error_code(0x0000) - not-present page
PGD 0 P4D 0
Oops: Oops: 0000 [#1] SMP
CPU: 20 UID: 0 PID: 791 Comm: less Not tainted 7.2.0-rc2 #401 PREEMPT
Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
RIP: 0010:strlen+0x0/0x20
Call Trace:
<TASK>
trace_seq_puts+0x18/0x80
trace_print_symbols_seq+0x68/0xa0
trace_raw_output_kvm_inj_exception+0x64/0xf0 [kvm]
s_show+0x47/0x110
seq_read_iter+0x2a5/0x4c0
seq_read+0xfd/0x130
vfs_read+0xb6/0x330
? vfs_write+0x2f2/0x3f0
ksys_read+0x61/0xd0
do_syscall_64+0xb7/0x570
entry_SYSCALL_64_after_hwframe+0x4b/0x53
RIP: 0033:0x7ff283714862
</TASK>
Simply drop the dummy array entirely, so that __print_symbolic() generates
a truly empty array.
Signed-off-by: Daniel Paziyski <danielpaziyski@gmail.com>
Fixes: 754e38d2d1 ("tracing: Use explicit array size instead of sentinel elements in symbol printing")
Reviewed-by: Thomas Weißschuh <thomas.weissschuh@linutronix.de>
Link: https://patch.msgid.link/20260710134055.16432-1-danielpaziyski@gmail.com
[sean: massage changelog, add splat, add Fixes, cc stable]
Signed-off-by: Sean Christopherson <seanjc@google.com>
Converting from int to long, back to int and then to unsigned int is
confusing at best.
None of this voodoo is required. Negative syscall numbers including -1
don't need any of this treatment and the low level ASM code already does
the sign extension to 64-bit on a 64-bit kernel.
The only point where signedness matters is the comparison against the
maximum syscall number, but that can be simplified by just using a unsigned
argument for the various syscall invocation functions.
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Tested-by: Mukesh Kumar Chaurasiya (IBM) <mkchauras@gmail.com>
Link: https://patch.msgid.link/20260707190254.545214398@kernel.org
Invoking sys_ni_syscall() from a code path, which already knows that the
syscall number is invalid just to assign -ENOSYS to regs->ax is a pointless
exercise. It's even redundant as the low level entry code already has set
regs->ax to -ENOSYS on entry.
Remove the extra conditionals and the function calls.
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Tested-by: Mukesh Kumar Chaurasiya (IBM) <mkchauras@gmail.com>
Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com>
Link: https://patch.msgid.link/20260707190254.493733289@kernel.org
They are only used in the respective source files. No point in exposing
them.
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Tested-by: Mukesh Kumar Chaurasiya (IBM) <mkchauras@gmail.com>
Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com>
Reviewed-by: Mukesh Kumar Chaurasiya (IBM) <mkchauras@gmail.com>
Link: https://patch.msgid.link/20260707190254.438361565@kernel.org
The return value of __secure_computing() currently uses 0 to indicate
that a system call should be allowed, and -1 to indicate that it should
be blocked/killed. This 0/-1 pattern is non-intuitive for a security
check function and makes the control flow at the call sites less readable.
Furthermore, any potential future changes to these return values would
require a high-risk, error-prone audit of all its users across different
architectures.
Sanitize this logic by converting the return type of __secure_computing()
to a proper boolean, where 'true' explicitly means 'allow' and 'false'
means 'fail/deny'.
Update all the two dozen or so call sites across the tree to align with
this new boolean semantic. No functional changes are intended, as the
callers still return -1 to the lower-level assembly entry code upon
seccomp denial.
Rename the function to __seccomp_permit_syscall() so that the purpose is
entirely clear.
[ tglx: Rename the function ]
Suggested-by: Thomas Gleixner <tglx@kernel.org>
Suggested-by: Mark Rutland <mark.rutland@arm.com>
Signed-off-by: Jinjie Ruan <ruanjinjie@huawei.com>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Tested-by: Mukesh Kumar Chaurasiya (IBM) <mkchauras@gmail.com>
Reviewed-by: Mukesh Kumar Chaurasiya (IBM) <mkchauras@gmail.com>
Acked-by: Oleg Nesterov <oleg@redhat.com>
Link: https://patch.msgid.link/20260707190254.230735780@kernel.org
These functions integrate the stack randomization.
syscall_enter_from_user_mode_randomize_stack() has the advantage that the
randomization happens early right after enter_from_user_mode().
In both cases also the overhead of get/put_cpu_var() in
add_random_kstack_offset() is avoided.
No functional change.
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Tested-by: Mukesh Kumar Chaurasiya (IBM) <mkchauras@gmail.com>
Reviewed-by: Radu Rendec <radu@rendec.net>
Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@oss.qualcomm.com>
Reviewed-by: Mukesh Kumar Chaurasiya (IBM) <mkchauras@gmail.com>
Link: https://patch.msgid.link/20260707190254.079478122@kernel.org
cmdline_find_option() returns the full length of the parsed acpi_rsdp=
value. get_cmdline_acpi_rsdp() then silently truncates values which do
not fit in the val[] buffer.
Prevent boot_kstrtoul() from parsing a truncated value and then the
kernel from silently using the wrong RSDP address, see discussion in
Link:.
Issue a warning so that the user is aware that s/he supplied a malformed
value and can get feedback instead of silent crashes.
[ bp: Make commit message more precise. ]
Fixes: 3c98e71b42 ("x86/boot: Add "acpi_rsdp=" early parsing")
Signed-off-by: Thorsten Blum <thorsten.blum@linux.dev>
Signed-off-by: Borislav Petkov (AMD) <bp@alien8.de>
Cc: stable@vger.kernel.org
Link: https://lore.kernel.org/all/20260617130417.36651-4-thorsten.blum@linux.dev
Support for UMC CPUs was removed in
7d328c5de4 ("x86/cpu: Remove CPU_SUP_UMC_32 support"),
but a Makefile rule for the support code remained. Remove it.
Fixes: 7d328c5de4 ("x86/cpu: Remove CPU_SUP_UMC_32 support")
Signed-off-by: Ethan Nelson-Moore <enelsonmoore@gmail.com>
Signed-off-by: Borislav Petkov (AMD) <bp@alien8.de>
Reviewed-by: Ahmed S. Darwish <darwi@linutronix.de>
Link: https://patch.msgid.link/20260610033252.164571-1-enelsonmoore@gmail.com
A user-only branch stack can contain branches that originate from
the kernel. As a result, kernel addresses are exposed to user space
even when PERF_SAMPLE_BRANCH_USER is requested. On AMD processors
supporting X86_FEATURE_BRS (Zen 3 only), perf can still report entries
such as SYSRET/interrupt returns for which the branch-from addresses
are in the kernel.
E.g.
$ perf record -j any,u -c 4000 -e branch-brs -o - -- \
perf bench syscall basic --loop 1000 | \
perf script -i - -F brstack|tr ' ' '\n'| \
grep -E '0x[89a-f][0-9a-f]{15}'
...
0xffffffff810001c4/0x72e2e32955eb/-/-/-/0//-
0xffffffff810001c4/0x72e2d94a9821/-/-/-/0//-
0xffffffff810001c4/0x72e2d94ffa1b/-/-/-/0//-
...
BRS provides no hardware branch filtering, so privilege level
filtering is performed entirely in software. However, amd_brs_match_plm()
only validates the branch-to address against the requested privilege
levels. For branches from the kernel to user space, the branch-from
address is left unchecked and is leaked. Extend the software filter to
also validate the branch-from address, so that any branch record whose
branch-from address is in the kernel is dropped when
PERF_SAMPLE_BRANCH_USER is requested.
Fixes: 8910075d61 ("perf/x86/amd: Enable branch sampling priv level filtering")
Reported-by: Sashiko <sashiko-bot@kernel.org>
Signed-off-by: Sandipan Das <sandipan.das@amd.com>
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Cc: stable@vger.kernel.org
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Stephane Eranian <eranian@google.com>
Link: https://patch.msgid.link/f05931c4f89a146c364bd5dc6b8170b1ac611c65.1783701239.git.sandipan.das@amd.com
Closes: https://lore.kernel.org/all/20260710110235.F3FD81F000E9@smtp.kernel.org/
Commit aebc3ca190 ("KVM: x86: Enable CMCI capability by default and
handle injected UCNA errors") introduced kvm_vcpu_x86_set_ucna(), which
accesses @vcpu->arch.mci_ctl2_banks[] using @mce->bank as the index. The
@mce struct is user-controlled, provided via the KVM_X86_SET_MCE ioctl.
The caller of this function, kvm_vcpu_ioctl_x86_set_mce(), bounds-checks
@mce->bank and applies array_index_nospec() to advance the @banks
pointer, but @mce->bank itself is passed through unclamped. On a
speculative path that bypasses the bounds check, the raw @mce->bank
value can index mci_ctl2_banks[] out-of-bounds.
In practice this is a very weak gadget, and would at most allow leaking
a single bit in a 64-bit integer, but prevent potential future issues by
clamping @mce->bank in place with array_index_nospec(), before passing
the struct to kvm_vcpu_x86_set_ucna().
Fixes: aebc3ca190 ("KVM: x86: Enable CMCI capability by default and handle injected UCNA errors")
Signed-off-by: Carlos López <clopez@suse.de>
Link: https://patch.msgid.link/20260609131856.2562222-3-clopez@suse.de
Signed-off-by: Sean Christopherson <seanjc@google.com>
Advertise the Zhaoxin big-number arithmetic engine to guests via
CPUID 0xC0000001 EDX bits 27 (RSA) and 28 (RSA_EN). The RSA feature
provides two unprivileged instructions for modular arithmetic on big
integers, documented in the Zhaoxin PadLock Instruction Reference,
chapter 4 ("Modular Multiplication and Exponentiation Engine"). Both
support operand sizes from 256 to 32768 bits (in 128-bit increments):
- REP XMODEXP (encoding F3 0F A6 F8, subsection 4.1) computes
A^B mod M
- REP MONTMUL2 (encoding F3 0F A6 F0, subsection 4.2) computes
A*B mod M
REP MONTMUL2 is the long-mode replacement of legacy REP MONTMUL, which
is restricted to compatibility and 32-bit protected modes. These
primitives accelerate RSA and related public-key operations.
Both instructions are unprivileged (no CPL restriction) and available
in all CPU modes, with no associated MSR control. The RSA and RSA_EN
bits are redundant by hardware design (set or cleared together) and
both serve purely as CPUID-level feature-presence reporting flags
requiring no KVM emulation. Both bits are advertised because different
software may probe either one when checking for RSA availability.
Reviewed-by: Binbin Wu <binbin.wu@linux.intel.com>
Signed-off-by: Ewan Hai <ewanhai-oc@zhaoxin.com>
Link: https://patch.msgid.link/20260610023512.3690734-6-ewanhai-oc@zhaoxin.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
Advertise the Zhaoxin PadLock Hash Engine v2 to guests via CPUID
0xC0000001 EDX bits 25 (PHE2) and 26 (PHE2_EN). PHE2 extends the
PadLock hash family with SHA-384 and SHA-512 support per FIPS 180-3,
complementing the existing PHE feature (SHA-1 and SHA-256).
Two unprivileged instructions are exposed, documented in the Zhaoxin
PadLock Instruction Reference, chapter 3 ("Hash Engine"):
- REP XSHA384 (encoding F3 0F A6 D8, subsection 3.3)
- REP XSHA512 (encoding F3 0F A6 E0, subsection 3.4)
Both consume software-padded 128-byte blocks (RCX = block count, RSI =
input, RDI = state) and produce hash output in the state buffer.
Both instructions are unprivileged (no CPL restriction) and available
in all CPU modes, with no associated MSR control. The PHE2 and PHE2_EN
bits are redundant by hardware design (set or cleared together) and
both serve purely as CPUID-level feature-presence reporting flags
requiring no KVM emulation. Both bits are advertised because different
software may probe either one when checking for PHE2 availability.
Reviewed-by: Binbin Wu <binbin.wu@linux.intel.com>
Signed-off-by: Ewan Hai <ewanhai-oc@zhaoxin.com>
Link: https://patch.msgid.link/20260610023512.3690734-5-ewanhai-oc@zhaoxin.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
Advertise the Zhaoxin second-generation hardware RNG to guests via
CPUID 0xC0000001 EDX bits 22 (RNG2) and 23 (RNG2_EN).
RNG2 is exposed by the REP XRNG2 instruction (encoding F3 0F A7 F8),
documented in the Zhaoxin PadLock Instruction Reference, subsection 1.3
("REP XRNG2"). It produces random bytes from two on-die RNG sources
selectable via RAX bits[10:9] and an output mode (raw vs post-processed)
controlled by RDX bits[1:0], providing high-quality entropy intended
for cryptographic operations.
REP XRNG2 is unprivileged (no CPL restriction) and available in all
CPU modes, with no associated MSR control. The RNG2 and RNG2_EN bits
are redundant by hardware design (set or cleared together) and both
serve purely as CPUID-level feature-presence reporting flags requiring
no KVM emulation. Both bits are advertised because different software
may probe either one when checking for RNG2 availability.
Reviewed-by: Binbin Wu <binbin.wu@linux.intel.com>
Signed-off-by: Ewan Hai <ewanhai-oc@zhaoxin.com>
Link: https://patch.msgid.link/20260610023512.3690734-4-ewanhai-oc@zhaoxin.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
Advertise the Zhaoxin CCS (Chinese Cryptography Standard) feature to
guests via CPUID 0xC0000001 EDX bits 4 (CCS) and 5 (CCS_EN). CCS groups
two unprivileged instructions for Chinese national cryptographic
primitives, documented in the Zhaoxin GMI Instruction Set Reference,
chapter 2 ("CCS instruction group"):
- SM3 (encoding F3 0F A6 E8, subsection 2.1) implements the SM3 hash
algorithm specified in GM/T 0004-2012. It supports two modes
selected by RAX: auto-padding stream mode (RAX=0) and pre-padded
block mode (RAX=-1).
- SM4 (encoding F3 0F A7 F0, subsection 2.2) implements the SM4 block
cipher specified in GM/T 0002-2012, supporting ECB / CBC / CFB /
OFB / CTR modes via a control word in RAX, and CBC-MAC / CFB-MAC
when RAX bit[11] is set.
Both instructions are unprivileged (no CPL restriction) and available
in all CPU modes, with no associated MSR control. The CCS and CCS_EN
bits are redundant by hardware design (set or cleared together) and
both serve purely as CPUID-level feature-presence reporting flags
requiring no KVM emulation. Both bits are advertised because different
software may probe either one when checking for CCS availability.
Reviewed-by: Binbin Wu <binbin.wu@linux.intel.com>
Signed-off-by: Ewan Hai <ewanhai-oc@zhaoxin.com>
Link: https://patch.msgid.link/20260610023512.3690734-3-ewanhai-oc@zhaoxin.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
Advertise the Zhaoxin SM2 instruction support to guests via CPUID
0xC0000001 EDX bits 0 (SM2) and 1 (SM2_EN).
The SM2 instruction (encoding F2 0F A6 C0) implements the SM2
elliptic-curve public-key cryptography algorithm specified in
GM/T 0003-2012; the hardware-level behavior is documented in the
Zhaoxin GMI Instruction Set Reference, chapter 1 ("SM2"). The
instruction multiplexes its sub-functions on the RDX[5:0] control
word: encryption (subsection 1.1), decryption (1.2), signing (1.3),
signature verification (1.4), the three key-exchange sub-operations
of section 1.5 (1.5.1 SM2 key-pair generation, which the spec also
uses for the initiator's ephemeral key; 1.5.2 responder shared-key
derivation; 1.5.3 initiator shared-key derivation), and two
preprocess steps for identity and message hashing (1.6.1 and 1.6.2).
The instruction is unprivileged (no CPL restriction) and available
in all CPU modes, with no associated MSR control. The SM2 and SM2_EN
bits are redundant by hardware design (set or cleared together) and
both serve purely as CPUID-level feature-presence reporting flags
requiring no KVM emulation. Both bits are advertised because
different software may probe either one when checking for SM2
availability.
Reviewed-by: Binbin Wu <binbin.wu@linux.intel.com>
Signed-off-by: Ewan Hai <ewanhai-oc@zhaoxin.com>
Link: https://patch.msgid.link/20260610023512.3690734-2-ewanhai-oc@zhaoxin.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
Rework KVM's handling of per-vendor nested ops to copy the vendor's ops
into a global structure owned by common x86, i.e. treat nested ops just
like x86 and PMU ops. In addition to providing consistency across all ops
implementations, making a copy of the ops prevents changes to the vendor's
ops after KVM is initialized, i.e. guards against goofs where KVM *thinks*
it is updating nested ops, but which won't take effect now that KVM uses
static calls to invoke vendor hooks.
Ignoring the side effects of tagging {svm,vmx}_nested_ops as __initdata,
no functional change intended.
Link: https://patch.msgid.link/20260630202828.440724-4-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
Use static calls to invoke nested virtualization ops, as many of the calls
are in relatively hot paths when L2 is active, e.g. checking for events,
and because there's no reason not use static calls these days.
Opportunistically use a RET0 static call for get_evmcs_version() instead
of manually checking for a non-NULL vendor hook.
Link: https://patch.msgid.link/20260630202828.440724-3-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
Add a flag to explicitly track if nested virtualization is enabled, and use
it enumerate that various nested CAPs are unsupported, and to reject
enablement of said CAPs. When the nested ops hooks were moved to their
own structure, KVM's NULL-by-default behavior was deliberately dropped,
with the changelog asserting that all was well. That wasn't quite true;
there is no danger to KVM, but now KVM is over-reporting support for
KVM_CAP_NESTED_STATE and KVM_CAP_HYPERV_ENLIGHTENED_VMCS.
Fixes: 33b2217245 ("KVM: x86: move nested-related kvm_x86_ops to a separate struct")
Reviewed-by: Vitaly Kuznetsov <vkuznets@redhat.com>
Link: https://patch.msgid.link/20260630202828.440724-2-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
Reject KVM_TDX_INIT_VM if userspace changes cpuid.nent between the
initial read and the subsequent copy of the initialization data.
tdx_td_init() first reads user_data->cpuid.nent to size the flexible
kvm_tdx_init_vm copy. The copied structure also contains cpuid.nent,
and that field can differ from the value used to size the allocation if
userspace modifies the input concurrently. setup_tdparams_cpuids() later
passes init_vm->cpuid.nent to kvm_find_cpuid_entry2(), which uses it as
the array bound for the copied entries.
Require the copied count to match the value used to size the allocation
so that CPUID parsing cannot access beyond the entries actually copied.
Fixes: 0bd0a4a142 ("KVM: TDX: Replace kmalloc + copy_from_user with memdup_user in tdx_td_init()")
Reported-by: Sashiko:gemini-3.1-pro-preview
Cc: <stable@vger.kernel.org>
Signed-off-by: Binbin Wu <binbin.wu@linux.intel.com>
Reviewed-by: Xiaoyao Li <xiaoyao.li@intel.com>
Reviewed-by: Thorsten Blum <thorsten.blum@linux.dev>
Link: https://patch.msgid.link/20260710035324.3170534-1-binbin.wu@linux.intel.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
Wrap the .gmem_prepare() declarations with HAVE_KVM_GMEM_PREPARE so that
non-SEV code doesn't try to wire up a callback without doing the necessary
enabling.
No functional change intended.
Fixes: 3bb2531e20 ("KVM: guest_memfd: Add hook for initializing memory")
Reviewed-by: Ackerley Tng <ackerleytng@google.com>
Reviewed-by: Michael Roth <michael.roth@amd.com>
Link: https://patch.msgid.link/20260709204948.1988414-13-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
Track the guest as having a guest-provided VMSA as soon as control.vmsa_pa
is invalidated, instead of waiting to see if the guest-provided VMSA is
usable, so that KVM doesn't switch back to the original VMSA instead of
exiting to userspace (due to an invalid VMSA). By the time a vCPU tries
to load a guest-provided VMSA, KVM has already communicated "success" for
AP creation, i.e. KVM has committed to using the guest-provided VMSA.
Reviewed-by: Michael Roth <michael.roth@amd.com>
Link: https://patch.msgid.link/20260709204948.1988414-12-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
Wire up a gmem_invalidate_range() call for SNP VMs, and use it to force
vCPUs to reload/recheck their guest-provided VMSA if the backing gmem
page is being invalidated, e.g. is being PUNCH_HOLE'd. Use the same core
logic to handle invalidations as VMX does for the APIC-access page, as the
two concepts are nearly identical: shove the physical address of a page
into the vCPU's control structure:
1. Snapshot the invalidation sequence counter
2. Grab the pfn (from guest_memfd in this case)
3. Acquire mmu_lock for read
4. Re-request reload if retry is needed, otherwise commit the change.
Note, the re-request action in #4 is necessary as KVM's retry logic is
fuzzy, i.e. can get false positives. If the guest_memfd page has been
dropped, at some point a subsequent reload will fail to get a PFN from
guest_memfd, and KVM will fail KVM_RUN. If the retry was due to a false
positive, KVM will retry until there are no relevant MMU notifier events
(and will retry in the "outer" loop, i.e. will drop locks and resched as
needed).
Note #2! Take care to invalidate the VMSA when a relevant memslot is
DELETED or MOVED, as invalidations in response to PUNCH_HOLE are predicated
on memslot bindings (KVM doesn't know what GFN range(s) to invalidate
without a binding). And more importantly, the VMSA mapping requires a
memslot, i.e. must be invalidated if its memslots disappears, regardless of
the state of the underlying guest_memfd inode.
Failure to invalidate the vCPU's control.vmsa_pa (which is checked by
pre_sev_run()) can prevent KVM from properly freeing the page as firmware
will reject the RMPUPDATE to reclaim the page with FAIL_INUSE if the vCPU
is actively running, i.e. if VMSA page is in-use. That in turn leads to an
RMP #PF on the next use, as the page will still be assigned to the SNP VM.
SEV-SNP: RMPUPDATE failed for PFN 78d198, pg_level: 1, ret: 3
SEV-SNP: PFN 0x78d198, RMP entry: [0xfff0000000144001 - 0x000000000000000f]
CPU: 3 UID: 0 PID: 31345 Comm: sev_snp_vmsa_pu Tainted: G U O
Tainted: [U]=USER, [O]=OOT_MODULE
Hardware name: Google, Inc. Arcadia_IT_80/Arcadia_IT_80, BIOS 34.86.0-102 01/25/2026
Call Trace:
<TASK>
dump_stack_lvl+0x54/0x70
rmpupdate+0x12c/0x140
rmp_make_shared+0x3b/0x60
sev_gmem_invalidate+0xe0/0x170 [kvm_amd]
delete_from_page_cache_batch+0x1d8/0x220
truncate_inode_pages_range+0x120/0x3d0
kvm_gmem_fallocate+0x19a/0x270 [kvm]
vfs_fallocate+0x1bc/0x1f0
__x64_sys_fallocate+0x48/0x70
do_syscall_64+0x10a/0x480
entry_SYSCALL_64_after_hwframe+0x4b/0x53
RIP: 0033:0x496c7e
</TASK>
------------[ cut here ]------------
SEV: Failed to update RMP entry for PFN 0x78d198 error -14
WARNING: arch/x86/kvm/svm/sev.c:5160 at sev_gmem_invalidate+0x126/0x170 [kvm_amd], CPU#3: sev_snp_vmsa_pu/31345
CPU: 3 UID: 0 PID: 31345 Comm: sev_snp_vmsa_pu Tainted: G U O
Tainted: [U]=USER, [O]=OOT_MODULE
Hardware name: Google, Inc. Arcadia_IT_80/Arcadia_IT_80, BIOS 34.86.0-102 01/25/2026
RIP: 0010:sev_gmem_invalidate+0x12b/0x170 [kvm_amd]
Call Trace:
<TASK>
delete_from_page_cache_batch+0x1d8/0x220
truncate_inode_pages_range+0x120/0x3d0
kvm_gmem_fallocate+0x19a/0x270 [kvm]
vfs_fallocate+0x1bc/0x1f0
__x64_sys_fallocate+0x48/0x70
do_syscall_64+0x10a/0x480
entry_SYSCALL_64_after_hwframe+0x4b/0x53
RIP: 0033:0x496c7e
</TASK>
irq event stamp: 20689
hardirqs last enabled at (20699): [<ffffffff8e76092c>] __console_unlock+0x5c/0x60
hardirqs last disabled at (20708): [<ffffffff8e760911>] __console_unlock+0x41/0x60
softirqs last enabled at (20722): [<ffffffff8e6cd74e>] __irq_exit_rcu+0x7e/0x140
softirqs last disabled at (20717): [<ffffffff8e6cd74e>] __irq_exit_rcu+0x7e/0x140
---[ end trace 0000000000000000 ]---
BUG: unable to handle page fault for address: ffff99a64d198000
#PF: supervisor write access in kernel mode
#PF: error_code(0x80000003) - RMP violation
PGD 13eb001067 P4D 13eb001067 PUD 78d1d1063 PMD 1184e0063 PTE 800000078d198163
SEV-SNP: PFN 0x78d198, RMP entry: [0x6030000000144001 - 0x000000000000000f]
Oops: Oops: 0003 [#1] SMP
CPU: 3 UID: 0 PID: 31407 Comm: highlanderd_hea Tainted: G U W O
Tainted: [U]=USER, [W]=WARN, [O]=OOT_MODULE
Hardware name: Google, Inc. Arcadia_IT_80/Arcadia_IT_80, BIOS 34.86.0-102 01/25/2026
RIP: 0010:prep_new_page+0x67/0x220
Call Trace:
<TASK>
get_page_from_freelist+0x1c40/0x1c70
__alloc_frozen_pages_noprof+0xca/0x1f0
alloc_pages_mpol+0x10b/0x1b0
alloc_pages_noprof+0x81/0x90
pte_alloc_one+0x1b/0xd0
do_pte_missing+0xdf/0x1020
handle_mm_fault+0x7c7/0xb20
do_user_addr_fault+0x268/0x6b0
exc_page_fault+0x67/0xa0
asm_exc_page_fault+0x26/0x30
RIP: 0033:0x4a6b1e
</TASK>
gsmi: Log Shutdown Reason 0x03
CR2: ffff99a64d198000
---[ end trace 0000000000000000 ]---
RIP: 0010:prep_new_page+0x67/0x220
Drop the pseudo-TODO comment about needing to pin the page if guest_memfd
every supports migration, as integrating with invalidations events means
KVM will Just Work if/when page migration is ever supported (assuming SNP
hardware supports migrating VMSA pages).
Note #3, invalidate() and invalidate_range() have _completely_ different
semantics; the new invalidate_range() is a true invalidation, whereas the
existing invalidate() is really a "make shared" operation. Ignore the
confusing naming and poor Kconfig bundling for the moment to minimize the
delta for LTS kernels, the mess will be cleaned up shortly.
Reported-by: Hyunwoo Kim <imv4bel@gmail.com>
Closes: https://lore.kernel.org/all/aimMWzAf5b3luM0b@v4bel
Fixes: e366f92ea9 ("KVM: SEV: Support SEV-SNP AP Creation NAE event")
Cc: stable@vger.kernel.org
Cc: Tom Lendacky <thomas.lendacky@amd.com>
Cc: Michael Roth <michael.roth@amd.com>
Cc: Jörg Rödel <joro@8bytes.org>
Cc: Fuad Tabba <tabba@google.com>
Cc: Ackerley Tng <ackerleytng@google.com>
Reviewed-by: Michael Roth <michael.roth@amd.com>
Link: https://patch.msgid.link/20260709204948.1988414-11-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
Manually invoke the front half and back half of the "zap all fast" flow
when invalidating a memslot so that mmu_lock is acquired at function scope
in kvm_arch_flush_shadow_memslot(). This will allow putting more code
inside the critical section without having to take mmu_lock twice in quick
succession.
Opportunistically open code checking whether or not to do the fast zap, to
discourage removing the local "zap_all" in a future cleanup, i.e. to ensure
the SLOT_ZAP_ALL quirk is queried exactly once. Processing the front half
but not the back half of the fast zap (if SLOT_ZAP_ALL were disabled
concurrently) would result in KVM unnecessarily keeping invalid TDP MMU
roots until the VM is destroyed.
No functional change intended.
Cc: stable@vger.kernel.org # 6.12.x
Reviewed-by: Michael Roth <michael.roth@amd.com>
Link: https://patch.msgid.link/20260709204948.1988414-10-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
Split kvm_mmu_zap_all_fast() into a "front half" and a "back half", where
the front half is everything that runs with mmu_lock held for write, and
the back half is the code that runs outside of mmu_lock. This will allow
putting more code inside kvm_arch_flush_shadow_memslot()'s critical section
without having to take mmu_lock twice in quick succession.
No functional change intended.
Cc: stable@vger.kernel.org # 6.12.x
Reviewed-by: Michael Roth <michael.roth@amd.com>
Link: https://patch.msgid.link/20260709204948.1988414-9-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
Fold kvm_mmu_zap_memslot() into its sole caller so that its GFN range
structure can be used to trigger guest_memfd invalidations regardless of
whether KVM will do a partial or full zap of the MMU.
No functional change intended.
Cc: stable@vger.kernel.org # 6.12.x
Reviewed-by: Michael Roth <michael.roth@amd.com>
Link: https://patch.msgid.link/20260709204948.1988414-8-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
Wrap the sole reader of disabled_quirks with READ_ONCE(), and wrap the
post-VM-creation write to disabled_quirks with WRITE_ONCE(), to ensure
checking the status of a quirk doesn't re-read disabled_quirks *if* the
caller needs such a guarantee. This will allow splitting the "fast" MMU
zap into front and back halves, without potentially skipping the back
half if SLOT_ZAP_ALL were concurrently disabled (which would be "fine" in
the current code base, but far from ideal).
Cc: stable@vger.kernel.org # 6.12.x
Reviewed-by: Michael Roth <michael.roth@amd.com>
Link: https://patch.msgid.link/20260709204948.1988414-7-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
Protect writes to disabled_quirks with kvm->lock to ensure KVM doesn't
clobber state in the unlikely scenario that userspace disables disparate
quirks from multiple tasks. More importantly, this will allow wrapping
accesses with {READ,WRITE}_ONCE without "needing" to also guard the writer
with a useless and confusing READ_ONCE (since the RMW wouldn't be atomic
anyways).
Ideally, KVM would disallow disabling quirks once quirks are "live", but
that would be a potentially breaking userspace ABI change, and while all
existing quirks are fully live only after vCPUs have been created, several
MMU-related quirks, IGNORE_GUEST_PAT and SLOT_ZAP_ALL, are partially live
at all times. Because populating MMUs requires a vCPU, the guest-visible
behavior of IGNORE_GUEST_PAT and SLOT_ZAP_ALL requires a vCPU, but for KVM
itself, processing the quirk (or not) has functional impact, i.e. for all
intents and purposes, KVM can't prevent those quirks from being disabled
after they've been consumed.
Cc: stable@vger.kernel.org # 6.12.x
Reviewed-by: Michael Roth <michael.roth@amd.com>
Link: https://patch.msgid.link/20260709204948.1988414-6-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
Wire up the SEV-SNP guest_memfd kvm_x86_ops hooks if and only if SEV is
actually enabled, and drop the now-unnecessary stubs. Leaving the hooks
NULL allows the static call infrastructure to elide the CALL+RET, and more
importantly, referencing the hooks if and only if SEV support is enabled
will allow conditionally definining the hooks using their corresponding
HAVE_KVM_ARCH_GMEM_XXX Kconfig.
No functional change intended.
Cc: stable@vger.kernel.org # 6.12.x
Reviewed-by: Ackerley Tng <ackerleytng@google.com>
Link: https://patch.msgid.link/20260709204948.1988414-5-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
Always mark the vCPU as RUNNABLE after responding to AP_CREATE, even if the
guest-specified VMSA is unusable, e.g. isn't backed by a memslot or doesn't
have a backing guest_memfd page. If the VMSA is unusable, leaving the vCPU
in a non-running state will effectively hang the vCPU instead of reporting
an error to userspace. This will also allow retrying the VMSA load in the
future, to fix a bug where KVM doesn't honor guest_memfd invalidation
events, e.g. if AP_CREATION races with PUNCH_HOLE.
Cc: stable@vger.kernel.org # 6.12.x
Reviewed-by: Michael Roth <michael.roth@amd.com>
Link: https://patch.msgid.link/20260709204948.1988414-4-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
Extract the loading/retrieval of a guest-provided VMSA to a separate helper
so that KVM can reuse the core logic when refreshing the VMSA after an MMU
invalidation from guest_memfd.
No functional change intended.
Cc: stable@vger.kernel.org # 6.12.x
Reviewed-by: Michael Roth <michael.roth@amd.com>
Link: https://patch.msgid.link/20260709204948.1988414-3-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
Track the GPA of the guest-provided VMSA used after AP_CREATION events when
running SNP guests, instead of simply tracking whether or not the vCPU is
using a guest-provided VMSA. KVM needs to know the GPA of the VMSA that's
actively being used so that it can react to MMU invalidation events, i.e.
so that KVM can drop the VMSA if its backing guest_memfd page is punched
out of existence.
Opportunistically rename snp_vmsa_gpa to clarify that it tracks the pending
VMSA GPA, whereas snp_guest_vmsa_gpa now tracks the in-use VMSA GPA.
Note! Take care to track the GPA, not the GFN, as VALID_PAGE() won't
behave correctly if an invalid GFN is converted to a GPA for checking.
Note #2! Keep snp_has_guest_vmsa so that switching to a guest-provided
VMSA is sticky, even if the guest-provided VMSA becomes invalid.
No functional change intended.
Cc: stable@vger.kernel.org # 6.12.x
Reviewed-by: Michael Roth <michael.roth@amd.com>
Link: https://patch.msgid.link/20260709204948.1988414-2-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
The intra-host migration/mirroring feature is not fully implemented for
SEV-SNP VMs. The proper migration requires additional SNP-specific
state such as guest_req_mutex, guest_req_buf, and guest_resp_buf to be
transferred or initialized on the destination.
The SNP VM mirroring requires vmsa features to be copied as well otherwise
ASID would be bound to SNP range while VM is detected as a SEV VM.
Reject SNP source VMs in migration/mirroring until proper SNP state
transfer is implemented.
Fixes: 1dfe571c12 ("KVM: SEV: Add initial SEV-SNP support")
Reported-by: Chris Mason <clm@meta.com>
Reported-by: Sashiko <sashiko-bot@kernel.org>
Assisted-by: Claude:claude-opus-4-6
Reviewed-by: Tom Lendacky <thomas.lendacky@amd.com>
Signed-off-by: Atish Patra <atishp@meta.com>
Link: https://patch.msgid.link/20260602-sev_snp_fixes-v3-1-24bfd3ae047c@meta.com
Cc: stable@vger.kernel.org
[sean: let lines poke past 80 chars, tag for stable]
Signed-off-by: Sean Christopherson <seanjc@google.com>
In svm_set_nested_state(), the success path reaches out_free with
ret already set to 0 from nested_svm_load_cr3(). The explicit
'ret = 0' assignment before out_free is therefore redundant.
Remove it.
No functional change.
Signed-off-by: Qiang Ma <maqianga@uniontech.com>
Link: https://patch.msgid.link/20260618085217.3934985-1-maqianga@uniontech.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
A user-only branch stack can contain branches that originate from
the kernel. As a result, kernel addresses are exposed to user space
even when PERF_SAMPLE_BRANCH_USER is requested. On AMD processors
supporting X86_FEATURE_AMD_LBR_V2, perf can still report SYSRET/ERET
entries for which the branch-from addresses are in the kernel.
E.g.
$ perf record -e cycles -o - -j any,save_type,u -- \
perf bench syscall basic --loop 1000 | \
perf script -i - -F brstack|tr ' ' '\n'| \
grep -E '0x[89a-f][0-9a-f]{15}'
...
0xffffffff81001268/0x717a90a38f1a/M/-/-/0/ERET/NON_SPEC_CORRECT_PATH
0xffffffff81001268/0x717a90a39157/M/-/-/0/ERET/NON_SPEC_CORRECT_PATH
0xffffffff81001268/0x717a90a2c628/M/-/-/0/ERET/NON_SPEC_CORRECT_PATH
0xffffffff81001268/0x717a90a41b60/M/-/-/0/ERET/NON_SPEC_CORRECT_PATH
0xffffffff81001268/0x717a90a260db/M/-/-/0/ERET/NON_SPEC_CORRECT_PATH
0xffffffff81001268/0x717a90a260db/M/-/-/0/ERET/NON_SPEC_CORRECT_PATH
0xffffffff81001268/0x717a8bef1c30/M/-/-/0/ERET/NON_SPEC_CORRECT_PATH
0xffffffff81001268/0x717a8e4d3c90/M/-/-/0/ERET/NON_SPEC_CORRECT_PATH
...
The reason is that the hardware filter only considers the privilege
level applicable to the branch target. Extend software filtering to
also validate the branch-from addresses against br_sel, so that any
branch record whose branch-from address is in the kernel is dropped
when PERF_SAMPLE_BRANCH_USER is requested.
Fixes: f4f925dae7 ("perf/x86/amd/lbr: Add LbrExtV2 hardware branch filter support")
Reported-by: Ian Rogers <irogers@google.com>
Signed-off-by: Sandipan Das <sandipan.das@amd.com>
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Cc: stable@vger.kernel.org
Cc: Peter Zijlstra <peterz@infradead.org>
Link: https://patch.msgid.link/a898a29725f6b2f30518354cdc2e432db66c43cf.1783680119.git.sandipan.das@amd.com
Add CPUID and MSR bit definitions required to support Intel Directed
Package Thermal Interrupt.
A CPU requests directed package-level thermal interrupts by setting bit 25
in IA32_THERM_INTERRUPT. Hardware acknowledges by setting bit 25 in
IA32_PACKAGE_THERM_STATUS, indicating that only CPUs that opted in will
receive the interrupt. If no CPU in the package requests it, delivery
falls back to broadcast.
Signed-off-by: Ricardo Neri <ricardo.neri-calderon@linux.intel.com>
Link: https://patch.msgid.link/20260613-rneri-directed-therm-intr-v3-2-3a26d1e47fc8@linux.intel.com
Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
Move the comment about setting the encryption mask above the line which
does that and, especially, inside the ifdeffery, where it belongs.
Move comments on top of the code lines they refer to and not on the
side, which impairs readability.
No functional changes.
Signed-off-by: Borislav Petkov (AMD) <bp@alien8.de>
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Reviewed-by: Tom Lendacky <thomas.lendacky@amd.com>
Link: https://patch.msgid.link/20260620043757.347076-1-bp@kernel.org
PTI requires the begin and end of ENTRY_TEXT be aligned to PMD_SIZE. SRSO
requires srso_alias_untrain_ret to be 2M aligned. This costs between 2-4 MiB
of RAM (depending on the size of the preceding section). So, only align when
either of the two are enabled.
Co-developed-by: Jared White <jaredwhite@microsoft.com>
Signed-off-by: Jared White <jaredwhite@microsoft.com>
Signed-off-by: Hamza Mahfooz <hamzamahfooz@linux.microsoft.com>
Signed-off-by: Borislav Petkov (AMD) <bp@alien8.de>
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Link: https://patch.msgid.link/20260623152430.1473481-1-hamzamahfooz@linux.microsoft.com
The driver sets acpi_device_name() and acpi_device_class() which are
never read afterward, so make it stop doing that and drop the symbols
defined specifically for this purpose.
No intentional functional impact.
This will facilitate the removal of device_name and device_class from
struct acpi_device_pnp in the future.
Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
Reviewed-by: Andy Shevchenko <andriy.shevchenko@linux.intel.com>
Link: https://patch.msgid.link/2866696.mvXUDI8C0e@rafael.j.wysocki
Reviewed-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
vmx_get_msr() and vmx_set_msr() already cache to_vmx(vcpu) in a local
'vmx' pointer, but a few cases still open-code to_vmx(vcpu). Use the
cached pointer for consistency.
Likewise, cache to_vmx(vcpu) in vmx_get_segment_base() instead of
open-coding it in both the real-mode check and the VMCS read path.
No functional change intended.
Signed-off-by: Hao Zhang <zhanghao1@kylinos.cn>
Link: https://patch.msgid.link/tencent_A78DC401911634111A3391650CB00FCD0409@qq.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
When a VM is being destroyed, delete it from the list used to process GA
Log interrupts before vCPUs are freed, otherwise avic_ga_log_notifier()
could theoretically hit a use-after-free if a GA Log notification arrives
for a vCPU after the last reference to the VM has been put.
Note, in practice, it's likely all but impossible to trigger UAF, as all
all irqfds and thus all IRTEs are cleaned up by:
kvm_irqfd_release()
|
|-> irqfd_deactivate()
|
|-> irqfd_shutdown()
|
|-> irq_bypass_unregister_consumer()
And kvm_irqfd_release() is guaranteed to run before the last reference to
the VM is put. KVM also configures GA Log interrupts only when a vCPU is
blocking (older versions of KVM configre GA Log interrupts at all times,
but AVIC is off by default on those kernels). Hitting UAF would require
tearing down a VM shortly after a vCPU stopped blocking, and with a very,
very delayed IRQ from hardware.
Opportunistically use guard() to avoid a local "flags" variable.
Fixes: 5881f73757 ("svm: Introduce AMD IOMMU avic_ga_log_notifier")
Cc: Naveen N Rao (AMD) <naveen@kernel.org>
Cc: Xiao Wu <xiaowu.417@qq.com>
Reviewed-by: Naveen N Rao (AMD) <naveen@kernel.org>
Link: https://patch.msgid.link/20260630210156.457151-4-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
Move all per-VM AVIC initialization from VM creation to vCPU pre-creation,
i.e. defer allocating the logical ID table and adding the VM to the GA Log
list until vCPUs are created. This will allow removing the VM from the GA
Log list before vCPUs are destroyed without needing yet another kvm_x86_ops
hook (.vm_pre_destroy() is very intentionally called if and only if VM
creation fully succeeds).
As a bonus, this re-unites physical and logic table allocation, and avoids
allocating a logical table in the unlikely scenario that userspace creates
a VM without an in-kernel local APIC.
Another bonus to hooking .vcpu_precreate() is that there is no need to
unwind on failure, as the VM has already been created, i.e. KVM will run
through all phases of VM destruction. In fact, unwinding is undesirable,
as KVM tries to keep VM-wide behavior idempotent/sticky across creaton of
multiple vCPUs.
Reviewed-by: Naveen N Rao (AMD) <naveen@kernel.org>
Link: https://patch.msgid.link/20260630210156.457151-3-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
In anticipation of deferring all per-VM AVIC initialization until a vCPU
is first created, move SVM's kvm_x86_ops.vcpu_precreate() hook into avic.c
as avic_vcpu_precreate() and nullify the hook if AVIC is disabled (and
WARN if the hook is somehow invoked without AVIC enabled).
Reviewed-by: Naveen N Rao (AMD) <naveen@kernel.org>
Link: https://patch.msgid.link/20260630210156.457151-2-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
Move "struct kvm_vcpu_hv" and all of its child structures to hyperv.h,
guarded by CONFIG_KVM_HYPERV=y, as "struct kvm_vcpu_arch" holds a pointer
to the structure, i.e. only needs the structure to be declared, not fully
defined.
No functional change intended.
Reviewed-by: Kai Huang <kai.huang@intel.com>
Link: https://patch.msgid.link/20260625220450.3354415-10-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
Move the definition of "struct kvm_apic_map", a.k.a. the optimized local
APIC map, to lapic.h, as it is very nearly an implementation details that's
internal to KVM's local APIC emulation (KVM also uses the map to do quick
lookups when a vCPU is yielding to a different vCPU).
No functional change intended.
Suggested-by: Kai Huang <kai.huang@intel.com>
Reviewed-by: Kai Huang <kai.huang@intel.com>
Link: https://patch.msgid.link/20260625220450.3354415-9-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
Relocate KVM's TASK_SWITCH_<reason> enums from kvm_host.h to x86.h, as the
enums are arbitrary values, i.e. not architectural, and are intended to be
used only to translate vendor specific information to a common x86 reason
when invoking kvm_task_switch().
Opportunistically name the overall enum to help document the role of the
values.
No functional change intended.
Reviewed-by: Kai Huang <kai.huang@intel.com>
Link: https://patch.msgid.link/20260625220450.3354415-8-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
Add static asserts to sanity check the I/O permission map and TSS size
macros against tss_segment_32. Alternatively, the macros could simply use
offsetof() and sizeof(), but having literal numbers makes it easier to
understand the bigger picture, and provides a good excuse for the sanity
checks.
Opportunistically add the necessary includes to make tss.h self sufficient.
No functional change intended.
Reviewed-by: Kai Huang <kai.huang@intel.com>
Link: https://patch.msgid.link/20260625220450.3354415-7-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
Move KVM_GUESTDBG_VALID_MASK into x86.c so that it's not globally visible.
As explained by commit 462474588b ("KVM: x86: Move misc "VALID MASK"
defines from kvm_host.h => x86.c"), which unintentionally missed GUESTDBG,
the set of valid flags/bits is very much a KVM-internal detail, as the
values from the hardcoded #defines are often captured and massaged by KVM's
setup code, i.e. *directly* using the macros outside of KVM x86 would be
actively dangerous.
No functional change intended.
Reviewed-by: Kai Huang <kai.huang@intel.com>
Link: https://patch.msgid.link/20260625220450.3354415-6-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
Relocate a variety of Control/Debug Register macros that unintentionally
got left behind when the related helper function prototypes were moved to
regs.h.
No functional change intended.
Reviewed-by: Kai Huang <kai.huang@intel.com>
Link: https://patch.msgid.link/20260625220450.3354415-5-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
Tag tdp_enabled with __read_mostly as the variable is only ever written
during vendor module load, same as all the other global MMU variables that
are handled by kvm_configure_mmu().
Opportunistically annotate the tdp_mmu_enabled and eager_page_split
declarations with __read_mostly, to match their definitions. The compiler
will warn if there are conflicting annotations, i.e. there's minimal risk
of the declaration annotation becoming stale.
No functional change intended.
Reviewed-by: Kai Huang <kai.huang@intel.com>
Link: https://patch.msgid.link/20260625220450.3354415-3-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
Move the macros that define the mostly-obsolete apic_attention bits into
lapic.c, as the gory details of PV EOIs and the pre-APICv TPR acceleration
are 100% internal to KVM's local APIC emulation.
No functional change intended.
Reviewed-by: Kai Huang <kai.huang@intel.com>
Link: https://patch.msgid.link/20260625220450.3354415-2-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
When activating Hyper-V's Synthetic Interrupt Controller (SynIC), mark it
active with WRITE_ONCE() and query it using READ_ONCE() in synic_get(),
the only known cross-task reader, to document that the flag is accessed
without holding the vCPU's mutex.
Note, there are no data dependencies on the SynIC being marked active,
e.g. the vector read by synic_set_irq() is set (usually in response to
guest activity) long after the SynIC is initially activated, and a false
negative on the SynIC being active would be benign (ignoring that such a
race is likely to be problematic for the guest irrespective of what KVM
does).
Link: https://patch.msgid.link/20260630225619.511632-12-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
Assert that either vcpu->mutex is held or the VM is otherwise unreachable
when using the normal vCPU => HyperV accessor to help detect improper
cross-task usage of the HyperV structure. When accessing the structure
without holding the vCPU's mutex, e.g. to send interrupts or to queue TLB
flushes, KVM needs to use the more paranoid to_hv_vcpu_safe() to guarantee
that it can't see a half-baked structure.
To avoid false positives, open code accesses to vcpu->arch.hyperv in the
Synthetic Timer callbacks (can be reached if and only if HyperV state is
fully initialized).
Link: https://patch.msgid.link/20260630225619.511632-11-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
Extract nVMX's lockdep assertion that a vCPU is locked or otherwise
unreachable into a common helper, as KVM x86 is about to gain another user,
but there is nothing x86-specific about the logic, i.e. the assertion may
be useful for other architectures.
No functional change intended.
Reviewed-by: Philippe Mathieu-Daudé <philmd@oss.qualcomm.com>
Link: https://patch.msgid.link/20260630225619.511632-9-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
Explicitly invalidate KVM's internal Xen vCPU ID during vCPU creation
instead of *trying* to set the Xen ID to the vCPU index by default, and
forward singleshot timer hypercalls to userspace if the VMM hasn't set the
Xen ID via KVM_XEN_VCPU_ATTR_TYPE_VCPU_ID. Using the vCPU's index as its
default Xen ID is reasonable in concept, but in practice is horribly flawed
as the index is left as '0' until after vCPU initialization completes, i.e.
every vCPU gets a Xen ID of '0' by default.
Forward hypercalls to userspace instead of trying to salvage any kind of
default behavior, as all userspace implementations that support multiple
vCPUs either don't enable the timer, are guaranteed to set Xen ID, or work
only because *all* guests also screw up the singleshot timer hypercalls.
The last scenarios is extremely unlikely given that Linux-as-a-guest uses
the actual Xen vCPU ID when making timer hypercalls. In other words, for
all intents and purposes, KVM's ABI is already that userspace must set the
Xen vCPU ID, so just commit to that ABI.
Note, KVM's handling of KVM_XEN_VCPU_ATTR_TYPE_VCPU_ID restricts the ID to
KVM_MAX_VCPUS, so there's no chance of a valid ID colliding with U32_MAX.
Add a compile-time assertion to ensure this holds true in the future (KVM
doesn't care what value is used for "invalid", only that there can't be a
collision).
Link: https://lore.kernel.org/all/20260612233017.1F9771F000E9@smtp.kernel.org
Suggested-by: David Woodhouse <dwmw2@infradead.org>
Reviewed-by: David Woodhouse <dwmw@amazon.co.uk>
Link: https://patch.msgid.link/20260630225619.511632-7-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
Hoist the checks on the Xen vCPU ID when handling set_singleshot_timer and
stop_singleshot_timer hypercalls out of their individual if-statements,
so that both checks on the ID are in common code. kvm_xen_hcall_vcpu_op()
is already doubly committed to handling only singleshot timer hypercalls,
and even if that were to change in the future, the function could simply
be renamed and turned into a helper specifically for timer hypercalls.
Opportunistically add a comment to explain why the check exists; the code
looks rather nonsensical without the knowledge that @vcpu_id is a common
param for all per-vCPU hypercalls.
No functional change intended.
Reviewed-by: David Woodhouse <dwmw@amazon.co.uk>
Reviewed-by: Philippe Mathieu-Daudé <philmd@oss.qualcomm.com>
Link: https://patch.msgid.link/20260630225619.511632-6-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
When handling Xen vCPU hypercalls, explicitly route non-singleshot-timer
commands to userspace, *before* checking if in-kernel emulation of the Xen
timer is enabled. Punting hypercalls that are never accelerated by KVM
because some other hypercall happens to be disabled is confusing and
actively dangerous, e.g. it's easy to miss that the only reason KVM can
bail early is because the timer-disabled case provides the same semantics
as the implicit "default" path in the switch-statement.
Opportunistically convert the switch-statement to an if-else-statement to
avoid having to carry code for an impossible "default" case.
For all intents and purposes, no functional change intended.
Link: https://patch.msgid.link/20260630225619.511632-5-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
When initializing a vCPU's Hyper-V object, ensure the object is fully
initialized prior to exposing it through the vCPU, and ensure accesses from
other tasks (e.g. other vCPUs) see the fully initialized object if
vcpu->arch.hyperv is non-NULL.
Lack of ordering manifests as a lockdep splat due to attempting to lock a
TLB flush FIFO before the spinlock is initialized.
INFO: trying to register non-static key.
The code is fine but needs lockdep annotation, or maybe
you didn't initialize this object before use?
turning off the locking correctness validator.
CPU: 1 PID: 5005 Comm: syz-executor189 Not tainted 6.6.120-smp-DEV #1
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 03/18/2026
Call Trace:
<TASK>
[<ffffffff810dd10c>] dump_stack_lvl+0xcc/0x130 lib/dump_stack.c:106
[<ffffffff8192bddd>] assign_lock_key+0x1fd/0x230 kernel/locking/lockdep.c:977
[<ffffffff8191cb97>] register_lock_class+0x187/0x7a0 kernel/locking/lockdep.c:1291
[<ffffffff8191e7a9>] __lock_acquire+0x179/0x7650 kernel/locking/lockdep.c:5016
[<ffffffff8191e28f>] lock_acquire+0x13f/0x3d0 kernel/locking/lockdep.c:5756
[<ffffffff8101a65b>] __raw_spin_lock include/linux/spinlock_api_smp.h:133 [inline]
[<ffffffff8101a65b>] _raw_spin_lock+0x2b/0x40 kernel/locking/spinlock.c:154
[<ffffffff81319d44>] spin_lock include/linux/spinlock.h:351 [inline]
[<ffffffff81319d44>] hv_tlb_flush_enqueue+0xb4/0x270 arch/x86/kvm/hyperv.c:1946
[<ffffffff813160c6>] kvm_hv_flush_tlb+0xa96/0x1dc0 arch/x86/kvm/hyperv.c:2145
[<ffffffff8131438b>] kvm_hv_hypercall+0x103b/0x1fe0 arch/x86/kvm/hyperv.c:-1
[<ffffffff8133bff3>] __vmx_handle_exit arch/x86/kvm/vmx/vmx.c:6624 [inline]
[<ffffffff8133bff3>] vmx_handle_exit+0x12e3/0x21f0 arch/x86/kvm/vmx/vmx.c:6641
[<ffffffff81215d11>] vcpu_enter_guest arch/x86/kvm/x86.c:11649 [inline]
[<ffffffff81215d11>] vcpu_run+0x4d01/0x79c0 arch/x86/kvm/x86.c:11832
[<ffffffff8120fe39>] kvm_arch_vcpu_ioctl_run+0xb49/0x1c80 arch/x86/kvm/x86.c:12179
[<ffffffff8119cd60>] kvm_vcpu_ioctl+0xc80/0xff0 virt/kvm/kvm_main.c:6029
[<ffffffff8226fefd>] vfs_ioctl fs/ioctl.c:52 [inline]
[<ffffffff8226fefd>] __do_sys_ioctl fs/ioctl.c:872 [inline]
[<ffffffff8226fefd>] __se_sys_ioctl+0xfd/0x170 fs/ioctl.c:858
[<ffffffff85ac97d9>] do_syscall_x64 arch/x86/entry/common.c:52 [inline]
[<ffffffff85ac97d9>] do_syscall_64+0x69/0xb0 arch/x86/entry/common.c:93
[<ffffffff85c000d0>] entry_SYSCALL_64_after_hwframe+0x68/0xd2
</TASK>
Use the "safe" variant in all paths that are known to access the Hyper-V
object, as detected by an upcoming lockdep assertion, with an assist or two
from Sashiko.
Link: https://lore.kernel.org/all/20260612232258.0D9131F000E9@smtp.kernel.org
Fixes: 0823570f01 ("KVM: x86: hyper-v: Introduce TLB flush fifo")
Fixes: fc08b628d7 ("KVM: x86: hyper-v: Allocate Hyper-V context lazily")
Reported-by: syzbot+5b32c49cd8f005e65654@syzkaller.appspotmail.com
Reported-by: syzbot+5d2b94b77112148d1744@syzkaller.appspotmail.com
Closes: https://lore.kernel.org/all/6a396a66.52ae72c2.136ac7.0002.GAE@google.com
Tested-by: syzbot+5d2b94b77112148d1744@syzkaller.appspotmail.com
Link: https://patch.msgid.link/20260630225619.511632-4-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
Check for a NULL Hyper-V object in kvm_hv_get_tlb_flush_fifo() instead of
relying on the caller to do so. This will allow fixing a cross-vCPU race
where KVM can access a vCPU's FIFO before it's fully initialized, without
having to jump through too many cognitive hoops to reason about the
correctness of the logic.
Ignoring changes in ordering that only affect the aforementioned race, no
functional change intended.
Reviewed-by: Philippe Mathieu-Daudé <philmd@oss.qualcomm.com>
Link: https://patch.msgid.link/20260630225619.511632-3-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
When handling Hyper-V PV TLB flushes, retrieve the to-be-used FIFO in
hv_tlb_flush_enqueue() instead of having the caller pass in the FIFO. This
will make it easier to fix a cross-vCPU race where KVM can access a vCPU's
FIFO before it's fully initialized.
No functional change intended.
Link: https://patch.msgid.link/20260630225619.511632-2-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
Micro-optimize emulation of MOV DR instructions by checking CR4.DE if and
only if DR4 or DR5 is being accessed.
No functional change intended.
Reviewed-by: Jim Mattson <jmattson@google.com>
Link: https://patch.msgid.link/20260612230113.684301-9-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
WARN if ->set_dr() => kvm_set_dr() fails when emulating a MOV DR write,
as the emulator _must_ pre-check for #GPs in order to get the event
priority right when emulating MOV DR for L2 on SVM (all exceptions have
higher priority than the instruction intercept).
Opportunistically update the comment as the blurb about "#UD" being
checked is incomplete and misleading.
Reviewed-by: Jim Mattson <jmattson@google.com>
Link: https://patch.msgid.link/20260612230113.684301-8-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
Use kvm_dr{6,7}_valid() to validate the incoming DR{4,5,6,7} value in the
emulator instead of open coding an equivalent check. In the unlikely event
that the behavior of DR6/7 (and their aliases) changes in the future, using
common helpers will hopefully make it less likely the emulator logic will
be overlooked.
No functional change intended.
Reviewed-by: Jim Mattson <jmattson@google.com>
Link: https://patch.msgid.link/20260612230113.684301-7-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
When emulating a MOV DR on Intel with DR7.GD=1 at CPL>0, prioritize the #DB
due to DR7.GD over the #GP due to CPL>0, as empirical testing shows that
Intel CPUs (Skylake, Icelake and Emerald Rapids) prioritize the DR7.GD #DB
over all #GPs, whereas AMD CPUs prioritize the CPL>0 #GP (but not illegal
value #GPs) over the #DB.
Outside of the emulator, don't bother trying to provide the "correct"
priority based on the virtual CPU model, as it's simply impossible to do
so without intercepting *all* MOV DR accesses, which would result in a
massive, unacceptable performance hit. Note, getting the priority right
when advertising Intel on AMD would also require intercepting #GP, as SVM
prioritizes all exceptions over the instruction intercept.
Note, neither Intel's SDM nor AMD's APM says anything about the relative
priority, hence the empirical testing. Arguably Intel's description of
DR7.GD:
causes a debug exception to be generated prior to any MOV instruction
that accesses a debug register.
implies that DR7.GD has higher priority. But that's a fairly weak argument
as the statement would still hold true if the #GP due to CPL>0 had higher
priority, as the #GP would prevent any access to a DR.
Fixes: 3b88e41a41 ("KVM: SVM: Add intercept check for accessing dr registers")
Link: https://patch.msgid.link/20260612230113.684301-6-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
Manually handle the CPL check for MOV DR instructions instead of using the
Priv flag, *after* checking for #UD scenarios, as #GP due to CPL>0 has
lower priority than all #UDs.
Fixes: 1e470be5a1 ("KVM: x86 emulator: fix mov dr to inject #UD when needed.")
Reviewed-by: Jim Mattson <jmattson@google.com>
Link: https://patch.msgid.link/20260612230113.684301-5-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
Manually (pre)check the values being written to DR4/5, i.e. the DR6/DR7
aliases, instead of relying on ->set_dr() => kvm_set_dr() to signal a #GP.
SVM unfortunately prioritizes all exceptions over an instruction intercept,
i.e. nSVM is relying on the emulator to perform *all* exception checks
prior to attempting to execute the instruction.
Fixes: 3b88e41a41 ("KVM: SVM: Add intercept check for accessing dr registers")
Reviewed-by: Jim Mattson <jmattson@google.com>
Link: https://patch.msgid.link/20260612230113.684301-4-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
When emulating a MOV DR, specifically a write to DR6 or DR7, treat a #DB
due to DR7.GD (General Detect) as higher priority than a #GP due to an
illegal value. While neither Intel's SDM nor AMD's APM says anything
about the relative priority, empirical testing on Intel and AMD shows that
the #DB has higher priority. And for VMX, where the instruction intercept
has priority over *all* exceptions, KVM already treats the #DB as having
higher priority.
Cc: Maciej W. Rozycki <macro@orcam.me.uk>
Fixes: 3b88e41a41 ("KVM: SVM: Add intercept check for accessing dr registers")
Reviewed-by: Jim Mattson <jmattson@google.com>
Link: https://patch.msgid.link/20260612230113.684301-3-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
When emulating a MOV to a debug register, em_dr_write() calls
@ctxt->ops->set_dr(), which is forwarded to emulator_set_dr() and
then kvm_set_dr(). The latter checks that the written value is valid,
otherwise returning an error, in which case the emulator is supposed to
inject a #GP fault into the guest.
Commit 996ff5429e ("KVM: x86: move kvm_inject_gp up from kvm_set_dr
to callers") changed the contract of kvm_set_dr() (and thus
emulator_set_dr()), returning 1 as an error instead of -1, but the
caller in em_dr_write() was never updated, checking only if the returned
value is negative. The end result is that em_dr_write() does not detect
the error, so an invalid write does not generate a #GP, but at the same
time the register value is not updated.
The practical impact is limited, as check_dr_write() already checks DR6
and DR7 manually. However, it misses DR4/DR5, which alias DR6/DR7 when
CR4.DE=0.
Fix the bug by treating any non-zero return from set_dr() as a reason to
inject #GP.
Note, the manual checks on DR6 and DR7 are flawed, as they incorrectly
prioritize the #GP over a DR7.GD=1 #DB (the General Detect #DB has
priority on both Intel and AMD).
Note #2, relying on ->set_dr() to detect #GP is also flawed as all
exceptions have higher priority than the instruction intercept on SVM,
i.e. the manual checks need to be extended to DR4 and DR5 (after the
priority bug is fixed).
Fixes: 996ff5429e ("KVM: x86: move kvm_inject_gp up from kvm_set_dr to callers")
Signed-off-by: Carlos López <clopez@suse.de>
Link: https://patch.msgid.link/20260601133320.91479-2-clopez@suse.de
[sean: drop explicit "!= 0", massage changelog]
Reviewed-by: Jim Mattson <jmattson@google.com>
Link: https://patch.msgid.link/20260612230113.684301-2-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
Pass the CPUID index into cpuid_func_emulated() and return no emulated
features for indexed CPUID leaves with a non-zero index.
KVM currently emulates CPUID features only for index 0, but
kvm_vcpu_after_set_cpuid() looks up emulated features by function alone.
As a result, reverse_cpuid[] entries that share a function but use a
non-zero index, e.g. CPUID.7.1:ECX, can inherit emulated features that
belong to index 0. For example, RDPID, which is CPUID.7.0:ECX[22], can
be incorrectly OR'd into CPUID.7.1:ECX.
This is benign today because the affected bits do not correspond to
features KVM cares about, but it can become a real bug as new CPUID
features are defined. Make the helper index-aware so emulated features
are applied only to the CPUID entry they actually describe.
Fixes: e592ec657d ("KVM: x86: Initialize guest cpu_caps based on KVM support")
Suggested-by: Sean Christopherson <seanjc@google.com>
Signed-off-by: Binbin Wu <binbin.wu@linux.intel.com>
Reviewed-by: Xiaoyao Li <xiaoyao.li@intel.com>
Link: https://patch.msgid.link/20260609075748.612704-1-binbin.wu@linux.intel.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
Return EINVAL instead of EOPNOTSUPP if userspace attempts to pass a NULL
pointer for the source page of INIT_MEM_REGION, so that KVM's ABI is
consistent between TDX and SNP (for LAUNCH_UPDATE). EOPNOTSUPP was chosen
to be a forward-looking error code for when guest_memfd supports in-place
conversion, but even when in-place conversion comes along, it's an awkward
error code as KVM is deliberately choosing to disallow virtual address '0',
which is technically a legal userspace address. I.e. it's not so much a
lack of support as it is that KVM reserves address '0' to simplify KVM's
internal implementation.
Opportunistically move the check so that it's co-located with the other
checks on the userspace address, and so that it's more obvious that a NULL
source address is explicitly disallowed.
Fixes: 2a62345b30 ("KVM: guest_memfd: GUP source pages prior to populating guest memory")
Cc: Yan Zhao <yan.y.zhao@intel.com>
Cc: Ackerley Tng <ackerleytng@google.com>
Reviewed-by: Xiaoyao Li <xiaoyao.li@intel.com>
Acked-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
Reviewed-by: Binbin Wu <binbin.wu@linxu.intel.com>
Reviewed-by: Yan Zhao <yan.y.zhao@intel.com>
Tested-by: Yan Zhao <yan.y.zhao@intel.com>
Reviewed-by: Ackerley Tng <ackerleytng@google.com>
Link: https://patch.msgid.link/20260630213711.479692-3-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
Explicitly reject a NULL userspace virtual address for the source page of
SNP_LAUNCH_UPDATE instead of relying on the post-populate callback to do
the check, and don't WARN on failure, as the scenario is blatantly user-
triggerable, as reported by Sashiko. Waiting until post-populate to check
the address "works", but makes it unnecessarily difficult to see that KVM's
ABI is to disallow a NULL source page for non-ZERO pages.
Note, several existing VMMs pass a valid userspace address for the ZERO
case, i.e. KVM can't *require* the userspace address to be NULL for ZERO
pages, at least not without breaking userspace.
Fixes: dee5a47cc7 ("KVM: SEV: Add KVM_SEV_SNP_LAUNCH_UPDATE command")
Reported-by: Sashiko Bot <sashiko-bot@kernel.org>
Closes: https://lore.kernel.org/all/20260611125849.9ED631F00893@smtp.kernel.org
Signed-off-by: Joerg Roedel <joerg.roedel@amd.com>
Co-developed-by: Sean Christopherson <seanjc@google.com>
Reviewed-by: Ackerley Tng <ackerleytng@google.com>
Link: https://patch.msgid.link/20260630213711.479692-2-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
When emulating INVVPID, KVM executes INVVPID on the physical CPU using
vpid02 (instead of the L1 assigned VPID), after doing some validations
on the operands. However, it is possible that the physical CPU KVM
executes INVVPID on is different from the CPU L2 is running on.
For example, in the following scenario:
- L2 runs on CPU #1 and exits to L1 (vmx->nested.vmcs02.cpu=1)
- L1 migrates to CPU #2 and executes INVVPID
- KVM executes INVVPID on CPU #2
- L1 migrates back to CPU #1 and runs L2 (vmx->nested.vmcs02.cpu=1)
The TLB entries on CPU #1 are never invalidated, because INVVPID was
executed on CPU #2, and vmcs02 never ran on a different pCPU (i.e.
vmx_vcpu_load_vmcs() will *not* request KVM_REQ_TLB_FLUSH).
Ensure that INVVPID is being executed on the same pCPU that L2 last ran
on, and if not, fallback to clearing last_vpid=0 to trigger a full VPID
flush on the next nested VM-Enter (as KVM will detect L1 using a
different VPID for L2). If L2 ends up running on a different pCPU, KVM
will flush the TLB anyway through vmx_vcpu_load_vmcs().
Cc: stable@vger.kernel.org
Signed-off-by: Yosry Ahmed <yosry@kernel.org>
Reviewed-by: Kai Huang <kai.huang@intel.com>
Link: https://patch.msgid.link/20260616214652.2157032-4-yosry@kernel.org
Signed-off-by: Sean Christopherson <seanjc@google.com>
Separate the INVVPID operand checks from the actual flushing of vpid02 so
the flushing can be adjusted to do the right thing when vmcs02 was last
loaded on a different pCPU, without having to duplicate the logic across
multiple case-statements.
Opportunistically let the VM-Fail paths poke out past 80 chars.
No functional change intended.
Cc: stable@vger.kernel.org
Signed-off-by: Yosry Ahmed <yosry@kernel.org>
Reviewed-by: Kai Huang <kai.huang@intel.com>
Link: https://patch.msgid.link/20260616214652.2157032-3-yosry@kernel.org
Signed-off-by: Sean Christopherson <seanjc@google.com>
Make sure vpid02 is always flushed on first use by setting last_vpid=0
when allocating vpid02. nested_vmx_transition_tlb_flush() will always
detect a VPID change on first VM-Enter after VMXON, because VPID=0 in
vmcs12 is not allowed if L1 enables VPID.
This avoids using stale TLB entries from a previous lifetime of the
VPID, that might have been associated with a different vCPU (or a
completely different VM).
Note that last_vpid is already being initialized as 0 when the vCPU is
created, but it is not reset when vpid02 is freed on VMXOFF. Hence, the
problem can only occur if L1 does VMXOFF -> VMXON, runs an L2, and KVM
happens to reuse a VPID that has TLB entries on the physical CPU.
Cc: stable@vger.kernel.org
Signed-off-by: Yosry Ahmed <yosry@kernel.org>
Reviewed-by: Kai Huang <kai.huang@intel.com>
Reviewed-by: Jim Mattson <jmattson@google.com>
Link: https://patch.msgid.link/20260616214652.2157032-2-yosry@kernel.org
Signed-off-by: Sean Christopherson <seanjc@google.com>
Add a dedicated field in "struct nested_vmx" to track L1's pre-VM-Enter CR3
instead of using vmcs01.GUEST_CR3, which isn't anywhere near as safe as the
comment purports it to be. E.g. in addition to the warn_on_missed_cc bug
(that was fixed by relocating the consistency check), if getting vmcs12
pages (during actual nested VM-Entry) fails and EPT is disabled (in KVM),
KVM will return control to userspace with vmcs01.GUEST_CR3 holding a guest-
controlled value.
Alternatively, KVM could force a reload of vmcs01.GUEST_CR3 by resetting
the MMU context in the error path, but as above, the safety of the vmcs01
approach is extremely questionable, e.g. it took all of ~4 months for the
code to break.
Fixes: 671ddc700f ("KVM: nVMX: Don't leak L1 MMIO regions to L2")
Cc: stable@vger.kernel.org
Cc: Jim Mattson <jmattson@google.com>
Link: https://patch.msgid.link/20260612145642.452392-3-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
Move the off-by-default consistency check for vmcs12.tpr_threshold vs.
the virtual APIC vTPR into the "normal" controls checks, as waiting until
KVM has loaded some amount of state is unnecessary and actively dangerous.
Specifically, failure to unwind vmcs01.GUEST_CR3 to KVM's value when EPT
is disabled results in KVM running L1 with an L1-controlled CR3, not with
KVM's CR3!
Alternatively, KVM could simply reset the MMU to force a reload of
vmcs01.GUEST_CR3, but the _only_ reason the check was shoved into a "late"
flow was to wait until the vmcs12 pages were retrieved. Rather than build
up more crusty code, simply access vTPR using a regular guest memory access
(performance isn't a concern). To circumvent the restrictions that led to
KVM deferring nested_get_vmcs12_pages(), (a) use a VM-scoped API to read
guest memory so that it always hits non-SMM memslots (for RSM), and (b)
skip the check (since its off-by-default anyways) when the vCPU doesn't
want to run, i.e. when userspace is restoring/stuffing state.
If reading guest memory fails, simply skip the consistency check, as KVM's
de facto ABI is that VMX instruction accesses to non-existent memory get
PCI Bus Error semantics, where reads return 0xFFs. And if vTPR=0xFF, then
the vTPR is guaranteed to be greater than or equal to TPR_THRESHOLD.
Fixes: 1100e4910a ("KVM: nVMX: Add an off-by-default module param to WARN on missed consistency checks")
Cc: stable@vger.kernel.org
Link: https://patch.msgid.link/20260612145642.452392-2-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
Nullify irqfd->producer if updating the IRTE for bypass fails, as leaving a
dangling pointer will result in a use-after-free if the irqfd is reachable
through KVM's routing, but the producer is freed separately. E.g. for VFIO
PCI, the producer is embedded in struct "vfio_pci_irq_ctx" and freed when
the vector is disabled, which can happen independent of routing updates.
Fixes: 77e1b8332d ("KVM: x86: Decouple device assignment from IRQ bypass")
Cc: stable@vger.kernel.org
Signed-off-by: leixiang <leixiang@kylinos.cn>
Link: https://patch.msgid.link/1782119051448443.14545.seg@mailgw.kylinos.cn
[sean: drop PPC change, massage changelog]
Signed-off-by: Sean Christopherson <seanjc@google.com>
Branch Sampling (BRS) and Last Branch Record (LBR) are mutually
exclusive hardware features, and users of both are tracked via
cpuc->lbr_users.
When SVM is toggled on a CPU, the host perf events are reprogrammed to
update the HostOnly filter bit (set when virtualization is enabled,
cleared when it is disabled). On PerfMonV2-capable processors, this
reprogramming is performed by calling amd_pmu_enable_all() to rewrite
the event selectors. However, amd_pmu_enable_all() also calls
amd_brs_enable_all(), which enables BRS whenever cpuc->lbr_users > 0.
Having active LBR events satisfies this gating on processors that have
LBR but not BRS. The kernel then tries to set the BRS enable bit in
DebugExtnCfg (MSR 0xc000010f). Since that bit is deprecated on such
hardware, the write results in a #GP:
Call Trace:
<IRQ>
amd_pmu_enable_all+0x1d/0x90
amd_pmu_disable_virt+0x62/0xb0
kvm_arch_disable_virtualization_cpu+0xa/0x40 [kvm]
hardware_disable_nolock+0x1a/0x30 [kvm]
__flush_smp_call_function_queue+0x9b/0x410
__sysvec_call_function+0x18/0xc0
sysvec_call_function+0x69/0x90
</IRQ>
<TASK>
asm_sysvec_call_function+0x16/0x20
RIP: 0010:cpuidle_enter_state+0xc4/0x450
? cpuidle_enter_state+0xb7/0x450
cpuidle_enter+0x29/0x40
cpuidle_idle_call+0xf5/0x160
do_idle+0x7b/0xe0
cpu_startup_entry+0x26/0x30
start_secondary+0x115/0x140
secondary_startup_64_no_verify+0x194/0x19b
</TASK>
Fix this by ensuring that BRS is not enabled from the event selector
reprogramming path even when cpuc->lbr_users > 0.
Fixes: bae19fdd7e ("perf/x86/amd/core: Fix reloading events for SVM")
Signed-off-by: Sandipan Das <sandipan.das@amd.com>
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Link: https://patch.msgid.link/702fa204d574b03d14e3664c7d4b201db048bbfd.1783506528.git.sandipan.das@amd.com
If CONFIG_COMPAT_32BIT_TIME is disabled then the vDSO should not
provide any 32-bit time related functionality. This is the intended
effect of the kconfig option and also the fallback system calls would
also not be implemented.
Signed-off-by: Thomas Weißschuh <thomas.weissschuh@linutronix.de>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Reviewed-by: Arnd Bergmann <arnd@arndb.de>
Link: https://patch.msgid.link/20260702-vdso-compat_32bit_time-v3-3-db9f36d8d432@linutronix.de
The usage of 'bool' requires linux/types.h, which is currently only
included through a transitive dependency chain.
Include linux/types.h as that chain is going to go away.
Signed-off-by: Thomas Weißschuh (Schneider Electric) <thomas.weissschuh@linutronix.de>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Link: https://patch.msgid.link/20260702-hrtimer-header-dependencies-v1-7-c50b19bda473@linutronix.de
Revert
99cf1fb58e ("x86/virt/sev: Drop WBINVD before setting MSR_AMD64_SYSCFG_SNP_EN").
Section 8.8 of the SNP spec says:
Before invoking SNP_INIT_EX with INIT_RMP set to 1, software must ensure
that no CPUs contain dirty cache lines for the memory containing the RMP.
Cachelines can be moved from cache to cache in a dirty state. The
wbinvd_on_all_cpus() before SNP_INIT_EX flushes the caches for each CPU, but
if the IPIs for WBINVD race with this dirty cacheline movement, it is possible
that they may not get flushed, violating the firmware requirement.
Doing wbinvd_on_all_cpus() before setting SNPEn is safer since the RMP
table is not yet in use.
[ Heroically bisected by Srikanth. ]
[ bp: Massage commit message. ]
Fixes: 99cf1fb58e ("x86/virt/sev: Drop WBINVD before setting MSR_AMD64_SYSCFG_SNP_EN")
Reported-by: Srikanth Aithal <Srikanth.Aithal@amd.com>
Signed-off-by: Tycho Andersen (AMD) <tycho@kernel.org>
Signed-off-by: Borislav Petkov (AMD) <bp@alien8.de>
Tested-by: Srikanth Aithal <Srikanth.Aithal@amd.com>
Reviewed-by: Tom Lendacky <thomas.lendacky@amd.com>
Cc: <stable@kernel.org>
Link: https://patch.msgid.link/20260707150033.2364758-1-tycho@kernel.org
Some multi GPU systems may have a VGA compatible device, but that might not be
used for display. If due to enumeration order this device is found before the
one actually used for display then multiple devices may show the boot_display
attribute, confusing userspace.
When screen info is valid, use it exclusively to find the primary device so
that only the device backing the framebuffer is reported. Only when no
framebuffer has been set up does it make sense to fall back to the default VGA
device. This ensures at most one primary graphics device, preferably the one
with the framebuffer.
Fixes: ad90860bd1 ("fbcon: Use screen info to find primary device")
Closes: https://lore.kernel.org/linux-pci/20260618081803.2790848-1-aaron.ma@canonical.com/#t
Reported-by: Aaron Ma <aaron.ma@canonical.com>
Suggested-by: Thomas Zimmermann <tzimmermann@suse.de>
Signed-off-by: Mario Limonciello <mario.limonciello@amd.com>
Signed-off-by: Borislav Petkov (AMD) <bp@alien8.de>
Reviewed-by: Thomas Zimmermann <tzimmermann@suse.de>
Tested-by: Aaron Ma <aaron.ma@canonical.com>
Cc: <stable@kernel.org>
Link: https://patch.msgid.link/20260623141505.1816786-1-mario.limonciello@amd.com
x86 asm/setup.h includes linux/kexec_handover.h. This is because it is
used by setup.c and kaslr.c. But this inclusion is problematic. The
header is included in many places, so it results in the KHO header being
propagated there. Also, the setup header is used by realmode code. If
KHO header includes things like mm.h, it causes a big dump of
compilation failures.
Nothing in setup.h uses anything from KHO. Remove the header from
setup.h, and directly include it in setup.c. which does use things from
KHO. Since kaslr.c is a part of the decompressor, avoid including linux
headers there directly. Instead, split out struct kho_scratch, which is
the only thing the kaslr.c uses, and move it to
include/asm-generic/kexec_handover.h.
This should also help reduce files recompiled when kexec_handover.h
changes.
Signed-off-by: Pratyush Yadav (Google) <pratyush@kernel.org>
Acked-by: Borislav Petkov (AMD) <bp@alien8.de>
Link: https://patch.msgid.link/20260706153751.1166003-1-pratyush@kernel.org
Signed-off-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
cpu_feature_enabled() is the one to use to test feature flags so hide
the static thing which doesn't pay attention to disabled mask bits
anyway.
Use the following command to do the replacement:
$ git grep --files-with-matches -w static_cpu_has -- ':(exclude)*cpufeature.h' \
| xargs sed -i 's/static_cpu_has(/cpu_feature_enabled\(/g'
There should be no functional changes resulting from this.
Signed-off-by: Borislav Petkov (AMD) <bp@alien8.de>
Reviewed-by: Nikolay Borisov <nik.borisov@suse.com>
Link: https://patch.msgid.link/20260620015041.336288-1-bp@kernel.org
CPUs when Intel SNC (Sub-NUMA Clustering) is enabled
(Reinette Chatre)
Signed-off-by: Ingo Molnar <mingo@kernel.org>
-----BEGIN PGP SIGNATURE-----
iQJFBAABCgAvFiEEBpT5eoXrXCwVQwEKEnMQ0APhK1gFAmpKEsURHG1pbmdvQGtl
cm5lbC5vcmcACgkQEnMQ0APhK1gcWhAApp+w5vUx0FXlou1Mzvn63bXey9/CMXyX
rrUeWA+ic7PpNvImuwixdyPyS1CnF2l7lMcmOtiNE4wnRvCf+0rsrwgYy4m4qd/2
aLAftFDp9WhKKH3Bb6gnVfMXiXmYb4eCQO0pRhJY26QtVJ2RGkbenW/TdTnAAAwS
KO8bJGsllufiF0+k4G5YMgiVcgZFByn/nuqmzvIc+oKQt5LSMvkKijoM7ozX9q5g
c1ABWvag3KWU2gFI00GfvzuQ7n7ckGhBVRwtO9DMox68liOlKOHedEFF0A0+KiDt
fk6b2LFbqm6hB7TpWRQqQ+rPusbDdAVxKG8ehIkjR8BD49VwzZUcr5kiWS5/xplm
a9F7cJpAJiuSiTi2HYtQVOlrCujonmjt11qqj3cCCjkton+IC9twhDkrj6+Kxygq
W3oc8Gia2JWoFBNt/6l0iKRJjluqgPJ8crPHpnDr4C5KEnsEIqCQsSPsVjoYM5mY
+xwS9jMJM7Dhrw3OZ6b5D8jY9oonBM/BB+jMomQ9ncDijUwKybgqp0JXB/3ydIw3
ci69tewFdVD+9EyNAz+dqANuhNyF1JTFza/5QxAU88E1wp2O2WjahSPvkNU/ekNK
jDGLf8VyiI08bH2pyYIhEhcvmTZ5Ocyuj9zA4Ufi2V3gaDMkvMXRipxm50dAhaWG
CMoZ+wTnI/8=
=mae/
-----END PGP SIGNATURE-----
Merge tag 'x86-urgent-2026-07-05' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
Pull x86 fix from Ingo Molnar:
- Prevent OOB access in the resctrl code while offlining
CPUs when Intel SNC (Sub-NUMA Clustering) is enabled
(Reinette Chatre)
* tag 'x86-urgent-2026-07-05' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip:
x86,fs/resctrl: Prevent out-of-bounds access while offlining CPU when SNC enabled
- Fix a perf_event_attr::remove_on_exec bug for group
events (Taeyang Lee)
- Fix uprobes CALL emulation interaction with shadow stacks,
and add a testcase for this (David Windsor)
- Fix uprobes unregister bug (Jiri Olsa)
Signed-off-by: Ingo Molnar <mingo@kernel.org>
-----BEGIN PGP SIGNATURE-----
iQJFBAABCgAvFiEEBpT5eoXrXCwVQwEKEnMQ0APhK1gFAmpKCX8RHG1pbmdvQGtl
cm5lbC5vcmcACgkQEnMQ0APhK1hwJRAAne1HniwuV9PV2GQHNKipeKV320tod2iR
WY9yy9ez5WJbPttS7Fy28NobQKxuTanNWnAJqqM3TJF0tPPSrrYqkhAGrpEb2ab0
G7gl4KlrO//5kKTZnnI86t/quSV5BDt00UUdMBFp+hbNYXNul5/AeUxqMhNoGqB6
DpJHrV+7kNQZ4I7tjatYWL11hZHpEhrx4QWLUsnd+nDwddpmdsRNXVZwpGh9+Dh4
+XXcgLD7M8FqNFN3GCVfhJKO8x8HRaBWv3FHeGqbCUL9k5viWcn+N91IbrcPVat1
HBk0JDtpLK286WHLFy7uROafCA59AlYp5DX7mobXi0VF1FdMqPtjaXEsZM7Ng/P5
/tpbV5P4irrUnMSCEDSDzqZJWXbcBSqCJ9p6z5/Tjzo3VegHyrXe29wjVoqvXjLx
og/9OnPZv/2QEsE37rBRwNC889ihFMUDZh3T+uUc1YKUEwYWFXwtUECTg+0Oi+lL
mLTdK05j/6NmVkcY77mFjQfTMFAZeD78g6cPY3yDHiHRFtxPOIhknCKBeLD5BjXD
tz08x3MN0ItVEBXfuKAlkb1KPhHy8x02IrpdfVojoQ7lUBRH/JqLvnSGgvVsRM+0
v9D3N9BAuB3s8DmxEdh0wyRzISl6jzBHJDJ83+uWSfZWLlEW+RVn9NbH2ubNU/jt
FO3U81r1MvE=
=Mb4z
-----END PGP SIGNATURE-----
Merge tag 'perf-urgent-2026-07-05' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
Pull perf events fixes from Ingo Molnar:
- Fix a perf_event_attr::remove_on_exec bug for group events
(Taeyang Lee)
- Fix uprobes CALL emulation interaction with shadow stacks, and
add a testcase for this (David Windsor)
- Fix uprobes unregister bug (Jiri Olsa)
* tag 'perf-urgent-2026-07-05' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip:
uprobes/x86: Use proper mm_struct in __in_uprobe_trampoline
selftests/x86: Add shadow stack uprobe CALL test
x86/uprobes: Keep shadow stack in sync for emulated CALLs
perf/core: Detach event groups during remove_on_exec
The amd_uncore driver currently does not validate event groups and
allows creation of groups with more events than the number of available
hardware counters. Because of this, pmu->event_init() succeeds but
counter assignment fails later in pmu->add() which returns -EBUSY once
all counters are exhausted.
Address this by introducing group validation in the pmu->event_init()
path. Since the uncore PMUs have no per-event constraints and all
counters of a PMU are interchangeable, validation is reduced to just
counting the group members that target a PMU and ensuring that they fit
within the available set of counters.
Fixes: c43ca5091a ("perf/x86/amd: Add support for AMD NB and L2I "uncore" counters")
Signed-off-by: Sandipan Das <sandipan.das@amd.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Link: https://patch.msgid.link/750877d66e208603c3047f13eed6399625d43969.1782884387.git.sandipan.das@amd.com
Fix all kernel-doc warnings in imr.c:
imr.c:300: warning: Function parameter or struct member 'rmask' not described in 'imr_add_range'
imr.c:300: warning: Function parameter or struct member 'wmask' not described in 'imr_add_range'
imr.c:300: warning: Excess function parameter 'read_mask' description in 'imr_add_range'
imr.c:300: warning: Excess function parameter 'write_mask' description in 'imr_add_range'
Fixes: 28a375df16 ("x86/intel/quark: Add Isolated Memory Regions for Quark X1000")
Signed-off-by: Randy Dunlap <rdunlap@infradead.org>
Signed-off-by: Borislav Petkov (AMD) <bp@alien8.de>
Link: https://patch.msgid.link/20250620065639.3348609-1-rdunlap@infradead.org
-----BEGIN PGP SIGNATURE-----
iJEEABYKADkWIQRTLbB6QfY48x44uB6AXGG7T9hjvgUCakeC1xsUgAAAAAAEAA5t
YW51MiwyLjUrMS4xMiwyLDIACgkQgFxhu0/YY74ZBQD/chB7mZrRK5FWdhtEYZuT
5is9FKdEe7P3rRChGuDxGZoBAISClrfxdh46C90D8jhjE3G3xWsN5DIajo1YaDbH
TroI
=wiih
-----END PGP SIGNATURE-----
Merge tag 'for-linus-7.2a-rc2-tag' of git://git.kernel.org/pub/scm/linux/kernel/git/xen/tip
Pull xen fixes from Juergen Gross:
- rename function parameters and a comment related to
xen_exchange_memory() (Jan Beulich)
- replace __ASSEMBLY__ with __ASSEMBLER__ (Thomas Huth)
- add some sanity checking to the Xen pvcalls frontend driver (Michael
Bommarito)
- fix error handling in the Xen gntdev driver (Wentao Liang)
- fix several minor bugs in Xen related drivers (Yousef Alhouseen)
* tag 'for-linus-7.2a-rc2-tag' of git://git.kernel.org/pub/scm/linux/kernel/git/xen/tip:
x86/Xen: correct commentary and parameter naming of xen_exchange_memory()
xenbus: reject unterminated directory replies
xen/gntalloc: validate grant count before allocation
xen/gntalloc: make grant counters unsigned
xen/front-pgdir-shbuf: free grant reference head on errors
xen/gntdev: fix error handling in ioctl
xen: Replace __ASSEMBLY__ with __ASSEMBLER__ in header files
xen/pvcalls: bound backend response req_id before indexing rsp[]
Andrii reported an issue with optimized uprobes [1] that can clobber
redzone area with call instruction storing return address on stack
where user code may keep temporary data without adjusting rsp.
Fixing this by moving the optimized uprobes on top of 10-bytes nop
instruction, so we can squeeze another instruction to escape the
redzone area before doing the call, like:
lea -0x80(%rsp), %rsp
call tramp
Note the lea instruction is used to adjust the rsp register without
changing the flags.
We use nop10 and following transformation to optimized instructions
above and back as suggested by Peterz [2].
Optimize path (int3_update_optimize):
1) Initial state after set_swbp() installed the uprobe:
cc 2e 0f 1f 84 00 00 00 00 00
From offset 0 this is INT3 followed by the tail of the original
10-byte NOP.
After a previous unoptimization bytes 5..9 may still contain the
old call instruction, which remains valid for threads already there.
2) Rewrite the LEA tail and call displacement:
cc [8d 64 24 80 e8 d0 d1 d2 d3]
From offset 0 this traps on the uprobe INT3. Bytes 1..9 are not
executable entry points while byte 0 is trapped.
3) Publish the first LEA byte:
[48] 8d 64 24 80 e8 d0 d1 d2 d3
From offset 0 this is:
lea -0x80(%rsp), %rsp
call <uprobe-trampoline>
Unoptimize path (int3_update_unoptimize):
1) Initial optimized state:
48 8d 64 24 80 e8 d0 d1 d2 d3
Same as 3) above.
2) Trap new entries before restoring the NOP bytes:
[cc] 8d 64 24 80 e8 d0 d1 d2 d3
From offset 0 this traps. A thread that had already executed the
LEA can still reach the intact CALL at offset 5.
3) Restore bytes 1..4 of the original NOP while keeping byte 0 trapped
and byte 5 as CALL.
cc [2e 0f 1f 84] e8 d0 d1 d2 d3
From offset 0 this still traps. Offset 5 is still the CALL for any
thread that was already past the first LEA byte.
4) Publish the first byte of the original NOP:
[66] 2e 0f 1f 84 e8 d0 d1 d2 d3
From offset 0 this is the restored 10-byte NOP; the CALL opcode and
displacement are now only NOP operands. Offset 5 still decodes as
CALL for a thread that was already there.
Tthere is only a single target uprobe-trampoline for the given nop10
instruction address, so the CALL instruction will not be changed across
unoptimization/optimization cycles.
Therefore, any task that is preempted at the CALL instruction is guaranteed
to observe that CALL and not anything else.
Note as explained in [2] we need to use following nop10:
PF1 PF2 ESC NOPL MOD SIB DISP32
NOP10: 0x66, 0x2e, 0x0f, 0x1f, 0x84, 0x00, 0x00, 0x00, 0x00, 0x00 -- cs nopw 0x00000000(%rax,%rax,1)
which means we need to allow 0x2e prefix which maps to INAT_PFX_CS
attribute in is_prefix_bad function.
Also changing the uprobe syscall error when called out of uprobe
trampoline to -EPROTO, so we are able to detect the fixed kernel.
The optimized uprobe performance stays the same:
uprobe-nop : 3.129 ± 0.013M/s
uprobe-push : 3.045 ± 0.006M/s
uprobe-ret : 1.095 ± 0.004M/s
--> uprobe-nop10 : 7.170 ± 0.020M/s
uretprobe-nop : 2.143 ± 0.021M/s
uretprobe-push : 2.090 ± 0.000M/s
uretprobe-ret : 0.942 ± 0.000M/s
--> uretprobe-nop10: 3.381 ± 0.003M/s
usdt-nop : 3.245 ± 0.004M/s
--> usdt-nop10 : 7.256 ± 0.023M/s
[1] https://lore.kernel.org/bpf/20260509003146.976844-1-andrii@kernel.org/
[2] https://lore.kernel.org/bpf/20260518104306.GU3102624@noisy.programming.kicks-ass.net/#t
Closes: https://lore.kernel.org/bpf/20260509003146.976844-1-andrii@kernel.org/
Fixes: ba2bfc97b4 ("uprobes/x86: Add support to optimize uprobes")
Reported-by: Andrii Nakryiko <andrii@kernel.org>
Assisted-by: Codex:GPT-5.5
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Oleg Nesterov <oleg@redhat.com>
Link: https://patch.msgid.link/20260703114917.238144-6-jolsa@kernel.org
When we do fork or clone without CLONE_VM the new process won't
have uprobe trampoline vma objects and at the same time it will
have optimized code calling that trampoline and crash.
Fixing this by allowing vma uprobe trampoline objects to be copied
on fork to the new process.
Fixes: ba2bfc97b4 ("uprobes/x86: Add support to optimize uprobes")
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Oleg Nesterov <oleg@redhat.com>
Link: https://patch.msgid.link/20260703114917.238144-5-jolsa@kernel.org
In case the optimization fails, we leak new-ly created trampoline
vma mapping (in case we just created it), let's unmap it.
Fixes: ba2bfc97b4 ("uprobes/x86: Add support to optimize uprobes")
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Oleg Nesterov <oleg@redhat.com>
Link: https://patch.msgid.link/20260703114917.238144-4-jolsa@kernel.org
Removing struct uprobe_trampoline object and it's tracking code,
because it's not needed. We can do same thing directly on top of
struct vm_area_struct objects.
This makes the code simpler and allows easy propagation of the
trampoline vma object into child process in following change.
Note the original code called destroy_uprobe_trampoline if the
optimiation failed, but it only freed the struct uprobe_trampoline
object, not the vma. The new vma leak is fixed in following change.
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Oleg Nesterov <oleg@redhat.com>
Acked-by: Andrii Nakryiko <andrii@kernel.org>
Link: https://patch.msgid.link/20260703114917.238144-3-jolsa@kernel.org
The 32-bit MSR interface rdmsr() is planned to be removed. Use the
related 64-bit variant instead.
Signed-off-by: Juergen Gross <jgross@suse.com>
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Link: https://patch.msgid.link/20260629060526.3638272-29-jgross@suse.com
The 32-bit MSR interface rdmsr() is planned to be removed. Use the
related 64-bit variant instead.
This conversion also simplifies the code a bit.
Signed-off-by: Juergen Gross <jgross@suse.com>
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Cc: K. Y. Srinivasan <kys@microsoft.com>
Cc: Dexuan Cui <decui@microsoft.com>
Cc: Haiyang Zhang <haiyangz@microsoft.com>
Cc: Long Li <longli@microsoft.com>
Cc: Wei Liu <wei.liu@kernel.org>
Link: https://patch.msgid.link/20260629060526.3638272-28-jgross@suse.com
numa_add_memblk() now sets the node in numa_nodes_parsed itself, so the
caller's own node_set() is redundant. Remove it.
No functional change.
Signed-off-by: Sang-Heon Jeon <ekffu200098@gmail.com>
Link: https://patch.msgid.link/20260703041329.2797584-5-ekffu200098@gmail.com
Signed-off-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
<linux/mod_devicetable.h> is included transitively in nearly every
driver in an x86_64 allmodconfig build of v7.1:
$ find drivers -name \*.o -not -name \*.mod.o | wc -l
21330
$ find drivers -name \*.o.cmd -not -name \*.mod.o.cmd | xargs grep -l mod_devicetable.h | wc -l
17038
The result of this mixture of different and unrelated subsystem details
is that even when touching an obscure device id struct most of the
kernel needs to be recompiled. Given that each driver typically only
needs one or two of these structures, splitting into per subsystem
headers and only including what is really needed reduces the amount of
needed recompilation.
This split is implemented in the first commit and then after some
preparatory work in the following commits, the last two replace
includes of <linux/mod_devicetable.h> by the actually needed more
specific headers. There are still a few instances left, but the ones
with high impact (that is in headers that are used a lot) and the easy
ones (.c files) are handled. These remaining includes will be addressed
during the next merge window.
-----BEGIN PGP SIGNATURE-----
iQEzBAABCgAdFiEEP4GsaTp6HlmJrf7Tj4D7WH0S/k4FAmpHSvIACgkQj4D7WH0S
/k6Rdgf/QqxMIt7DEAN6KvyCnwVnNnvEWQhac/DtYA3NF1JaynieJIoFcimrRr/j
cntxjz2zlMhLH4CYZ87IunZ7nGZgkMlsnl/OVMm5NY0wq0vVYX6VmpiSvhMAgvjV
QxKHOAyuVqPslwVE8DF2Vh9mgJILviH5t/yIy8DHF85QBrDSjcd/n6csx09uTjIy
+D6NuxQ7ZeOi+O2wZuiDhYpD1kwgO1sXAg56ogN1OSTQoo1kkKqj66b5fgbPVm4/
MX3EBOjKIytX8csfWBJjNSB7aRyWVMkMKmmDO9+7MIq6tPRbc3yYwhu2kZBAw5Pt
mQfy1oyUi+0jwew4yKHeoVLyWAc4bg==
=YhFg
-----END PGP SIGNATURE-----
Merge tag 'device-id-rework' of git://git.kernel.org/pub/scm/linux/kernel/git/ukleinek/linux
Pull mod_devicetable.h header split from Uwe Kleine-König:
"Split <linux/mod_devicetable.h> in per subsystem headers
<linux/mod_devicetable.h> is included transitively in nearly every
driver in an x86_64 allmodconfig build of v7.1:
$ find drivers -name \*.o -not -name \*.mod.o | wc -l
21330
$ find drivers -name \*.o.cmd -not -name \*.mod.o.cmd | xargs grep -l mod_devicetable.h | wc -l
17038
The result of this mixture of different and unrelated subsystem
details is that even when touching an obscure device id struct most of
the kernel needs to be recompiled. Given that each driver typically
only needs one or two of these structures, splitting into per
subsystem headers and only including what is really needed reduces the
amount of needed recompilation.
This split is implemented in the first commit and then after some
preparatory work in the following commits, the last two replace
includes of <linux/mod_devicetable.h> by the actually needed more
specific headers.
There are still a few instances left, but the ones with high impact
(that is in headers that are used a lot) and the easy ones (.c files)
are handled. These remaining includes will be addressed during the
next merge window"
* tag 'device-id-rework' of git://git.kernel.org/pub/scm/linux/kernel/git/ukleinek/linux:
Replace <linux/mod_devicetable.h> by more specific <linux/device-id/*.h> (c files)
Replace <linux/mod_devicetable.h> by more specific <linux/device-id/*.h> (headers)
parisc: #include <linux/compiler.h> for unlikely() in <asm/ptrace.h>
media: em28xx: Add include for struct usb_device_id
LoongArch: KVM: Add include defining struct cpu_feature
ALSA: hda/core: Add include defining struct hda_device_id
usb: dwc2: Add include defining struct pci_device_id
platform/x86: int3472: Add include defining struct dmi_system_id
platform/x86: x86-android-tablets: Add include defining struct dmi_system_id
i2c: Let i2c-core.h include <linux/i2c.h>
of: Explicitly include <linux/types.h> and <linux/err.h>
platform/x86: msi-ec: Ensure dmi_system_id is defined
usb: serial: Include <linux/usb.h> in <linux/usb/serial.h>
driver core: platform: Include header for struct platform_device_id
driver: core: Include headers for acpi_device_id and of_device_id for struct device_driver
media: ti: vpe: #include <linux/platform_device.h> explicitly
mod_devicetable.h: Split into per subsystem headers
Replace the #include of <linux/mod_devicetable.h> by the more specific
<linux/device-id/*.h> where applicable. For most cases the include
can be dropped completely, only a few drivers need one or two headers
added.
Acked-by: Danilo Krummrich <dakr@kernel.org>
Acked-by: Takashi Sakamoto <o-takashi@sakamocchi.jp>
Acked-by: Bjorn Helgaas <bhelgaas@google.com>
Link: https://patch.msgid.link/1a3f2007c5c5dcf555c09a4035ce3ae8ef1b6c49.1782808461.git.u.kleine-koenig@baylibre.com
Signed-off-by: Uwe Kleine-König (The Capable Hub) <u.kleine-koenig@baylibre.com>
<linux/mod_devicetable.h> is included in a many files:
$ git grep '<linux/mod_devicetable.h>' ef0c9f75a1 | wc -l
1598
; some of them are widely used headers. To stop mixing up different and
unrelated driver( type)s let the subsystem headers only use the subset
of the recently split <linux/mod_devicetable.h> that are relevant for
them.
The fallout (I hope) is addressed in the previous commits that handle
sources relying on e.g. <linux/i2c.h> pulling in the full legacy header
and thus providing pci_device_id.
Acked-by: Danilo Krummrich <dakr@kernel.org>
Acked-by: Takashi Sakamoto <o-takashi@sakamocchi.jp>
Link: https://patch.msgid.link/199fe46b624ba07fb9bd3e0cd6ff13757932cb5f.1782808461.git.u.kleine-koenig@baylibre.com
Signed-off-by: Uwe Kleine-König (The Capable Hub) <u.kleine-koenig@baylibre.com>
Call xbc_prepend_embedded_cmdline() in setup_arch() right after the
CONFIG_CMDLINE merge and before strscpy(command_line, ...) so the
build-time-rendered embedded bootconfig "kernel" subtree is part of
boot_command_line by the time parse_early_param() runs. early_param()
handlers (mem=, earlycon=, loglevel=, ...) now see values supplied via
CONFIG_BOOT_CONFIG_EMBED_FILE without parsing bootconfig at runtime.
Gate the prepend on the same opt-in the runtime parser uses: prepend
when "bootconfig" is present on the command line, or when
CONFIG_BOOT_CONFIG_FORCE is set. Detect it with parse_args(), exactly
as setup_boot_config() does, so both agree on what counts as opt-in:
any "bootconfig" key regardless of value (bare, =0, =1, ...), and only
before the "--" that separates init arguments. Sharing the parser keeps
the early and late paths from diverging -- e.g. "bootconfig=0" or a
"-- bootconfig" meant for init must not apply the embedded keys early
while the runtime parser skips them.
The prepend necessarily runs before setup_boot_config() detects an
initrd bootconfig, so an initrd cannot override the embedded "kernel"
keys for early_param(). This is intentional: the embedded cmdline acts
like a build-time CONFIG_CMDLINE. An initrd bootconfig's "kernel" keys
never reached early_param() anyway (they apply late via
extra_command_line), so nothing is lost -- the initrd keys still apply
late, with last-wins keeping the embedded values in effect.
Link: https://lore.kernel.org/all/20260626-bootconfig_using_tools-v7-7-24ab72139c29@debian.org/
Signed-off-by: Breno Leitao <leitao@debian.org>
Signed-off-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
In the unregister path we use __in_uprobe_trampoline check with
current->mm for the VMA lookup, which is wrong, because we are
in the tracer context, not the traced process.
Add mm_struct pointer argument to __in_uprobe_trampoline and
changing related callers to pass proper mm_struct pointer.
Fixes: ba2bfc97b4 ("uprobes/x86: Add support to optimize uprobes")
Reported-by: syzbot+61ce80689253f42e6d80@syzkaller.appspotmail.com
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Oleg Nesterov <oleg@redhat.com>
Acked-by: Andrii Nakryiko <andrii@kernel.org>
Tested-by: syzbot+61ce80689253f42e6d80@syzkaller.appspotmail.com
Link: https://patch.msgid.link/20260701111337.53943-2-jolsa@kernel.org
Uprobe CALL emulation updates the normal user stack, but not the CET user
shadow stack. The subsequent RET then sees a stale shadow stack entry and
raises #CP.
Update the relative CALL emulation and XOL CALL fixup paths to keep the
shadow stack in sync.
Fixes: 488af8ea71 ("x86/shstk: Wire in shadow stack interface")
Signed-off-by: David Windsor <dwindsor@gmail.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Acked-by: Oleg Nesterov <oleg@redhat.com>
Acked-by: Jiri Olsa <jolsa@kernel.org>
Tested-by: Jiri Olsa <jolsa@kernel.org>
Link: https://patch.msgid.link/8b5b1c7407b98f31664ad7b6a6faf20d2d4a6cad.1782777969.git.dwindsor@gmail.com
The 32-bit MSR interfaces rdmsr() and wrmsr() are planned to be
removed. Use the related 64-bit variants instead.
Signed-off-by: Juergen Gross <jgross@suse.com>
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Link: https://patch.msgid.link/20260629060526.3638272-20-jgross@suse.com
The 32-bit MSR interfaces rdmsr() and wrmsr() are planned to be
removed. Use the related 64-bit variants instead.
Signed-off-by: Juergen Gross <jgross@suse.com>
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Link: https://patch.msgid.link/20260629060526.3638272-19-jgross@suse.com
The 32-bit MSR interfaces rdmsr() and wrmsr() are planned to be
removed. Use the related 64-bit variants instead.
Signed-off-by: Juergen Gross <jgross@suse.com>
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Acked-by: Reinette Chatre <reinette.chatre@intel.com>
Link: https://patch.msgid.link/20260629060526.3638272-18-jgross@suse.com
The 32-bit MSR interfaces rdmsr() and rdmsr_safe() are planned to be
removed. Use the related 64-bit variants instead.
Signed-off-by: Juergen Gross <jgross@suse.com>
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Link: https://patch.msgid.link/20260629060526.3638272-15-jgross@suse.com
The 32-bit MSR interfaces rdmsr(), wrmsr() and rdmsr_safe() are
planned to be removed. Use the related 64-bit variants instead.
Signed-off-by: Juergen Gross <jgross@suse.com>
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Link: https://patch.msgid.link/20260629060526.3638272-13-jgross@suse.com
The 32-bit MSR interface rdmsr_safe()is planned to be removed. Use the
related 64-bit variant instead.
Signed-off-by: Juergen Gross <jgross@suse.com>
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Cc: Pu Wen <puwen@hygon.cn>
Link: https://patch.msgid.link/20260629060526.3638272-11-jgross@suse.com
The 32-bit MSR interfaces rdmsr(), wrmsr() and rdmsr_safe() are
planned to be removed. Use the related 64-bit variants instead.
Signed-off-by: Juergen Gross <jgross@suse.com>
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Link: https://patch.msgid.link/20260629060526.3638272-9-jgross@suse.com
The 32-bit MSR interfaces rdmsr(), wrmsr(), rdmsr_safe() and
wrmsr_safe() are planned to be removed. Use the related 64-bit variants
instead.
Signed-off-by: Juergen Gross <jgross@suse.com>
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Cc: H. Peter Anvin <hpa@zytor.com>
Link: https://patch.msgid.link/20260629060526.3638272-7-jgross@suse.com
