populate_extable() encodes "there is no destination register to clear" as
DONT_CLEAR in the DST_REG field of the exception table metadata, and later
ex_handler_bpf() then reuses that very value to derive the direction it
reports the fault with is_write = (reg == DONT_CLEAR). The two coincide
for a plain load or store, but not for a RMW carrying BPF_FETCH. Such an
atomic writes memory, so it has to be reported as a WRITE, and it also reads
the old value into a register, src_reg for BPF_ADD | BPF_FETCH and BPF_XCHG,
r0 for BPF_CMPXCHG, so that register has to be cleared on fault. A single
DONT_CLEAR cannot say both, and the store branch picks it unconditionally:
[...]
} else {
arena_reg = reg2pt_regs[dst_reg];
fixup_reg = DONT_CLEAR;
}
[...]
The reported direction is therefore right, but on a fault over an unmapped
arena page the fetch destination keeps whatever it held before the atomic,
where every other BPF_PROBE_* access delivers 0. Give the metadata its own
ARENA_WRITE bit so that the reported direction no longer depends on whether
there is a register to clear, and fill DST_REG in from bpf_atomic_load_reg().
BPF_{AND,OR,XOR} | BPF_FETCH need no handling here, bpf_jit_supports_insn()
already rejects those in the arena.
Fixes: d503a04f8b ("bpf: Add support for certain atomics in bpf_arena to x86 JIT")
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Reviewed-by: Puranjay Mohan <puranjay@kernel.org>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://patch.msgid.link/20260811131600.506721-3-daniel@iogearbox.net
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
The TODO comment in alix.c and geos.c mentions moving leds-net5501.c to
platform code. But, leds-net5501.c was already replaced by
arch/x86/platform/geode/net5501.c.
Signed-off-by: Eshaan Deshmukh <eshaan2031@icloud.com>
Signed-off-by: Dave Hansen <dave.hansen@linux.intel.com>
Link: https://patch.msgid.link/20260810064953.2048373-1-eshaan2031@icloud.com
With the recently found INVLPGB / TLBSYNC issue, there has been some
interest in disabling INVLPGB-based TLB flushing, in order to rule out
that CPU issue as a cause of userspace crashes.
Add a kernel command line option to control the TLB flushing behavior.
If the need arises, we will add a "tlbi=broadcast" for the case when TLB
invalidation broadcasts need to be explicitly selected, but this is not
needed now yet.
[ bp: Rewrite commit message, move to cpu/common.c, add documentation. ]
Fixes: 767ae437a3 ("x86/mm: Add INVLPGB feature and Kconfig entry")
Suggested-by: Borislav Petkov <bp@alien8.de>
Signed-off-by: Rik van Riel <riel@surriel.com>
Signed-off-by: Borislav Petkov (AMD) <bp@alien8.de>
Cc: <stable@kernel.org>
Link: https://patch.msgid.link/20260729204341.3eb0b5ea@fangorn
match_prev_assignment() currently forces a mismatch for ACR events, so
ACR counter indices are reprogrammed on every scheduling pass. That
causes avoidable overhead because disable and enable paths must touch
multiple MSRs.
The previous ACR assignment is already cached in acr_cfg_b[]. Use that
state to compare the newly computed ACR counter indices in hwc->config1
against the cached value in acr_cfg_b[hwc->idx]. If they match, skip
unnecessary disable and enable work.
Also tighten is_acr_self_reload_event() so it first verifies the event
is an ACR event before testing for the self-reload case.
Signed-off-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Thomas Falcon <thomas.falcon@intel.com>
Reviewed-by: Zide Chen <zide.chen@intel.com>
Link: https://patch.msgid.link/20260717080342.1879573-8-dapeng1.mi@linux.intel.com
intel_cap (IA32_PERF_CAPABILITIES) updates are currently tied to
X86_FEATURE_ARCH_PERFMON_EXT, but these are independent feature paths.
As a result, hybrid PMU capability state can be updated under the wrong
condition.
Also, intel_pmu_broken_perf_cap() is too narrow. Per RPL018, the missing
PERF_METRICS_AVAILABLE bit affects both Raptor Lake and Meteor Lake
parts, not only the currently covered subset.
Move intel_cap updates out of the ARCH_PERFMON_EXT-gated path, extend
intel_pmu_broken_perf_cap() coverage to both RPL and MTL families, and
introduce intel_update_pmu_caps() to centralize PMU capability updates.
Signed-off-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Thomas Falcon <thomas.falcon@intel.com>
Reviewed-by: Zide Chen <zide.chen@intel.com>
Link: https://patch.msgid.link/20260717080342.1879573-7-dapeng1.mi@linux.intel.com
On hybrid systems, init_hw_perf_events() can call check_hw_exists() with
the global PMU pointer after perf_is_hybrid is set. In that case,
fixed_counter_disabled() uses hybrid() on a non-hybrid PMU object, so the
intel_ctrl access is taken from the wrong layout and can read out of
bounds.
fixed_counter_disabled() was added in commit 32451614da
("perf/x86/intel: Support CPUID 10.ECX to disable fixed counters"), when
fixed counters were tracked via num_fixed_counters. Today fixed counters
are represented by fixed_cntr_mask, so this helper is obsolete.
Remove fixed_counter_disabled() and its callers, and rely directly on the
fixed-counter bitmask. With the helper gone, check_hw_exists() no longer
needs a PMU argument, so drop that parameter as well. This removes the
invalid hybrid access and closes the out-of-bounds read risk.
Signed-off-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Thomas Falcon <thomas.falcon@intel.com>
Reviewed-by: Zide Chen <zide.chen@intel.com>
Link: https://patch.msgid.link/20260717080342.1879573-6-dapeng1.mi@linux.intel.com
intel_pmu_cpu_prepare() allocates per-CPU perf state first and then sets
up the arch PEBS buffer. If alloc_arch_pebs_buf_on_cpu() fails,
the previously allocated cpuc resources are left behind.
Make the failure path call intel_cpuc_finish(cpuc) to release the per-CPU
state allocated by intel_cpuc_prepare().
Signed-off-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Thomas Falcon <thomas.falcon@intel.com>
Reviewed-by: Zide Chen <zide.chen@intel.com>
Link: https://patch.msgid.link/20260717080342.1879573-5-dapeng1.mi@linux.intel.com
In failure paths, cpuc->pmu can still point to the global static pmu
instead of an embedded x86_hybrid_pmu::pmu. Calling hybrid_pmu() on
that pointer causes an invalid container conversion and may lead to
out-of-bounds access.
This can happen in at least two cases:
- init_hybrid_pmu() fails check_hw_exists() and leaves cpuc->pmu as-is.
- CPU hotplug fails between CPUHP_PERF_X86_PREPARE and
CPUHP_AP_PERF_X86_STARTING, and rollback invokes intel_pmu_cpu_dead().
Fix both paths by:
- Clear cpuc->pmu to NULL when check_hw_exists() fails.
- Validat that cpuc->pmu is not the global static pmu before calling
hybrid_pmu() in intel_pmu_cpu_dead().
A new helper x86_get_static_pmu() is added to get the global static pmu.
Signed-off-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Thomas Falcon <thomas.falcon@intel.com>
Reviewed-by: Zide Chen <zide.chen@intel.com>
Link: https://patch.msgid.link/20260717080342.1879573-4-dapeng1.mi@linux.intel.com
If PMU initialization fails, for example in check_hw_exists(), hybrid
state can be left partially initialized: x86_pmu.hybrid_pmu is not freed
and perf_is_hybrid remains set. This can leak memory and leave stale
hybrid state reachable after a failed init path.
Add x86_pmu_free_hybrid() and use it on PMU init failure paths so all
hybrid-related state is consistently reset.
Signed-off-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Thomas Falcon <thomas.falcon@intel.com>
Reviewed-by: Zide Chen <zide.chen@intel.com>
Link: https://patch.msgid.link/20260717080342.1879573-3-dapeng1.mi@linux.intel.com
Fix an NMI handler leak in init_hw_perf_events(). When PMU
initialization fails after register_nmi_handler(), the error path
exits without calling unregister_nmi_handler(), leaving a stale
NMI_LOCAL "PMI" handler registered. Add the missing call before
clearing x86_pmu state.
Signed-off-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Thomas Falcon <thomas.falcon@intel.com>
Reviewed-by: Zide Chen <zide.chen@intel.com>
Link: https://patch.msgid.link/20260717080342.1879573-2-dapeng1.mi@linux.intel.com
With the changes that enable preempt count to track IRQ disabling
nesting, we don't have enough bits in 32-bit preempt count
implementation, as a result we move NMI nesting bits out of the 32-bit
preempt count. However on the architectures that can support 64-bit
preempt count implementation, we can keep the NMI nesting bits in the
32-bit preempt count and avoid maintaining NMI nesting bits outside of
the same cache line.
Therefore HAS_SEPARATE_PREEMPT_RESCHED_BITS is introduced to allow
architectures to select this. Note that under this Kconfig, preempt
count is maintained in a 64-bit word however preempt_count() still
remains as an int because all the effective bits still fit in
(previously we mask out NEED_RESCHED bit in preempt_count()). This
should make no functional changes for existing preempt_count() users.
Enable this for x86_64 along with the introduction of the Kconfig.
[boqun: Undo the __preempt_count_{add,sub}() optimization in 32-bit
preempt count since it may introduce {over,under}flow]
Originally-by: Peter Zijlstra <peterz@infradead.org>
Signed-off-by: Boqun Feng <boqun@kernel.org>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/20260804161447.84806-11-boqun@kernel.org
In order to use preempt_count() to track the interrupt disable nesting
level, __preempt_count_{add,sub}_return() are introduced, as their names
suggest, these primitives return the new value of the preempt_count()
after changing it. The following example shows the usage of it in
local_interrupt_disable():
// increase the HARDIRQ_DISABLE bit
new_count = __preempt_count_add_return(HARDIRQ_DISABLE_OFFSET);
// if it's the first-time increment, then disable the interrupt
// at hardware level.
if ((new_count & HARDIRQ_DISABLE_MASK) == HARDIRQ_DISABLE_OFFSET) {
local_irq_save(flags);
raw_cpu_write(local_interrupt_disable_state, flags);
}
Having these primitives will avoid a read of preempt_count() after
changing preempt_count() on certain architectures.
Signed-off-by: Boqun Feng <boqun@kernel.org>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Acked-by: Heiko Carstens <hca@linux.ibm.com> # s390
Link: https://patch.msgid.link/20260804161447.84806-4-boqun@kernel.org
save_args() reads stack-passed arguments relative to rbp assuming two
return addresses sit between the saved rbp and the arguments, which
holds when the trampoline is entered through the fentry call from a
traced function. An indirect trampoline is called through a function
pointer, so only the caller's return address is on the stack and the
arguments start at rbp + 16, not rbp + 24. Every stack-passed argument
of a struct_ops callback with more than six argument slots is read one
slot off.
This has gone unnoticed because no in-tree struct_ops member passes
arguments on the stack. The jmp-entry form already accounts for having
a single return address; treat BPF_TRAMP_F_INDIRECT the same way.
Fixes: 473e3150e3 ("bpf, x86: allow function arguments up to 12 for TRACING")
Signed-off-by: Tejun Heo <tj@kernel.org>
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Tested-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://patch.msgid.link/20260808003938.3486067-12-memxor@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Implement the struct_ops arena argument conversion on x86. save_args()
receives the arena base from bpf_tramp_arena_base() and consults the
btf_func_model argument flags as it copies each native argument into the
BPF ctx, routing a marked argument through RAX:
movl %esrc, %eax /* truncate and clear the upper 32 bits */
subl $base_lo, %eax
movq %rax, ctx_slot
A nullable argument tests the full 64-bit kernel pointer first:
movq %rsrc, %rax
testq %rax, %rax
jz 1f
subl $base_lo, %eax
1:
movq %rax, ctx_slot
The 32-bit subtraction is sufficient since (u32)(kaddr - base) ==
(u32)kaddr - (u32)base, and it clears the upper half as the JITs require
of arena pointer registers. Stack-passed arguments already reload
through RAX, so only the subtraction (and the NULL test) is inserted
there.
Keep arena and nullable classification in btf_func_model.
bpf_tramp_arena_base() returns a base only for a single-program
struct_ops indirect trampoline; other trampolines pass zero and perform
no conversion. The size probe reruns the same emission with the same
model and nodes, so the image size matches by construction.
With both the kfunc and struct_ops directions implemented, flip
bpf_jit_supports_arena_args() on for x86.
Signed-off-by: Tejun Heo <tj@kernel.org>
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://patch.msgid.link/20260808003938.3486067-8-memxor@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Implement arena argument rebasing for kfunc calls on x86. R12 already
holds kern_vm_start whenever the prog has an arena, so each tagged
argument costs two instructions emitted right before the call:
movl %eN, %eN /* truncate, clear the upper 32 bits */
addq %r12, %rN
A nullable argument tests the truncated value and jumps over the add:
movl %eN, %eN
testl %eN, %eN
jz 1f
addq %r12, %rN
1:
addq carries a REX prefix for every argument register and is always
three bytes, so the jz displacement is constant. The sequence is native
code generated after constant blinding has run on the BPF instruction
stream, so blinding never sees the rebase and needs no special handling.
bpf_jit_supports_arena_args() is not flipped yet; that happens when the
struct_ops trampoline side is in place as well.
Signed-off-by: Tejun Heo <tj@kernel.org>
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://patch.msgid.link/20260808003938.3486067-7-memxor@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Merge updates of the thermal core, thermal drivers for Intel platforms
and the thermometer utility for 7.3-rc1:
- Use sysfs_emit_at() in trans_table_show() (Thorsten Blum)
- Remove hwmon class devices created for thermal zones when the thermal
zone devices holding them are removed (Rafael Wysocki)
- Add support for the Directed Package-level Thermal Interrupt to
the Intel thermal throttling driver to allow package-level thermal
interrupts to go to one specific CPU in a processor package instead
of going to all of the CPUs in it (Ricardo Neri)
- Clean up RFIM groups on DVFS failure and clean up ODVP on probe
failures in the int340x thermal driver (Pengpeng Hou)
- Remove redundant dev_err() from the int340x thermal driver and the
bxt_pmic driver (Pan Chuang)
- Simplify ptc_temperature_write() in the int340x thermal driver by
using kstrtou32_from_user() (Dmitry Antipov)
- Close fd on realloc() failure in the thermometer utility (Amarjeet)
* thermal-core:
thermal: sysfs: Use sysfs_emit_at() in trans_table_show()
* thermal-hwmon:
thermal: hwmon: Remove hwmon class device along with its parent
* thermal-intel:
thermal: intel: int340x: simplify ptc_temperature_write()
thermal: intel: bxt_pmic: Remove redundant dev_err()
thermal: intel: int340x: Remove redundant dev_err()
thermal: intel: int3400: clean up ODVP on probe failures
thermal: intel: int340x: clean up RFIM groups on DVFS failure
thermal: intel: Add a syscore shutdown callback for kexec reboot
thermal: intel: Add syscore callbacks for suspend and resume
thermal: intel: Enable the Directed Package-level Thermal Interrupt
thermal: intel: Add resources to handle directed package-level thermal interrupts
x86/thermal: Add bit definitions for Intel Directed Package Thermal Interrupt
* thermal-tools:
tools/thermal/thermometer: close fd on realloc() failure
When adding cc clobber to wmb()'s definition, the alternative()
condition to use sfence was incorrectly raised from X86_FEATURE_XMM to
X86_FEATURE_XMM2.
Restore the correct constraint for potential better performance on
machines without SSE2.
Fixes: bd922477d9 ("locking/x86: Add cc clobber for ADDL")
Signed-off-by: Yao Zi <me@ziyao.cc>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Cc: stable@vger.kernel.org
Link: https://patch.msgid.link/20260801182953.15069-1-me@ziyao.cc
If pt_event_stop() is called without PERF_EF_UPDATE flag, then
perf_aux_output_end() is not called. A subsequent call to pt_event_start()
will call perf_aux_output_begin() again which violates the rule against
nesting and triggers a WARNING in perf_aux_output_begin().
Originally, pt_event_stop() was never called without PERF_EF_UPDATE,
because the only code paths to do so are from event overflow, and Intel PT
does not do that.
However the introduction of group throttling by commit 9734e25fbf
("perf: Fix the throttle logic for a group") meant that an Intel PT event
could be throttled if it was part of a group. Throttling calls PMU
->stop() / ->start() callbacks without flags.
An example is when AUX area sampling is used. The following commands
hit the issue:
echo 10000 > /proc/sys/kernel/perf_event_max_sample_rate
perf record -F32000 --aux-sample -e '{intel_pt//u,cycles:u}' \
-- bash -c 'for i in `seq 1 100000` ; do true ; done'
Use PERF_HES_UPTODATE to track whether perf_aux_output_begin() and
perf_aux_output_end() are balanced. A cleared PERF_HES_UPTODATE bit
indicates that an AUX output context is still open.
Amend pt_event_start() / pt_event_stop() accordingly so that begin/end
stay balanced:
- In non-snapshot mode, stop() always closes the buffer (the buffer may
have run out of space, and that accounting is done by the update), so
a following start() opens a fresh one as before.
- In snapshot/overwrite mode, stop() without PERF_EF_UPDATE leaves the
buffer open so that pt_event_snapshot_aux() can still copy from it,
and start() then only re-enables tracing instead of calling
perf_aux_output_begin() again.
Note that pt_event_del() calls pt_event_stop() with PERF_EF_UPDATE flag set
(as is required by the documentation), so a final call to
perf_aux_output_end() is assured.
Fixes: 52ca9ced3f ("perf/x86/intel/pt: Add Intel PT PMU driver")
Signed-off-by: Adrian Hunter <adrian.hunter@intel.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Tested-by: Yi Lai <yi1.lai@intel.com>
Link: https://patch.msgid.link/20260721070254.13557-4-adrian.hunter@intel.com
The Intel PT driver reads and writes event->hw.state as a whole value,
assuming it is either 0 or PERF_HES_STOPPED. That is true today, but a
subsequent fix needs to also track an open AUX output buffer using the
PERF_HES_UPTODATE bit of the same field.
When more than one bit can be set, whole-value assignments would
overwrite the other bits and whole-value comparisons would fail to match.
Convert all accesses to set, clear and test the PERF_HES_STOPPED bit
individually, in preparation for that change.
No functional change intended: event->hw.state currently only ever
holds 0 or PERF_HES_STOPPED, so the bitwise forms are equivalent.
Signed-off-by: Adrian Hunter <adrian.hunter@intel.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Tested-by: Yi Lai <yi1.lai@intel.com>
Link: https://patch.msgid.link/20260721070254.13557-3-adrian.hunter@intel.com
pt_config() enables tracing by allowing NMIs and pause/resume, issuing
the necessary barriers, and calling pt_config_start(). A later change
needs to re-enable tracing on a (re-)start path without repeating the
full pt_config() setup (filters, RTIT_CTL, buffer configuration).
Factor that enabling sequence out into a new helper, pt_config_enable(),
so it can be called on its own.
No functional change intended.
Signed-off-by: Adrian Hunter <adrian.hunter@intel.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Tested-by: Yi Lai <yi1.lai@intel.com>
Link: https://patch.msgid.link/20260721070254.13557-2-adrian.hunter@intel.com
Remove the code in kvm_cpuid() that dynamically updates the Xen TSC
info CPUID leaf at runtime, as KVM is updating the wrong sub-leaf anyway
(0x40000x03/2 EAX is the *host* TSC frequency per the Xen ABI, not the
guest frequency which belongs in 0x40000x03/0 ECX).
Simply drop the code instead of fixing it to fill the correct sub-leaf, as
modifying guest CPUID entries/output at runtime is generally undesirable,
and providing userspace the necessary data to fill the sub-leaf itself is
useful for other reasons, e.g. to fill the generic 0x40000010 timing leaf
and to provide exact scaling information to aid save/restore.
Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
Link: https://patch.msgid.link/20260728144954.355376-23-dwmw2@infradead.org
[sean: tweak changelog to take this patch before the new uAPI]
Signed-off-by: Sean Christopherson <seanjc@google.com>
When kvm_xen_update_runstate() is invoked to set a vCPU's runstate, the
time spent in the previous runstate is accounted. This is based on the
delta between the current KVM clock time, and the previous value stored
in vcpu->arch.xen.runstate_entry_time.
If the KVM clock goes backwards, that delta will be negative. Or, since
it's an unsigned 64-bit integer, very *large*. Linux guests deal with
that particularly badly, reporting 100% steal time for ever more (well,
for *centuries* at least, until the delta has been consumed).
So when a negative delta is detected, just refrain from updating the
runstate times until the KVM clock catches up with runstate_entry_time
again.
Also clamp steal_ns to delta_ns to prevent steal time from exceeding
the total elapsed time, and handle negative steal_ns (which can happen
if run_delay goes backwards across a scheduler update).
The userspace APIs for setting the runstate times do not allow them to
be set past the current KVM clock, but userspace can still adjust the
KVM clock *after* setting the runstate times, which would cause this
situation to occur.
Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
Reviewed-by: Paul Durrant <paul@xen.org>
Link: https://patch.msgid.link/20260728144954.355376-21-dwmw2@infradead.org
Signed-off-by: Sean Christopherson <seanjc@google.com>
On PARAVIRT_SPINLOCKS=y kernels queued_spin_unlock() is dispatched
through a static_call(). Those PARAVIRT_SPINLOCKS=y kernels are quite
popular. Gating contended_release behind a static branch would leave a
NOP on the unlock hot path even, when the tracepoint is disabled.
Since the static_call() is already present, swap its target to a traced
unlock, when the tracepoint is enabled instead. When contended_release
tracepoint is disabled the target is the plain unlock (an inline store
on native x86_64), so the unlock path is unchanged and the tracepoint is
truly zero-cost.
Provide two traced variants, native_queued_spin_unlock_traced() and
pv_queued_spin_unlock_traced(), so each tail-calls its own base unlock
directly rather than recursing through the now-traced static_call().
Teach pv_is_native_spin_unlock() that the traced native variant still
counts as native.
Only PARAVIRT_SPINLOCKS=y is affected. PARAVIRT_SPINLOCKS=n keeps the
generic static-branch path.
Suggested-by: Peter Zijlstra <peterz@infradead.org>
Signed-off-by: Dmitry Ilvokhin <d@ilvokhin.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Acked-by: Juergen Gross <jgross@suse.com>
Link: https://patch.msgid.link/17fa67f9fa4cf93f1150725e89f5f916e41a9b6f.1785778551.git.d@ilvokhin.com
queued_spin_lock_slowpath() and queued_spin_unlock() are dispatched
through pv_ops_lock via the paravirt-ops ALTERNATIVE machinery, which
picks the target (native inline store / hypervisor call) once at boot
and cannot change at runtime.
Convert both to static_call(). The site becomes a direct call patched in
place (one byte smaller), and on native the unlock still collapses to
the inline "movb $0, (%rdi)" store, so the fast path is unchanged.
Unlike the ALTERNATIVE mechanism, a static_call() target can also be
updated at runtime via static_call_update(). This is a prerequisite for
the contended_release tracepoint, which has to swap in a traced unlock
while the system is running.
[ ilvokhin: commit message; fix PARAVIRT_SPINLOCKS=n build; teach
__static_call_validate() about the inline unlock insn; make the
slowpath site module-safe: static_call_mod() +
EXPORT_STATIC_CALL_TRAMP(); pass @lock to the callee-save unlock,
fixing a boot hang under CALL_DEPTH_TRACKING. Boot tested native + KVM
PV guest. ]
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Co-developed-by: Dmitry Ilvokhin <d@ilvokhin.com>
Signed-off-by: Dmitry Ilvokhin <d@ilvokhin.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Acked-by: Juergen Gross <jgross@suse.com>
Link: https://lore.kernel.org/all/20260603120811.GW3493090@noisy.programming.kicks-ass.net/
Link: https://patch.msgid.link/9a32ae399eb804a02a31af04dcabe7e7ee4f3fdf.1785778551.git.d@ilvokhin.com
- fix a lot of small bugs and races
x86:
- fix missing locking related to KVM_CAP_MOVE_ENC_CONTEXT_FROM
- warn on creating a new page table that is the child of an invalid one,
and limit damage before it's too late
- disable use of INVLPGA when NPT is enabled, because it doesn't seem
to flush TLBs correctly
-----BEGIN PGP SIGNATURE-----
iQFIBAABCgAyFiEE8TM4V0tmI4mGbHaCv/vSX3jHroMFAmp14g0UHHBib256aW5p
QHJlZGhhdC5jb20ACgkQv/vSX3jHroPzrAf+KGSkR+qBbqgsSy0I86kw6VvX6h8i
TySes/l0JaNbVQDDt3vYzw/ZXhrm6dRA58i1mwjIVLIY3pflXzUrwqqXjLwvcGNf
1eX0jOZOg68fXPW22okR+7mNJdHSiRiX3Ozw20TrS5+33xN3dKa1xsTtNFRN99f6
h8mtQIP5L9dYn6y9lI27y51lkx3Ojd8SWAUJyXn8d3OA28mBcIShpq6BJsroHT34
HzibCnNAodFMGDsUJr93mpJisUV4EsWPKO2hzMm5FUUkV/JBD3GnqwF550Kwpc8a
ojuaHCEAnZXLoDuOTBiT8iQBa1fDUHizERlTkHD91s0ktwqf+e3p0azjMw==
=734T
-----END PGP SIGNATURE-----
Merge tag 'for-linus' of git://git.kernel.org/pub/scm/virt/kvm/kvm
Pull vkm fixes from Paolo Bonzini:
"s390:
- fix a lot of small bugs and races
x86:
- fix missing locking related to KVM_CAP_MOVE_ENC_CONTEXT_FROM
- warn on creating a new page table that is the child of an invalid
one, and limit damage before it's too late
- disable use of INVLPGA when NPT is enabled, because it doesn't seem
to flush TLBs correctly"
* tag 'for-linus' of git://git.kernel.org/pub/scm/virt/kvm/kvm: (26 commits)
KVM: x86/mmu: WARN and clear role.invalid when creating a child shadow page
KVM: SVM: Serialize accesses to the owner and mirror list with separate lock
KVM: SVM: make svm_flush_tlb_gva do a full asid flush if NPT enabled
KVM: s390: Fix cleanup in kvm_s390_pv_create_cpu()
KVM: s390: Fix ordering when adding to SCA
KVM: s390: Return -EINTR if a signal is pending while faulting-in
KVM: s390: Free the mmu cache when kvm_arch_vcpu_create() fails
KVM: s390: ucontrol: Add missing locking around gmap_remove_child()
KVM: s390: cmma: Fix dirty tracking when removing memslot
KVM: s390: Fix race in __do_essa()
KVM: s390: Fix leaking of PGM_ADDRESSING to userspace
KVM: s390: ucontrol: Fix sca_clear_ext_call()
KVM: s390: Fix overclearing ESCA in case of error
KVM: s390: Fix kvm_s390_vcpu_unsetup_cmma()
KVM: s390: Do not free SCA if it was not allocated
KVM: s390: Fix unlikely NULL gmap dereference
s390/vfio_ccw: Implement a crw lock
s390/vfio_ccw: Selectively expand io_mutex
s390/vfio_ccw: Move cp cleanup out of not operational
s390/vfio_ccw: Cancel existing workqueues
...
A load-acquire from an arena pointer is converted to BPF_PROBE_ATOMIC and
gets an exception table entry, but the entry is filled in as if it were a
store, since populate_extable() decides based on instruction class alone
and a load-acquire is of BPF_STX class:
if (BPF_CLASS(insn->code) == BPF_LDX) {
arena_reg = reg2pt_regs[src_reg];
fixup_reg = reg2pt_regs[dst_reg];
} else {
arena_reg = reg2pt_regs[dst_reg];
fixup_reg = DONT_CLEAR;
}
For a load-acquire dst_reg holds the loaded value and src_reg holds the
address, so both assignments in the else branch are wrong. On a fault
over an unmapped arena page ex_handler_bpf() then:
- computes the reported address from the value register instead
of the address register
- reports the access as a WRITE, since it derives the direction
from fixup_reg == DONT_CLEAR
- leaves dst_reg untouched, so the program continues with a stale
value instead of the 0 that BPF_PROBE_* loads deliver
The access itself is emitted correctly, emit_atomic_ld_st_index() uses
src_reg as the address, so this is a broken probe contract and a wrong
diagnostic rather than a memory safety issue.
Use bpf_atomic_is_load_acq() helper so a load-acquire takes the load path.
Fixes: 5341c9a4d8 ("bpf, x86: Support load-acquire and store-release instructions")
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Link: https://lore.kernel.org/bpf/20260806201047.333389-3-daniel@iogearbox.net
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Explicitly clear role.invalid when deriving a child shadow page's role from
its parent to harden against bugs elsewhere in KVM, as violating KVM's
invariant that invalid pages are NOT on the list of active MMU pages leads
to use-after-free due to __kvm_mmu_prepare_zap_page() using list_add()
instead of list_move() when processing an invalid shadow page, i.e. makes a
bad situation far worse.
Yell loudly if the parent is invalid, as it means KVM has missed a validity
check, i.e. KVM is attempting to map memory using an invalid/obsolete root,
but continue on as the child is otherwise still a valid shadow page.
==================================================================
BUG: KASAN: slab-use-after-free in __kvm_mmu_get_shadow_page+0x1817/0x1860 [kvm]
Write of size 8 at addr ff11000153dd1368 by task repro/853
CPU: 1 UID: 1000 PID: 853 Comm: repro Not tainted 7.2.0-rc2-3aec122bdcaf-next-vm #5 PREEMPT
Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
Call Trace:
<TASK>
dump_stack_lvl+0x4b/0x70
print_report+0x153/0x49c
kasan_report+0xbc/0xf0
__kvm_mmu_get_shadow_page+0x1817/0x1860 [kvm]
mmu_alloc_root+0x141/0x320 [kvm]
kvm_mmu_load+0x612/0x20f0 [kvm]
kvm_arch_vcpu_ioctl_run+0x3dd5/0x6150 [kvm]
kvm_vcpu_ioctl+0x5e4/0x10d0 [kvm]
__x64_sys_ioctl+0x131/0x1b0
do_syscall_64+0x67/0x5f0
entry_SYSCALL_64_after_hwframe+0x4b/0x53
</TASK>
Allocated by task 853:
kasan_save_stack+0x20/0x40
kasan_save_track+0x14/0x30
__kasan_slab_alloc+0x5f/0x70
kmem_cache_alloc_noprof+0xfe/0x2e0
__kvm_mmu_topup_memory_cache+0x135/0x530 [kvm]
paging64_page_fault+0x318/0x1e30 [kvm]
kvm_mmu_do_page_fault+0x21d/0x630 [kvm]
kvm_mmu_page_fault+0x18c/0x17b0 [kvm]
kvm_arch_vcpu_ioctl_run+0x1f35/0x6150 [kvm]
kvm_vcpu_ioctl+0x5e4/0x10d0 [kvm]
__x64_sys_ioctl+0x131/0x1b0
do_syscall_64+0x67/0x5f0
entry_SYSCALL_64_after_hwframe+0x4b/0x53
Freed by task 853:
kasan_save_stack+0x20/0x40
kasan_save_track+0x14/0x30
kasan_save_free_info+0x3b/0x60
__kasan_slab_free+0x43/0x70
kmem_cache_free+0xe2/0x400
kvm_mmu_commit_zap_page.part.0+0x1e2/0x310 [kvm]
kvm_mmu_free_roots+0x283/0x560 [kvm]
kvm_arch_vcpu_ioctl_run+0x33c8/0x6150 [kvm]
kvm_vcpu_ioctl+0x5e4/0x10d0 [kvm]
__x64_sys_ioctl+0x131/0x1b0
do_syscall_64+0x67/0x5f0
entry_SYSCALL_64_after_hwframe+0x4b/0x53
Reported-by: Hyunwoo Kim <imv4bel@gmail.com>
Fixes: a770f6f28b ("KVM: MMU: Inherit a shadow page's guest level count from vcpu setup")
Cc: stable@vger.kernel.org
Signed-off-by: Sean Christopherson <seanjc@google.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Interaction between KVM_CAP_VM_MOVE_ENC_CONTEXT_FROM and
KVM_CAP_VM_COPY_ENC_CONTEXT_FROM can cause two separate issues:
- in sev_migrate_from(), when the destination KVM is a mirror, the mirror
entry is moved from the source's list to the owner's mirror_vms list,
without holding the owner's lock unlike other writers of the owner's
mirror list (sev_vm_copy_enc_context_from(), sev_vm_destroy()).
A concurrent COPY or destroy can race with sev_migrate_from() and
corrupt the list.
- In sev_vm_destroy(), the *owner* is still active and could receive
concurrently a KVM_CAP_VM_MOVE_ENC_CONTEXT_FROM that causes
sev->enc_context_owner to change. In this case the incorrect VM
receives kvm_put_kvm().
The second issue needs particular care because the owner could disappear
altogether (even though the race window is impossibly small) between
reading it and locking it. There is thus no way to perform the checks
under the owner lock without putting struct kvm under SLAB_TYPESAFE_BY_RCU
(which would allow kvm_get_kvm_safe() under RCU critical section).
It is much simpler to just use a global lock, since the critical
sections are so small and the new lock is always a leaf lock.
Fixes: b2125513df ("KVM: SEV: Allow SEV intra-host migration of VM with mirrors")
Cc: stable@vger.kernel.org
Reported-by: Shen Yongchao <grayhat@foxmail.com>
Link: https://lore.kernel.org/kvm/tencent_625C0F42824E542C72B34733392AF2C49709@qq.com/
Link: https://lore.kernel.org/kvm/tencent_DDC4E4352EC91CAC05A9A8F4E55E8C96730A@qq.com/
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Red Hat is seeing multiple reports of Windows memory corruptions
(and consequent BSODs) with hv-tlbflush=on, on AMD processors only.
The crashes, while extremely rare, happen even with a stock configuration,
but with Driver Verifier enabled they can be detected after approximately
200 VM hours. In particular, Alexander Lougovski measured the following:
- on AMD Turin, 15 crashes in 3300 VM hours
- on AMD Milan, 2 crashes in 500 VM hours (there are fewer hours
here due to the host being smaller)
- on Intel Sapphire Rapids, 0 crashes in 8000 VM hours
- on AMD Turin with full TLB flush (not exactly this patch but
similar), no crashes in ~2 weeks of run time which should also
be ~7000 VM hours
For Turin, the microcode version was 0x0b002162, which (assuming
this is the same issue) should not be affected by the problem listed in
https://knowledge.broadcom.com/external/article/419026/bsod-on-virtual-machines-running-on-amd.html;
on the other hand that problem should not apply to earlier processors.
AMD has not provided any information or analysis yet, and when we asked
we didn't know yet that it reproduced on Milan as well.
As to the workload, Alexander threw more or less everything at the same
time at the VM:
- a full Windows Defender scan every 30 minutes
- a disk I/O job
- a loop doing repeated mmap of system files (mostly to hope that
it triggers some consistency check in the Windows memory manager)
- SQL Express 2022 + StressDB (1.6M rows), with the host doing queries
(75% write/25% read) via sqlcmd
Driver Verifier is able to detect BSODs more or less at the same time as
the pages are freed. They mostly happen in the Windows Defender filter
driver, but occasionally also in the networking stack (e.g., afd.sys)
or elsewhere in the filesystem stack (e.g., fltmgr.sys).
The flush is issued from kvm_hv_vcpu_flush_tlb(), which receives the
cross-CPU requests from the Hyper-V TLB flush hypercalls via a kfifo
and is invoked by the KVM_REQ_HV_TLB_FLUSH request. The mechanism is
the same for both Intel and AMD, and the handler for both vendors is
a simple INVVPID(ADDR)/INVLPGA instruction.
Because the request is handled on the destination CPU, there is a question
of what happens if the VM is migrated across physical CPUs. In that case,
the INVLPGA instruction would use a stale svm->vmcb->control.asid; but
if anything that might do an *unnecessary* flush (on an asid that's being
used for another VM) and then pre_svm_run() would force a full TLB rebuild.
So, for lack of better ideas, this patch forces a full ASID bump in
svm_flush_tlb_gva(). To avoid paying the price on Intel and also to
avoid unnecessary loops on AMD, the flush_tlb_gva op now returns whether
it did a full flush or not; kvm_hv_vcpu_flush_tlb() takes note and exits
its loops immediately. While there is an obvious performance impact,
about half of the benefit from Hyper-V tlbflush is preserved (10% vs. 20%
on the SQL Server workload).
kvm_mmu_invalidate_addr() is the only other caller of the flush_tlb_gva op.
The change would have a performance impact on every intercepted INVLPG and,
for nested SVM, on every L1 INVLPGA. For INVLPGA specifically, this covers
the same suspected issue but for nested hypervisors, so it is correct to
apply the workaround; for INVLPG on shadow paging, instead, the impact
would be stronger and, due to lack of data, for now the use of INVLPGA is
left in place in svm_flush_tlb_gva().
Analyzed-by: Vitaly Kuznetsov <vkuznets@redhat.com>
Analyzed-by: Alexander Lougovski <alougovs@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
CONFIG_XEN_PVHVM_SMP is referenced only on x86 in Xen specific code,
so it can be replaced with CONFIG_SMP.
Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
Signed-off-by: Juergen Gross <jgross@suse.com>
Message-ID: <20260805082137.1214967-5-jgross@suse.com>
On x86 CONFIG_XEN_PVHVM is now a synonym of CONFIG_XEN.
In Xen specific x86 code it can be just dropped, in non-Xen specific
x86 code it can be replaced with CONFIG_XEN.
In architecture independent code it is used only where CONFIG_XEN is
defined, so it can be replaced with CONFIG_X86 there.
Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
Signed-off-by: Juergen Gross <jgross@suse.com>
Message-ID: <20260805082137.1214967-3-jgross@suse.com>
CONFIG_XEN depends on CONFIG_X86_LOCAL_APIC already, so the dependency
of CONFIG_XEN_PVHVM on CONFIG_X86_LOCAL_APIC can be dropped.
Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
Reviewed-by: Jan Beulich <jbeulich@suse.com>
Signed-off-by: Juergen Gross <jgross@suse.com>
Message-ID: <20260805082137.1214967-2-jgross@suse.com>
sequence used in the SRSO mitigation and still poisoning the RSB.
Do that by emulating the saferet sequence and thus avoiding executing
a RET instruction.
-----BEGIN PGP SIGNATURE-----
iQIzBAABCgAdFiEEzv7L6UO9uDPlPSfHEsHwGGHeVUoFAmpmGc8ACgkQEsHwGGHe
VUow/Q/+KJxaQyhYwINEUE2M/mzAjraCJBHbTWIsbpUgwHzC2EjVQluV91Kg5O5s
P3cvbxeAcfx9SWIJ9e0vi5JdxAHDd0Idy0FMccg0xWOPJOzGjXpQZzeg+4yvm4a4
URgr7P+IFNgdnfDsZoTbf510AczlNXFQS4YcmgVDOvdhZjlcLK2/7CYdRtgQ+B3E
J1NkaD9RPW9HaCr5XbCgw+lI8XMQNQEdQ3MGwfCgQERl5i5iM6D33Sh/0bq6zfNy
HiMk27LthH7woHRuTrp2DjqeiygQdAPl+87/NND95KWz73fkqFofMfFkMfTVDvkZ
klibXv1oVkEAnGD2/QUVUrn6EbLU/yXB+7LwHC7O7Du7fMV60xyQhEkxWHYV/Szx
kSOtwaTYHc84KdqsBVFLoS7Gr6sMb0cIGcHwC4dQdWT5ZqztwBZJJklsLjJNoPNG
prFCq1ER3282DJ76C8DUseIKXdLzcMcPZK4gdi1g/g0ESaCRxH2e4aRRMaXMQC2W
uZT6EPkyMl+eR5r0gSFJEXLxFnBVguFeJV2uSh3crnXvrJqKEyJDyHgodDxUzQ9P
wFa6KD+yILiDXMwRDLw8C5EhApQ4MqRmys2HEC65pXWYAm14ivEaOXZHDdQ8pd0W
eaWa90nFe0dwQz00g/A0XfxnEmh4AMIGBsSwAIFsMjKIunizRUk=
=PpIN
-----END PGP SIGNATURE-----
Merge tag 'x86_bugs_saferet' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
- Add a mitigation for the attack vector of interrupting the saferet
sequence used in the SRSO mitigation and still poisoning the RSB.
Do that by emulating the saferet sequence and thus avoiding executing
a RET instruction.
* tag 'x86_bugs_saferet' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip:
x86/bugs: Make Safe-RET robust against interrupt injection
Tidy up the x86 vPMU emulation code and its selftests by fixing comments
that no longer reflect the code along with assorted spelling and grammar
mistakes, and by dropping redundant blank lines reported by checkpatch.
No functional change intended.
Signed-off-by: Like Xu <likexu@tencent.com>
Link: https://patch.msgid.link/20260625090155.6326-1-likexu@tencent.com
[sean: keep quotes around "hit", shove sequences fix into separate patch]
Signed-off-by: Sean Christopherson <seanjc@google.com>
There's no CPUID "write" operation so there's no need to have
a cpuid_read_output thing - cpuid_output is perfectly clear. And
shortens the code just fine.
Remove the funky function signature line breaks while at it.
No functional changes.
Signed-off-by: Borislav Petkov (AMD) <bp@alien8.de>
Link: https://lore.kernel.org/r/20260805025323.GCanKlox0DF7Gwhpf1@fat_crate.local
CONFIG_SYSCTL used to make sense as a separate hidden bool before commit
61a47c1ad3 ("sysctl: Remove the sysctl system call"); it was selected
by both CONFIG_SYSCTL_SYSCALL and CONFIG_PROC_SYSCTL.
Today CONFIG_PROC_SYSCTL is the only selector, so the two are always
equal. Kill the hidden bool, rename the PROC_SYSCTL prompt to SYSCTL,
and s/CONFIG_PROC_SYSCTL/CONFIG_SYSCTL/ tree-wide.
Signed-off-by: Oleg Nesterov <oleg@redhat.com>
Signed-off-by: Joel Granados <joel.granados@kernel.org>
It turns out that commit 5fce67641a ("x86/mm/pat: Don't gate
cpa_lock on debug_pagealloc_enabled()") was a little too quick to
remove the debug_pagealloc exception for cpa_lock.
Notably __kernel_map_pages() is used by the page-allocator from any
context the page-allocator itself is used, which violates the cpa_lock
rules.
Re-instate the exception, except make it specific to the
__kernel_map_pages() such that any other cpa() usage is still fully
serialized by cpa_lock. Also note that since cpa() should not be used
on memory that isn't allocated, the page-allocator locking and cpa are
infact mutually exclusive and all cpa usage in fully serialized.
Add a comment explaining this and other 'funnies' surrounding
DEBUG_PAGEALLOC, including how pgd_lock is not affected and the TLB
trickery.
Fixes: 5fce67641a ("x86/mm/pat: Don't gate cpa_lock on debug_pagealloc_enabled()")
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/20260729111119.604452135@infradead.org
There are a number of places where we open code what linear_page_index()
and linear_page_delta() calculate.
Replace this code with the appropriate functions for consistency.
No functional change intended.
Link: https://lore.kernel.org/20260710-b4-pre-scalable-cow-v2-21-2a5aa403d977@kernel.org
Signed-off-by: Lorenzo Stoakes <ljs@kernel.org>
Reviewed-by: Thomas Zimmermann <tzimmermann@suse.de> [DRM]
Acked-by: Kai Huang <kai.huang@intel.com> # for sgx
Reviewed-by: Gregory Price <gourry@gourry.net>
Reviewed-by: Pedro Falcato <pfalcato@suse.de> # for mm
Reviewed-by: Ackerley Tng <ackerleytng@google.com> [guest_memfd]
Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Cc: David Hildenbrand (Arm) <david@kernel.org>
Cc: Marek Szyprowski <m.szyprowski@samsung.com>
Cc: SJ Park <sj@kernel.org>
Cc: Liam R. Howlett (Oracle) <liam@infradead.org>
Cc: Zi Yan <ziy@nvidia.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
There is no good reason to mark boot page tables as MIX_SECTION_INFO: we
only free boot page tables when they are completely empty, and memory
offlining/hotunplug doesn't benefit from it in any way.
So just stop marking page tables as MIX_SECTION_INFO. In
free_pagetable(), we can now simply free reserved pages directly.
Link: https://lore.kernel.org/20260716-bootmem_info_part2-v2-6-4afc76c73d61@kernel.org
Signed-off-by: David Hildenbrand (Arm) <david@kernel.org>
Reviewed-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Reviewed-by: Dave Hansen <dave.hansen@linux.intel.com>
Cc: Alexander Gordeev <agordeev@linux.ibm.com>
Cc: Andy Lutomirski <luto@kernel.org>
Cc: "Borislav Petkov (AMD)" <bp@alien8.de>
Cc: Brendan Jackman <jackmanb@google.com>
Cc: Brendan Jackman <brendan.jackman@linux.dev>
Cc: Christian Borntraeger <borntraeger@linux.ibm.com>
Cc: Gerald Schaefer <gerald.schaefer@linux.ibm.com>
Cc: Heiko Carstens <hca@linux.ibm.com>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Muchun Song <muchun.song@linux.dev>
Cc: Oscar Salvador <osalvador@suse.de>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Sven Schnelle <svens@linux.ibm.com>
Cc: Vasily Gorbik <gor@linux.ibm.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Zi Yan <ziy@nvidia.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
We added the whole bootmem registration machinery in commit 0475327876
("memory hotplug: register section/node id to free").
The main use case was to remember to which memory section memmap pages
belonged, so the memmap could be handled accordingly when freeing memory.
However, all that machinery is not required anymore: a memory section can
only get offlined if *all* pages can get offlined; and it can only get
unplugged once offline. If some of these pages are unmovable memmap
pages: bad luck, doesn't work. Offlining will fail.
Further, a lot of this machinery was required for pre-vmemmap support.
Now we only support the vmemmap with memory hotplug.
So the whole machinery is useless today. Let's start by removing the last
pieces by first stopping to mark vmemmap pages as SECTION_INFO. In
free_vmemmap_pages(), we can now always just free the reserved pages
directly.
Link: https://lore.kernel.org/20260716-bootmem_info_part2-v2-5-4afc76c73d61@kernel.org
Signed-off-by: David Hildenbrand (Arm) <david@kernel.org>
Reviewed-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Reviewed-by: Dave Hansen <dave.hansen@linux.intel.com>
Cc: Alexander Gordeev <agordeev@linux.ibm.com>
Cc: Andy Lutomirski <luto@kernel.org>
Cc: "Borislav Petkov (AMD)" <bp@alien8.de>
Cc: Brendan Jackman <jackmanb@google.com>
Cc: Brendan Jackman <brendan.jackman@linux.dev>
Cc: Christian Borntraeger <borntraeger@linux.ibm.com>
Cc: Gerald Schaefer <gerald.schaefer@linux.ibm.com>
Cc: Heiko Carstens <hca@linux.ibm.com>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Muchun Song <muchun.song@linux.dev>
Cc: Oscar Salvador <osalvador@suse.de>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Sven Schnelle <svens@linux.ibm.com>
Cc: Vasily Gorbik <gor@linux.ibm.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Zi Yan <ziy@nvidia.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Let's extend free_reserved_page() in page_alloc.c to
free_reserved_pages(), dropping the custom x86 variant. The common-code
variant will consume an order, so adjust the x86 callers accordingly.
Make free_reserved_pages() assume that we are freeing ordinary high-order
pages, just with the special "reserved" flavor. The target use case for
now is freeing vmemmap PMD pages.
Set the refcount directly to 0 (instead of 1) and call
__free_frozen_pages(). Set the page count to 0 before clearing
PG_reserved, so someone checking PG_reserved (and not finding it set) to
then try grabbing a ref would not suddenly have that ref be dropped. That
is arguably cleaner and safer than the old way of doing it.
Add some kerneldoc. Use a single adjust_managed_page_count() call.
Link: https://lore.kernel.org/20260716-bootmem_info_part2-v2-2-4afc76c73d61@kernel.org
Signed-off-by: David Hildenbrand (Arm) <david@kernel.org>
Reviewed-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Reviewed-by: Muchun Song <muchun.song@linux.dev>
Reviewed-by: Zi Yan <ziy@nvidia.com>
Reviewed-by: Dave Hansen <dave.hansen@linux.intel.com>
Cc: Alexander Gordeev <agordeev@linux.ibm.com>
Cc: Andy Lutomirski <luto@kernel.org>
Cc: "Borislav Petkov (AMD)" <bp@alien8.de>
Cc: Brendan Jackman <jackmanb@google.com>
Cc: Brendan Jackman <brendan.jackman@linux.dev>
Cc: Christian Borntraeger <borntraeger@linux.ibm.com>
Cc: Gerald Schaefer <gerald.schaefer@linux.ibm.com>
Cc: Heiko Carstens <hca@linux.ibm.com>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Oscar Salvador <osalvador@suse.de>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Sven Schnelle <svens@linux.ibm.com>
Cc: Vasily Gorbik <gor@linux.ibm.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Patch series "mm: remove CONFIG_HAVE_BOOTMEM_INFO_NODE (Part 2)", v2.
Let's remove the remaining pieces of CONFIG_HAVE_BOOTMEM_INFO_NODE,
performing some smaller cleanups around freeing of reserved vmemmap pages
on the way.
This patch (of 10):
All callers pass 0, so let's drop the parameter.
Link: https://lore.kernel.org/20260716-bootmem_info_part2-v2-0-4afc76c73d61@kernel.org
Link: https://lore.kernel.org/20260716-bootmem_info_part2-v2-1-4afc76c73d61@kernel.org
Signed-off-by: David Hildenbrand (Arm) <david@kernel.org>
Reviewed-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Reviewed-by: Muchun Song <muchun.song@linux.dev>
Reviewed-by: Zi Yan <ziy@nvidia.com>
Reviewed-by: Dave Hansen <dave.hansen@linux.intel.com>
Cc: Alexander Gordeev <agordeev@linux.ibm.com>
Cc: Andy Lutomirski <luto@kernel.org>
Cc: "Borislav Petkov (AMD)" <bp@alien8.de>
Cc: Brendan Jackman <jackmanb@google.com>
Cc: Brendan Jackman <brendan.jackman@linux.dev>
Cc: Christian Borntraeger <borntraeger@linux.ibm.com>
Cc: Gerald Schaefer <gerald.schaefer@linux.ibm.com>
Cc: Heiko Carstens <hca@linux.ibm.com>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Oscar Salvador <osalvador@suse.de>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Sven Schnelle <svens@linux.ibm.com>
Cc: Vasily Gorbik <gor@linux.ibm.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Userfaultfd RWP will reuse the uffd-wp PTE bit to mark access-tracking
PTEs, alongside the write-protected ones it already marks. The bit's
meaning now depends on the VMA flag (WP or RWP), not on its name.
Rename the kernel-internal names that describe the bit:
- pte/pmd/huge_pte accessors (and swap variants)
- pgtable_supports_uffd() capability query
- SCAN_PTE_UFFD khugepaged enum
The ftrace string emitted by mm_khugepaged_scan_pmd for this enum is
kept as "pte_uffd_wp" so existing trace-based tooling keeps matching.
Pure mechanical rename -- no behavior change.
Link: https://lore.kernel.org/20260708111417.173443-4-kirill@shutemov.name
Signed-off-by: Kiryl Shutsemau <kas@kernel.org>
Assisted-by: Claude:claude-opus-4-6
Reviewed-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Reviewed-by: SeongJae Park <sj@kernel.org>
Cc: Andrea Arcangeli <aarcange@redhat.com>
Cc: David Hildenbrand <david@kernel.org>
Cc: James Houghton <jthoughton@google.com>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Liam Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: Peter Xu <peterx@redhat.com>
Cc: Sean Christopherson <seanjc@google.com>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Zi Yan <ziy@nvidia.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
The uffd-wp PTE bit is about to gain a second consumer: userfaultfd RWP
will use the same bit to mark access-tracking PTEs, distinct from
mprotect(PROT_NONE) or NUMA-hinting PTEs. WP vs RWP semantics come from
the VMA flag; the bit is just "uffd has claimed this entry." Drop the
"_wp" suffix from the arch-private bit macros so they reflect that.
x86: _PAGE_BIT_UFFD_WP -> _PAGE_BIT_UFFD
_PAGE_UFFD_WP -> _PAGE_UFFD
_PAGE_SWP_UFFD_WP -> _PAGE_SWP_UFFD
arm64: PTE_UFFD_WP -> PTE_UFFD
PTE_SWP_UFFD_WP -> PTE_SWP_UFFD
riscv: _PAGE_UFFD_WP -> _PAGE_UFFD
_PAGE_SWP_UFFD_WP -> _PAGE_SWP_UFFD
Pure mechanical rename -- no behavior change.
Link: https://lore.kernel.org/20260708111417.173443-3-kirill@shutemov.name
Signed-off-by: Kiryl Shutsemau <kas@kernel.org>
Assisted-by: Claude:claude-opus-4-6
Reviewed-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Reviewed-by: SeongJae Park <sj@kernel.org>
Cc: Andrea Arcangeli <aarcange@redhat.com>
Cc: David Hildenbrand <david@kernel.org>
Cc: James Houghton <jthoughton@google.com>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Liam Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: Peter Xu <peterx@redhat.com>
Cc: Sean Christopherson <seanjc@google.com>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Zi Yan <ziy@nvidia.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Patch series "userfaultfd: working set tracking for VM guest memory", v10.
This series adds userfaultfd support for tracking the working set of VM
guest memory, so a VMM can identify hot pages and reclaim cold ones to
tiered or remote storage.
This patch (of 15):
pte_protnone() and pmd_protnone() detect present-but-inaccessible page
table entries. This capability is useful beyond NUMA balancing -- for
example, userfaultfd working set tracking uses protnone PTEs to track page
access without unmapping pages.
Introduce CONFIG_ARCH_HAS_PTE_PROTNONE to decouple the protnone PTE
infrastructure from CONFIG_NUMA_BALANCING. The six architectures that
support protnone PTEs (x86_64, arm64, powerpc, s390, riscv, loongarch) now
select this option, and CONFIG_NUMA_BALANCING depends on it.
No functional change -- the same set of architectures continues to have
working protnone support, but the infrastructure is now available
independently of NUMA balancing.
Link: https://lore.kernel.org/20260708111417.173443-1-kirill@shutemov.name
Link: https://lore.kernel.org/20260708111417.173443-2-kirill@shutemov.name
Signed-off-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
Assisted-by: Claude:claude-opus-4-6
Acked-by: SeongJae Park <sj@kernel.org>
Acked-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Cc: Andrea Arcangeli <aarcange@redhat.com>
Cc: David Hildenbrand <david@kernel.org>
Cc: James Houghton <jthoughton@google.com>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Liam Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: Peter Xu <peterx@redhat.com>
Cc: Sean Christopherson <seanjc@google.com>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Zi Yan <ziy@nvidia.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
I hit the following on one of my machines:
mce: CPU0 BANK15 CMCI inherited storm
------------[ cut here ]------------
ODEBUG: assert_init not available (active state 0) object: (____ptrval____) object type: timer_list hint: 0x0
WARNING: lib/debugobjects.c:632 at debug_object_assert_init+0x178/0x230, CPU#0: swapper/0/0
CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Not tainted 7.2.0-rc5 #3 PREEMPTLAZY
RIP: 0010:debug_object_assert_init+0x18f/0x230
Call Trace:
<TASK>
__mod_timer
mce_timer_kick
cmci_discover
intel_init_cmci
mce_intel_feature_init
mcheck_cpu_init
identify_cpu
identify_boot_cpu
arch_cpu_finalize_init
start_kernel
A second splat follows right after, from timer_setup() finding that same
timer already queued:
ODEBUG: init active (active state 0) object: (____ptrval____) object type: timer_list hint: stub_timer+0x0/0x10
This is happening because CMCI storm detection is trying to modify the timer
before latter was properly set up.
Set up the timer first. __mcheck_cpu_setup_timer() only calls timer_setup(),
and depends on neither the generic nor the vendor init.
[ bp: Massage commit message. ]
Fixes: 1f68ce2a02 ("x86/mce: Handle Intel threshold interrupt storms")
Signed-off-by: Breno Leitao <leitao@debian.org>
Signed-off-by: Borislav Petkov (AMD) <bp@alien8.de>
Cc: stable@vger.kernel.org
Link: https://patch.msgid.link/20260803-mce_timer_init-v1-1-9539db424330@debian.org
The KVM clock is an interesting thing. It is defined as "nanoseconds
since the guest was created", but in practice it runs at two *different*
rates — or three different rates, if you count implementation bugs.
Definition A is that it runs synchronously with the CLOCK_MONOTONIC_RAW
of the host, with a delta of kvm->arch.kvmclock_offset.
But that version doesn't actually get used in the common case, where the
host has a reliable TSC and the guest TSCs are all running at the same
rate and in sync with each other, and kvm->arch.use_master_clock is set.
In that common case, definition B is used: There is a reference point in
time at kvm->arch.master_kernel_ns (again a CLOCK_MONOTONIC_RAW time),
and a corresponding host TSC value kvm->arch.master_cycle_now. This
fixed point in time is converted to guest units (the time offset by
kvmclock_offset and the TSC Value scaled and offset to be a guest TSC
value) and advertised to the guest in the pvclock structure. While in
this 'use_master_clock' mode, the fixed point in time never needs to be
changed, and the clock runs precisely in time with the guest TSC, at the
rate advertised in the pvclock structure.
The third definition C is implemented in kvm_get_wall_clock_epoch() and
__get_kvmclock(), using the master_cycle_now and master_kernel_ns fields
but converting the *host* TSC cycles directly to a value in nanoseconds
instead of scaling via the guest TSC.
One might naïvely think that all three definitions are identical, since
CLOCK_MONOTONIC_RAW is not skewed by NTP frequency corrections; all
three are just the result of counting the host TSC at a known frequency,
or the scaled guest TSC at a known precise fraction of the host's
frequency. The problem is with arithmetic precision, and the way that
frequency scaling is done in a division-free way by multiplying by a
scale factor, then shifting right. In practice, all three ways of
calculating the KVM clock will suffer a systemic drift from each other.
Eventually, definition C should just be eliminated. Commit 451a707813
("KVM: x86/xen: improve accuracy of Xen timers") worked around it for
the specific case of Xen timers, which are defined in terms of the KVM
clock and suffered from a continually increasing error in timer expiry
times. That commit notes that get_kvmclock_ns() is non-trivial to fix
and says "I'll come back to that", which remains true.
Definitions A and B do need to coexist, the former to handle the case
where the host or guest TSC is suboptimally configured. But KVM should
be more careful about switching between them, and the discontinuity in
guest time which could result.
In particular, KVM_REQ_MASTERCLOCK_UPDATE will take a new snapshot of
time as the reference in master_kernel_ns and master_cycle_now, yanking
the guest's clock back to match definition A at that moment.
When invoked from in 'use_master_clock' mode, kvm_update_masterclock()
should probably *adjust* kvm->arch.kvmclock_offset to account for the
drift, instead of yanking the clock back to definition A. But in the
meantime there are a bunch of places where it just doesn't need to be
invoked at all.
To start with: there is no need to do such an update when a Xen guest
populates the shared_info page. This seems to have been a hangover from
the very first implementation of shared_info which automatically
populated the vcpu_info structures at their default locations, but even
then it should just have raised KVM_REQ_CLOCK_UPDATE on each vCPU
instead of using KVM_REQ_MASTERCLOCK_UPDATE. And now that userspace is
expected to explicitly set the vcpu_info even in its default locations,
there's not even any need for that either.
Fixes: 629b534884 ("KVM: x86/xen: update wallclock region")
Reviewed-by: Paul Durrant <paul@xen.org>
Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
Signed-off-by: Sean Christopherson <seanjc@google.com>
In query_edd(), use bool for do_edd and initialize it directly from
CONFIG_EDD_OFF via IS_ENABLED(). Also use bool for do_mbr and be_quiet.
No change in functionality intended.
Signed-off-by: Thorsten Blum <thorsten.blum@linux.dev>
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Link: https://patch.msgid.link/20260804115227.62158-5-thorsten.blum@linux.dev
- Fix the boot-time memcmp() asm implementation's constraints
and optimization properties (Mauricio Faria de Oliveira)
- Move the 0xd0...0xd7 AMD Zen5 model range from the Zen6
range where it mistakenly ended up (Pratik Vishwakarma)
Signed-off-by: Ingo Molnar <mingo@kernel.org>
-----BEGIN PGP SIGNATURE-----
iQJFBAABCgAvFiEEBpT5eoXrXCwVQwEKEnMQ0APhK1gFAmpu9+0RHG1pbmdvQGtl
cm5lbC5vcmcACgkQEnMQ0APhK1j8Ig//WBoOGw9bbFkDJFUdIS59EdJS+4l7KTJh
eRxiExnwQluOpo1x4/mKAMmgqz7IoFCc0PjLXIBC3+rN+AvM5+L/qkBGtHmcaHEu
uXdU5zAYk6a7QfvK9C6k1C/v4HQKb3FLvUliJShKfLAv7ekLLvVHWxExmJYoBsrn
VxPIW3+rpnBJn4v5bLUVs/Z6h99aB4nicA4i6p7WKiy+Vr1qOSIu5T4piff27FJE
btT61yOfTKFoKgum5dI4ZYJTX29edXcJUMPidN/egJAxMQnUdI38fO/joYMreYsx
rJ3zuv3mdV+LiusPO7ZHcNVZtc09PPMM2krLrOyjSEHzhE/WGqLXNn6FKK5H+j86
xKyawqhD+2ANktj+09bFfu3cpZEz5bdifyKx6BtvgYkSEVaX055E1zzqYXZCIaVs
DqdAs3aSKNWAbJkfMONJMZ4P28ZGikppeefihG6hcAnBjAyQC9SaTOA37fHT6WQ/
bDf3NU5c+TX0TrWd696E9jy5cLjXL60jbVPL7SF7C3d8/QPPWCbOU14Dw3s4bWh+
WfptSuB9Iuk+aalRnAVWpUBI9DDUcomC3oDLLHttztXojdZPvP1zdUhDzjaJ0Im8
N8xC30PBMGIrFGJpXN4XRM3BlSxXBVGrmS/WAqE/KE51RcA5WzQBFMGBe4V3gRbO
OV3slKkCD9k=
=EHql
-----END PGP SIGNATURE-----
Merge tag 'x86-urgent-2026-08-02' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
Pull misc x86 fixes from Ingo Molnar:
- Fix the boot-time memcmp() asm implementation's constraints
and optimization properties (Mauricio Faria de Oliveira)
- Move the 0xd0...0xd7 AMD Zen5 model range from the Zen6
range where it mistakenly ended up (Pratik Vishwakarma)
* tag 'x86-urgent-2026-08-02' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip:
x86/CPU/AMD: Carve out a Zen5 models range
x86/boot: Add volatile, clobbers and zero-length test in memcmp()
Sort it according to the following precedence of struct cpuid_bit fields:
(level) -> (sub_leaf) -> (reg) -> (bit).
No functional changes.
Signed-off-by: Borislav Petkov (AMD) <bp@alien8.de>
btf_distill_func_proto() accepts a function argument up to 16 bytes, so a
128-bit scalar such as __int128 reaches the x86 trampoline with
arg_size == 16. But the current implementation assumes an __int128
argument only needs one register, so the register save area is
under-allocated and save_args() overwrites adjacent stack slots.
Compute the register count from arg_size for all arguments to fix it.
Fixes: a9c5ad31fb ("bpf: x86: Support in-register struct arguments in trampoline programs")
Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
Acked-by: Leon Hwang <leon.hwang@linux.dev>
Link: https://lore.kernel.org/bpf/20260729050204.2586457-1-yonghong.song@linux.dev
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
vCPU creation in kvm_vm_ioctl_create_vcpu() may fail after
kvm_arch_vcpu_create() -> vmx_vcpu_create() already succeeded. In such
case kvm_vm_ioctl_create_vcpu() destroys the newly created vCPU in the
failure path. However, that leaves a side effect: the IPIv pid_table
entry remains configured with this vCPU's pi_desc address. As a result,
when another vCPU sends an IPI to the APIC ID of this failed-to-create
vCPU, it will cause HW to write to this (freed!) pi_desc memory. [*]
Fix this by clearing the pid_table entry when destroying the vCPU.
Note that the same issue exists for SVM AVIC as well [1], to be fixed.
[*] Although, since this memory is freed into the kvm_vcpu_cache kmem
cache which is only used for allocating kvm_vcpus, _maybe_ this
memory will only be reused for pi_desc of another vCPU, not for
anything else. So _maybe_ this will only result in delivering the
IPI to a wrong vCPU (possibly of another VM) in the worst case, not
in a random corruption of kernel memory.
Link: https://lore.kernel.org/kvm/al4rNqpBYy8FGKPw@blrnaveerao1 [1]
Signed-off-by: Dmytro Maluka <dmaluka@chromium.org>
Reviewed-by: Kai Huang <kai.huang@intel.com>
Link: https://patch.msgid.link/20260729170621.308809-3-dmaluka@chromium.org
Signed-off-by: Sean Christopherson <seanjc@google.com>
On early boot CPU detection, use parsed CPUID(0x1) instead of a direct CPUID
query.
Beside the parser's centralization benefits, use the auto generated CPUID data
types, and their C99 bitfields, instead of doing ugly bitwise operations.
[ bp: - Use a common code pattern for l1 of fetching and checking it
right after that
- flip the check to save an indentation level
- align assignments vertically.
]
Signed-off-by: Ahmed S. Darwish <darwi@linutronix.de>
Signed-off-by: Borislav Petkov (AMD) <bp@alien8.de>
Link: https://patch.msgid.link/20260528153923.403473-12-darwi@linutronix.de
Rework .vcpu_pre_run() into a more specific .vcpu_needs_initialization() to
consolidate the SNP and TDX control flows, and to eliminate the potentially
confusing almost-collision between svm_vcpu_pre_run() and pre_sev_run(): the
former is SEV specific, but is pre-KVM_RUN, whereas the latter is pre-VMRUN.
Link: https://patch.msgid.link/20260731173340.2644656-4-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
Synthesize SHUTDOWN (for L1) if L2 requires unhandleable emulation after
loading guest state from SMRAM during RSM to prevent a misbehaving L1 (or
userspace via L1) from tripping the sanity check that KVM doesn't try to
cancel a pending nested VM-Enter. If SMRAM is modified such that RSM will
load what should be impossible state for L2, then KVM will detect that it
needs to emulate the current code stream and will abort VM-Entry to L2.
And because KVM (rightly) expects such a scenario to be impossible, KVM
WARNs and bugs the VM.
__ret && !(vcpu->kvm)->vm_bugged
WARNING: arch/x86/kvm/vmx/vmx.c:6741 at vmx_handle_exit+0x65/0x790 [kvm_intel], CPU#13: vmx_invalid_nes/2902
Modules linked in: kvm_intel kvm irqbypass [last unloaded: kvm]
CPU: 13 UID: 1000 PID: 2902 Comm: vmx_invalid_nes Tainted: G W 7.2.0-rc2 #124 PREEMPT
Tainted: [W]=WARN
Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
RIP: 0010:vmx_handle_exit+0x65/0x790 [kvm_intel]
Call Trace:
<TASK>
kvm_arch_vcpu_ioctl_run+0xdf8/0x1d00 [kvm]
kvm_vcpu_ioctl+0x2d5/0x960 [kvm]
__x64_sys_ioctl+0x8a/0xd0
do_syscall_64+0xb7/0x570
entry_SYSCALL_64_after_hwframe+0x4b/0x53
</TASK>
Alternatively, KVM could suppress the WARN for the RSM case, but that would
still leave the vCPU in a "bad" state that KVM doesn't know how to handle
(which is also why KVM rejects attempts to do KVM_RUN when the vCPU is
loaded with invalid state). And architecturally, the Intel SDM explicitly
states that RSM leads to shutdown if the CPU detects invalid state.
Fixes: 2bb8cafea8 ("KVM: vVMX: signal failure for nested VMEntry if emulation_required")
Reported-by: Hao Zhang <zhanghao1@kylinos.cn>
Link: https://patch.msgid.link/20260731173340.2644656-3-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
Expose VMX's check for unhandleable emulation as its own kvm_x86_ops hook,
and move the actual pre-KVM_RUN check into common x86. This will allow
sharing the core logic with KVM's RSM emulation without needed to add a
post-RSM hook, and is a step towards removing the .vcpu_pre_run() hook
entirely.
Alternatively, KVM could provide a post-RSM hook as mentioned, but pre/post
hooks tend to be unwieldy as the exact "timing" of the call often matters
greatly. E.g. in this case, the call must slot in exactly between loading
guest state from SMRAM and the hack to force the vCPU out of L2 on SHUTDOWN.
Link: https://patch.msgid.link/20260731173340.2644656-2-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
The x86 library provides x86_family() and x86_model(). They take raw
CPUID register output and calculate the CPU family and model from it.
In follow-up work, the x86 subsystem will use APIs which access
previously-parsed CPUID leafs structure instead of doing direct CPUID queries.
These new APIs force using the auto generated leaf data types at
<asm/cpuid/leaf_types.h>.
Introduce x86 family and model calculation functions that take these
auto-generated data types. Refactor the original code so that no logic is
duplicated.
[ bp: Massage commit message, unbreak too long line. ]
Signed-off-by: Ahmed S. Darwish <darwi@linutronix.de>
Signed-off-by: Borislav Petkov (AMD) <bp@alien8.de>
Link: https://patch.msgid.link/20260528153923.403473-11-darwi@linutronix.de
Due to a widespread Intel erratum (e.g. EMR158), programming the
VMX-preemption timer with certain large values may cause the timer to
expire earlier than expected. The recommended workaround is to cap the
VMX-preemption timer value to strictly less than:
2^25 * CPUID.15H:EBX[31:0] / CPUID.15H:EAX[31:0].
Calculate the maximum "safe" preemption timer value during hardware setup
based on CPUID 15H when available, and use the adjusted max value in all
locations where KVM currently hardcodes the max architectural value,
including in the subtle case where KVM soft-disables the timer.
Don't apply the workaround when running as a VM, because absent explicit
enumeration to state the bug is present (or not), it's L0's responsibility
to faithfully emulate/virtualize the VMX preemption timer.
WARN if the above logic would result in a max value of zero and fall back
to the maximum architectural value, as the expectation is that real
hardware will never provide problematic EAX/EBX values (which is another
reason to ignore the erratum when running as a VM; there's less chance of
a false positive on the WARN due to L0 providing an unanticipated ratio).
Reported-by: Sean Christopherson <seanjc@google.com>
Closes: https://lore.kernel.org/all/Zn9X0yFxZi_Mrlnt@google.com/
Suggested-by: Chao Gao <chao.gao@intel.com>
Assisted-by: Gemini:Gemini-Next
Reviewed-by: Chao Gao <chao.gao@intel.com>
Signed-off-by: Jim Mattson <jmattson@google.com>
Reviewed-by: Binbin Wu <binbin.wu@linux.intel.com>
[sean: track inclusive max instead of exclusive limit, massage changelog]
Link: https://patch.msgid.link/20260731171926.2629627-3-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
Double down on using the VMX preemption timer only for 64-bit kernels, and
bury the setup and runtime adjustment code, and all global variables, under
CONFIG_X86_64=y. This will allow addressing a widespread Intel erratum
without running afoul of unused-but-set-variable and __udivdi3() errors on
32-bit kernels.
No functional change intended.
Reviewed-by: Binbin Wu <binbin.wu@linux.intel.com>
Reviewed-by: Chao Gao <chao.gao@intel.com>
Link: https://patch.msgid.link/20260731171926.2629627-2-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
Use parsed CPUID(0x0) instead of a direct CPUID query.
[ bp: Drop the unnecessary NULL check. ]
Signed-off-by: Ahmed S. Darwish <darwi@linutronix.de>
Signed-off-by: Borislav Petkov (AMD) <bp@alien8.de>
Link: https://patch.msgid.link/20260528153923.403473-10-darwi@linutronix.de
@lost and @topa_index are no longer part of struct pt_buffer so
delete their comment lines.
Warning: ../arch/x86/events/intel/pt.h:86 Excess struct member 'lost' description in 'pt_buffer'
Warning: ../arch/x86/events/intel/pt.h:86 Excess struct member 'topa_index' description in 'pt_buffer'
Signed-off-by: Randy Dunlap <rdunlap@infradead.org>
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Link: https://patch.msgid.link/20260730233429.285788-2-rdunlap@infradead.org
Add a short description for imr_self_test() and drop the comments for
the non-existent imr_self_test_exit() to avoid kernel-doc warnings:
Warning: arch/x86/platform/intel-quark/imr_selftest.c:52 missing initial short description on line:
* imr_self_test
Warning: ../arch/x86/platform/intel-quark/imr_selftest.c:128 function parameter 'imr_self_test_init' not described in 'device_initcall'
Warning: ../arch/x86/platform/intel-quark/imr_selftest.c:128 expecting prototype for imr_self_test_exit(). Prototype was for device_initcall() instead
Signed-off-by: Randy Dunlap <rdunlap@infradead.org>
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Link: https://patch.msgid.link/20260730233429.285788-6-rdunlap@infradead.org
These function parameters don't exist so drop them to avoid kernel-doc
warnings:
Warning: ../arch/x86/include/asm/spec-ctrl.h:28 Excess function parameter 'guest_spec_ctrl' description in 'x86_spec_ctrl_set_guest'
Warning: ../arch/x86/include/asm/spec-ctrl.h:42 Excess function parameter 'guest_spec_ctrl' description in 'x86_spec_ctrl_restore_host'
Signed-off-by: Randy Dunlap <rdunlap@infradead.org>
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Link: https://patch.msgid.link/20260730233429.285788-5-rdunlap@infradead.org
- correct missing function parameter names
- add missing function return value sections
- use the correct function name in comments
to avoid these warnings (samples):
Warning: arch/x86/include/asm/checksum_64.h:23 No description found for return value of 'csum_fold'
Warning: arch/x86/include/asm/checksum_64.h:46 function parameter 'iph' not described in 'ip_fast_csum'
Warning: arch/x86/include/asm/checksum_64.h:46 function parameter 'ihl' not described in 'ip_fast_csum'
Warning: arch/x86/include/asm/checksum_64.h:46 No description found for return value of 'ip_fast_csum'
Warning: arch/x86/include/asm/checksum_64.h:89 expecting prototype for csum_tcpup_nofold(). Prototype was for csum_tcpudp_nofold() instead
Warning: arch/x86/include/asm/checksum_64.h:115 expecting prototype for csum_tcpup_magic(). Prototype was for csum_tcpudp_magic() instead
Warning: arch/x86/include/asm/checksum_64.h:119 No description found for return value of 'csum_tcpudp_magic'
Warning: arch/x86/include/asm/checksum_64.h:129 No description found for return value of 'csum_partial'
Warning: ./arch/x86/include/asm/checksum_64.h:175 function parameter '_saddr' not described in 'csum_ipv6_magic'
Warning: ./arch/x86/include/asm/checksum_64.h:175 function parameter '_daddr' not described in 'csum_ipv6_magic'
Warning: ./arch/x86/include/asm/checksum_64.h:175 Excess function parameter 'saddr' description in 'csum_ipv6_magic'
Warning: ./arch/x86/include/asm/checksum_64.h:175 Excess function parameter 'daddr' description in 'csum_ipv6_magic'
Signed-off-by: Randy Dunlap <rdunlap@infradead.org>
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Link: https://patch.msgid.link/20260730233429.285788-4-rdunlap@infradead.org
Use the correct macro parameter names in the kernel-doc comments
to avoid kernel-doc warnings:
Warning: arch/x86/include/asm/barrier.h:35 function parameter 'idx' not described in 'array_index_mask_nospec'
Warning: arch/x86/include/asm/barrier.h:35 function parameter 'sz' not described in 'array_index_mask_nospec'
Warning: arch/x86/include/asm/barrier.h:35 Excess function parameter 'index' description in 'array_index_mask_nospec'
Warning: arch/x86/include/asm/barrier.h:35 Excess function parameter 'size' description in 'array_index_mask_nospec'
Signed-off-by: Randy Dunlap <rdunlap@infradead.org>
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Link: https://patch.msgid.link/20260730233429.285788-3-rdunlap@infradead.org
Teach dma_direct_map_phys() to select the DMA address encoding based on
DMA_ATTR_CC_SHARED.
Use phys_to_dma_unencrypted() for decrypted mappings and
phys_to_dma_encrypted() otherwise. If a device requires unencrypted DMA
but the source physical address is still encrypted, force the mapping
through swiotlb so the DMA address and backing memory attributes remain
consistent.
Update the arm64, x86, s390 and powerpc secure-guest setup to not use
swiotlb force option
Tested-by: Jiri Pirko <jiri@nvidia.com>
Tested-by: Michael Kelley <mhklinux@outlook.com>
Tested-by: Mostafa Saleh <smostafa@google.com>
Signed-off-by: Aneesh Kumar K.V (Arm) <aneesh.kumar@kernel.org>
Link: https://lore.kernel.org/r/20260717180442.110954-18-aneesh.kumar@kernel.org
[mszyprow: rebased onto latest changes in arch/arm64/mm/init.c]
Signed-off-by: Marek Szyprowski <m.szyprowski@samsung.com>
Teach dma_capable() about DMA_ATTR_CC_SHARED so the capability
check can reject encrypted DMA addresses for devices that require
unencrypted/shared DMA.
Also propagate DMA_ATTR_CC_SHARED in swiotlb_map() when the selected
SWIOTLB pool is decrypted so the capability check sees the correct DMA
address attribute.
Reviewed-by: Jason Gunthorpe <jgg@nvidia.com>
Tested-by: Jiri Pirko <jiri@nvidia.com>
Tested-by: Michael Kelley <mhklinux@outlook.com>
Tested-by: Mostafa Saleh <smostafa@google.com>
Reviewed-by: Petr Tesarik <ptesarik@suse.com>
Signed-off-by: Aneesh Kumar K.V (Arm) <aneesh.kumar@kernel.org>
Link: https://lore.kernel.org/r/20260717180442.110954-16-aneesh.kumar@kernel.org
Signed-off-by: Marek Szyprowski <m.szyprowski@samsung.com>
The difference between __alloc_pages_node() and alloc_pages_node() is
that the latter allows you to pass NUMA_NO_NODE.
The former is going away and the latter works fine here so switch over.
No functional change intended.
Link: https://lore.kernel.org/20260703-alloc-trylock-v5-10-c87b714e19d3@google.com
Signed-off-by: Brendan Jackman <jackmanb@google.com>
Reviewed-by: Suren Baghdasaryan <surenb@google.com>
Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Reviewed-by: Zi Yan <ziy@nvidia.com>
Cc: Thomas Gleixner <tglx@kernel.org>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Dave Hansen <dave.hansen@linux.intel.com>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Assisted-by: Gemini:unknown-version
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
The difference between __alloc_pages_node() and alloc_pages_node() is
that the latter allows you to pass NUMA_NO_NODE.
The former is going away and the latter works fine here so switch over.
No functional change intended.
Link: https://lore.kernel.org/20260703-alloc-trylock-v5-9-c87b714e19d3@google.com
Signed-off-by: Brendan Jackman <jackmanb@google.com>
Reviewed-by: Suren Baghdasaryan <surenb@google.com>
Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Reviewed-by: Zi Yan <ziy@nvidia.com>
Cc: Sean Christopherson <seanjc@google.com>
Cc: Paolo Bonzini <pbonzini@redhat.com>
Assisted-by: Gemini:unknown-version
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
The difference between __alloc_pages_node() and alloc_pages_node() is that
the latter allows you to pass NUMA_NO_NODE.
The former is going away and the latter works fine here so switch over.
No functional change intended.
Link: https://lore.kernel.org/20260703-alloc-trylock-v5-8-c87b714e19d3@google.com
Signed-off-by: Brendan Jackman <jackmanb@google.com>
Reviewed-by: Suren Baghdasaryan <surenb@google.com>
Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Reviewed-by: Zi Yan <ziy@nvidia.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Arnaldo Carvalho de Melo <acme@kernel.org>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Ian Rogers <irogers@google.com>
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: James Clark <james.clark@linaro.org>
Assisted-by: Gemini:unknown-version
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
CONFIG_ARCH_ENABLE_THP_MIGRATION gates PMD-level migration entries.
PMD-level device-private entries use the same migration mechanism and
therefore require the same architecture support.
Upcoming PMD-level swap entries can use the same PMD softleaf helpers
without depending on page migration, so rename the architecture gate to
CONFIG_ARCH_HAS_PMD_SOFTLEAVES. This describes the PMD entry capability
rather than one current user of it.
This is a pure rename: the set of selecting architectures (x86, arm64,
s390, riscv, loongarch, and powerpc on PPC_BOOK3S_64) and the gating
semantics are unchanged.
No functional change intended.
Link: https://lore.kernel.org/20260706114320.1643046-7-usama.arif@linux.dev
Signed-off-by: Usama Arif <usama.arif@linux.dev>
Reviewed-by: Zi Yan <ziy@nvidia.com>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Cc: Alexandre Ghiti <alex@ghiti.fr>
Cc: Baolin Wang <baolin.wang@linux.alibaba.com>
Cc: Baoquan He <baoquan.he@linux.dev>
Cc: Barry Song <baohua@kernel.org>
Cc: Chris Li <chrisl@kernel.org>
Cc: Dev Jain <dev.jain@arm.com>
Cc: "Huang, Ying" <ying.huang@linux.alibaba.com>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Kairui Song <kasong@tencent.com>
Cc: Kemeng Shi <shikemeng@huaweicloud.com>
Cc: Kiryl Shutsemau <kas@kernel.org>
Cc: Lance Yang <lance.yang@linux.dev>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Matthew Wilcox (Oracle) <willy@infradead.org>
Cc: Nhat Pham <nphamcs@gmail.com>
Cc: Nico Pache <npache@redhat.com>
Cc: Rik van Riel <riel@surriel.com>
Cc: Ryan Roberts <ryan.roberts@arm.com>
Cc: Shakeel Butt <shakeel.butt@linux.dev>
Cc: Vlastimil Babka <vbabka@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
As noted in commit 85a2d4a890 ("x86,ibt: Use UDB instead of 0xEA") Jcc should
be assumed not-taken, however the normal kCFI (ABI) emits the following sequence:
movl $(-hash), %r10d
addl -15(%r11), %r10d
je 1f
ud2
1: cs call __x86_indirect_thunk_r11
(when used in conjunction with -mretpoline-external-thunk).
Notably, the Jcc here is always taken, resulting in lower throughput than would
be ideal. Replace it with the following sequence on boot:
movl $(-hash), %r10d
addl -15(%r11), %r10d
jne . + 3
test $0xd6, %al
cs call __x86_indirect_thunk_r11
This jumps to the UDB instruction used as an immediate byte in the test
instruction. The test instruction will clobber eflags, but that is immaterial,
eflags is already changed by the preceding addl.
Intel recommends the FineIBT sequence on platforms that support IBT; older
platforms are still widely used and would benefit from this.
An earlier PoC was benchmarked by Scott:
Indirect branch miss rate (br_misp_retired.indirect:k / br_inst_retired.indirect:k)
BHI_DIS_S=1
Benchmark Baseline IBT kCFI kCFI-opt
-----------------------------------------------------------------------------
iperf3 UDP 0.103764 0.103180 0.104311 0.102945
hackbench 0.000885 0.000876 0.001996 0.000826
lmbench syscall 0.005089 0.004486 0.016990 0.005852
lmbench fork+exit 0.018454 0.019176 0.031085 0.015153
lmbench fork+exec 0.017147 0.021613 0.029129 0.016337
redis 0.032220 0.032655 0.045540 0.027946
nginx+wrk 0.109033 0.112765 0.132557 0.102417
fio randread 0.009704 0.009620 0.008548 0.000962
fio seqwrite 0.006927 0.006707 0.019372 0.004590
kbuild 0.056748 0.057324 0.064640 0.048136
BHI_DIS_S=0
Benchmark Baseline IBT kCFI kCFI-opt
-----------------------------------------------------------------------------
iperf3 UDP 0.000077 0.000106 0.000186 0.000073
hackbench 0.000123 0.000132 0.000367 0.000097
lmbench syscall 0.023259 0.018319 0.040903 0.012772
lmbench fork+exit 0.011494 0.011887 0.029079 0.016415
lmbench fork+exec 0.037782 0.038994 0.055378 0.026381
redis 0.002481 0.003152 0.017073 0.000184
nginx+wrk 0.015478 0.016266 0.033637 0.000268
fio randread 0.009836 0.007949 0.007096 0.000143
fio seqwrite 0.014587 0.014165 0.041792 0.002157
kbuild 0.055774 0.055249 0.062590 0.046546
Suggested-by: Scott D Constable <scott.d.constable@intel.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Acked-by: Borislav Petkov (AMD) <bp@alien8.de>
Tested-by: Nathan Chancellor <nathan@kernel.org>
Link: https://patch.msgid.link/20260612071506.GQ187714%40noisy.programming.kicks-ass.net
Always update x2APIC MSR intercepts for L1 when AVIC is deactivated, even
if L2 is active and KVM is using a separate MSR bitmap to run L2. If AVIC
is fully enabled prior to running L2, and is then inhibited while L2 is
active (for a VM-scoped inhibit), then KVM will run L1 with AVIC disabled,
but with x2APIC MSR intercepts disabled, i.e. will allow L1 to read most of
the host's APIC state, send arbitrary interrupts, change task priority, and
ultimately trivially DoS the host.
E.g. sending a self-IPI in L1 on HYPERV_REENLIGHTENMENT_VECTOR, 0xee, with
CONFIG_HYPERV=n in the host kernel as a "safe" PoC, yields:
Spurious interrupt (vector 0xee) on CPU#425. Acked
And hacking KVM to abuse kvm_set_posted_intr_wakeup_handler() to register a
handler and WARN on POSTED_INTR_WAKEUP_VECTOR yields:
------------[ cut here ]------------
WARNING: arch/x86/kvm/svm/svm.c:5594 at pi_wakeup_handler+0x9/0x10 [kvm_amd], CPU#156: nested_x2apic_t/316940
CPU: 156 UID: 0 PID: 316940 Comm: nested_x2apic_t Tainted: G S U
Tainted: [S]=CPU_OUT_OF_SPEC, [U]=USER
Hardware name: Google Astoria-Turin/astoria, BIOS 0.20260209.0-0 02/09/2026
RIP: 0010:pi_wakeup_handler+0x9/0x10 [kvm_amd]
Call Trace:
<IRQ>
sysvec_kvm_posted_intr_wakeup_ipi+0x64/0x80
</IRQ>
<TASK>
asm_sysvec_kvm_posted_intr_wakeup_ipi+0x1a/0x20
RIP: 0010:vcpu_run+0x1430/0x1e40 [kvm]
kvm_arch_vcpu_ioctl_run+0x2c1/0x600 [kvm]
kvm_vcpu_ioctl+0x580/0x6b0 [kvm]
__se_sys_ioctl+0x6d/0xb0
do_syscall_64+0x10a/0x480
entry_SYSCALL_64_after_hwframe+0x4b/0x53
RIP: 0033:0x46ff4b
</TASK>
---[ end trace 0000000000000000 ]---
Fixes: 091abbf578 ("KVM: x86: nSVM: optimize svm_set_x2apic_msr_interception")
Cc: stable@vger.kernel.org
Cc: Yosry Ahmed <yosry@kernel.org>
Signed-off-by: Sean Christopherson <seanjc@google.com>
Link: https://patch.msgid.link/20260729213558.639074-1-pbonzini@redhat.com/
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Family 0x1a, model 0xd0..0xd7 belongs to the Zen5 generation. Carve it
out from the larger, Zen6 range where former doesn't belong.
[ bp: Rewrite commit message, add tags. ]
Fixes: b5f53e6d3d ("x86/CPU/AMD: Add more Zen6 models")
Signed-off-by: Pratik Vishwakarma <Pratik.Vishwakarma@amd.com>
Signed-off-by: Borislav Petkov (AMD) <bp@alien8.de>
Cc: <stable@kernel.org>
Link: https://patch.msgid.link/20260729055459.15904-1-Pratik.Vishwakarma@amd.com
Use CMPXCHG instead of clear_bit(), which currently emits a LOCK BTR since
the to-be-cleared bit isn't a compile-time constant, when aging SPTEs in
the shadow MMU to align with the approach taken by the TDP MMU, and because
using CMPXCHG is far more robust against bugs in KVM. E.g. if the SPTE is
somehow no longer an SPTE due to a KVM bug, CMPXCHG will fail gracefully,
whereas clear_bit() would potentially corrupt/clobber memory.
Clearing the Accessed bit without atomically ensuring the SPTE is still the
old SPTE is "fine", as holding the rmap's lock ensures zapping the old SPTE
can't fully complete, which in turn ensures a new, different SPTE can't be
installed. But that chain of logic isn't exactly obvious, and there's zero
reason to avoid CMPXCHG as its cost on modern hardware is within ~1-2 uops
of LOCK BTR (and may even be cheaper on some microarchitectures). Doing a
64-bit CMPXCHG on 32-bit kernels does require a more expensive CMPXCHG8B,
but 32-bit KVM is all but dead at this point.
Cc: James Houghton <jthoughton@google.com>
Reviewed-by: Kai Huang <kai.huang@intel.com>
Reviewed-by: James Houghton <jthoughton@google.com>
Link: https://patch.msgid.link/20260728002236.869865-3-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
Use LOCK CMPXCHG instead of LOCK AND to clear the Accessed bit when aging
SPTEs in the TDP MMU, as doing a LOCK AND can corrupt a FROZEN SPTE and
allow a third CPU to effectively overwrite the FROZEN SPTE. As pointed
out by AI of some kind, because the magic FROZEN_SPTE value is a "full"
SPTE, not a single bit, and includes the Accessed bit, clearing the
Accessed bit in a FROZEN SPTE will result in is_frozen_spte() getting a
false negative.
E.g. if CPU0 freezes an SPTE, and CPU1 clears the Accessed bit in the
frozen SPTE, then CPU2 could come along and overwrite the frozen SPTE with
a shadow-present SPTE.
Thankfully, the false negative is largely benign, because outside of TDX,
which doesn't support aging, KVM only freezes leaf SPTEs when removing an
upper level shadow page. So while KVM could clobber a frozen SPTE back to
a shadow-present SPTE, and could even use the new SPTE, the subsequent TLB
flush will make the orphaned, shadow-present SPTE unreachable. Failure to
ever zap the orphaned leaf SPTE would show up in KVM's stats, but otherwise
is benign (because KVM no longer keeps an elevated refcount for leaf SPTEs).
Opportunistically add a comment to warn future developers away from using
kvm_tdp_mmu_write_spte_atomic() and tdp_mmu_clear_spte_bits_atomic(), as
they are generally unsafe. Keep the helpers, e.g. instead of open-coding
the atomic64_fetch_and() in tdp_mmu_clear_spte_bits(), as scary warnings
usually are more effective deterrent against recidivism than removal of the
dangerous code.
Alternatively, KVM could use different bits for the magic FROZEN_SPTE value,
e.g. setting the Dirty bits (with effective IPAT and Global aliases) would
likely be "ok", as IPAT/Global are extremely unlikely to be cleared without
doing a full SPTE write, and KVM's clearing of Dirty bits shares logic with
Write-Protection, which must do a full SPTE write (via cmpxchg64() in the
TDP MMU) to ensure KVM isn't clobbering state. But there is zero reason to
carry that risk (beyond stubbornness in wanting to preserve a "cute" idea),
as the cost of LOCK CMPXCHG and LOCK AND are within 1-2 uops of each other
on modern hardware.
Fixes: b146a9b34a ("KVM: x86/mmu: Age TDP MMU SPTEs without holding mmu_lock")
Cc: stable@vger.kernel.org
Reviewed-by: Kai Huang <kai.huang@intel.com>
Reviewed-by: James Houghton <jthoughton@google.com>
Link: https://patch.msgid.link/20260728002236.869865-2-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
Add the volatile qualifier and clobbers parameter to prevent bugs with
instruction reordering and optimization.
Also add TEST for the zero-length case to set ZF, as, if the count register
is zero, the REPE prefix does not run the CMPSB instruction, leaving the ZF
flag undetermined.
[ bp: Add a comment about the len==0 case. ]
Fixes: 62bd0337d0 ("Top header file for new x86 setup code")
Closes: https://sashiko.dev/#/patchset/20260701-pvh-kasan-inline-v6-0-ba99045dfa9f%40igalia.com
Suggested-by: Borislav Petkov <bp@alien8.de>
Signed-off-by: Mauricio Faria de Oliveira <mfo@igalia.com>
Signed-off-by: Borislav Petkov (AMD) <bp@alien8.de>
Link: https://lore.kernel.org/all/20260721-pvh-kasan-inline-v7-2-38979a50cef0@igalia.com
Convert the kvm_vcpu_map_readonly() usage in nVMX's MSR bitmap merging to
the new CLASS(kvm_vcpu_map_local_readonly) implementation, to eliminate the
last of the open-coded on-stack "struct kvm_host_map" declarations (in x86,
PPC still has one more to convert).
No functional change intended.
Link: https://patch.msgid.link/20260724004757.131420-5-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
Convert the kvm_vcpu_map() usage in the enter/leave SMM flows to the new
CLASS(kvm_vcpu_map_local) implementations, to eliminate the need to
manually do unmap() in error paths, and more importantly to eliminate more
of the open-coded on-stack "struct kvm_host_map" declarations.
No functional change intended.
Link: https://patch.msgid.link/20260724004757.131420-4-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
Add CLASS() definitions for locally mapping a PFN using kvm_vcpu_map()
given a vCPU+gfn pair. In addition to eliminating the need to manually
do unmap(), e.g. in error paths, this will allow hardening KVM against
double-mapping without having to manually ensure every on-stack declaration
is zero-initialized.
Use "map local" as the primary terminology as the basic concept is more or
less the same as kmap_local(): ensure the current context has a kernel
mapping to the underlying memory.
Immediately convert the relatively straightforward nested SVM flows, and
defer converting the more involved SMM flows to a separate change.
No functional change intended.
Cc: Yosry Ahmed <yosry@kernel.org>
Link: https://patch.msgid.link/20260724004757.131420-3-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
Always check and clear KVM_REQ_GET_NESTED_STATE_PAGES when emulating a
nested VM-Exit to ensure the request is cleared, even when KVM was built
with CONFIG_KVM_HYPERV=n, as KVM subtly relies on the "check" to clear
the flag and thus avoid double-mapping the vmcs12 pages, e.g. if KVM
manages to bail from VM-Enter without processing the request, and then
emulates VMLAUNCH or VMRESUME.
Fixes: b4f69df0f6 ("KVM: x86: Make Hyper-V emulation optional")
Cc: stable@vger.kernel.org
Reported-by: Yosry Ahmed <yosry@kernel.org>
Reviewed-by: Yosry Ahmed <yosry@kernel.org>
Link: https://patch.msgid.link/20260724004757.131420-2-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
The metric table needs to be delivered to userspace as a single
atomic snapshot, but the current sysfs metrics_bin path is a file
read: userspace can read it in chunks and observe a torn snapshot
if an SMU refresh happens between read() calls. The same path is
also bounded by PAGE_SIZE, so the ~13 KB table used by HSMP protocol
version 7 on Family 1Ah Model 50h-5Fh cannot be returned at all,
regardless of how userspace reads it. Rather than extend sysfs to
lift both restrictions, expose the metric table through the
existing HSMP character device using a new ioctl that always copies
the table in one shot.
Add struct hsmp_telemetry_data and HSMP_IOCTL_GET_TELEMETRY_DATA
to the UAPI header. Under the surrounding #pragma pack(4), placing
the __u64 user pointer first gives a tight 16-byte layout that is
identical for 32- and 64-bit callers, and the trailing __u16
reserved field is rejected with -EINVAL if non-zero so future
kernels can repurpose it without breaking already-deployed
userspace. The command is encoded with _IOW because the kernel only
reads the request struct; the snapshot travels through the user
pointer it carries.
The requested size may be anything from one byte up to the size
firmware reported for that socket's table. A short request returns
the leading bytes of the snapshot, so userspace built against an
older table layout keeps working on firmware that grew the table,
mirroring the relaxed response_sz rule applied to HSMP messages
earlier in this series. A request larger than the firmware table is
rejected with -EINVAL rather than short-written, so a caller can
never mistake a partial copy for a full one.
Dispatch hsmp_ioctl() on the ioctl command: the existing message
handler is factored out as hsmp_ioctl_msg() for HSMP_IOCTL_CMD, and
HSMP_IOCTL_GET_TELEMETRY_DATA goes to a new
hsmp_ioctl_get_telemetry() helper.
/dev/hsmp is a singleton character device that outlives an
individual socket unbind, so an ioctl issued on an already-open fd
can run concurrently with socket teardown. hsmp_sock_rwsem is the
driver's contract for that: the data plane takes it for read, and
probe and remove take it for write to drain the data plane before
freeing the socket array, unmapping the metric tables and
destroying the per-socket mutexes. hsmp_ioctl_get_telemetry() takes
it for read across the socket lookup, the checks on that socket's
metric-table state and the table read itself, so none of that state
can be torn down underneath it. Without this the handler would
sleep in its kvmalloc() holding no lock at all, and could resume
with a freed socket, locking a destroyed mutex and reading from an
unmapped iomem region.
The lock is dropped before the copy_to_user(), because faulting in
the destination can block indefinitely on a userfaultfd-backed
buffer and would otherwise leave a socket unbind waiting for the
write lock.
Since hsmp_metric_tbl_read() reached the mailbox through
hsmp_send_message(), which takes hsmp_sock_rwsem itself, calling it
with the lock already held would recursively take the read side and
can deadlock against a queued writer. Split out
hsmp_metric_tbl_read_locked(), which asserts the lock and uses
hsmp_send_message_locked(), and leave hsmp_metric_tbl_read() as a
wrapper that takes the read lock for the sysfs callers. This also
brings the whole fill-and-copy under the rwsem for those callers,
where the memcpy_fromio() previously ran outside it, and makes the
lock order uniformly hsmp_sock_rwsem -> metric_read_lock ->
hsmp_sem.
The user-controlled socket index in HSMP_IOCTL_GET_TELEMETRY_DATA is
clamped with array_index_nospec() before indexing hsmp_pdev.sock[],
mitigating Spectre v1 (CVE-2017-5753). Include linux/nospec.h, which
the file relied on getting transitively.
Co-developed-by: Muthusamy Ramalingam <muthusamy.ramalingam@amd.com>
Signed-off-by: Muthusamy Ramalingam <muthusamy.ramalingam@amd.com>
Signed-off-by: Muralidhara M K <muralidhara.mk@amd.com>
Link: https://patch.msgid.link/20260727141542.3370108-5-muralidhara.mk@amd.com
Reviewed-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
The driver hard-codes the metric-table region size to
sizeof(struct hsmp_metric_table). That is correct for HSMP protocol
version 6 but mis-sizes the ioremap of the SMU DRAM region on newer
platforms: Family 1Ah Model 50h-5Fh exposes a ~13 KB table under
protocol version 7, and the table is expected to keep growing on
future firmware. The same hard-coded value also forces
hsmp_metric_tbl_read() to reject any read that follows the actual
firmware layout.
Pick up the table size from firmware instead. SMU on Family 1Ah
Model 50h and later populates HSMP_GET_METRIC_TABLE_DRAM_ADDR's
args[2] with the DRAM region size in bytes; older firmware leaves it
0. Bump the descriptor's response_sz to 3 so the field is read, and
store the result in the new per-socket hsmp_socket.metric_tbl_size,
which is then used both for the ioremap() of the region and as the
expected size in hsmp_metric_tbl_read().
The size is stored per socket rather than per platform because
hsmp_get_tbl_dram_base() runs once per socket and each socket maps
its own region. A single platform-wide field would let the last
socket's size be used to copy out of an earlier socket's smaller
mapping.
Bump DRIVER_VERSION to 2.6.
Behaviour on existing protocol-version-6 hardware is unchanged.
Reading a third response word is safe there: for this command SMU
leaves args[2] as 0 rather than a stale value from an earlier
mailbox transaction, so the fallback always applies, yielding the
same value as the previous hard-coded one, and both the ioremap and
the size check produce the same result as before.
Co-developed-by: Muthusamy Ramalingam <muthusamy.ramalingam@amd.com>
Signed-off-by: Muthusamy Ramalingam <muthusamy.ramalingam@amd.com>
Signed-off-by: Muralidhara M K <muralidhara.mk@amd.com>
Link: https://patch.msgid.link/20260727141542.3370108-4-muralidhara.mk@amd.com
Reviewed-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
Family 1Ah Model 50h-5Fh firmware exposes new HSMP messages (0x29-0x2A,
0x33-0x3A) for PC6/CC6 control, CCD power/thermal monitoring, DIMM
sideband access, floor- and SDPS-limit control, and command-enable
discovery. The same firmware extends three existing SET-only messages
(HSMP_SET_XGMI_LINK_WIDTH 0x0C, HSMP_SET_DF_PSTATE 0x0D,
HSMP_SET_PSTATE_MAX_MIN 0x22) with a read-back path selected by bit[31]
of args[0] (0 = set, 1 = get). Add the new IDs and convert the three
messages to HSMP_SET_GET.
Also add PQoS-related HSMP messages HSMP_PQOS_TRAFFIC_PRIORITY (0x3B)
and HSMP_PQOS_FLOATING_BW (0x3C) with matching hsmp_msg_desc_table[]
descriptors so userspace can reach the new functionality.
Backward compatibility is preserved on prior platforms: new IDs
previously occupied HSMP_RSVD slots, and existing userspace that leaves
bit[31] = 0 continues to take a pure SET path. Converting the three
SET messages to HSMP_SET_GET also keeps them accepted by
validate_message(), which already applies a relaxed upper-bound check
on response_sz for that type.
Co-developed-by: Muthusamy Ramalingam <muthusamy.ramalingam@amd.com>
Signed-off-by: Muthusamy Ramalingam <muthusamy.ramalingam@amd.com>
Signed-off-by: Muralidhara M K <muralidhara.mk@amd.com>
Link: https://patch.msgid.link/20260727141542.3370108-2-muralidhara.mk@amd.com
Reviewed-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
mmu_try_to_unsync_pages() skips the shadow-page lookup when the
supplied memslot allows a hugepage, because a shadow page would disallow
hugepages. But hugepage metadata is per-address-space while shadow pages
are shared across all address spaces. With SMM, the other address space
can therefore have a shadow page even when the supplied memslot allows a
hugepage.
Check the corresponding memslot in the other address space before
taking the fast path. Skip the shadow-page lookup only when all address
spaces allow a hugepage.
Fixes: b3ae3ceb55 ("KVM: x86/mmu: KVM: x86/mmu: Skip unsync when large pages are allowed")
Assisted-by: Codex:GPT-5
Signed-off-by: Jinu Kim <kimjw04271234@gmail.com>
[invert direction of the conditional. - Paolo]
Message-ID: <20260721103512.2136240-3-kimjw04271234@gmail.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
kvm_gfn_is_write_tracked() checks only the supplied memslot, but page
tracking is per-address-space and shadow pages are shared across all
address spaces. With SMM, a GFN can therefore be write-tracked in one
address space and appear untracked through the other.
Check the supplied slot first, then the slot for the other address space.
This ensures all callers honor write tracking regardless of the active
address space. In particular, it prevents mmu_try_to_unsync_pages() from
marking an upper-level shadow page unsync and eventually triggering the
BUG in pte_list_remove().
Fixes: 699023e239 ("KVM: x86: add SMM to the MMU role, support SMRAM address space")
Assisted-by: Codex:GPT-5
Signed-off-by: Jinu Kim <kimjw04271234@gmail.com>
Message-ID: <20260721103512.2136240-2-kimjw04271234@gmail.com>
[invert direction of the conditional. - Paolo]
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Cancel (and flush) the I/O APIC's delayed EOI handling work during the
"pre VM destroy" phase, before vCPUs are destroyed, as processing the EOI
broadcast will inject another IRQ if the line is asserted, i.e. will try
to deliver an IRQ to the target vCPU(s). Canceling the work after vCPUs
are destroyed leads to UAF if the delayed work is processed after vCPUs are
destroyed.
BUG: KASAN: slab-use-after-free in __kvm_irq_delivery_to_apic_fast+0x9bf/0xa20 arch/x86/kvm/lapic.c:1250
Read of size 8 at addr ffff8880499abea0 by task kworker/1:2/1218
CPU: 1 UID: 0 PID: 1218 Comm: kworker/1:2 Not tainted 7.1.0-rc7 #5 PREEMPT(lazy)
Hardware name: QEMU Ubuntu 25.10 PC v2 (i440FX + PIIX, + 10.1 machine, 1996), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
Workqueue: events kvm_ioapic_eoi_inject_work
Call Trace:
<TASK>
__dump_stack lib/dump_stack.c:94
dump_stack_lvl+0x100/0x190 lib/dump_stack.c:120
print_address_description mm/kasan/report.c:378
print_report+0x139/0x4ad mm/kasan/report.c:482
kasan_report+0xe4/0x1d0 mm/kasan/report.c:595
__kvm_irq_delivery_to_apic_fast+0x9bf/0xa20 arch/x86/kvm/lapic.c:1250
__kvm_irq_delivery_to_apic+0xd8/0xbf0 arch/x86/kvm/lapic.c:1345
kvm_irq_delivery_to_apic arch/x86/kvm/lapic.h:129
ioapic_service+0x308/0x590 arch/x86/kvm/ioapic.c:492
kvm_ioapic_eoi_inject_work+0x13c/0x190 arch/x86/kvm/ioapic.c:532
process_one_work+0xa59/0x19a0 kernel/workqueue.c:3314
process_scheduled_works kernel/workqueue.c:3397
worker_thread+0x5eb/0xe50 kernel/workqueue.c:3478
kthread+0x370/0x450 kernel/kthread.c:436
ret_from_fork+0x72b/0xd30 arch/x86/kernel/process.c:158
ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:245
</TASK>
Note, the VM is unreachable once kvm_destroy_vm() starts, and scheduling
new work via kvm_ioapic_send_eoi() can only be done via KVM_RUN, i.e.
requires a live vCPU.
Alternatively, KVM could simply destroy the I/O APIC during the "pre" phase
of VM destruction, but that gets more than a bit sketchy as KVM expects the
I/O APIC to exist if ioapic_in_kernel() is true, and nested virtualization
in particular has a bad habit of touching VM-scope state during vCPU
destruction. E.g. attempting to free the PIC during the pre phase would
lead to a NULL pointer dereference in kvm_cpu_has_extint(), and it's not
hard to imagine the I/O APIC having a similar flaw.
Fixes: 17bcd71442 ("KVM: x86: Free vCPUs before freeing VM state")
Reported-by: <zdi-disclosures@trendmicro.com>
Reported-by: Zhong Wang <wangzhong.c0ss4ck@bytedance.com>
Reported-by: Xuanqing Shi <shixuanqing.11@bytedance.com>
Cc: stable@vger.kernel.org
Signed-off-by: Weiming Shi <bestswngs@gmail.com>
Co-developed-by: Sean Christopherson <seanjc@google.com>
Signed-off-by: Sean Christopherson <seanjc@google.com>
Message-ID: <20260727171718.543491-1-seanjc@google.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
VMCLEAR/VMREAD/VMWRITE/VMPTRLD access the internal VMCS cache, which
is not visible to the compiler; without a memory clobber, the compiler
can reorder them in troublesome ways because "asm volatile" and "asm goto"
only protect against removal of the asm. For example, placing a VMWRITE
before the corresponding VMCS pointer is loaded can lead to corruption.
While none of this has been observed, it is better to prevent than cure.
Likewise, INVEPT and INVVPID access the TLB and, even though in their
case the effect is only visible to the next VMLAUNCH/VMRESUME, it is
technically correct to add the clobber there too. So avoid any urge to
special case them, and simply hardcode "memory" into the clobber list
of vmx_asm1() and vmx_asm2(). __vmcs_readl() open-codes its own asm,
so add the clobber there as well.
Link: https://lore.kernel.org/kvm/CABgObfbL3t21yVeSwiLSjjOUER+rTYDPHYAH9YU4TWGRjx6XHg@mail.gmail.com/
Cc: Sean Christopherson <seanjc@google.com>
Cc: stable@vger.kernel.org
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Futex hash computation requires a mask operation with read-only after
init data that will be converted to a runtime constant in the subsequent
commit.
Introduce runtime_const_mask_32 to further optimize the mask operation
in the futex hash computation hot path.
[ prateek: Broke off the x86 chunk, commit message. ]
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Signed-off-by: K Prateek Nayak <kprateek.nayak@amd.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/20260227161841.GH606826@noisy.programming.kicks-ass.net
Link: https://patch.msgid.link/20260728052540.4728-2-kprateek.nayak@amd.com
Transmeta CPUs allow masking CPUID(0x1).EDX feature flags via MSR writes. If
a bit is cleared in the 0x80860004 MSR, its corresponding feature flag is not
reported by CPUID.
Refresh the CPUID parser's CPUID(0x1) cache after all MSR bits are unmasked.
Note, the MSR 0x80860004 semantics are documented in the "BIOS Programmer's
Guide: Transmeta Crusoe Processor" document, dated June 14, 2002.
Signed-off-by: Ahmed S. Darwish <darwi@linutronix.de>
Signed-off-by: Borislav Petkov (AMD) <bp@alien8.de>
Link: https://patch.msgid.link/20260528153923.403473-9-darwi@linutronix.de
When getting a to-be-injected IRQ, don't WARN if the IRQ disappeared and
Xen emulation is supported, as a guest could concurrently toggle its
evtchn_upcall_pending flag in shared memory and deassert the IRQ. Even
more annoyingly, userspace could disable Xen emulation for the entire VM
KVM_XEN_HVM_CONFIG. So, suppress WARNs on lost IRQs if Xen emulation is
supported to prevent false positives.
Alternatively, KVM could track if the VM has ever used Xen emulation, but
the added complexity isn't worth carrying given that the vast majority of
deployments can and should disable Xen emulation.
Fixes: bf672720e8 ("KVM: x86: check the kvm_cpu_get_interrupt result before using it")
Reported-by: Sashiko Bot <sashiko-bot@kernel.org>
Closes: https://lore.kernel.org/all/20260625212001.3B6561F000E9@smtp.kernel.org
Link: https://patch.msgid.link/20260724173425.278753-3-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
When getting a to-be-injected IRQ, don't WARN if the IRQ disappeared and
the VM has an in-kernel PIC, as the ExtINT handling that's routed through
KVM's virtual PIC is tracked per-VM, not per-vCPU. If another vCPU grabs
the IRQ, or deasserts the interrupt (which is level-triggered), then it's
both expected and "fine" for a
Keep the assert for split IRQCHIP VMs to help detect KVM bugs, as userspace
is responsible for routing ExtINT to the intended vCPU, i.e. once an ExtINT
is pending, it can't be cleared without holding the vCPU's mutex, and thus
false positives are impossible.
Fixes: bf672720e8 ("KVM: x86: check the kvm_cpu_get_interrupt result before using it")
Debugged-by: Alexander Potapenko <glider@google.com>
Reported-by: syzbot+dd769db18693736eee89@syzkaller.appspotmail.com
Closes: https://syzkaller.appspot.com/bug?extid=dd769db18693736eee89
Closes: https://lore.kernel.org/all/6a360fdf.871e809a.2d6dda.0000.GAE@google.com
Link: https://patch.msgid.link/20260724173425.278753-2-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
The default case in __apic_accept_irq() has carried a printk("TODO:
unsupported delivery mode") since the original LAPIC emulation was
introduced in commit 97222cc831 ("KVM: Emulate local APIC in
kernel").
The switch now handles all eight delivery modes defined by the x86
architecture and is always constrained to the three-bit field
defined by the architecture: Either by masking with APIC_MODE_MASK,
by three-bit bitfield widths in the IOAPIC and MSI structs, or by
using APIC_DM_* constants directly.
Replace the unreachable printk with WARN_ON_ONCE(1) to match the
existing pattern for impossible defaults elsewhere in the same file.
Signed-off-by: Tim Wiederhake <twiederh@redhat.com>
Link: https://patch.msgid.link/20260715120341.2661873-1-twiederh@redhat.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
Tweak the guest_memfd make_private() hook to explicitly pass the number of
pages to align with the signature of the make_shared() hook, and because
the existing code is outright broken if a guest_memfd folio is comprised of
more than one page (which can't happen, yet). The SNP code *tries* to
create a corresponding huge entry, but if the RMP must use 4KiB entries for
whatever reason, KVM will only convert the first pfn, and not the entire
range of pfns that will be mapped into the guest.
Alternatively, @max_order could simply be repurposed as _the_ @order, but
that will fall apart when in-place conversion comes along, at which point
KVM will need to deal with conversions that aren't bound 1:1 to a folio.
I.e. the number of pages to convert may not be exactly a power-of-2 (and
folios don't strictly guarantee power-of-2 pages anyways).
WARN in the SNP code if the number of pages to prepare is anything other
than '1', i.e. if guest_memfd is trying to prepare/convert more than a
single 4KiB page, as sev_gmem_prepare() doesn't actually handle conversion
greater than order-0 folios.
Opportunistically swap the ordering of @pfn and @gfn params for
kvm_x86_ops.gmem_make_private() to match kvm_arch_gmem_make_private().
Fixes: b85524314a ("KVM: guest_memfd: delay kvm_gmem_prepare_folio() until the memory is passed to the guest")
Reviewed-by: Xiaoyao Li <xiaoyao.li@intel.com>
Reviewed-by: Ackerley Tng <ackerleytng@google.com>
Link: https://patch.msgid.link/20260723210811.72720-9-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
Rework guest_memfd's prepare() hook into a more accurate make_private(),
and rework its Kconfig from PREPARE to a more generic CONVERT. This will
allow x86 to share (pun intended) a kvm_x86_ops.gmem_make_shared() hook
between the "convert to shared" and "reclaim" flows, which are one and the
same for SNP.
No functional change intended.
Reviewed-by: Ackerley Tng <ackerleytng@google.com>
Reviewed-by: Fuad Tabba <fuad.tabba@linux.dev>
Link: https://patch.msgid.link/20260723210811.72720-8-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
Rename kvm_x86_ops's gmem_invalidate() hook to gmem_make_shared(), as the
hook doesn't invalidate anything, and so that KVM doesn't need to add yet
another vendor callback to support "convert to shared" once in-place
conversion comes along.
Opportunistically wrap the ops declarations with a GMEM_RECLAIM guard so
that attempting to wire up a .gmem_make_shared() hook without selecting
CONFIG_HAVE_KVM_ARCH_GMEM_RECLAIM will result in a build failure.
No functional change intended.
Reviewed-by: Xiaoyao Li <xiaoyao.li@intel.com>
Reviewed-by: Ackerley Tng <ackerleytng@google.com>
Reviewed-by: Fuad Tabba <fuad.tabba@linux.dev>
Link: https://patch.msgid.link/20260723210811.72720-4-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
Rename guest_memfd's invalidate() hook to reclaim() and isolate it via its
own RECLAIM Kconfig, as the hook is called when a folio is freed, which is
far too late and lacks sufficient information for KVM to actually
invalidate its usage of the memory. E.g. SNP uses the hook to convert
memory back to SHARED so that it can be safely accessed by the host, there
is no invalidation of guest mappings anywhere. Isolating the hook will
also allow pKVM on arm64 to opt-in to reclaim() without also having to
differentiate between reclaim and conversions to shared for active VMs.
Keep guest_memfd's trampoline, even though it would be trivial to wire up
.free_folio() directly to an arch callback, to avoid bleeding guest_memfd
internals into arch code (specifically, avoid referencing folios in arch
code).
Leave the kvm_x86_ops hook as-is for the moment, as "reclaim" on SNP is
the same as convert-to-shared, i.e. using a different name for the x86 hook
will allow reusing it for in-place conversion.
Reviewed-by: Xiaoyao Li <xiaoyao.li@intel.com>
Reviewed-by: Fuad Tabba <fuad.tabba@linux.dev>
Reviewed-by: Ackerley Tng <ackerleytng@google.com>
Link: https://patch.msgid.link/20260723210811.72720-3-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
Pass the number of pages to "invalidate", i.e. reclaim, instead of the end
pfn, as a first step towards aligning the function prototypes between the
de facto "to private" and "to shared" arch hooks. Eventually, the goal
is to end up with kvm_gmem_arch_make_{private,shared}(), and in both cases,
providing the number of pages makes the call sites slightly nicer, and also
avoids any confusion over whether the end pfn is inclusive or exclusive.
Opportunistically rename "start" to "pfn", again to align with the expected
signature of make_private() (which needs to pass a starting gfn as well, at
which point the "start" becomes noise).
No functional change intended.
Cc: Fuad Tabba <fuad.tabba@linux.dev>
Cc: Ackerley Tng <ackerleytng@google.com>
Reviewed-by: Xiaoyao Li <xiaoyao.li@intel.com>
Reviewed-by: Ackerley Tng <ackerleytng@google.com>
Reviewed-by: Fuad Tabba <fuad.tabba@linux.dev>
Link: https://patch.msgid.link/20260723210811.72720-2-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
Wire up the fchroot() system call as number 472 on (nearly) all
architectures and sync the mirrored copies of the syscall tables and
the asm-generic unistd.h under tools/.
Link: https://patch.msgid.link/20260724-work-failfs-v2-5-485dabbae185@kernel.org
Signed-off-by: Christian Brauner (Amutable) <brauner@kernel.org>
These are the remaining patches for legacy gpio interface
cleanup, ensuring that files referencing them are sure to
have CONFIG_GPIOLIB_LEGACY set, and use linux/gpio/legacy.h
instead of the old linux/gpio.h header.
There are a few more patches already merged in other
branches for 7.3, including soc, led, net and media. Once
everything is in, we can apply the final patches to turn
off CONFIG_GPIOLIB_LEGACY by default and remove the obsolete
header file.
This branch is based on the immutable ib-mfd-legacy-gpio-7.3
branch from mfd.git to avoid a conflict in the rohm drivers.
-----BEGIN PGP SIGNATURE-----
iQIzBAABCgAdFiEEo6/YBQwIrVS28WGKmmx57+YAGNkFAmpjrCwACgkQmmx57+YA
GNlhuA//eRNxgyDFRUtuRchUUnIjpCzPdZOXTvnTmDTVYIsHaKS6J/KTBZcxRG7g
tA749mgpQ2tT0hZFhT1XHCaxHwwBfmV0lomIO65pNcmqmEAUODxTwEy3t4ON4loi
Ee0SqEXhI+NhMbEoQkrpcCk5y+89Xo+iABbEv12UL/V/K0PKfRDCRs2ZbARnE0yu
L9wiqt2TJn3baHMabxBZJSWXrYpHCaPqm+vQgIhELQZ4TGswfi6OvCEPhUkk1QUa
mVvWH7LtUi1A0Ya1HvAhgncvboY6LZYjDHMw7meQmUOSTIu7XoQ6+XjEJ6LJSgtH
rLAdrFUHFezoHd0UaklnB3taEa7KOZIljNIY8IGRMSqv7jYF8fYXTPuxnS9fuj2o
u+GxDriWYLp1DfH7HlkrVbmVeX6zunH8qYx91z5aRl7YHNKfwg0wRv8mxAgeBTJy
1HO/su+ZT0X21hvVibiQNy08VWiQ5+9F5DV+SPTKtaT8chGjjx6uw7oyZkgvzWyu
xdHuwOzH9WAhL7WnGcJ7PkJzWDoLVkl3KzRCCaYMXBCyLRwCIJTfC5CH7hJNh41i
fW8b5Nfh7DRAhEuapQk87/NE/sRIFKZW2kfyzbCzsJHm+TW/5X+O6OwoJ+KUSdWQ
SqVJOVUKDWjg0S8+jxu0mr1ikzwKgrFv2gzWFxaffb6Oi0rqBc4=
=Ybrr
-----END PGP SIGNATURE-----
Merge tag 'config-gpio-legacy-for-7.3' of https://git.kernel.org/pub/scm/linux/kernel/git/arnd/playground into gpio/for-next
gpio: legacy interface cleanups for 7.3
These are the remaining patches for legacy gpio interface
cleanup, ensuring that files referencing them are sure to
have CONFIG_GPIOLIB_LEGACY set, and use linux/gpio/legacy.h
instead of the old linux/gpio.h header.
There are a few more patches already merged in other
branches for 7.3, including soc, led, net and media. Once
everything is in, we can apply the final patches to turn
off CONFIG_GPIOLIB_LEGACY by default and remove the obsolete
header file.
This branch is based on the immutable ib-mfd-legacy-gpio-7.3
branch from mfd.git to avoid a conflict in the rohm drivers.
a bit more widely, because newer versions of LLVM started
optimizing it a bit better and introduced run-time relocations
in PIE code. (Nathan Chancellor)
Signed-off-by: Ingo Molnar <mingo@kernel.org>
-----BEGIN PGP SIGNATURE-----
iQJFBAABCgAvFiEEBpT5eoXrXCwVQwEKEnMQ0APhK1gFAmpl0uARHG1pbmdvQGtl
cm5lbC5vcmcACgkQEnMQ0APhK1hmpxAAltRv2TwPA4MeZt10iYTG4BdwMp4OleR4
hDxqYL9T1OJev4wLujlri3/Vb1MVKG1S3YQjeG6hWAB0L+//hbx2e/nEU24uW5xb
j+0kAWSSZPUIP3KRc5Cy4mb1EXiumjk3GHnwXZpRV5XA4RRoTmpexd09J5v5KNvt
/q/c8ZzVMB8p8S1K98BdL/wMAyEHLg+iSywTdNsa8Zyw8qB8qAlRgLHTlhYjkcnO
E1+g3MY7QDEjMYDgHAMP1GHvtXTktXfp2Neo4s0WLmY/C6LEyWE/+HIDrMVuzjat
AU5IWbE0tDxGYdeyVvVoxRafFWmVg7mfnazkORp6iojbfJeVBB+IkNAVX0fbKEBT
Yx6Gcm/x0Xb2pf8zaUWE2q02Sz4WYpWrkcLZrLIVnCphyAK7K7V+I7ZkFP1reQR9
fr9RIsKC+6K5mWivFpOjkEzYp+D9Yp2OJyAgK0oE7F7Zd6u0+qpCWaM+VPHsYp41
wsVGTPgD5JP871o6ln38YskBct70E3BnzO5p10a5PbyRaaBCSBZC1XGCglIs7PS9
TmsZOI4Wc8dHzobUTXNdgOymfqXV/bQEITL8qRt9Ge2GUiUROFLp8IVzmJIADWKv
oqtoNo/vjthpFTXVZ3c4beACEaNu71bDbLjwjOdIb2W8NPY+Z+WeSkHDcg5Gt7qN
2cYWBUyhc3o=
=3e+9
-----END PGP SIGNATURE-----
Merge tag 'x86-urgent-2026-07-26' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
Pull x86 fix from Ingo Molnar:
- Disable jump/lookup tables in the x86 boot decompressor code
a bit more widely, because newer versions of LLVM started
optimizing it a bit better and introduced run-time relocations
in PIE code (Nathan Chancellor)
* tag 'x86-urgent-2026-07-26' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip:
x86/boot/compressed: Disable jump tables
An attacker injecting interrupts while the Safe-RET mitigation executes
on machines affected by SRSO can neutralize the safe return sequence,
potentially leading to data leakage through speculative execution.
Fixup register state as if the Safe-RET sequence executed successfully
by "emulating" it, in a manner of speaking, and avoid executing a RET
instruction after returning from the interrupt.
Co-developed-by: David Kaplan <David.Kaplan@amd.com>
Signed-off-by: David Kaplan <David.Kaplan@amd.com>
Signed-off-by: Borislav Petkov (AMD) <bp@alien8.de>
Unconditionally intercept and emulate ICEBP, i.e. INT1 #DBs, on SVM to
workaround a bug/misfeature in current AMD CPUs. AMD CPUs don't provide
information to allow software to distinguish instruction-induced #DBs (i.e.
ICEBP) from exception-induced #DBs (all others), and also don't report an
instruction length for an ICEBP-induced TASK_SWITCH.
On an intercepted TASK_SWITCH (which always has fault semantics), KVM
(any hypervisor, really) looks at the vectoring event type to decide
whether it was logically caused by a trap, and therefore whether to
advance guest RIP before entering the new task. If the guest IDT is
configured to deliver #DBs via a task gate, then the guest will see broken
behavior for ICEBP #DB because KVM doesn't have enough information to
detect and skip the ICEBP.
The typical workaround is to intercept ICEBP unconditionally and handle the
FAULT=>TRAP conversion in the hypervisor, at which point the #DB-induced
TASK_SWITCH occurs with RIP on the correct instruction boundary regardless
of whether it was instruction-induced or exception-induced.
As a bonus, intercepting ICEBP more or less aligns SVM with VMX (KVM always
intercepts #DBs on VMX, and ICEBP #DB VM-Exits on Intel have fault-like
behavior).
Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
Link: https://patch.msgid.link/e03f092dfbb7d391a6bf2797ba01e122ba080bcd.camel@infradead.org
[sean: drop selftest, reword changelog to provide more details]
Signed-off-by: Sean Christopherson <seanjc@google.com>
The OLPC GPIO controller sets up a fixed number space that is used
by at least two drivers:
arch/x86/platform/olpc/olpc-xo1-sci.c: In function 'setup_ec_sci':
arch/x86/platform/olpc/olpc-xo1-sci.c:358:13: error: implicit declaration of function 'gpio_request' [-Wimplicit-function-declaration]
358 | r = gpio_request(OLPC_GPIO_ECSCI, "OLPC-ECSCI");
| ^~~~~~~~~~~~
sound/pci/cs5535audio/cs5535audio_olpc.c: In function 'olpc_analog_input':
sound/pci/cs5535audio/cs5535audio_olpc.c:41:9: error: implicit declaration of function 'gpio_set_value'; did you mean 'gpiod_set_value'? [-Wimplicit-function-declaration]
41 | gpio_set_value(OLPC_GPIO_MIC_AC, on);
The AMD Geode platform that this is based on is now marked as
'Orphaned' in Kconfig, and it is likely that there are no XO1
users on modern kernels, but so far there is no consensus on
removing it entirely.
Select CONFIG_GPIOLIB_LEGACY for this platform and make sure the
sound driver portion cannot be compiled without this.
Acked-by: Borislav Petkov (AMD) <bp@alien8.de>
Acked-by: Takashi Iwai <tiwai@suse.de>
Reviewed-by: Bartosz Golaszewski <bartosz.golaszewski@oss.qualcomm.com>
Reviewed-by: Linus Walleij <linusw@kernel.org>
Acked-by: Dmitry Torokhov <dmitry.torokhov@gmail.com> # for input
Signed-off-by: Arnd Bergmann <arnd@arndb.de>
KVM services local TLB flushes on "full" nested VM-Exits (through
__nested_vmx_vmexit()), but not if a nested VM-Enter fails (e.g. due to
failed VMCS checks in nested_vmx_enter_non_root_mode()).
However, it is possible that KVM had queued TLB flushes that need to be
performed, even if the nested VM-Enter was not successful. For example,
if VPID is disabled for L2 (via nested_vmx_transition_tlb_flush(), or if
via the MSR load lists, as the SDM says:
If any MSR is being loaded in such a way that would architecturally
require a TLB flush, the TLBs are updated so that, after VM entry, the
logical processor will not use any translations that were cached before
the transition.
The SDM is unclear about when the TLB flush should occur, and whether or
not a failed VM entry would flush the TLB, so it is safer to always
do the TLB flush in this case.
More concretely, KVM also updates the last VPID L1 used for L2 in
nested_vmx_transition_tlb_flush() (i.e. last_vpid), even if the VM entry
ultimately fails. With the current code, KVM could miss a TLB flush if
L1 changes L2's VPID, then does a failed VM entry followed by a
successful one, as the failed VM entry would update last_vpid but not
actually flush the TLB. Servicing local TLB flushes on failed VM entries
makes sure that the TLB is always flushed when last_vpid is updated.
Fixes: 5c614b3583 ("KVM: nVMX: nested VPID emulation")
Cc: stable@vger.kernel.org
Reported-by: Sashiko <sashiko-bot@kernel.org> # Internal review
Suggested-by: Sean Christopherson <seanjc@google.com>
Signed-off-by: Yosry Ahmed <yosry@kernel.org>
Link: https://patch.msgid.link/20260722230128.1587363-1-yosry@kernel.org
Signed-off-by: Sean Christopherson <seanjc@google.com>
When {de,en}crypting memory of an SEV or SEV-ES guest on an SNP-enabled
host via a temporary buffer, allocate a full 4KiB page for the buffer to
ensure the page containing the buffer is wholly owned by KVM, i.e. won't
be concurrently allocated and accessed by other kernel code while KVM is
using the buffer to {de,en}crypt memory. On SNP-enabled platforms, when
sending SEV/SEV-ES commands that trigger firmware writes to memory, the
to-be-written page(s) must be (temporarily) assigned to Firmware (as
required by the SNP architecture, to guard against using such commands as
gadgets to attack SNP guests). See snp_map_cmd_buf_desc() and friends.
Unfortunately, transferring ownership of a page to Firmware makes the page
inaccessible to software, and thus writes generate RMP #PF violations. If
KVM uses a sub-page allocation for its temporary buffer, some other actor
in the kernel can allocate and use the other portions of the page, and thus
trigger unexpected (and seemingly spurious) RMP #PF violations due to
software attempting to access a Firmware-owned page.
BUG: unable to handle page fault for address: ffff906ae30f0300
#PF: supervisor write access in kernel mode
#PF: error_code(0x80000003) - RMP violation
PGD 6b1b80d067 P4D 6b1b80d067 PUD 100231e2063 PMD 10055a88063 PTE 80000100630f0163
SEV-SNP: PFN 0x100630f0 unassigned, dumping non-zero entries in 2M PFN region: [0x10063000 - 0x10063200]
Oops: Oops: 0003 [#1] SMP
CPU: 70 UID: 0 PID: 10658 Comm: svw_WaiterThrea Tainted: G U W O 7.1.0-smp--c22293789940-seanjc-next #1 PREEMPTLAZY
Tainted: [U]=USER, [W]=WARN, [O]=OOT_MODULE
Hardware name: Google, Inc. Arcadia_IT_80/Arcadia_IT_80, BIOS 34.86.0-102 01/25/2026
RIP: 0010:memset+0xf/0x20
Call Trace:
<TASK>
__kvmalloc_node_noprof+0x2a4/0x710
do_getxattr+0x4e/0x130
path_getxattrat+0x125/0x1b0
do_syscall_64+0x10a/0x480
entry_SYSCALL_64_after_hwframe+0x4b/0x53
RIP: 0033:0x7f3a22cb6daa
</TASK>
Modules linked in: kvm_amd kvm irqbypass vfat fat ccp k10temp sha3 libsha3 i2c_piix4 gq(O) cdc_acm xhci_pci xhci_hcd
gsmi: Log Shutdown Reason 0x03
CR2: ffff906ae30f0300
---[ end trace 0000000000000000 ]---
RIP: 0010:memset+0xf/0x20
Kernel panic - not syncing: Fatal exception
Kernel Offset: 0x39e00000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
gsmi: Log Shutdown Reason 0x02
Fixes: 4c735bf1bc ("KVM: SEV: Allocate only as many bytes as needed for temp crypt buffers")
Cc: stable@vger.kernel.org
Cc: Michael Roth <michael.roth@amd.com>
Debugged-by: Michael Roth <michael.roth@amd.com>
Link: https://patch.msgid.link/20260723211306.75397-1-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
AMD heterogeneous parts report a third core type via CPUID Fn0x80000026
EBX[31:28] (Extended CPU Topology, Core Type):
0 - Performance
1 - Efficiency
2 - Low Power
get_topology_cpu_type() only translates the first two, so on parts that ship
low-power cores the third type falls through to TOPO_CPU_TYPE_UNKNOWN.
Add TOPO_CPU_TYPE_LOW_POWER which takes care of this new type.
[ bp: Massage commit message, zap the "what" explanation. ]
Signed-off-by: Vishal Badole <Vishal.Badole@amd.com>
Signed-off-by: Borislav Petkov (AMD) <bp@alien8.de>
Link: https://patch.msgid.link/20260721154514.2506972-4-Vishal.Badole@amd.com
Replace the bare 0/1 in get_topology_cpu_type() with named constants that
mirror what the AMD APM publishes for CPUID Fn0x80000026 EBX[31:28] (Extended
CPU Topology, Core Type):
0 - Performance core
1 - Efficient core
2 - Low-power core (used by a follow-up)
No functional change.
Suggested-by: Borislav Petkov (AMD) <bp@alien8.de>
Signed-off-by: Vishal Badole <Vishal.Badole@amd.com>
Signed-off-by: Borislav Petkov (AMD) <bp@alien8.de>
Link: https://patch.msgid.link/20260721154514.2506972-3-Vishal.Badole@amd.com
Sashiko reported¹ that a confusion could ensue if a vendor-specific CPU
type 0 (performance) gets attempted to be used in a x86_cpu_id match
table. The current logic treats 0 as the wildcard X86_CPU_TYPE_ANY and
such a thing would end up matching the wrong CPUs.
This is all backwards because we started using the vendor-specific CPU
type number instead of using a Linux-defined, generic CPU type which
is agnostic.
Convert the current logic to it before users start appearing.
¹https://sashiko.dev/#/patchset/20260629094349.533301-1-Vishal.Badole%40amd.com
Signed-off-by: Borislav Petkov (AMD) <bp@alien8.de>
Signed-off-by: Vishal Badole <Vishal.Badole@amd.com>
Link: https://patch.msgid.link/20260721154514.2506972-2-Vishal.Badole@amd.com
Virtual addresses used with the AES-GCM library don't have to be in the
linear mapping. Remove the virt_addr_valid() checks from
snp_send_guest_request(), which were a workaround left over from when
this code used crypto_aead, which required scatterlists.
Link: https://lore.kernel.org/all/20251010151036.42241-2-ardb+git@google.com/
Reviewed-by: Alexey Kardashevskiy <aik@amd.com>
Tested-by: Alexey Kardashevskiy <aik@amd.com>
Tested-by: Nikunj A Dadhania <nikunj@amd.com>
Reviewed-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://patch.msgid.link/20260722025338.33354-3-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
The old AES-GCM library code is being replaced as part of an overhaul
that is adding support for all the common AES encryption modes with
consistent conventions. The SEV code is the only user of the old
AES-GCM library. Update it to use the new API instead.
Besides adjustments to the key struct name, function names, return
value, and parameter order, the only notable changes are:
- Replace the direct accesses to the auth tag length field (which should
be considered private until someone truly needs it) with the
AUTHTAG_LEN constant which is already defined in <asm/sev.h>.
- Replace kfree() with kfree_sensitive() on lines being changed anyway.
Tested-by: Nikunj A Dadhania <nikunj@amd.com>
Reviewed-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://patch.msgid.link/20260722025338.33354-2-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
Change the function's return type from 'int' to 'bool' and return the result
of memcmp() directly to simplify the code. While at it, cast ->start to
const u8 * to better match the expected type.
Also inline the verify_sha256_digest() check and remove the redundant
local 'ret' variable in purgatory().
No change in functionality expected.
Signed-off-by: Thorsten Blum <thorsten.blum@linux.dev>
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Link: https://patch.msgid.link/20260723133634.161598-3-thorsten.blum@linux.dev
flush_tlb_mm_range() and arch_tlbbatch_flush() pin the current CPU while
they decide whether the flush can be handled locally or must be sent to
remote CPUs. The CPU pinning is needed for the current CPU number and for
the local TLB flush path, which reads per-CPU TLB state.
The caller does not need to remain pinned while waiting for a remote TLB
flush to complete. After the remote-flush path has been selected,
flush_tlb_info is caller-private stack storage, so the caller no longer
has to stay on the same CPU to protect a shared per-CPU flush_tlb_info
object.
flush_tlb_multi() may also route through x86 PV backends. Those backends
must protect their own CPU-local scratch state instead of relying on the
caller to stay pinned. Hyper-V already does this by disabling interrupts
while using hyperv_pcpu_input_arg, and Xen's multicall path brackets its
per-CPU multicall buffer with xen_mc_batch() and xen_mc_issue().
kvm_flush_tlb_multi() also disables preemption while using __pv_cpu_mask.
Remote TLB flushes may synchronously wait for many CPUs, and the wait can
take tens of milliseconds when remote CPUs have interrupts disabled or
when many CPUs are involved. Keeping preemption disabled for that whole
wait unnecessarily increases scheduling latency on the initiating CPU.
Drop the CPU pinning before calling flush_tlb_multi() in the remote paths
of flush_tlb_mm_range() and arch_tlbbatch_flush(). Keep the local paths
inside the pinned section because they still access this CPU's TLB state.
Signed-off-by: Chuyi Zhou <zhouchuyi@bytedance.com>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Tested-by: Paul E. McKenney <paulmck@kernel.org>
Reviewed-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Link: https://patch.msgid.link/20260709122933.4021501-15-zhouchuyi@bytedance.com
kvm_flush_tlb_multi() is installed as an x86 PV TLB flush backend, so
flush_tlb_multi() can reach it through pv_ops when running as a KVM
guest.
kvm_flush_tlb_multi() uses the per-CPU scratch cpumask __pv_cpu_mask.
That buffer must remain tied to the current CPU until the mask has been
copied, filtered, and consumed by native_flush_tlb_multi().
The x86/mm callers currently enter flush_tlb_multi() while pinned to a
CPU. To let those callers drop CPU pinning before issuing the remote TLB
flush, each PV backend must protect its own CPU-local scratch state.
Make the KVM backend protect its per-CPU scratch cpumask by disabling
preemption locally. This is harmless with the current callers, where the
preemption disable is nested, and makes the KVM pv_ops dependency
explicit before changing the x86/mm call sites.
Signed-off-by: Chuyi Zhou <zhouchuyi@bytedance.com>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Reviewed-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Link: https://patch.msgid.link/20260709122933.4021501-14-zhouchuyi@bytedance.com
flush_tlb_info benefits from cacheline alignment, but using
cacheline-aligned stack storage directly can grow stack usage too much on
configurations with large SMP_CACHE_BYTES values. Commit 515ab7c413
("x86/mm: Align TLB invalidation info") attempted to align stack storage,
and commit 780e0106d4 ("x86/mm/tlb: Revert "x86/mm: Align TLB
invalidation info"") reverted it because using SMP_CACHE_BYTES led to 320
bytes of stack consumption. Commit 3db6d5a5ec ("x86/mm/tlb: Remove
'struct flush_tlb_info' from the stack") moved flush_tlb_info to per-CPU
storage, which avoided the stack growth problem while preserving
cacheline alignment. That was a good fit while the callers kept
preemption disabled for the whole flush operation.
However, a single per-CPU flush_tlb_info also requires all flush_tlb*()
operations to keep preemption disabled while the object is in use, so
that it cannot be overwritten by another flush on the same CPU.
flush_tlb*() may send IPIs to remote CPUs and synchronously wait for all
remote CPUs to complete their local TLB flushes. That wait can take tens
of milliseconds when interrupts are disabled on a remote CPU or when a
large number of remote CPUs are involved.
To shorten the CPU-pinned and preemption-disabled section around those
remote TLB flush waits, move flush_tlb_info back to caller-private stack
storage. The caller then does not have to stay on the same CPU until the
remote flush completes.
The type alignment is capped at 64 bytes. This keeps the alignment
benefit for stack objects without reintroducing the old large-cacheline
stack usage problem.
To evaluate the performance impact, use the following script to reproduce
the microbenchmark mentioned in commit 3db6d5a5ec ("x86/mm/tlb: Remove
'struct flush_tlb_info' from the stack"). The test environment is an Ice
Lake system (Intel(R) Xeon(R) Platinum 8336C) with 128 CPUs and 2 NUMA
nodes. During the test, the threads were bound to specific CPUs, and both
pti and mitigations were disabled:
#include <stdio.h>
#include <stdlib.h>
#include <pthread.h>
#include <sys/mman.h>
#include <sys/time.h>
#include <unistd.h>
#define NUM_OPS 1000000
#define NUM_THREADS 3
#define NUM_RUNS 5
#define PAGE_SIZE 4096
volatile int stop_threads = 0;
void *busy_wait_thread(void *arg) {
while (!stop_threads) {
__asm__ volatile ("nop");
}
return NULL;
}
long long get_usec() {
struct timeval tv;
gettimeofday(&tv, NULL);
return tv.tv_sec * 1000000LL + tv.tv_usec;
}
int main() {
pthread_t threads[NUM_THREADS];
char *addr;
int i, r;
addr = mmap(NULL, PAGE_SIZE, PROT_READ | PROT_WRITE, MAP_PRIVATE
| MAP_ANONYMOUS, -1, 0);
if (addr == MAP_FAILED) {
perror("mmap");
exit(1);
}
for (i = 0; i < NUM_THREADS; i++) {
if (pthread_create(&threads[i], NULL, busy_wait_thread, NULL))
exit(1);
}
printf("Running benchmark: %d runs, %d ops each, %d background\n"
"threads\n", NUM_RUNS, NUM_OPS, NUM_THREADS);
for (r = 0; r < NUM_RUNS; r++) {
long long start, end;
start = get_usec();
for (i = 0; i < NUM_OPS; i++) {
addr[0] = 1;
if (madvise(addr, PAGE_SIZE, MADV_DONTNEED)) {
perror("madvise");
exit(1);
}
}
end = get_usec();
double duration = (double)(end - start);
double avg_lat = duration / NUM_OPS;
printf("Run %d: Total time %.2f us, Avg latency %.4f us/op\n",
r + 1, duration, avg_lat);
}
stop_threads = 1;
for (i = 0; i < NUM_THREADS; i++)
pthread_join(threads[i], NULL);
munmap(addr, PAGE_SIZE);
return 0;
}
base on-stack-aligned on-stack-not-aligned
---- --------- -----------
avg (usec/op) 2.5278 2.5261 2.5508
stddev 0.0007 0.0027 0.0023
The benchmark results show that the average latency difference between
the baseline (base) and the properly aligned stack variable
(on-stack-aligned) is within the standard deviation (stddev). This
indicates that the variations are caused by testing noise, and reverting
to a stack variable with proper alignment causes no performance
regression compared to the per-CPU implementation. The unaligned version
(on-stack-not-aligned) shows a minor performance drop. The
CPU-pinned/preemption-disabled section can therefore be shortened without
sacrificing performance.
With caller-private storage there is no shared per-CPU object to protect,
so remove the DEBUG_VM reentrancy counter as well.
Signed-off-by: Chuyi Zhou <zhouchuyi@bytedance.com>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Tested-by: Paul E. McKenney <paulmck@kernel.org>
Reviewed-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Acked-by: Nadav Amit <nadav.amit@gmail.com>
Link: https://patch.msgid.link/20260709122933.4021501-13-zhouchuyi@bytedance.com
A stack allocated flush_tlb_info should keep cacheline alignment to avoid
the regression that motivated the per-CPU storage, but using
SMP_CACHE_BYTES directly can make the stack frame grow excessively on
configurations with large cache lines. This was addressed by
commit 780e0106d4 ("x86/mm/tlb: Revert "x86/mm: Align TLB
invalidation info""), where the stack consumption reached 320 bytes.
Add FLUSH_TLB_INFO_ALIGN and cap the type alignment at 64 bytes. The
existing per-CPU flush_tlb_info instance remains
DEFINE_PER_CPU_SHARED_ALIGNED(), so its per-CPU shared-cacheline
alignment is unchanged.
This prepares for moving flush_tlb_info back to stack storage without
reintroducing the old large-cacheline stack usage problem.
Signed-off-by: Chuyi Zhou <zhouchuyi@bytedance.com>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Reviewed-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Link: https://patch.msgid.link/20260709122933.4021501-12-zhouchuyi@bytedance.com
get_flush_tlb_info() has two responsibilities: it reserves the per-CPU
flush_tlb_info storage and it initializes the fields that describe the
flush operation. The per-CPU storage also carries the DEBUG_VM
reentrancy check and the matching put_flush_tlb_info() lifetime rules.
Moving flush_tlb_info back to caller-provided storage requires the same
field initialization without tying the caller to the per-CPU object.
Leaving the field setup embedded in get_flush_tlb_info() would either
keep those callers tied to the per-CPU object or duplicate the
initialization logic.
Split the field setup into init_flush_tlb_info(). Keep the per-CPU
storage selection, DEBUG_VM reentrancy check and put_flush_tlb_info()
lifetime rules in get_flush_tlb_info().
No functional change intended.
Signed-off-by: Chuyi Zhou <zhouchuyi@bytedance.com>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Reviewed-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Link: https://patch.msgid.link/20260709122933.4021501-11-zhouchuyi@bytedance.com
After a recent upstream LLVM change to start generating jump and lookup
tables in switch statements in more instances [1], linking the
compressed x86 boot image when CONFIG_KERNEL_ZSTD is enabled fails with:
ld.lld: error: Unexpected run-time relocations (.rela) detected!
Dumping the relocations in misc.o, which is the only file influenced by
CONFIG_KERNEL_ZSTD in the decompressor, shows dynamic relocations to
some string constants, which correspond to the string literals in the
switch statement in handle_zstd_error():
Relocation section '.rela.data.rel.ro' at offset 0x277b0 contains 31 entries:
Offset Info Type Symbol's Value Symbol's Name + Addend
0000000000000000 0000006600000001 R_X86_64_64 0000000000000000 .rodata.str1.1 + 73a
0000000000000008 0000006600000001 R_X86_64_64 0000000000000000 .rodata.str1.1 + 78e
0000000000000010 0000006600000001 R_X86_64_64 0000000000000000 .rodata.str1.1 + 78e
0000000000000018 0000006600000001 R_X86_64_64 0000000000000000 .rodata.str1.1 + 78e
...
This optimization is problematic for the decompressor environment, as it
is built as -fPIE without any explicit absolute references (as described
at the top of misc.c) while not applying any dynamic relocations, hence
the linker assertion. To opt out of this optimization, which is of
little value in this special early boot code, and to mirror the other
x86 startup code in arch/x86/boot/startup, disable jump tables in the
decompressor.
Signed-off-by: Nathan Chancellor <nathan@kernel.org>
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Acked-by: Ard Biesheuvel <ardb@kernel.org>
Cc: Bill Wendling <morbo@google.com>
Cc: Justin Stitt <justinstitt@google.com>
Cc: Nick Desaulniers <ndesaulniers@google.com>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: stable@vger.kernel.org
Link: fa02a6ed66 [1]
Link: https://patch.msgid.link/20260722-x86-boot-compressed-disable-jt-clang-v2-1-7373d38482fb@kernel.org
Closes: https://github.com/ClangBuiltLinux/linux/issues/2165
Commit 913bd90601 ("[PATCH] x86_64: Increase the variability of the
process stack on 64bit architectures"), merged in 2006, increased stack
randomisation from 8 MiB to an ostensible 1 GiB for 64-bit systems.
However the commit actually introduced a mask of 0x3fffff * 4 KiB pages
which provides 16 GiB of randomisation, not 1 GiB.
It appears Andi perhaps typo'd an additional 'f', as 0x3ffff would indeed
provide the stated 1 GiB.
Since this value has been in use for 20 years it seems sensible to retain
it, so correct the comment instead.
Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Acked-by: SJ Park <sj@kernel.org>
Link: https://patch.msgid.link/20260722-fix-x86-comment-v1-1-605080af988d@kernel.org
Signed-off-by: Kees Cook <kees@kernel.org>
__kvm_rmap_lock() deliberately elides the rmap lock when it observes an
empty rmap. In that case kvm_rmap_lock_readonly() also re-enables
preemption and returns zero, so the caller holds neither the rmap lock
nor a preemption reference. The elision documents the invariant it
relies on:
* Elide the lock if the rmap is empty, as lockless walkers (read-only
* mode) don't need to (and can't) walk an empty rmap, nor can they add
* entries to the rmap. I.e. the only paths that process empty rmaps
* do so while holding mmu_lock for write, and are mutually exclusive.
kvm_rmap_age_gfn_range() ignores the returned value and unconditionally
enters for_each_rmap_spte_lockless(). The iterator started with
rmap_get_first(), which re-reads rmap_head->val rather than using the
value returned by the lock. If a writer populates the rmap between the
lock's read and the iterator's re-read, the aging path walks the newly
installed rmap without holding its lock.
For a KVM_RMAP_MANY rmap this leaves the walker following a
pte_list_desc chain that it never locked. A writer holding mmu_lock for
write may free that chain (e.g. kvm_zap_all_rmap_sptes() on the recycle
path, or any rmap zap) via kmem_cache_free() while the walk is in
progress, giving a slab use-after-free. Nothing serialises the two: the
aging path runs without mmu_lock when CONFIG_KVM_MMU_LOCKLESS_AGING=y,
and the rmap lock that would otherwise exclude the writer was elided.
Because the empty path re-enables preemption, the interval between the
two reads can span an arbitrary scheduling delay.
Fix the class of bug by having the lockless walk consume the value
returned by the lock instead of re-reading the rmap. Split
rmap_get_first() into __rmap_get_first(), which starts an iterator from
an already-read rmap value, and make for_each_rmap_spte_lockless() take
that value and call __rmap_get_first() directly.
kvm_rmap_age_gfn_range() passes the value returned by
kvm_rmap_lock_readonly(): when the lock was elided the value is zero,
__rmap_get_first() returns NULL, and the walk is skipped. No lockless
walker re-reads the rmap, so the lock-elision invariant cannot be
violated, and no lock()-without-paired-unlock() path is added to the
aging code.
Fixes: af3b6a9eba ("KVM: x86/mmu: Walk rmaps (shadow MMU) without holding mmu_lock when aging gfns")
Suggested-by: Sean Christopherson <seanjc@google.com>
Cc: stable@vger.kernel.org
Signed-off-by: Phil Rosenthal <phil@phil.gs>
Link: https://patch.msgid.link/20260720-rmap-age-elided-submit-v2-1-668973030d47@phil.gs
Signed-off-by: Sean Christopherson <seanjc@google.com>
The size of the bandwidth field is enumerated from AMD hardware. resctrl uses
this field width to determine the maximum bandwidth supported that is stored
in resctrl_membw::max_bw. User space allocation requests ("control values")
are compared against this maximum for validity before being programmed to
hardware.
resctrl filesystem and resctrl x86 architecture code only support u32 control
values: resctrl_membw::max_bw is a u32, the control value provided by user
space is parsed into u32 local variables, and after validity checks, the
control value is staged into the u32 resctrl_staged_config::new_ctrl for
architecture consumption. The resctrl x86 architecture code in turn caches the
new control value into the u32 array rdt_hw_ctrl_domain::ctrl_val[].
The AMD bandwidth field to which control values are written can be up to
64 bits wide. While not an issue with current hardware (bandwidths that
require more than a u32, more than 536870911.875 GB/s, seem unreasonable
today), it is theoretically possible that enumeration of maximum bandwidth
field width will return values that are according to specification but
cannot be supported by resctrl.
Static checkers complain about this size mismatch.
Fix the static checker complaint by explicitly encoding the fact that resctrl
is unable to support all values that the hardware specification allows. Switch
to BIT() instead of open-coding the bitshift to avoid signed integer overflow
if the number of bits is a valid 31.
Signed-off-by: Reinette Chatre <reinette.chatre@intel.com>
Signed-off-by: Borislav Petkov (AMD) <bp@alien8.de>
Reviewed-by: Tony Luck <tony.luck@intel.com>
Link: https://patch.msgid.link/dd9fc3505c0ed250c5f14898b0ed6d7460a3cb82.1784753375.git.reinette.chatre@intel.com
When pinning SEV guest memory, drop FOLL_WRITE and rely on FOLL_LONGTERM
to break CoW, as *KVM* doesn't actually to the memory using the GUP'd
pages. Omitting FOLL_WRITE fixes a regression when using file-backed guest
memory that was introduced when KVM (correctly) added FOLL_LONG (e.g. to
ensure anonymous memory is migrated out of MIGRATE_CMA/ZONE_MOVABLE before
a long term pin).
Unfortunately, as of commits:
8ac268436e ("mm/gup: disallow FOLL_LONGTERM GUP-nonfast writing to file-backed mappings")
a6e79df92e ("mm/gup: disallow FOLL_LONGTERM GUP-fast writing to file-backed mappings")
GUP uses FOLL_LONGTERM as a canary of sorts to detect pins that are likely
to be problematic, and disallows WRITE+LONGTERM pins for file-backed memory.
As a result, backing SEV+ guests with file-backed memory, e.g. virtio-pmem,
fails due to the disallowed FOLL_LONGTERM+FOLL_WRITE combination.
Note, in the past, FOLL_WRITE was required to trigger CoW unsharing, to
prevent replacing the page in the (primary MMU's) page tables during a
later write fault after already having pinned a (shared) page in
MAP_PRIVATE mappings. FOLL_LONGTERM does that nowadays, even without
FOLL_WRITE (see gup_must_unshare()).
Fixes: 7e066cb9b7 ("KVM: SEV: Use long-term pin when registering encrypted memory regions")
Cc: stable@vger.kernel.org
Suggested-by: "David Hildenbrand (Arm)" <david@kernel.org>
Link: https://lore.kernel.org/all/ad784f05-b36c-4e91-9f17-4c5b826735d0@kernel.org/
Signed-off-by: Pankaj Gupta <pankaj.gupta@amd.com>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Acked-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Link: https://patch.msgid.link/20260715063626.65899-1-pankaj.gupta@amd.com
[sean: massage changelog, add comment about CoW unsharing]
Signed-off-by: Sean Christopherson <seanjc@google.com>
When handling userspace SREGS writes, check the validity of EFER (i.e.
allowed bits) before writing the new value of EFER through the
per-vendor set_efer callbacks. This prevents userspace from writing
bogus values (e.g. EFER.SVME=1 with nested=0).
Note: on KVM_SET_MSRS, KVM only checks EFER validity in terms of KVM
caps, not guest caps, so it is possible to set EFER bits that are
supported by KVM but not by the guest CPUID. Potentially allowing
userspace to set msrs before CPUID.
However, for KVM_SET_SREGS*, check the validity of the set bits against
both KVM and guest caps. This is consistent with other validity checks
(e.g. for CR4) that check validity against guest caps, which already
imposes the need to set CPUID before SREGS.
Cc: stable@vger.kernel.org
Signed-off-by: Yosry Ahmed <yosry@kernel.org>
Link: https://patch.msgid.link/20260713180153.2728382-2-yosry@kernel.org
Signed-off-by: Sean Christopherson <seanjc@google.com>
Supported EFER bits naturally fits into kvm_caps because it gets
recomputed during vendor initialization (e.g. to account for EFER.SVME
being allowed/disallowed based on nested being enabled/disabled). Move
efer_supported_bits into kvm_caps as supported_efer_bits (for naming
consistency).
As the bitmask is now globally visible as part of kvm_caps, there's
little use for helpers to enable/disable specific bits, so drop them and
open-code updates to kvm_caps.supported_efer_bits.
No functional change intended.
Suggested-by: Sean Christopherson <seanjc@google.com>
Reviewed-by: Nikolay Borisov <nik.borisov@suse.com>
Signed-off-by: Yosry Ahmed <yosry@kernel.org>
Link: https://patch.msgid.link/20260713181020.2735367-6-yosry@kernel.org
Signed-off-by: Sean Christopherson <seanjc@google.com>
In preparation for moving efer_reserved_bits into kvm_caps, reverse its
polarity and make it efer_supported_bits, to be more consistent with
other fields in kvm_caps.
No functional change intended.
Reviewed-by: Nikolay Borisov <nik.borisov@suse.com>
Signed-off-by: Yosry Ahmed <yosry@kernel.org>
Link: https://patch.msgid.link/20260713181020.2735367-5-yosry@kernel.org
Signed-off-by: Sean Christopherson <seanjc@google.com>
EFER reserved bits are statically initialized, and do not reset if a
vendor module is re-loaded. For example, loading kvm_amd with nested=1
removes EFER.SVME (and potentially EFER.LMSLE) from the reserved bits.
Reloading kvm_amd with nested=0 does not add them back, allowing
userspace to set EFER.SVME with nested=0.
Re-initializing EFER reserved bits before configuring them on vendor
initialization.
Cc: stable@vger.kernel.org
Signed-off-by: Yosry Ahmed <yosry@kernel.org>
Link: https://patch.msgid.link/20260713181020.2735367-4-yosry@kernel.org
Signed-off-by: Sean Christopherson <seanjc@google.com>
Remove EFER.LME and EFER.LMA from EFER reserved bits only if long mode
is actually supported. KVM does check long-mode support before allowing
the bits for guest writes and userspace writes through KVM_SET_SREGS*
(in __kvm_valid_efer()), but userspace writes through KVM_SET_MSRS only
check reserved bits.
In practice, this doesn't really matter. The true motiviation is getting
rid of the #ifdeffery when initializing efer_reserved_bits.
Cc: stable@vger.kernel.org
Signed-off-by: Yosry Ahmed <yosry@kernel.org>
Link: https://patch.msgid.link/20260713181020.2735367-3-yosry@kernel.org
Signed-off-by: Sean Christopherson <seanjc@google.com>
Move SVM-specific EFER bit enablement to generic x86 code, with the rest
of EFER bit enablement. Unifying the code for EFER bit enablement allows
for a later change to re-initialize EFER bits on module init.
No functional change intended.
Cc: stable@vger.kernel.org
Suggested-by: Sean Christopherson <seanjc@google.com>
Signed-off-by: Yosry Ahmed <yosry@kernel.org>
Link: https://patch.msgid.link/20260713181020.2735367-2-yosry@kernel.org
Signed-off-by: Sean Christopherson <seanjc@google.com>
Fix an issue where userspace or the guest can program an Hyper-V
synthetic timer to have a deadline in the past via integer overflow,
preventing the CPU from making progress and triggering an RCU stall.
Hyper-V's SynIC exposes 4 per-vCPU synthetic timers to the
guest, which are emulated by KVM. Each is programmed through the
HV_X64_MSR_STIMERi_CONFIG and HV_X64_MSR_STIMERi_COUNT MSRs. Depending
on CONFIG, COUNT represents either the absolute expiration time or the
period of a periodic timer, both expressed in 100ns ticks. These timers
may be set both by the guest (WRMSR) and the host (KVM_SET_MSRS).
When the timer is enabled, stimer_start() translates COUNT to an
absolute monotonic deadline and arms an hrtimer. If COUNT is set to a
value close to U64_MAX, the deadline calculation can overflow.
ktime_add_ns(ktime_now, 100 * (stimer->exp_time - time_now))
This can result in a CPU livelock. stimer_start() arms the timer
via hrtimer_start() with a deadline in the past, which causes it to
immediately fire. The stimer callback then raises KVM_RQ_HV_STIMER, with
the intention of causing KVM to deliver a synthetic interrupt on the
next vCPU guest enter.
Then, once userspace issues KVM_RUN, vcpu_enter_guest() consumes the
request, calling kvm_hv_process_stimers(). This would normally disable
the timer via stimer_expiration() once the deadline is in the past.
However, the deadline comparison is done between the KVM reference
counter and stime->exp_time, which is a big value close to U64_MAX, so
this never happens for a few thousand years.
kvm_hv_process_timers() then re-arms the timer via stimer_start(), since
it was not disabled, which again fires immediately. Before entering
the guest, kvm_vcpu_exit_request() checks kvm_request_pending(),
which returns true due to the newly raised KVM_REQ_HV_STIMER. Then
vcpu_enter_guest() aborts the guest entry, returning early into
vcpu_run(), which loops back again into vcpu_enter_guest(), restarting
the cycle.
Since there are no manual yields in this loop, a task with SCHED_FIFO
may starve RCU grace-period kthreads, which exposes the stalls found
by syzcaller:
rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
rcu: (detected by 1, t=10502 jiffies, g=14269, q=1142 ncpus=2)
rcu: All QSes seen, last rcu_preempt kthread activity 10500 (4294965239-4294954739), jiffies_till_next_fqs=1, root ->qsmask 0x0
rcu: rcu_preempt kthread starved for 10500 jiffies! g14269 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0
rcu: Unless rcu_preempt kthread gets sufficient CPU time, OOM is now expected behavior.
( ... )
Call Trace:
<IRQ>
__run_hrtimer kernel/time/hrtimer.c:1773 [inline]
__hrtimer_run_queues+0x408/0xc30 kernel/time/hrtimer.c:1841
hrtimer_interrupt+0x45b/0xaa0 kernel/time/hrtimer.c:1903
local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1045 [inline]
__sysvec_apic_timer_interrupt+0x102/0x3e0 arch/x86/kernel/apic/apic.c:1062
instr_sysvec_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1056 [inline]
sysvec_apic_timer_interrupt+0xa1/0xc0 arch/x86/kernel/apic/apic.c:1056
</IRQ>
<TASK>
asm_sysvec_apic_timer_interrupt+0x1a/0x20 arch/x86/include/asm/idtentry.h:697
RIP: 0010:__raw_spin_unlock_irqrestore include/linux/spinlock_api_smp.h:152 [inline]
RIP: 0010:_raw_spin_unlock_irqrestore+0xa8/0x110 kernel/locking/spinlock.c:194
Code: 74 05 e8 0b f4 5f f6 48 c7 44 24 20 00 00 00 00 9c 8f 44 24 20 f6 44 24 21 02 75 4f f7 c3 00 02 00 00 74 01 fb bf 01 00 00 00 <e8> 23 6b 27 f6 65 8b 05 7c 60 5a 07 85 c0 74 40 48 c7 04 24 0e 36
RSP: 0018:ffffc900040a7320 EFLAGS: 00000206
RAX: 5de15cb931505900 RBX: 0000000000000a06 RCX: 5de15cb931505900
RDX: 0000000000000007 RSI: ffffffff8daa9dc3 RDI: 0000000000000001
RBP: ffffc900040a73b0 R08: ffffffff8fc3d077 R09: 1ffffffff1f87a0e
R10: dffffc0000000000 R11: fffffbfff1f87a0f R12: dffffc0000000000
R13: 0000000000000000 R14: ffff8880b8628240 R15: 1ffff92000814e64
hrtimer_start include/linux/hrtimer.h:259 [inline]
stimer_start arch/x86/kvm/hyperv.c:682 [inline]
kvm_hv_process_stimers+0xd0a/0x16a0 arch/x86/kvm/hyperv.c:893
vcpu_enter_guest arch/x86/kvm/x86.c:11193 [inline]
vcpu_run+0x2240/0x76b0 arch/x86/kvm/x86.c:11639
kvm_arch_vcpu_ioctl_run+0x1148/0x1c90 arch/x86/kvm/x86.c:11984
kvm_vcpu_ioctl+0x99a/0xed0 virt/kvm/kvm_main.c:4492
vfs_ioctl fs/ioctl.c:51 [inline]
__do_sys_ioctl fs/ioctl.c:597 [inline]
__se_sys_ioctl+0xfc/0x170 fs/ioctl.c:583
do_syscall_x64 arch/x86/entry/syscall_64.c:63 [inline]
do_syscall_64+0xfa/0xf80 arch/x86/entry/syscall_64.c:94
entry_SYSCALL_64_after_hwframe+0x77/0x7f
RIP: 0033:0x7f635278f749
Code: ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 a8 ff ff ff f7 d8 64 89 01 48
RSP: 002b:00007f635365c038 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
RAX: ffffffffffffffda RBX: 00007f63529e5fa0 RCX: 00007f635278f749
RDX: 0000000000000000 RSI: 000000000000ae80 RDI: 0000000000000005
RBP: 00007f6352813f91 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
R13: 00007f63529e6038 R14: 00007f63529e5fa0 R15: 00007ffd5b219358
</TASK>
Fix this by clamping the deadline computation to KTIME_MAX, which
preserves the intent of arming a timer very far in the future.
ktime_add_safe() already does this type of clamping, so use it after
checking that that multiplying by the 100ns time tick also does not
overflow.
Reviewed-by: Vitaly Kuznetsov <vkuznets@redhat.com>
Reported-by: syzbot+3d5461510f8dc4adfe30@syzkaller.appspotmail.com
Closes: https://syzkaller.appspot.com/bug?extid=3d5461510f8dc4adfe30
Fixes: 1f4b34f825 ("kvm/x86: Hyper-V SynIC timers")
Cc: stable@vger.kernel.org
Signed-off-by: Carlos López <clopez@suse.de>
Link: https://patch.msgid.link/20260714133212.3916611-3-clopez@suse.de
[sean: tag for stable]
Signed-off-by: Sean Christopherson <seanjc@google.com>
Change to using KVM_X86_OP() instead of KVM_X86_OP_OPTIONAL() for the
.pi_update_irte() hook in kvm-x86-ops.h since now both VMX and SVM have
implemented it.
For the Fixes tag:
This hook was introduced for VMX posted-interrupt support. SVM later
added its implementation, but at this point KVM_X86_OP* had not been
introduced yet.
Initially KVM introduced KVM_X86_OP_NULL (and KVM_X86_OP) and used it
for this hook. But this was correct, because the use of KVM_X86_OP_NULL
was "to mark calls that do not follow the [svm|vmx]_func_name naming
convention" and the VMX one was named pi_update_irte(), i.e., did not
follow the convention. See commit 9af5471bdb ("KVM: x86: introduce
definitions to support static calls for kvm_x86_ops").
KVM later removed KVM_X86_OP_NULL (due to "the naming convention is not
in use anymore"), and added KVM_X86_OP_OPTIONAL for the hooks that can
be NULL pointer. It used KVM_X86_OP_OPTIONAL for this hook, but should
use KVM_X86_OP instead. See commit e4fc23bad8 ("KVM: x86: remove
KVM_X86_OP_NULL and mark optional kvm_x86_ops").
Note this hook was named .update_pi_irte() when it was introduced, but
got renamed to .pi_update_irte() at some point between the above two
commits.
Fixes: e4fc23bad8 ("KVM: x86: remove KVM_X86_OP_NULL and mark optional kvm_x86_ops")
Signed-off-by: Kai Huang <kai.huang@intel.com>
Link: https://patch.msgid.link/20260715090505.601174-1-kai.huang@intel.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
These callbacks are sysfs show paths.
Use sysfs_emit() and cpumask_pr_args() to emit the masks.
This prepares for removing cpumap_print_to_pagebuf().
Signed-off-by: Yury Norov <ynorov@nvidia.com>
BPF core and x86 JIT use text poking and temporary writable buffers and
thus can handle ROX memory.
Enable ROX cache for EXECMEM_BPF when configuration and CPU features allow
that.
Signed-off-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Acked-by: Song Liu <song@kernel.org>
Link: https://lore.kernel.org/bpf/20260716-execmem-x86-rox-bpf-v0-v3-5-4e76158c01c5@kernel.org
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
arch_bpf_trampoline_size() allocates a buffer to get actual size required
for a trampoline.
This buffer must be in the module address space because
__arch_prepare_bpf_trampoline() calculates rel32 offsets relatively to
that buffer.
In preparation for enabling ROX mode for EXECMEM_BPF make sure that the
allocated memory is writable.
Add bpf_jit_alloc_exec_rw() wrapper for execmem_alloc_rw() and use it for
buffer allocation in arch_bpf_trampoline_size().
Signed-off-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Acked-by: Song Liu <song@kernel.org>
Link: https://lore.kernel.org/bpf/20260716-execmem-x86-rox-bpf-v0-v3-4-4e76158c01c5@kernel.org
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
- Avoid redundant allocations when allocating IMSIC page tables
- Apply SBI FWFT LOCK flag only on successful set
- Bound SBI PMU counter mask scan to BITS_PER_LONG, since on RV32
the PMU SBI start/stop helper can only access 32 PMU counters.
- Skip TLB flush when G-stage PTE becomes valid if the Svvptc
extension is available.
- Always show Zicbo[m|z|p] block sizes in ONE_REG
- Inject instruction access fault on unmapped guest fetch
- Use raw spinlock for irqs_pending and irqs_pending_mask
- Fix Spectre-v1 in vector register access via ONE_REG
x86:
- Fixes to SEV selftests.
- Once free_nested() did a VMCLEAR of shadow VMCS, there's no need to VMCLEAR it
again if the kernel is preempted and thread migration happens
- Preserve nested TDP shadow page tables if they are used as roots, instead of
clearing them unnecessarily
- Fix use of stale data if out-of-memory happens after vendor module reload
- Check for invalid/obsolete root *after* making MMU pages available, because
the latter can make a page invalid
- Only reset TSC Deadline Timer in apic_timer_expired on KVM_RUN
-----BEGIN PGP SIGNATURE-----
iQFIBAABCgAyFiEE8TM4V0tmI4mGbHaCv/vSX3jHroMFAmpfSUwUHHBib256aW5p
QHJlZGhhdC5jb20ACgkQv/vSX3jHroOrWQf/S8Za6t7bZbQbJHDV0NXCKufpyKyz
l6WMsLGdn4LFLNqCtKhvQf5lEZ5PKRHFTGoWrO1wQSbREHRUCnfMD5lsSKQA4Grf
0QpQ6YBjXFv6zg0M456/Q/S1oyusyqkXXBS8BxlBQyfqR+ByKx8f4PXTGazfUua3
4SPHr5TUxr6CXy8715OQUGa1Xdf0/5d3OseiAgBwqM31yZPhK0tsar77Cd9P93bs
DC+FzfphHg0gb3s/xSfuVR0BPnqHvQMDF9rS/YKgJgJdZkzHXCo7muWRQmHWK+FR
Q2zOTWSlJATQucC0+2j1yj2Wdhg1itgZTfj/8RaIDRKbmjk0S1bLc3bamg==
=Svds
-----END PGP SIGNATURE-----
Merge tag 'for-linus' of git://git.kernel.org/pub/scm/virt/kvm/kvm
Pull kvm fixes from Paolo Bonzini:
"RISC-V:
- Avoid redundant allocations when allocating IMSIC page tables
- Apply SBI FWFT LOCK flag only on successful set
- Bound SBI PMU counter mask scan to BITS_PER_LONG, since on RV32 the
PMU SBI start/stop helper can only access 32 PMU counters.
- Skip TLB flush when G-stage PTE becomes valid if the Svvptc
extension is available.
- Always show Zicbo[m|z|p] block sizes in ONE_REG
- Inject instruction access fault on unmapped guest fetch
- Use raw spinlock for irqs_pending and irqs_pending_mask
- Fix Spectre-v1 in vector register access via ONE_REG
x86:
- Fixes to SEV selftests
- Once free_nested() did a VMCLEAR of shadow VMCS, there's no need to
VMCLEAR it again if the kernel is preempted and thread migration
happens
- Preserve nested TDP shadow page tables if they are used as roots,
instead of clearing them unnecessarily
- Fix use of stale data if out-of-memory happens after vendor module
reload
- Check for invalid/obsolete root *after* making MMU pages available,
because the latter can make a page invalid
- Only reset TSC Deadline Timer in apic_timer_expired on KVM_RUN"
* tag 'for-linus' of git://git.kernel.org/pub/scm/virt/kvm/kvm:
KVM: x86: Only reset TSC Deadline Timer in apic_timer_expired on KVM_RUN
KVM: selftests: sev_init2_tests: Derive SEV availability from KVM
KVM: selftests: sev_smoke_test: Only run VM types the host offers
KVM: x86/mmu: Fix use-after-free on vendor module reload
KVM: x86/mmu: Preserve nested TDP shadow page tables if they are used as roots
KVM: x86: Check for invalid/obsolete root *after* making MMU pages available
KVM: nVMX: Hide shadow VMCS right after VMCLEAR
KVM: riscv: Fix Spectre-v1 in vector register access
RISC-V: KVM: Serialize virtual interrupt pending state updates
RISC-V: KVM: Inject instruction access fault on unmapped guest fetch
RISC-V: KVM: Zicbo[m|z|p] block sizes should be always present in ONE_REG
riscv: kvm: Skip TLB flush when G-stage PTE becomes valid with Svvptc
KVM: riscv: PMU: Bound counter mask scan to BITS_PER_LONG
KVM: riscv: SBI FWFT: Apply LOCK flag only on successful set
RISC-V: KVM: Avoid redundant page-table allocations in ioremap topup
On Intel platforms with a VMX preemption timer and APICv, if a VMM
calls KVM_GET_LAPIC before KVM_GET_MSRS to save the vCPU state, it is
possible to lose a pending timer interrupt.
If the thread running these ioctls is migrated to another core after
calling KVM_GET_LAPIC but before KVM_GET_MSRS and the guest is using
their LAPIC timer in TSC-deadline mode, not only does the save LAPIC
state not carry the pending interrupt, the TSCDEADLINE MSR will be
zeroed.
After migration across CPUs, KVM_GET_MSRS calls vcpu_load, posting the
interrupt and clearing the MSR:
vcpu_load() ->
kvm_arch_vcpu_load() ->
kvm_lapic_restart_hv_timer() ->
start_hv_timer() ->
apic_timer_expired() ->
kvm_apic_inject_pending_timer_irqs()
. post interrupt into the LAPIC state
. clear IA32_TSCDEADLINE
The saved LAPIC state will be missing the pending interrupt and the saved
MSR will be zero. Oops.
Fix by only posting an interrupt when we're attempting to enter the guest
(vcpu->wants_to_run == true), not for vcpu_load from other paths.
Assisted-by: gemini:gemini-3.1-pro-preview
Debugged-by: David Matlack <dmatlack@google.com>
Debugged-by: Sean Christopherson <seanjc@google.com>
Debugged-by: Jim Mattson <jmattson@google.com>
Debugged-by: James Houghton <jthoughton@google.com>
Signed-off-by: Venkatesh Srinivas <venkateshs@chromium.org>
Message-ID: <20260715234234.15382-2-venkateshs@chromium.org>
Reviewed-by: James Houghton <jthoughton@google.com>
Reviewed-by: Chao Gao <chao.gao@intel.com>
Cc: stable@vger.kernel.org
Fixes: ae95f566b3 ("KVM: X86: TSCDEADLINE MSR emulation fastpath", 2020-05-15)
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
mmu_destroy_caches() destroys pte_list_desc_cache and
mmu_page_header_cache, but leaves both pointers unchanged. The pointers
live in kvm.ko, and therefore survive when a vendor module is unloaded
while kvm.ko remains loaded.
If creation of pte_list_desc_cache fails during a subsequent vendor
module load, its assignment sets pte_list_desc_cache to NULL and the
error path calls mmu_destroy_caches(). mmu_page_header_cache still
points to the cache destroyed during the preceding vendor module
unload. Passing that stale pointer to kmem_cache_destroy() causes a
slab use-after-free.
Reproduce the issue on a v7.1.3 kernel with CONFIG_KASAN=y,
CONFIG_KASAN_GENERIC=y, CONFIG_KVM=m, and CONFIG_KVM_INTEL=m. A
one-shot test hook forces pte_list_desc_cache to NULL on the second
invocation of kvm_mmu_vendor_module_init():
1. Load kvm.ko and kvm-intel.ko, creating both caches.
2. Unload only kvm_intel, leaving kvm.ko loaded.
3. Reload kvm_intel and force initialization through the -ENOMEM path.
KASAN reports:
BUG: KASAN: slab-use-after-free in
kvm_mmu_vendor_module_init+0x5b/0x170 [kvm]
...
kmem_cache_destroy+0x21/0x1d0
kvm_mmu_vendor_module_init+0x5b/0x170 [kvm]
...
Allocated by task 16817:
__kmem_cache_create_args+0x12c/0x3b0
__kmem_cache_create.constprop.0+0xb6/0xf0 [kvm]
kvm_mmu_vendor_module_init+0x13b/0x170 [kvm]
...
Freed by task 16820:
kmem_cache_destroy+0x117/0x1d0
kvm_mmu_vendor_module_exit+0x21/0x30 [kvm]
Clear both pointers immediately after destroying their caches so that
the stored state reflects the caches' lifetime and repeated cleanup is
safe.
With the fix applied, the same injected vendor module reload fails with
-ENOMEM as expected and produces no KASAN report.
Fixes: cb498ea2ce ("KVM: Portability: Combine kvm_init and kvm_init_x86")
Cc: stable@vger.kernel.org
Signed-off-by: Phil Rosenthal <phil@phil.gs>
Message-ID: <20260718-kvm-mmu-cache-uaf-v3-1-e103b93c74e1@phil.gs>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
kvm_mmu_zap_oldest_mmu_pages() excludes a shadow page whose root_count
is non-zero from top-level reclaim, because such a page cannot be
freed. The path in mmu_page_zap_pte() that recursively zaps a parentless
nested TDP child has no such check. As a result, a shadow page can
be zapped even if the page itself can't be freed; as the comment in
kvm_mmu_zap_oldest_mmu_pages() notes, zapping it will just force vCPUs
to rebuild the page.
As in top-level reclaim, do not recursively prepare zapping of a
nested TDP child whose root_count is non-zero.
Fixes: 2de4085ccc ("KVM: x86/MMU: Recursively zap nested TDP SPs when zapping last/only parent")
Signed-off-by: Hyunwoo Kim <imv4bel@gmail.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Check for a "stale" page fault, i.e. for an invalid and/or obsolete root,
after making MMU pages available for the shadow MMU. If reclaiming shadow
pages zaps an in-use root, i.e. marks it invalid, then KVM will attempt to
map memory into an invalid root. On its own, populating an invalid root is
"fine", but because child shadow pages inherit their parent's role, any
children created during the map/fetch will be created as invalid pages,
thus violating KVM's invariant that invalid pages are never on the list of
active MMU pages.
Note, the underlying flaw has existed since KVM first started tracking
invalid roots in 2008 (commit 2e53d63acb, "KVM: MMU: ignore zapped root
pagetables"), but the true badness only came along in 2020 (Linux 5.9)
with the invariant that invalid shadow pages can't be on the list of
active pages.
Note #2, inheriting role.invalid when creating child shadow pages is also
far from ideal; that flaw will be addressed separately.
Reported-by: Hyunwoo Kim <imv4bel@gmail.com>
Fixes: f95eec9bed ("KVM: x86/mmu: Don't put invalid SPs back on the list of active pages")
Cc: stable@vger.kernel.org
Signed-off-by: Sean Christopherson <seanjc@google.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
free_nested() frees the shadow VMCS while vmcs01 still points to it. But
because it is asynchronous with respect to loaded_vmcs_clear(), the vCPU
might migrate before the pointer is cleared and __loaded_vmcs_clear()
may then execute VMCLEAR.
The VMCS needs to stay attached until its explicit VMCLEAR completes, but
then it can be hidden and the page safely freed.
Fixes: 355f4fb140 ("kvm: nVMX: VMCLEAR an active shadow VMCS after last use")
Cc: stable@vger.kernel.org
Signed-off-by: Hyunwoo Kim <imv4bel@gmail.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
The low 32-bit ICR data is prepared by __prepare_ICR(), which returns a 32-bit
value. However, when this value is assigned to a new variable, it's easy to
mistakenly declare that variable with a different width.
To avoid this class of mistakes, use __prepare_ICR() directly as the function
argument instead of storing its result in an intermediate variable. This also
shaves off a bunch of lines in the code.
There should be no functionality change resulting from this.
Signed-off-by: Melody Wang <huibo.wang@amd.com>
Signed-off-by: Borislav Petkov (AMD) <bp@alien8.de>
Acked-by: Thomas Gleixner <tglx@kernel.org>
Link: https://patch.msgid.link/20260708012117.177959-1-huibo.wang@amd.com
The return values of syscall_enter_from_user_mode[_work]() are
non-intuitive. Both functions return the syscall number which should be
invoked by the architecture specific syscall entry code. The returned
number can be:
- the unmodified syscall number which was handed in by the caller
- a modified syscall number (ptrace, seccomp, trace/probe/bpf)
That has an additional twist. If the return value is -1L then the caller is
not allowed to modify the return value as that indicates that the modifying
entity requests to abort the syscall and set the return value already. That
can obviously not be differentiated from a syscall which handed in -1 as
syscall number.
The most trivial way to deal with that is:
set_return_value(regs, -ENOSYS);
nr = syscall_enter_from_user_mode(regs, nr);
if (valid(nr))
handle_syscall(regs, nr);
That's what LOONGARCH, RISCV, and X86 do. But PowerPC and S390 do not
preset the return value, so when user space hands in -1 and there is
nothing setting the return value in the entry work code, then the syscall
is skipped but the return value is whatever random data has been in the
return value register.
Change the return values of syscall_enter_from_user_mode[_work]() to
boolean and return false, when either ptrace or seccomp request to skip the
syscall. If they return true, update the syscall number as it might have
been changed.
That results in slightly different behaviour of the architectures versus
tracing.
If the syscall tracepoint has probe/BPF attached, those might set the
syscall number to -1 and also set the return value. PowerPC and S390 will
then overwrite that value with -ENOSYS. The other architectures will just
ignore it like any other invalid syscall and use the modified one.
Originally-by: Michal Suchánek <msuchanek@suse.de>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Tested-by: Michal Suchánek <msuchanek@suse.de>
Link: https://patch.msgid.link/20260712141346.772209074@kernel.org
Thomas Gleixner pointed out that cpu-type is a per-CPU property while hybrid
is a system property; conflating the two in the CPU matching infrastructure is
wrong. Currently, on a hybrid system x86_match_cpu() matches any cpu-type.
This works if the intent is to find the possibility of a cpu-type in a system.
But fails if matching for the cpu-type of a given CPU.
Borislav posted a cleanup here:
https://lore.kernel.org/all/20260703193222.GFakgORjvxwnZTPRnI@fat_crate.local
To make way for the cleanup stop matching cpu-type in cpu_vuln_blacklist.
RFDS is the only user, so drop the VULNBL_INTEL_TYPE entries and fold their
RFDS bit into the base Alder Lake (0x97) and Raptor Lake (0xB7) blacklist
entries. For now open-code cpu-type check in vulnerable_to_rfds(). In the
future, if more vulnerabilities need cpu-type matching a helper can be added.
No functional change intended.
Fixes: 722fa0dba7 ("x86/rfds: Exclude P-only parts from the RFDS affected list")
Signed-off-by: Pawan Gupta <pawan.kumar.gupta@linux.intel.com>
Signed-off-by: Borislav Petkov (AMD) <bp@alien8.de>
Link: https://patch.msgid.link/20260708-cpu-type-vuln-v1-1-85c1d3c704db@linux.intel.com
- Reject too long acpi_rsdp= boot parameter values
(Thorsten Blum)
- Validate console=uart8250 baud rate to fix early boot hang
(Thorsten Blum)
- Remove dead Makefile rule (Ethan Nelson-Moore)
Signed-off-by: Ingo Molnar <mingo@kernel.org>
-----BEGIN PGP SIGNATURE-----
iQJFBAABCgAvFiEEBpT5eoXrXCwVQwEKEnMQ0APhK1gFAmpcjhMRHG1pbmdvQGtl
cm5lbC5vcmcACgkQEnMQ0APhK1gVUw//UmbQ3jby1pvjDfAeJeqI8x2iCY2F9N5l
AqPD3371RPNzyx5uqyz4dLY94M31JW4jqzxntbIhGJrm9aXPLRTVzfMjF6sd9Lmj
lkMyHW+a4DVtA7oVSk2INebGUX2t+cpGLcF+9/yH3QavpLIXL9BNz5vDYqS0POMc
EitNNNqmO4+/f6oLf7Xxfl8Z2z2cVhcPCj1EG8b/rOWKeFmw1teuVu+dSagnV7Cn
/Hi4S8Q5HYUvvFBIFxgtuPW5S8HlTINhaE8hiat+Z0BENlAnbNThGEpz7hdogXAq
VDFwGPsjg3AF2agTWnF3e7Vs2CdDLHgcdTxUGII2lhZ9wCXzQ72+qXajf9PPynzH
YV9xtPw0a4sTDcR4QtdgBneCLf320WmbV6aH+tm9aZ1LIcPUgBO0qxYI/Ff1Frm8
NI1LxZJMiVfNNpFBk4JW/V1QKo3Padjhzfc23Rb4/8Xpu+S+Cu31pligIH3Rtnhm
pNt6Bv8eV61QJMZvjy4+5MouFfK2rL3YyEiE/WMP5cRhOCaK10hgDDEqCbuL5IRV
AgYMqGQPLTbZbdZZ1oSub+OPIRidMDfITlLoR9ggcJgNrV5J4tGysgAJWU2fCDoR
5dvRlzeQ0cjmCVMC+FVjQAahjBX5KqAQolDuDGB/vAAvxiir711ZO7g9UTCCRWQg
Bq1slSZ0WEc=
=upW0
-----END PGP SIGNATURE-----
Merge tag 'x86-urgent-2026-07-19' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
Pull x86 fixes from Ingo Molnar:
- Reject too long acpi_rsdp= boot parameter values (Thorsten Blum)
- Validate console=uart8250 baud rate to fix early boot hang (Thorsten
Blum)
- Remove dead Makefile rule (Ethan Nelson-Moore)
* tag 'x86-urgent-2026-07-19' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip:
x86/boot: Validate console=uart8250 baud rate to fix early boot hang
x86/boot: Reject too long acpi_rsdp= values
x86/cpu: Remove Makefile rule for removed UMC CPU support
The recent switch to 64-bit MSR interfaces introduced a build warning.
Work it around with a __maybe_unused annotation. After the full conversion
of the APIs we'll be able to address this more robustly by making sure
the APIs consume all the values as far the compiler is concerned, so that
the compiler doesn't emit such nuisance warnings.
[ mingo: Updated the changelog. ]
Fixes: cff219368b ("x86/mce: Stop using 32-bit MSR interfaces")
Closes: https://lore.kernel.org/oe-kbuild-all/202607031726.ZOwu4snu-lkp@intel.com/
Reported-by: kernel test robot <lkp@intel.com>
Signed-off-by: Juergen Gross <jgross@suse.com>
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Link: https://patch.msgid.link/20260703105555.1758819-1-jgross@suse.com
The 32-bit MSR interfaces rdmsr() and rdmsr_safe() are planned to be
removed. Use the related 64-bit variants instead.
Signed-off-by: Juergen Gross <jgross@suse.com>
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Cc: H. Peter Anvin <hpa@zytor.com>
Link: https://patch.msgid.link/20260703112426.1763049-1-jgross@suse.com
The 32-bit MSR interfaces rdmsr(), wrmsr() and rdmsr_safe() are
planned to be removed. Use the related 64-bit variants instead.
No change in functionality intended.
Signed-off-by: Juergen Gross <jgross@suse.com>
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Cc: H. Peter Anvin <hpa@zytor.com>
Cc: Kiryl Shutsemau <kas@kernel.org>
Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: Rick Edgecombe <rick.p.edgecombe@intel.com>
Cc: Sean Christopherson <seanjc@google.com>
Cc: kvm@vger.kernel.org
Link: https://patch.msgid.link/20260629060526.3638272-10-jgross@suse.com
The 32-bit MSR interfaces rdmsr(), wrmsr(), rdmsr_safe() and
wrmsr_safe() are planned to be removed. Use the related 64-bit variants
instead.
Signed-off-by: Juergen Gross <jgross@suse.com>
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Cc: H. Peter Anvin <hpa@zytor.com>
Link: https://patch.msgid.link/20260703112344.1762985-1-jgross@suse.com
The 32-bit MSR interfaces rdmsr(), wrmsr(), rdmsr_safe() and
wrmsr_safe() are planned to be removed. Use the related 64-bit variants
instead.
In processor_throttling.c drop needless initializers.
Signed-off-by: Juergen Gross <jgross@suse.com>
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Acked-by: Rafael J. Wysocki (Intel) <rafael@kernel.org>
Link: https://patch.msgid.link/20260629060526.3638272-5-jgross@suse.com
If a vCPU stays scheduled out (or blocked) while the last pCPU it ran
on goes through a hotplug cycle (online->offline->online), and the vCPU
then resumes execution on the same pCPU, then it is possible for it to
run with an ASID that has now been assigned to a different vCPU,
resulting in stale TLB translations being used.
svm_enable_virtualization_cpu() resets asid_generation to 1 and sets
next_asid to max_asid + 1 on every CPU online event, including hotplug
cycles. Because next_asid starts beyond the pool boundary, the first
call to new_asid() after an online event always wraps the pool,
incrementing asid_generation to 2 and assigning ASIDs starting from
min_asid.
Consider two vCPUs from different VMs, vCPU-A pinned to CPU-X holding
asid_generation=2 and ASID=N from before the hotplug event:
1. CPU-X goes offline and back online: asid_generation resets to 1,
next_asid = max_asid + 1.
2. One or more vCPUs migrate to CPU-X and call new_asid(), wrapping
the pool and consuming ASIDs starting from min_asid. Eventually
vCPU-B from a different VM is assigned asid_generation=2, ASID=N
— the same ASID that vCPU-A held before the hotplug.
3. vCPU-A enters pre_svm_run() on CPU-X: current_vmcb->cpu is
unchanged so the migration branch is skipped. Its saved
asid_generation=2 matches sd->asid_generation=2, so the generation
check silently passes and vCPU-A continues running with ASID=N —
the same ASID just freshly assigned to vCPU-B.
Both vCPUs from different VMs now run on CPU-X with the same ASID,
causing them to share NPT TLB entries and producing stale translations.
The collision manifests as a KVM internal error (Suberror: 1, emulation
failure). The NPT page fault reports a faulting GPA far outside the
VM's physical memory range — a sign of stale TLB translations being
used. KVM falls back to instruction emulation, which fails on
FPU/XSave instructions (XRSTOR, STMXCSR) that the emulator does not
implement.
Fix this by incrementing asid_generation instead of resetting it to 1
in svm_enable_virtualization_cpu(). On module load, asid_generation
starts at 0 (memset) and the increment produces 1, identical to the
old behaviour. On subsequent hotplug cycles the generation advances
beyond any value a vCPU previously observed on this CPU, so the
generation check in pre_svm_run() reliably forces new_asid() on every
vCPU after every hotplug cycle.
Fixes: 774c47f1d7 ("[PATCH] KVM: cpu hotplug support")
Reported-by: Chandrakanth Silveru <Chandrakanth.Silveru@amd.com>
Tested-by: Srikanth Aithal <Srikanth.Aithal@amd.com>
Reviewed-by: K Prateek Nayak <kprateek.nayak@amd.com>
Reviewed-by: Tom Lendacky <thomas.lendacky@amd.com>
Signed-off-by: Nikunj A Dadhania <nikunj@amd.com>
Message-ID: <20260715063506.672432-1-nikunj@amd.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Loading and unloading modules concurrently on several CPUs on a KASAN
build, with a short delay injected at the CPA page-table lookup to
widen the window, faults within minutes:
BUG: KASAN: use-after-free in __change_page_attr+0x7cc/0x7e0
Write of size 8 at addr ffff888181139718 by task modprobe
...
The buggy address belongs to the physical page:
pfn:0x181139 ... page_type: f2(table)
cpa_collapse_large_pages() rebuilds a leaf PMD from its 4K PTEs and
frees the old PTE-table pages, while __change_page_attr() fetches a
PTE pointer from a lockless lookup_address_in_pgd_attr() and writes
it with set_pte_atomic() only later. When module text is served from
a shared large ROX mapping the two run on the same PMD:
CPU A (module load) CPU B (module finalize)
------------------- -----------------------
execmem_make_temp_rw
set_memory_nx
__change_page_attr
split 2M -> 4K table P
kpte = &P[i] (lockless)
execmem_restore_rox
set_memory_rox (CPA_COLLAPSE)
cpa_collapse_large_pages
rebuild leaf PMD
flush_tlb_all
pagetable_free(P)
set_pte_atomic(kpte, ...)
-> writes into freed P
P is a page-table page (page_type: table), reused at once, so the
write corrupts whatever got the page next: a bad-pte or bad-page
splat, or a fatal fault once P has been turned into read-only text.
The flush_tlb_all() before the free does not close this: its IPI only
serializes against page-table walkers that run with interrupts off
(e.g. GUP-fast); the walk in __change_page_attr() runs with interrupts
on, so nothing stops it from holding a stale pointer into P.
Serialize the collapse - the PMD rebuild, TLB flush and PTE-table
free - under cpa_lock, the same lock __change_page_attr() now takes
unconditionally since commit ("x86/mm/pat: stop gating cpa_lock on
debug_pagealloc_enabled()"), so a concurrent walker can no longer
hold a pointer into a table the collapse is about to free.
Fixes: 41d88484c7 ("x86/mm/pat: restore large ROX pages after fragmentation")
Signed-off-by: Denis V. Lunev <den@openvz.org>
Signed-off-by: Dave Hansen <dave.hansen@linux.intel.com>
Acked-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
Link: https://patch.msgid.link/20260715183453.2381141-1-den@openvz.org
The splitting and merging of kernel page table mappings between small and
large is protected by cpa_lock. The merging is relatively new but the
splitting is ancient.
The splitting has a locking optimization: since DEBUG_PAGEALLOC forces all
mappings to 4k, there are no large pages to split. So the code that *might*
cause a split can just skip the locking (and a few other things).
This is entertaining, but it adds complexity and makes for weird locking
rules. Plus it's all for a debugging feature which makes the kernel super
slow in the first place. Optimizing something which is already super slow
and not used in production is not the best way to spend our complexity
budget.
Stop gating cpa_lock on debug_pagealloc_enabled() to simplify the code
and the locking rules.
[ dhansen: flesh out changelog ]
Suggested-by: Dave Hansen <dave.hansen@intel.com>
Signed-off-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Signed-off-by: Dave Hansen <dave.hansen@linux.intel.com>
Link: https://patch.msgid.link/20260715144519.934289-1-rppt@kernel.org
Link: https://lore.kernel.org/all/aab44f08-89f8-47fe-bee4-0ab6b25968c6@intel.com/
The header define in asm/timex,h and the naming of the function to read the
delay timer are confusing at best.
Convert it to a config switch selected by the archictures which provide the
functionality and rename the function to delay_read_timer(), which makes
the purpose clear. Move the declaration to linux/delay.h where it belongs.
Remove the resulting empty asm/timex.h files as well.
No functional change.
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Link: https://patch.msgid.link/20260410120317.978403520@kernel.org
This has been scheduled for removal more than a decade ago and the comments
related to it have been dutifully ignored. The last dependencies are gone.
Remove it along with various now empty asm/timex.h files.
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Reviewed-by: Christophe Leroy (CS GROUP) <chleroy@kernel.org>
Link: https://patch.msgid.link/20260410120317.910770161@kernel.org
CLOCK_TICK_RATE is only used in x86 but defined all over the tree for no
reason with comments that it's scheduled for removal for more than a
decade.
Use PIT_TICK_RATE for registering refined jiffies to get rid of the last
dependency.
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Link: https://patch.msgid.link/20260410120317.843419270@kernel.org
Put all vmcs12 pages if KVM synthesizes a nested VM-Exit due to invalid
guest while emulating VMLAUNCH or VMRESUME. The invalid guest state path
doesn't use nested_vmx_vmexit() as that API is intended to be used if and
only if L2 is active, and the open coded equivalent neglects to put the
vmcs12 pages. Failure to put the vmcs12 pages leaks any pinned pages
(and/or mappings) if L1 retries VMLAUNCH/VMRESUME.
Note, the !from_vmenter scenario doesn't suffer the same problem, as
vmx_get_nested_state_pages() only gets/pins/maps the vmcs12 pages if L2 is
active, i.e. if a "full" VM-Exit is guaranteed before KVM will retry
getting vmcs12 pages.
Fixes: 96c66e87de ("KVM/nVMX: Use kvm_vcpu_map when mapping the virtual APIC page")
Fixes: 3278e04925 ("KVM/nVMX: Use kvm_vcpu_map when mapping the posted interrupt descriptor table")
Fixes: fe1911aa44 ("KVM: nVMX: Use kvm_vcpu_map() to get/pin vmcs12's APIC-access page")
Reported-by: Minh Nguyen <minhnguyen.080505@gmail.com>
Cc: stable@vger.kernel.org
Signed-off-by: Sean Christopherson <seanjc@google.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
- Fix a bug where KVM will trigger a UAF if updating IOMMU IRTEs fails when
registering an IRQ-bypass producer.
- Ignore pending PV EOI instead of BUG()ing the host if the feature was
disabled by the guest.
- Fix nVMX bugs where KVM would run L1 with an L1-controlled CR3 after a
failed "late" consistency check when KVM is NOT using EPT.
- Disallow intra-host migration/mirroring of SNP VMs as KVM doesn't yet
support moving/mirroring SNP state.
- Fix a TOCTOU bug in KVM's handling of the "trusted" CPUID for TDX guests.
- Fix a NULL pointer deref in trace_kvm_inj_exception() where a change to the
core infrastructure missed KVM's unique (ab)use of __print_symbolic().
-----BEGIN PGP SIGNATURE-----
iQIzBAABCgAdFiEEKTobbabEP7vbhhN9OlYIJqCjN/0FAmpWb/kACgkQOlYIJqCj
N/1eAA//RDKkj5grTOKN7n9aY+uBrEFSH0VsEoyfg0GgiFZNOFWq4J6yiPTYb9kk
5Jxnl0LObI1sQ1Ini7LvB2sNa7YR5M3za3WNzUvynjw2gBvQGxU/7JMUqwcFlcXt
85RH/Wg7eNPCUvRUSsKEQwtdfi2NiGCxdg/AVnHs8/IPl8Ce7n3tQeKq4zm44mN9
eJ5hV5cCp6ytV7iM3N89yXo2LgLg2FWTg4vPA6MwxmTjNaouft3XH0428sNyI4zw
4NhLAnPLL8zmUs2zCxMSBYNkKiJ8Ou7jwD+1hny8MdHFbnrkNeNbk4Pbpol9AQ+Z
PhYeSsL0rk4bw+l5OEH5mxvtqjPV0qsH/ntN99W6kDh/gYDgJr7d70DsWxHWvluV
1Ewu26fjaUhKKwBoi4pDsXJVSHK2rTfeufeKegVW5IqEKLXYDzG2Kaj41UdPcFbo
HDv2FdUiqQo8PG2PUZzTduMOpcWggDjVgiqhAr4dL/Mf+IH+pDtPdCHM4XpvYHkQ
a+lFO76ZA/7fBVl8QgeZy8qRkbqSZLh00isUXXSRBamSL/rD27ODGOFrw+R9ct4s
hfUlC5GQDgrMmkqdweILfVip3/4B3Im2UI/5/QJB1bouf4r6w5A5fseTnMek445j
9zcFWquD17H/hg2DaS7+J0eZMKxhOZXegVLcE3UVqZLYw4VUBEw=
=+cUz
-----END PGP SIGNATURE-----
Merge tag 'kvm-x86-fixes-7.2-rc4' of https://github.com/kvm-x86/linux into HEAD
KVM x86 fixes for 7.2-rcN
- Fix a bug where KVM will trigger a UAF if updating IOMMU IRTEs fails when
registering an IRQ-bypass producer.
- Ignore pending PV EOI instead of BUG()ing the host if the feature was
disabled by the guest.
- Fix nVMX bugs where KVM would run L1 with an L1-controlled CR3 after a
failed "late" consistency check when KVM is NOT using EPT.
- Disallow intra-host migration/mirroring of SNP VMs as KVM doesn't yet
support moving/mirroring SNP state.
- Fix a TOCTOU bug in KVM's handling of the "trusted" CPUID for TDX guests.
- Fix a NULL pointer deref in trace_kvm_inj_exception() where a change to the
core infrastructure missed KVM's unique (ab)use of __print_symbolic().
Now that GENERIC_GETTIMEOFDAY and VDSO_GETRANDOM enable
HAVE_GENERIC_VDSO automatically, the architecture specific kconfig files
do not need to do so anymore.
Remove the corresponding lines.
Keep them for riscv, loongarch and powerpc as those have their own
explicit usage of the vDSO datastore provided by HAVE_GENERIC_VDSO.
Signed-off-by: Thomas Weißschuh <thomas.weissschuh@linutronix.de>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Reviewed-by: Vincenzo Frascino <vincenzo.frascino@arm.com>
Link: https://patch.msgid.link/20260709-vdso-have_generic_vdso-v1-5-d2e1061f268b@linutronix.de
Add missing ENCODE_FRAME_POINTER macro invocation into FRED_ENTER macro,
to prevent the unwinder from encountering a NULL stack frame pointer
when CONFIG_UNWINDER_FRAME_POINTER is enabled
Fixes: 14619d912b ("x86/fred: FRED entry/exit and dispatch code")
Signed-off-by: David Stevens <stevensd@google.com>
Signed-off-by: Dave Hansen <dave.hansen@linux.intel.com>
Acked-by: H. Peter Anvin (Intel) <hpa@zytor.com>
Link: https://patch.msgid.link/20260424191456.2679717-12-stevensd@google.com
Currently, only x86 genuinely implements and supports Syscall User Dispatch
(SUD).
Multiple architectures provide a stub arch_syscall_is_vdso_sigreturn()
returning 'false' simply to satisfy GENERIC_ENTRY compilation, which
creates a false impression of feature support.
Introduce ARCH_SUPPORTS_SYSCALL_USER_DISPATCH to decouple this mechanism
from GENERIC_ENTRY. Select it exclusively on x86 and remove the redundant
stub functions from other architectures.
Suggested-by: Mark Rutland <mark.rutland@arm.com>
Signed-off-by: Jinjie Ruan <ruanjinjie@huawei.com>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Reviewed-by: Thomas Weißschuh <thomas.weissschuh@linutronix.de>
Link: https://patch.msgid.link/20260713035422.582771-1-ruanjinjie@huawei.com
When the baud rate is empty, 0, invalid, or overflows to 0 when stored
as an int, the system will hang during early boot because of a division
by zero in early_serial_init().
Fall back to DEFAULT_BAUD when the resulting baud rate is 0 to prevent
an early system hang.
Fixes: ce0aa5dd20 ("x86, setup: Make the setup code also accept console=uart8250")
Signed-off-by: Thorsten Blum <thorsten.blum@linux.dev>
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Cc: stable@vger.kernel.org
Link: https://patch.msgid.link/20260713194924.126472-3-thorsten.blum@linux.dev
Convert xen_mm_unpin_all() to ptdescs in preparation for the eventual
splitting of ptdescs from struct page. Continue checking PagePinned
through the underlying page as we do not have a per-memdesc api yet.
Signed-off-by: Vishal Moola <vishal.moola@gmail.com>
Signed-off-by: Dave Hansen <dave.hansen@linux.intel.com>
Reviewed-by: William Kucharski <william.kucharski@linux.dev>
Reviewed-by: Juergen Gross <jgross@suse.com>
Acked-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Link: https://patch.msgid.link/20260629185742.126987-10-vishal.moola@gmail.com
Convert xen_mm_pin_all() to ptdescs in preparation for the eventual
splitting of ptdescs from struct page. Continue checking PagePinned
through the underlying page as we do not have a per-memdesc api for
page flags yet.
Signed-off-by: Vishal Moola <vishal.moola@gmail.com>
Signed-off-by: Dave Hansen <dave.hansen@linux.intel.com>
Reviewed-by: William Kucharski <william.kucharski@linux.dev>
Reviewed-by: Juergen Gross <jgross@suse.com>
Acked-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Link: https://patch.msgid.link/20260629185742.126987-9-vishal.moola@gmail.com
Convert pgd_page_get_mm() to ptdescs. Define struct ptdesc in our
pgtable_types so that our declarations recognize ptdesc as an
appropriate page table type.
Now that all callers are using ptdescs, we can pass in that
ptdesc to get the underlying mm_struct.
Signed-off-by: Vishal Moola <vishal.moola@gmail.com>
Signed-off-by: Dave Hansen <dave.hansen@linux.intel.com>
Reviewed-by: William Kucharski <william.kucharski@linux.dev>
Acked-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Link: https://patch.msgid.link/20260629185742.126987-8-vishal.moola@gmail.com
Convert sync_global_pgds_l4() to ptdescs in preparation for the
eventual splitting of ptdescs from struct page.
Signed-off-by: Vishal Moola <vishal.moola@gmail.com>
Signed-off-by: Dave Hansen <dave.hansen@linux.intel.com>
Reviewed-by: William Kucharski <william.kucharski@linux.dev>
Acked-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Link: https://patch.msgid.link/20260629185742.126987-7-vishal.moola@gmail.com
Convert sync_global_pgds_l5() to ptdescs in preparation for the
eventual splitting of ptdescs from struct page.
Signed-off-by: Vishal Moola <vishal.moola@gmail.com>
Signed-off-by: Dave Hansen <dave.hansen@linux.intel.com>
Reviewed-by: William Kucharski <william.kucharski@linux.dev>
Acked-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Link: https://patch.msgid.link/20260629185742.126987-6-vishal.moola@gmail.com
Convert arch_sync_kernel_mappings() to ptdescs in preparation for
the eventual splitting of ptdescs from struct page.
Following this patch, we can successfully boot a 32-bit x86 kernel with
separately allocated ptdescs.
Signed-off-by: Vishal Moola <vishal.moola@gmail.com>
Signed-off-by: Dave Hansen <dave.hansen@linux.intel.com>
Reviewed-by: William Kucharski <william.kucharski@linux.dev>
Acked-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Link: https://patch.msgid.link/20260629185742.126987-5-vishal.moola@gmail.com
Convert collapse_pmd_page() to ptdescs in preparation for the
eventual splitting of ptdescs from struct page.
Signed-off-by: Vishal Moola <vishal.moola@gmail.com>
Signed-off-by: Dave Hansen <dave.hansen@linux.intel.com>
Reviewed-by: William Kucharski <william.kucharski@linux.dev>
Acked-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Link: https://patch.msgid.link/20260629185742.126987-4-vishal.moola@gmail.com
Convert __set_pmd_pte() to ptdescs in preparation for the eventual
splitting of ptdescs from struct page.
Signed-off-by: Vishal Moola <vishal.moola@gmail.com>
Signed-off-by: Dave Hansen <dave.hansen@linux.intel.com>
Reviewed-by: William Kucharski <william.kucharski@linux.dev>
Acked-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Link: https://patch.msgid.link/20260629185742.126987-3-vishal.moola@gmail.com
Use IS_ENABLED() to check if we are on 32 bit. This standardizes this
check with the other 32 bit check in the file. No functional changes.
Signed-off-by: Vishal Moola <vishal.moola@gmail.com>
Signed-off-by: Dave Hansen <dave.hansen@linux.intel.com>
Reviewed-by: William Kucharski <william.kucharski@linux.dev>
Acked-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Link: https://patch.msgid.link/20260629185742.126987-2-vishal.moola@gmail.com
According to x86 architecture rules, 32-bit operations zero-extend the
result to 64 bits. The current implementation of handle_in() only masks
the lower 32 bits, which preserves the upper 32 bits of RAX when a
32-bit port IN instruction is emulated.
Use insn_assign_reg() to write the result back into RAX with proper
partial-register-write semantics: 1- and 2-byte forms leave the upper
bits untouched, the 4-byte form zero-extends to the full register.
Fixes: 0314994883 ("x86/tdx: Port I/O: Add runtime hypercalls")
Reported-by: Borys Tsyrulnikov <tsyrulnikov.borys@gmail.com>
Signed-off-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
Signed-off-by: Dave Hansen <dave.hansen@linux.intel.com>
Reviewed-by: Binbin Wu <binbin.wu@linux.intel.com>
Link: https://lore.kernel.org/all/CAKw_Dz96rfSQc6Rn+9QBcUFHhmkK+9zu+P=bxowfZwxrATCBRg@mail.gmail.com/
Cc:stable@vger.kernel.org
Link: https://patch.msgid.link/20260713133753.223947-4-kirill@shutemov.name
KVM's instruction emulator has a small helper, assign_register(), that
writes a value into a register following the x86 rules for writes to
general-purpose registers: an 8- or 16-bit write leaves the rest of the
register untouched, a 32-bit write zero-extends the result to 64 bits,
and a 64-bit write replaces the whole register.
The TDX guest #VE handler needs the same logic for port I/O emulation
to get 32-bit zero-extension right. Rather than add a third copy of
the same switch, move the helper verbatim to <asm/insn-eval.h>, rename
it to insn_assign_reg(), and route KVM's callers through it.
Add <asm/insn.h> to the header's includes so it builds standalone in
callers that have not pulled it in transitively.
No functional change.
Signed-off-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
Signed-off-by: Dave Hansen <dave.hansen@linux.intel.com>
Acked-by: Sean Christopherson <seanjc@google.com>
Cc:stable@vger.kernel.org
Link: https://patch.msgid.link/20260713133753.223947-3-kirill@shutemov.name
handle_in() and handle_out() in arch/x86/coco/tdx/tdx.c use:
u64 mask = GENMASK(BITS_PER_BYTE * size, 0);
GENMASK(h, l) includes bit h. For size=1 (INB), this produces
GENMASK(8, 0) = 0x1FF (9 bits) instead of GENMASK(7, 0) = 0xFF (8
bits). The mask is one bit too wide for all I/O sizes.
Fix the mask calculation.
Fixes: 0314994883 ("x86/tdx: Port I/O: Add runtime hypercalls")
Reported-by: Borys Tsyrulnikov <tsyrulnikov.borys@gmail.com>
Signed-off-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
Signed-off-by: Dave Hansen <dave.hansen@linux.intel.com>
Reviewed-by: Kai Huang <kai.huang@intel.com>
Reviewed-by: Kuppuswamy Sathyanarayanan <sathyanarayanan.kuppuswamy@linux.intel.com>
Reviewed-by: Binbin Wu <binbin.wu@linux.intel.com>
Reviewed-by: Rick Edgecombe <rick.p.edgecombe@intel.com>
Link: https://lore.kernel.org/all/CAKw_Dz96rfSQc6Rn+9QBcUFHhmkK+9zu+P=bxowfZwxrATCBRg@mail.gmail.com/
Cc:stable@vger.kernel.org
Link: https://patch.msgid.link/20260713133753.223947-2-kirill@shutemov.name
A resctrl domain consists of the domain structure self that includes
pointers to dynamically allocated filesystem as well as architecture
specific data. For example, the L3 monitoring domain structure consists
of the architecture specific struct rdt_hw_l3_mon_domain that contains
the dynamically allocated rdt_hw_l3_mon_domain::arch_mbm_states
architectural state and the embedded struct rdt_l3_mon_domain contains
the dynamically allocated rdt_l3_mon_domain::mbm_states resctrl fs state.
The domains are added to and removed from an RCU protected list while
cpus_write_lock() is held so that readers could access domains via
cpus_read_lock() or from an RCU read-side critical section. A reader
accessing a domain via the RCU list expects that the domain and all its
dynamically allocated data is accessible.
Only place the domain on the RCU list when all its dynamically allocated
data is ready, similarly unlink it from RCU list (again with cpus_write_lock()
held) before removing any of its dynamically allocated data.
Calling resctrl_online_mon_domain() before adding the domain to the RCU
list creates the kernfs files that expose the domain's monitoring data to
user space before adding the domain to the RCU list. This is safe because
rdtgroup_mondata_show() acquires cpus_read_lock() before it traverses the
RCU list and will thus block until the domain is added to the RCU list.
There are no readers accessing a domain via RCU list. Ensure safety of
access when such a reader arrives.
Signed-off-by: Reinette Chatre <reinette.chatre@intel.com>
Signed-off-by: Borislav Petkov (AMD) <bp@alien8.de>
Reviewed-by: Tony Luck <tony.luck@intel.com>
Reviewed-by: Chen Yu <yu.c.chen@intel.com>
Link: https://patch.msgid.link/31ae67084c983e8cb8c5ef2c65e1096de5e8f9b0.1783963505.git.reinette.chatre@intel.com
rdt_resource::ctrl_domains and rdt_resource::mon_domains are RCU lists with
entries added and removed by architecture from CPU hotplug callbacks that are
run with cpus_write_lock() held. These lists can be traversed safely from
resctrl fs by either holding cpus_read_lock() or relying on an RCU read-side
critical section.
resctrl fs traversals of rdt_resource::ctrl_domains and
rdt_resource::mon_domains are done using list_for_each_entry() with
cpus_read_lock() held. Similarly, x86 architecture callbacks use
list_for_each_entry() expecting that resctrl fs makes the call with
cpus_read_lock() held. Inconsistently, a lockdep_assert_cpus_held() precedes
the list_for_each_entry() call with varying distance to document this safe RCU
list traversal.
In preparation for an upcoming traversal of rdt_resource::ctrl_domains that
needs to be done from RCU read-side critical section there is a requirement
for developers to always know exactly in which context the list is being
traversed.
Replace the list_for_each_entry() traversals of RCU list with
list_for_each_entry_rcu() to document that an RCU list is being traversed
while making use of the built-in lockdep expression that additionally
documents that it is cpus_read_lock() that enables the list to be
traversed from non-RCU protection. Only revert to documenting the
safety of traversal using a comment when lockdep does not have needed
visibility in functions called via smp_call*().
The lockdep expression within list_for_each_entry_rcu() depends on
RCU_EXPERT that is not set in a typical debug kernel so keep the existing
lockdep_assert_cpus_held() that is active with CONFIG_LOCKDEP=y found in
typical debug kernel.
Signed-off-by: Reinette Chatre <reinette.chatre@intel.com>
Signed-off-by: Borislav Petkov (AMD) <bp@alien8.de>
Link: https://patch.msgid.link/d9373f8da8ffde667740e186ffc96ab69628ac9a.1783963505.git.reinette.chatre@intel.com
This was there to be able to patch out locking instructions when running
a SMP kernel build on a UP CPU. The times are long gone when single-CPU
x86 machines were relevant so drop that machinery and simplify the code
considerably.
LOCK_PREFIX needs to stay for when one wants to do a UP build for
whatever reason. That'll go away when CONFIG_SMP becomes unconditional.
Kill a bunch of leftover, unused prototypes while at it.
Signed-off-by: Borislav Petkov (AMD) <bp@alien8.de>
Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Acked-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Link: https://lore.kernel.org/r/20260328081634.797552-1-rppt@kernel.org
The trace_kvm_inj_exception tracepoint takes as arguments the exception
vector, whether the exception has an error code (and subsequently, the
error code), and whether it is being reinjected. Because '0' is a valid
error code, KVM uses __print_symbolic() to format the error code as a
string to avoid printing the error code entirely if the exception doesn't
have an error code (see commit 21d4c575eb ("KVM: x86: Print error code
in exception injection tracepoint iff valid").
KVM's abuse of __print_symbolic() was all fine and dandy, until commit
754e38d2d1 ("tracing: Use explicit array size instead of sentinel
elements in symbol printing") reworked the printing to avoid terminating
the arrays with NULL/0 values, and missed KVM's clever use of not-quite
empty array of symbols.
BUG: kernel NULL pointer dereference, address: 0000000000000000
#PF: supervisor read access in kernel mode
#PF: error_code(0x0000) - not-present page
PGD 0 P4D 0
Oops: Oops: 0000 [#1] SMP
CPU: 20 UID: 0 PID: 791 Comm: less Not tainted 7.2.0-rc2 #401 PREEMPT
Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
RIP: 0010:strlen+0x0/0x20
Call Trace:
<TASK>
trace_seq_puts+0x18/0x80
trace_print_symbols_seq+0x68/0xa0
trace_raw_output_kvm_inj_exception+0x64/0xf0 [kvm]
s_show+0x47/0x110
seq_read_iter+0x2a5/0x4c0
seq_read+0xfd/0x130
vfs_read+0xb6/0x330
? vfs_write+0x2f2/0x3f0
ksys_read+0x61/0xd0
do_syscall_64+0xb7/0x570
entry_SYSCALL_64_after_hwframe+0x4b/0x53
RIP: 0033:0x7ff283714862
</TASK>
Simply drop the dummy array entirely, so that __print_symbolic() generates
a truly empty array.
Signed-off-by: Daniel Paziyski <danielpaziyski@gmail.com>
Fixes: 754e38d2d1 ("tracing: Use explicit array size instead of sentinel elements in symbol printing")
Reviewed-by: Thomas Weißschuh <thomas.weissschuh@linutronix.de>
Link: https://patch.msgid.link/20260710134055.16432-1-danielpaziyski@gmail.com
[sean: massage changelog, add splat, add Fixes, cc stable]
Signed-off-by: Sean Christopherson <seanjc@google.com>
Converting from int to long, back to int and then to unsigned int is
confusing at best.
None of this voodoo is required. Negative syscall numbers including -1
don't need any of this treatment and the low level ASM code already does
the sign extension to 64-bit on a 64-bit kernel.
The only point where signedness matters is the comparison against the
maximum syscall number, but that can be simplified by just using a unsigned
argument for the various syscall invocation functions.
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Tested-by: Mukesh Kumar Chaurasiya (IBM) <mkchauras@gmail.com>
Link: https://patch.msgid.link/20260707190254.545214398@kernel.org
Invoking sys_ni_syscall() from a code path, which already knows that the
syscall number is invalid just to assign -ENOSYS to regs->ax is a pointless
exercise. It's even redundant as the low level entry code already has set
regs->ax to -ENOSYS on entry.
Remove the extra conditionals and the function calls.
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Tested-by: Mukesh Kumar Chaurasiya (IBM) <mkchauras@gmail.com>
Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com>
Link: https://patch.msgid.link/20260707190254.493733289@kernel.org
They are only used in the respective source files. No point in exposing
them.
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Tested-by: Mukesh Kumar Chaurasiya (IBM) <mkchauras@gmail.com>
Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com>
Reviewed-by: Mukesh Kumar Chaurasiya (IBM) <mkchauras@gmail.com>
Link: https://patch.msgid.link/20260707190254.438361565@kernel.org
The return value of __secure_computing() currently uses 0 to indicate
that a system call should be allowed, and -1 to indicate that it should
be blocked/killed. This 0/-1 pattern is non-intuitive for a security
check function and makes the control flow at the call sites less readable.
Furthermore, any potential future changes to these return values would
require a high-risk, error-prone audit of all its users across different
architectures.
Sanitize this logic by converting the return type of __secure_computing()
to a proper boolean, where 'true' explicitly means 'allow' and 'false'
means 'fail/deny'.
Update all the two dozen or so call sites across the tree to align with
this new boolean semantic. No functional changes are intended, as the
callers still return -1 to the lower-level assembly entry code upon
seccomp denial.
Rename the function to __seccomp_permit_syscall() so that the purpose is
entirely clear.
[ tglx: Rename the function ]
Suggested-by: Thomas Gleixner <tglx@kernel.org>
Suggested-by: Mark Rutland <mark.rutland@arm.com>
Signed-off-by: Jinjie Ruan <ruanjinjie@huawei.com>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Tested-by: Mukesh Kumar Chaurasiya (IBM) <mkchauras@gmail.com>
Reviewed-by: Mukesh Kumar Chaurasiya (IBM) <mkchauras@gmail.com>
Acked-by: Oleg Nesterov <oleg@redhat.com>
Link: https://patch.msgid.link/20260707190254.230735780@kernel.org
These functions integrate the stack randomization.
syscall_enter_from_user_mode_randomize_stack() has the advantage that the
randomization happens early right after enter_from_user_mode().
In both cases also the overhead of get/put_cpu_var() in
add_random_kstack_offset() is avoided.
No functional change.
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Tested-by: Mukesh Kumar Chaurasiya (IBM) <mkchauras@gmail.com>
Reviewed-by: Radu Rendec <radu@rendec.net>
Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@oss.qualcomm.com>
Reviewed-by: Mukesh Kumar Chaurasiya (IBM) <mkchauras@gmail.com>
Link: https://patch.msgid.link/20260707190254.079478122@kernel.org
cmdline_find_option() returns the full length of the parsed acpi_rsdp=
value. get_cmdline_acpi_rsdp() then silently truncates values which do
not fit in the val[] buffer.
Prevent boot_kstrtoul() from parsing a truncated value and then the
kernel from silently using the wrong RSDP address, see discussion in
Link:.
Issue a warning so that the user is aware that s/he supplied a malformed
value and can get feedback instead of silent crashes.
[ bp: Make commit message more precise. ]
Fixes: 3c98e71b42 ("x86/boot: Add "acpi_rsdp=" early parsing")
Signed-off-by: Thorsten Blum <thorsten.blum@linux.dev>
Signed-off-by: Borislav Petkov (AMD) <bp@alien8.de>
Cc: stable@vger.kernel.org
Link: https://lore.kernel.org/all/20260617130417.36651-4-thorsten.blum@linux.dev
Support for UMC CPUs was removed in
7d328c5de4 ("x86/cpu: Remove CPU_SUP_UMC_32 support"),
but a Makefile rule for the support code remained. Remove it.
Fixes: 7d328c5de4 ("x86/cpu: Remove CPU_SUP_UMC_32 support")
Signed-off-by: Ethan Nelson-Moore <enelsonmoore@gmail.com>
Signed-off-by: Borislav Petkov (AMD) <bp@alien8.de>
Reviewed-by: Ahmed S. Darwish <darwi@linutronix.de>
Link: https://patch.msgid.link/20260610033252.164571-1-enelsonmoore@gmail.com
A user-only branch stack can contain branches that originate from
the kernel. As a result, kernel addresses are exposed to user space
even when PERF_SAMPLE_BRANCH_USER is requested. On AMD processors
supporting X86_FEATURE_BRS (Zen 3 only), perf can still report entries
such as SYSRET/interrupt returns for which the branch-from addresses
are in the kernel.
E.g.
$ perf record -j any,u -c 4000 -e branch-brs -o - -- \
perf bench syscall basic --loop 1000 | \
perf script -i - -F brstack|tr ' ' '\n'| \
grep -E '0x[89a-f][0-9a-f]{15}'
...
0xffffffff810001c4/0x72e2e32955eb/-/-/-/0//-
0xffffffff810001c4/0x72e2d94a9821/-/-/-/0//-
0xffffffff810001c4/0x72e2d94ffa1b/-/-/-/0//-
...
BRS provides no hardware branch filtering, so privilege level
filtering is performed entirely in software. However, amd_brs_match_plm()
only validates the branch-to address against the requested privilege
levels. For branches from the kernel to user space, the branch-from
address is left unchecked and is leaked. Extend the software filter to
also validate the branch-from address, so that any branch record whose
branch-from address is in the kernel is dropped when
PERF_SAMPLE_BRANCH_USER is requested.
Fixes: 8910075d61 ("perf/x86/amd: Enable branch sampling priv level filtering")
Reported-by: Sashiko <sashiko-bot@kernel.org>
Signed-off-by: Sandipan Das <sandipan.das@amd.com>
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Cc: stable@vger.kernel.org
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Stephane Eranian <eranian@google.com>
Link: https://patch.msgid.link/f05931c4f89a146c364bd5dc6b8170b1ac611c65.1783701239.git.sandipan.das@amd.com
Closes: https://lore.kernel.org/all/20260710110235.F3FD81F000E9@smtp.kernel.org/
Commit aebc3ca190 ("KVM: x86: Enable CMCI capability by default and
handle injected UCNA errors") introduced kvm_vcpu_x86_set_ucna(), which
accesses @vcpu->arch.mci_ctl2_banks[] using @mce->bank as the index. The
@mce struct is user-controlled, provided via the KVM_X86_SET_MCE ioctl.
The caller of this function, kvm_vcpu_ioctl_x86_set_mce(), bounds-checks
@mce->bank and applies array_index_nospec() to advance the @banks
pointer, but @mce->bank itself is passed through unclamped. On a
speculative path that bypasses the bounds check, the raw @mce->bank
value can index mci_ctl2_banks[] out-of-bounds.
In practice this is a very weak gadget, and would at most allow leaking
a single bit in a 64-bit integer, but prevent potential future issues by
clamping @mce->bank in place with array_index_nospec(), before passing
the struct to kvm_vcpu_x86_set_ucna().
Fixes: aebc3ca190 ("KVM: x86: Enable CMCI capability by default and handle injected UCNA errors")
Signed-off-by: Carlos López <clopez@suse.de>
Link: https://patch.msgid.link/20260609131856.2562222-3-clopez@suse.de
Signed-off-by: Sean Christopherson <seanjc@google.com>
Advertise the Zhaoxin big-number arithmetic engine to guests via
CPUID 0xC0000001 EDX bits 27 (RSA) and 28 (RSA_EN). The RSA feature
provides two unprivileged instructions for modular arithmetic on big
integers, documented in the Zhaoxin PadLock Instruction Reference,
chapter 4 ("Modular Multiplication and Exponentiation Engine"). Both
support operand sizes from 256 to 32768 bits (in 128-bit increments):
- REP XMODEXP (encoding F3 0F A6 F8, subsection 4.1) computes
A^B mod M
- REP MONTMUL2 (encoding F3 0F A6 F0, subsection 4.2) computes
A*B mod M
REP MONTMUL2 is the long-mode replacement of legacy REP MONTMUL, which
is restricted to compatibility and 32-bit protected modes. These
primitives accelerate RSA and related public-key operations.
Both instructions are unprivileged (no CPL restriction) and available
in all CPU modes, with no associated MSR control. The RSA and RSA_EN
bits are redundant by hardware design (set or cleared together) and
both serve purely as CPUID-level feature-presence reporting flags
requiring no KVM emulation. Both bits are advertised because different
software may probe either one when checking for RSA availability.
Reviewed-by: Binbin Wu <binbin.wu@linux.intel.com>
Signed-off-by: Ewan Hai <ewanhai-oc@zhaoxin.com>
Link: https://patch.msgid.link/20260610023512.3690734-6-ewanhai-oc@zhaoxin.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
Advertise the Zhaoxin PadLock Hash Engine v2 to guests via CPUID
0xC0000001 EDX bits 25 (PHE2) and 26 (PHE2_EN). PHE2 extends the
PadLock hash family with SHA-384 and SHA-512 support per FIPS 180-3,
complementing the existing PHE feature (SHA-1 and SHA-256).
Two unprivileged instructions are exposed, documented in the Zhaoxin
PadLock Instruction Reference, chapter 3 ("Hash Engine"):
- REP XSHA384 (encoding F3 0F A6 D8, subsection 3.3)
- REP XSHA512 (encoding F3 0F A6 E0, subsection 3.4)
Both consume software-padded 128-byte blocks (RCX = block count, RSI =
input, RDI = state) and produce hash output in the state buffer.
Both instructions are unprivileged (no CPL restriction) and available
in all CPU modes, with no associated MSR control. The PHE2 and PHE2_EN
bits are redundant by hardware design (set or cleared together) and
both serve purely as CPUID-level feature-presence reporting flags
requiring no KVM emulation. Both bits are advertised because different
software may probe either one when checking for PHE2 availability.
Reviewed-by: Binbin Wu <binbin.wu@linux.intel.com>
Signed-off-by: Ewan Hai <ewanhai-oc@zhaoxin.com>
Link: https://patch.msgid.link/20260610023512.3690734-5-ewanhai-oc@zhaoxin.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
Advertise the Zhaoxin second-generation hardware RNG to guests via
CPUID 0xC0000001 EDX bits 22 (RNG2) and 23 (RNG2_EN).
RNG2 is exposed by the REP XRNG2 instruction (encoding F3 0F A7 F8),
documented in the Zhaoxin PadLock Instruction Reference, subsection 1.3
("REP XRNG2"). It produces random bytes from two on-die RNG sources
selectable via RAX bits[10:9] and an output mode (raw vs post-processed)
controlled by RDX bits[1:0], providing high-quality entropy intended
for cryptographic operations.
REP XRNG2 is unprivileged (no CPL restriction) and available in all
CPU modes, with no associated MSR control. The RNG2 and RNG2_EN bits
are redundant by hardware design (set or cleared together) and both
serve purely as CPUID-level feature-presence reporting flags requiring
no KVM emulation. Both bits are advertised because different software
may probe either one when checking for RNG2 availability.
Reviewed-by: Binbin Wu <binbin.wu@linux.intel.com>
Signed-off-by: Ewan Hai <ewanhai-oc@zhaoxin.com>
Link: https://patch.msgid.link/20260610023512.3690734-4-ewanhai-oc@zhaoxin.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
