github-mirror/linux
1
0
Fork 0
mirror of https://github.com/torvalds/linux.git synced 2026-11-06 21:58:01 +01:00
Code Issues Actions Packages Projects Releases Wiki Activity Graph

Merge git://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf 7.3-rc5

Browse Source 
Cross-merge BPF and other fixes after downstream PR.

Conflicts:
  arch/arm64/net/bpf_jit_comp.c
  arch/x86/net/bpf_jit_comp.c

Signed-off-by: Alexei Starovoitov <ast@kernel.org>
This commit is contained in:
Alexei Starovoitov 2026-10-02 21:24:53 +00:00
commit 2b5440b31c
No known key found for this signature in database
716 changed files with 11678 additions and 3769 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

3
.mailmap
View File

@ -286,6 +286,8 @@ Ed L. Cashin <ecashin@coraid.com>
Elliot Berman <quic_eberman@quicinc.com> <eberman@codeaurora.org>
Enric Balletbo i Serra <eballetbo@kernel.org> <enric.balletbo@collabora.com>
Enric Balletbo i Serra <eballetbo@kernel.org> <eballetbo@iseebcn.com>
Eric Dumazet <edumazet@kernel.org> <edumazet@google.com>
Eric Dumazet <edumazet@kernel.org> <eric.dumazet@gmail.com>
Erik Kaneda <erik.kaneda@intel.com> <erik.schmauss@intel.com>
Ethan Carter Edwards <ethan@ethancedwards.com> Ethan Edwards <ethancarteredwards@gmail.com>
Eugen Hristev <ehristev@kernel.org> <eugen.hristev@microchip.com>
@ -977,6 +979,7 @@ Vladimir Davydov <vdavydov.dev@gmail.com> <vdavydov@virtuozzo.com>
Vlastimil Babka <vbabka@kernel.org> <vbabka@suse.cz>
WangYuli <wangyuli@aosc.io> <wangyl5933@chinaunicom.cn>
WangYuli <wangyuli@aosc.io> <wangyuli@deepin.org>
Wei Wang <weiwan@google.com> <weibunny@fb.com>
Weiwen Hu <huweiwen@linux.alibaba.com> <sehuww@mail.scut.edu.cn>
WeiXiong Liao <gmpy.liaowx@gmail.com> <liaoweixiong@allwinnertech.com>
Wen Gong <quic_wgong@quicinc.com> <wgong@codeaurora.org>

10
Documentation/admin-guide/kernel-parameters.txt
View File

@ -3510,7 +3510,7 @@ Kernel parameters
* SATA link speed limit: 1.5Gbps or 3.0Gbps.
* Transfer mode: pio[0-7], mwdma[0-4] and udma[0-7].
* Transfer mode: pio[0-6], mwdma[0-4] and udma[0-7].
udma[/][16,25,33,44,66,100,133] notation is also
allowed.
@ -3528,7 +3528,7 @@ Kernel parameters
* [no]ncqtrim: Enable or disable queued DSM TRIM.
* [no]ncqati: Enable or disable NCQ trim on ATI chipset.
* [no]ncqati: Enable or disable NCQ on ATI chipsets.
* [no]trim: Enable or disable (unqueued) TRIM.
@ -3559,11 +3559,9 @@ Kernel parameters
* max_sec_128: Set transfer size limit to 128 sectors.
* max_sec_1024: Set or clear transfer size limit to
1024 sectors.
* max_sec_1024: Set transfer size limit to 1024 sectors.
* max_sec_lba48: Set or clear transfer size limit to
65535 sectors.
* max_sec_lba48: Set transfer size limit to 65535 sectors.
* external: Mark port as external (hotplug-capable).

2
Documentation/arch/s390/pci.rst
View File

@ -67,7 +67,7 @@ Entries specific to zPCI functions and entries that hold zPCI information.
A physical function that currently supports a virtual function cannot be
powered off until all virtual functions are removed with:
echo 0 > /sys/bus/pci/devices/DDDD:BB:dd.f/sriov_numvf
echo 0 > /sys/bus/pci/devices/DDDD:BB:dd.f/sriov_numvfs
* /sys/bus/pci/devices/DDDD:BB:dd.f/:

26
Documentation/filesystems/netfs_library.rst
View File

@ -451,6 +451,32 @@ one.
The inode should be marked ``NETFS_ICTX_SINGLE_NO_UPLOAD`` if this API is to be
used. The writeback function requires the buffer to be of ITER_FOLIOQ type.
Clearing Stale Post-EOF Pagecache
---------------------------------
When a file is extended, data left in the pagecache past the old EOF by a write
through an mmap must not be exposed as file content. Netfslib clears this on
its own write paths, and exports a helper so a filesystem can do the same from
a resize path (truncate, setattr, fallocate and the like) that holds the
inode's ``i_rwsem`` exclusively across the whole resize::
void netfs_clear_stale_post_isize(struct inode *inode, uoff_t from,
uoff_t to);
This zeroes any such data within the ``[from, to)`` hole to be made, where
@from is the old EOF and @to is the new one, and the caller must have already
updated ``i_size`` to @to before calling it. Only the folio straddling @from
can hold data written past the EOF through an mmap, as pages wholly beyond the
EOF can't be faulted in, so the zeroing is limited to that folio. The folio is
zeroed rather than dropped so that a concurrent extending write can't lose
data.
This overlaps with ``pagecache_isize_extended()`` but can't reuse it: that
helper is keyed on a sub-page block size and is a no-op when the block size is
``>= PAGE_SIZE`` (as on network filesystems), and it doesn't wait for
writeback. As both address the same problem, a change to one should probably
be reflected in the other to keep them in sync.
High-Level VM API
==================

2
Documentation/netlink/specs/netdev.yaml
View File

@ -851,6 +851,8 @@ operations:
name: bind-tx
doc: Bind dmabuf to netdev for TX
attribute-set: dmabuf
# Intentionally unprivileged (no admin-perm / uns-admin-perm); see
# comment above netdev_nl_bind_tx_doit().
do:
request:
attributes:

52
Documentation/trace/events-landlock.rst
View File

@ -6,7 +6,7 @@ Landlock Trace Events
=====================
:Author: Mickaël Salaün
:Date: August 2026
:Date: September 2026
Landlock emits trace events for sandbox lifecycle operations and access
denials. These events can be consumed by ftrace (for human-readable
@ -31,8 +31,8 @@ Landlock trace events are organized in four categories:
**Syscall events** are emitted during Landlock system calls:
- ``landlock_create_ruleset``: a new ruleset is created
- ``landlock_add_rule_fs``: a filesystem rule is added to a ruleset
- ``landlock_add_rule_net``: a network port rule is added to a ruleset
- ``landlock_add_rule_path_beneath``: a filesystem rule is added to a ruleset
- ``landlock_add_rule_net_port``: a network port rule is added to a ruleset
- ``landlock_create_domain``: a new domain is created from a ruleset
- ``landlock_enforce_domain``: a domain is enforced on a thread
@ -47,8 +47,8 @@ Landlock trace events are organized in four categories:
**Rule evaluation events** are emitted during rule matching:
- ``landlock_check_rule_fs``: a filesystem rule is evaluated
- ``landlock_check_rule_net``: a network port rule is evaluated
- ``landlock_check_rule_inode``: an inode-keyed rule is evaluated
- ``landlock_check_rule_net_port``: a network-port-keyed rule is evaluated
**Lifecycle events**:
@ -141,10 +141,10 @@ in some field formats:
treated as untrusted input and escaped in the trace text output so it
cannot inject field separators or control characters.
- **Other party's domain**: A scope or ptrace denial compares the
subject's denying domain (``domain=``, always the enforcing domain and
never the current task) with the other party's domain, so these
tracepoints also report the other party's domain as a scalar ID:
- **Other party's domain**: A scope or ptrace denial compares the subject's
denying domain (``domain=``), which is the enforcing domain and not
necessarily the current task's domain, with the other party's domain.
These tracepoints also report the other party's domain as a scalar ID:
``tracee_domain=`` (ptrace), ``target_domain=`` (signal), and
``peer_domain=`` (abstract unix socket). It is ``0`` when the other
party is unsandboxed, and otherwise a domain ID that a consumer resolves
@ -189,7 +189,7 @@ rather than the caller's, so correlate those to the syscall by domain ID.
Interpreting check_rule events
==============================
The ``check_rule_fs`` and ``check_rule_net`` events expose the per-layer
The ``check_rule_inode`` and ``check_rule_net_port`` events expose the per-layer
rule evaluation, which is useful for understanding *why* a specific
access is allowed or denied.
@ -231,9 +231,10 @@ contribution, not the final decision:
through another matching rule, or the right is denied and appears in the
``blockers=`` field of the corresponding ``deny_access`` event.
To reconstruct the decision for an object, aggregate the ``grants=``
groups of all ``check_rule`` events emitted for that object during the
check.
For an access check that a consumer can delimit, aggregate the ``grants=``
groups of all matching ``check_rule`` events. These events do not carry a
request ID; use their execution context and generic tracepoints to separate
concurrent or successive checks of the same object.
.. note::
@ -244,29 +245,31 @@ check.
For example, a program sandboxed with read and execute access to the
whole filesystem reads ``/etc/passwd``; both the ``execve()`` and the
read match the rule covering ``/`` (inode 2), so ``check_rule_fs`` fires
read match the rule covering ``/`` (inode 2), so ``check_rule_inode`` fires
with the requested rights intersected against what that rule grants.
The ``access_request=`` mask includes ``truncate`` because the file-open hook
evaluates that optional right alongside the required access, but the
rule does not grant it, so ``truncate`` never appears in ``grants=``::
cat-127 [...] landlock_check_rule_fs: domain=1e40cb56f access_request=execute|read_file|truncate dev=0:17 ino=2 grants={execute|read_file}
cat-127 [...] landlock_check_rule_fs: domain=1e40cb56f access_request=read_file|truncate dev=0:17 ino=2 grants={read_file}
cat-127 [...] landlock_check_rule_inode: domain=1e40cb56f access_request=execute|read_file|truncate dev=0:17 ino=2 grants={execute|read_file}
cat-127 [...] landlock_check_rule_inode: domain=1e40cb56f access_request=read_file|truncate dev=0:17 ino=2 grants={read_file}
The ``[...]`` replaces the ftrace CPU, flags, and timestamp columns. A
single ``grants=`` group means the enforcing domain has one layer. With
two nested sandboxes that each grant the same rights, the rule spans both
layers, so ``grants=`` has one group per layer::
cat-128 [...] landlock_check_rule_fs: domain=184788b52 access_request=execute|read_file|truncate dev=0:17 ino=2 grants={execute|read_file,execute|read_file}
cat-128 [...] landlock_check_rule_fs: domain=184788b52 access_request=read_file|truncate dev=0:17 ino=2 grants={read_file,read_file}
cat-128 [...] landlock_check_rule_inode: domain=184788b52 access_request=execute|read_file|truncate dev=0:17 ino=2 grants={execute|read_file,execute|read_file}
cat-128 [...] landlock_check_rule_inode: domain=184788b52 access_request=read_file|truncate dev=0:17 ino=2 grants={read_file,read_file}
eBPF access
===========
eBPF programs attached via ``BPF_RAW_TRACEPOINT`` can access the
tracepoint arguments directly through BTF. The arguments include both
standard kernel objects and Landlock-internal objects:
BTF-enabled raw tracepoint programs attached through libbpf
``SEC("tp_btf/...")`` sections receive typed callback arguments. The
event prototypes in `Event reference`_ document their argument layouts.
The arguments include both standard kernel objects and Landlock-internal
objects:
- Standard kernel objects (``struct task_struct``, ``struct sock``,
``struct path``, ``struct dentry``) can be used with existing BPF
@ -277,7 +280,7 @@ standard kernel objects and Landlock-internal objects:
Internal struct layouts may change between kernel versions; use CO-RE
for field relocation.
A stateful eBPF program observes the full event stream and maintains
A stateful eBPF program attached before sandbox construction can maintain
per-domain state in BPF maps:
1. On ``landlock_create_domain``: record the domain ID and parent (the
@ -293,7 +296,10 @@ per-domain state in BPF maps:
final statistics.
This approach requires no kernel modification and no Landlock-specific
BPF helpers. The Landlock IDs serve as correlation keys across events.
BPF helpers. Landlock IDs serve as correlation keys within one boot.
Records exported through tracing or BPF buffers can be lost, and records
from different CPUs are not globally ordered, so consumers must detect and
reconcile incomplete state.
Audit filtering equivalence
===========================

83
MAINTAINERS
View File

@ -5141,6 +5141,7 @@ L: netdev@vger.kernel.org
L: openwrt-devel@lists.openwrt.org (subscribers-only)
S: Supported
F: Documentation/devicetree/bindings/net/dsa/brcm,b53.yaml
F: Documentation/networking/dsa/b53.rst
F: drivers/net/dsa/b53/*
F: drivers/net/dsa/bcm_sf2*
F: include/linux/dsa/brcm.h
@ -5452,6 +5453,7 @@ F: include/linux/brcmphy.h
BROADCOM GENET ETHERNET DRIVER
M: Doug Berger <opendmb@gmail.com>
M: Florian Fainelli <florian.fainelli@broadcom.com>
M: Nicolai Buchwitz <nb@tipi-net.de>
R: Broadcom internal kernel review list <bcm-kernel-feedback-list@broadcom.com>
L: netdev@vger.kernel.org
S: Maintained
@ -14903,7 +14905,7 @@ M: Damien Le Moal <dlemoal@kernel.org>
M: Niklas Cassel <cassel@kernel.org>
L: linux-ide@vger.kernel.org
S: Maintained
T: git git://git.kernel.org/pub/scm/linux/kernel/git/libata/linux.git
T: git git://git.kernel.org/pub/scm/linux/kernel/git/libata/linux.git for-next
F: Documentation/ABI/testing/sysfs-ata
F: Documentation/devicetree/bindings/ata/
F: drivers/ata/
@ -17206,31 +17208,6 @@ F: include/trace/events/ksm.h
F: mm/ksm.c
F: mm/mm_slot.h
MEMORY MANAGEMENT - MEMORY POLICY AND MIGRATION
M: Andrew Morton <akpm@linux-foundation.org>
M: David Hildenbrand <david@kernel.org>
R: Zi Yan <ziy@nvidia.com>
R: Matthew Brost <matthew.brost@intel.com>
R: Joshua Hahn <joshua.hahnjy@gmail.com>
R: Rakie Kim <rakie.kim@sk.com>
R: Byungchul Park <byungchul@sk.com>
R: Gregory Price <gourry@gourry.net>
R: Ying Huang <ying.huang@linux.alibaba.com>
R: Alistair Popple <apopple@nvidia.com>
L: linux-mm@kvack.org
S: Maintained
W: http://www.linux-mm.org
T: git git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm
F: Documentation/ABI/testing/sysfs-kernel-mm-mempolicy
F: Documentation/ABI/testing/sysfs-kernel-mm-mempolicy-weighted-interleave
F: include/linux/mempolicy.h
F: include/uapi/linux/mempolicy.h
F: include/linux/migrate.h
F: include/linux/migrate_mode.h
F: mm/mempolicy.c
F: mm/migrate.c
F: mm/migrate_device.c
MEMORY MANAGEMENT - MGLRU (MULTI-GEN LRU)
M: Andrew Morton <akpm@linux-foundation.org>
R: Kairui Song <kasong@tencent.com>
@ -17253,6 +17230,25 @@ F: include/linux/mmzone.h
F: mm/vmscan.c
F: mm/workingset.c
MEMORY MANAGEMENT - MIGRATION
M: Andrew Morton <akpm@linux-foundation.org>
M: David Hildenbrand <david@kernel.org>
R: Zi Yan <ziy@nvidia.com>
R: Matthew Brost <matthew.brost@intel.com>
R: Joshua Hahn <joshua.hahnjy@gmail.com>
R: Byungchul Park <byungchul@sk.com>
R: Gregory Price <gourry@gourry.net>
R: Ying Huang <ying.huang@linux.alibaba.com>
R: Alistair Popple <apopple@nvidia.com>
L: linux-mm@kvack.org
S: Maintained
W: http://www.linux-mm.org
T: git git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm
F: include/linux/migrate.h
F: include/linux/migrate_mode.h
F: mm/migrate.c
F: mm/migrate_device.c
MEMORY MANAGEMENT - MISC
M: Andrew Morton <akpm@linux-foundation.org>
M: David Hildenbrand <david@kernel.org>
@ -17268,7 +17264,6 @@ W: http://www.linux-mm.org
T: git git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm
F: Documentation/ABI/testing/sysfs-kernel-mm
F: Documentation/ABI/testing/sysfs-kernel-mm-cma
F: Documentation/ABI/testing/sysfs-kernel-mm-memory-tiers
F: Documentation/ABI/testing/sysfs-kernel-mm-numa
F: Documentation/admin-guide/mm/
F: Documentation/mm/
@ -17276,7 +17271,6 @@ F: drivers/char/mem.c
F: include/linux/cma.h
F: include/linux/dmapool.h
F: include/linux/ioremap.h
F: include/linux/memory-tiers.h
F: include/linux/page_idle.h
F: mm/backing-dev.c
F: mm/cma.c
@ -17288,7 +17282,6 @@ F: mm/early_ioremap.c
F: mm/fadvise.c
F: mm/ioremap.c
F: mm/mapping_dirty_helpers.c
F: mm/memory-tiers.c
F: mm/page_idle.c
F: mm/pgalloc-track.h
F: mm/process_vm_access.c
@ -17305,6 +17298,27 @@ F: mm/numa.c
F: mm/numa_emulation.c
F: mm/numa_memblks.c
MEMORY MANAGEMENT - NUMA PLACEMENT
M: Andrew Morton <akpm@linux-foundation.org>
M: David Hildenbrand <david@kernel.org>
M: Gregory Price <gourry@gourry.net>
R: Zi Yan <ziy@nvidia.com>
R: Joshua Hahn <joshua.hahnjy@gmail.com>
R: Rakie Kim <rakie.kim@sk.com>
R: Ying Huang <ying.huang@linux.alibaba.com>
L: linux-mm@kvack.org
S: Maintained
W: http://www.linux-mm.org
T: git git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm
F: Documentation/ABI/testing/sysfs-kernel-mm-memory-tiers
F: Documentation/ABI/testing/sysfs-kernel-mm-mempolicy
F: Documentation/ABI/testing/sysfs-kernel-mm-mempolicy-weighted-interleave
F: include/linux/memory-tiers.h
F: include/linux/mempolicy.h
F: include/uapi/linux/mempolicy.h
F: mm/memory-tiers.c
F: mm/mempolicy.c
MEMORY MANAGEMENT - OOM KILLER
M: Michal Hocko <mhocko@suse.com>
R: David Rientjes <rientjes@google.com>
@ -18882,7 +18896,7 @@ F: net/core/drop_monitor.c
NETWORKING DRIVERS
M: Andrew Lunn <andrew+netdev@lunn.ch>
M: "David S. Miller" <davem@davemloft.net>
M: Eric Dumazet <edumazet@google.com>
M: Eric Dumazet <edumazet@kernel.org>
M: Jakub Kicinski <kuba@kernel.org>
M: Paolo Abeni <pabeni@redhat.com>
L: netdev@vger.kernel.org
@ -18999,7 +19013,7 @@ K: struct\s+phy_port|phy_port_
NETWORKING [GENERAL]
M: "David S. Miller" <davem@davemloft.net>
M: Eric Dumazet <edumazet@google.com>
M: Eric Dumazet <edumazet@kernel.org>
M: Jakub Kicinski <kuba@kernel.org>
M: Paolo Abeni <pabeni@redhat.com>
R: Simon Horman <horms@kernel.org>
@ -19225,7 +19239,7 @@ F: net/ipv6/seg6*
F: tools/testing/selftests/net/srv6*
NETWORKING [TCP]
M: Eric Dumazet <edumazet@google.com>
M: Eric Dumazet <edumazet@kernel.org>
M: Neal Cardwell <ncardwell@google.com>
R: Kuniyuki Iwashima <kuniyu@google.com>
L: netdev@vger.kernel.org
@ -19258,7 +19272,7 @@ F: net/tls/
F: tools/testing/selftests/net/tls.c
NETWORKING [SOCKETS]
M: Eric Dumazet <edumazet@google.com>
M: Eric Dumazet <edumazet@kernel.org>
M: Kuniyuki Iwashima <kuniyu@google.com>
M: Paolo Abeni <pabeni@redhat.com>
M: Willem de Bruijn <willemb@google.com>
@ -20554,6 +20568,7 @@ ORACLE CLUSTER FILESYSTEM 2 (OCFS2)
M: Mark Fasheh <mark@fasheh.com>
M: Joel Becker <jlbec@evilplan.org>
M: Joseph Qi <joseph.qi@linux.alibaba.com>
R: Heming Zhao <heming.zhao@suse.com>
L: ocfs2-devel@lists.linux.dev
S: Supported
W: http://ocfs2.wiki.kernel.org
@ -25182,7 +25197,7 @@ R: David Rientjes <rientjes@google.com>
R: Roman Gushchin <roman.gushchin@linux.dev>
L: linux-mm@kvack.org
S: Maintained
T: git git://git.kernel.org/pub/scm/linux/kernel/git/vbabka/slab.git
T: git git://git.kernel.org/pub/scm/linux/kernel/git/mm/slab.git
F: Documentation/admin-guide/mm/slab.rst
F: Documentation/mm/slab.rst
F: include/linux/mempool.h

2
Makefile
View File

@ -2,7 +2,7 @@
VERSION = 7
PATCHLEVEL = 3
SUBLEVEL = 0
EXTRAVERSION = -rc4
EXTRAVERSION = -rc5
NAME = Baby Opossum Posse
# *DOCUMENTATION*

4
arch/arc/Kconfig
View File

@ -331,13 +331,9 @@ config ARC_FPU_SAVE_RESTORE
for floating point exceptions and rounding modes. These are
preserved across task context switch when enabled.
config ARC_CANT_LLSC
def_bool n
config ARC_HAS_LLSC
bool "Insn: LLOCK/SCOND (efficient atomic ops)"
default y
depends on !ARC_CANT_LLSC
if ISA_ARCV2

6
arch/arc/include/asm/atomic64-arcv2.h
View File

@ -156,6 +156,12 @@ static inline u64 __arch_cmpxchg64_relaxed(volatile void *ptr, u64 old, u64 new)
}
#define arch_cmpxchg64_relaxed __arch_cmpxchg64_relaxed
static inline s64 arch_atomic64_cmpxchg_relaxed(atomic64_t *v, s64 old, s64 new)
{
return arch_cmpxchg64_relaxed(&v->counter, old, new);
}
#define arch_atomic64_cmpxchg_relaxed arch_atomic64_cmpxchg_relaxed
static inline s64 arch_atomic64_xchg(atomic64_t *ptr, s64 new)
{
s64 prev;

4
arch/arc/include/asm/cmpxchg.h
View File

@ -46,7 +46,7 @@
__typeof__(*(ptr)) _n_ = (new); \
__typeof__(*(ptr)) _prev_; \
\
switch(sizeof((_p_))) { \
switch (sizeof(*(_p_))) { \
case 1: \
_prev_ = (__typeof__(*(ptr)))cmpxchg_emu_u8((volatile u8 *__force)_p_, (uintptr_t)_o_, (uintptr_t)_n_); \
break; \
@ -102,7 +102,7 @@
__typeof__(ptr) _p_ = (ptr); \
__typeof__(*(ptr)) _val_ = (val); \
\
switch(sizeof(*(_p_))) { \
switch (sizeof(*(_p_))) { \
case 4: \
_val_ = __arch_xchg(_p_, _val_); \
break; \

1
arch/arc/kernel/setup.c
View File

@ -593,6 +593,7 @@ static int show_cpuinfo(struct seq_file *m, void *v)
cpu_id);
} else {
freq = clk_get_rate(cpu_clk);
clk_put(cpu_clk);
}
if (freq)
seq_printf(m, "CPU speed\t: %lu.%02lu Mhz\n",

1
arch/arc/kernel/smp.c
View File

@ -12,7 +12,6 @@
#include <linux/spinlock.h>
#include <linux/sched/mm.h>
#include <linux/interrupt.h>
#include <linux/profile.h>
#include <linux/mm.h>
#include <linux/cpu.h>
#include <linux/irq.h>

2
arch/arm64/include/asm/kvm_host.h
View File

@ -322,7 +322,7 @@ struct kvm_arch {
* Stage 2 paging state for VMs with nested S2 using a virtual
* VMID.
*/
struct kvm_s2_mmu *nested_mmus;
struct kvm_s2_mmu **nested_mmus;
size_t nested_mmus_size;
int nested_mmus_next;

3
arch/arm64/include/asm/kvm_nested.h
View File

@ -66,7 +66,8 @@ static inline u64 translate_ttbr0_el2_to_ttbr0_el1(u64 ttbr0)
extern bool forward_smc_trap(struct kvm_vcpu *vcpu);
extern bool forward_debug_exception(struct kvm_vcpu *vcpu);
extern void kvm_init_nested(struct kvm *kvm);
extern int kvm_init_nested(struct kvm *kvm);
extern void kvm_destroy_nested(struct kvm *kvm);
extern int kvm_vcpu_init_nested(struct kvm_vcpu *vcpu);
extern void kvm_init_nested_s2_mmu(struct kvm_s2_mmu *mmu);
extern struct kvm_s2_mmu *lookup_s2_mmu(struct kvm_vcpu *vcpu);

3
arch/arm64/include/asm/kvm_pkvm.h
View File

@ -65,8 +65,7 @@ static inline bool kvm_pkvm_ioctl_allowed(struct kvm *kvm, unsigned int ioctl)
int r;
r = kvm_get_cap_for_kvm_ioctl(ioctl, &ext);
if (WARN_ON_ONCE(r < 0))
if (r < 0)
return false;
return kvm_pkvm_ext_allowed(kvm, ext);

2
arch/arm64/kernel/vdso/Makefile
View File

@ -41,7 +41,7 @@ CC_FLAGS_REMOVE_VDSO := $(CC_FLAGS_FTRACE) -Os $(CC_FLAGS_SCS) \
$(CC_FLAGS_LTO) $(CC_FLAGS_CFI) \
-Wmissing-prototypes -Wmissing-declarations
CC_FLAGS_ADD_VDSO := -O2 -mcmodel=tiny -fasynchronous-unwind-tables
CC_FLAGS_ADD_VDSO := -O2 -mcmodel=tiny -fasynchronous-unwind-tables $(CONFIG_CC_OPT_INLINE_MEMSET)
CFLAGS_REMOVE_vgettimeofday.o = $(CC_FLAGS_REMOVE_VDSO)
CFLAGS_REMOVE_vgetrandom.o = $(CC_FLAGS_REMOVE_VDSO)

10
arch/arm64/kvm/arm.c
View File

@ -236,8 +236,6 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned long type)
mutex_unlock(&kvm->lock);
#endif
kvm_init_nested(kvm);
ret = kvm_share_hyp(kvm, kvm + 1);
if (ret)
return ret;
@ -252,6 +250,10 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned long type)
if (ret)
goto err_free_cpumask;
ret = kvm_init_nested(kvm);
if (ret)
goto err_uninit_mmu;
if (is_protected_kvm_enabled()) {
/*
* If any failures occur after this is successful, make sure to
@ -280,6 +282,7 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned long type)
err_uninit_mmu:
kvm_uninit_stage2_mmu(kvm);
kvm_destroy_nested(kvm);
err_free_cpumask:
free_cpumask_var(kvm->arch.supported_cpus);
err_unshare_kvm:
@ -337,6 +340,7 @@ void kvm_arch_destroy_vm(struct kvm *kvm)
kvm_unshare_hyp(kvm, kvm + 1);
kvm_destroy_nested(kvm);
kvm_arm_teardown_hypercalls(kvm);
}
@ -1129,7 +1133,7 @@ static int kvm_vcpu_suspend(struct kvm_vcpu *vcpu)
static int check_vcpu_requests(struct kvm_vcpu *vcpu)
{
if (kvm_request_pending(vcpu)) {
if (kvm_check_request(KVM_REQ_VM_DEAD, vcpu))
if (kvm_test_request(KVM_REQ_VM_DEAD, vcpu))
return -EIO;
if (kvm_check_request(KVM_REQ_SLEEP, vcpu))

5
arch/arm64/kvm/emulate-nested.c
View File

@ -1445,7 +1445,7 @@ static const struct encoding_to_trap_config encoding_to_fgt[] __initconst = {
SR_FGT(OP_AT_S1E1A, HFGITR, ATS1E1A, 1),
SR_FGT(OP_COSP_RCTX, HFGITR, COSPRCTX, 1),
SR_FGT(OP_GCSPUSHX, HFGITR, nGCSEPP, 0),
SR_FGT(OP_GCSPOPX, HFGITR, nGCSEPP, 0),
SR_FGT(OP_GCSPOPCX, HFGITR, nGCSEPP, 0),
SR_FGT(OP_GCSPUSHM, HFGITR, nGCSPUSHM_EL1, 0),
SR_FGT(OP_BRB_IALL, HFGITR, nBRBIALL, 0),
SR_FGT(OP_BRB_INJ, HFGITR, nBRBINJ, 0),
@ -2386,9 +2386,6 @@ int __init populate_nv_trap_config(void)
print_nv_trap_error(fgt, "FGT bit is reserved", ret);
}
if (!cpus_have_final_cap(ARM64_HAS_FGT))
continue;
prev = xa_store(&sr_forward_xa, enc,
xa_mk_value(tc.val), GFP_KERNEL);

2
arch/arm64/kvm/hyp/include/nvhe/mem_protect.h
View File

@ -52,8 +52,10 @@ int __pkvm_host_test_clear_young_guest(u64 gfn, u64 nr_pages, bool mkold, struct
int __pkvm_host_mkyoung_guest(u64 gfn, struct pkvm_hyp_vcpu *vcpu);
bool addr_is_memory(phys_addr_t phys);
bool addr_is_hyp_text(phys_addr_t phys);
int host_stage2_idmap_locked(phys_addr_t addr, u64 size, enum kvm_pgtable_prot prot);
int host_stage2_set_owner_locked(phys_addr_t addr, u64 size, u8 owner_id);
bool host_stage2_pte_is_hyp_owned(kvm_pte_t pte);
int kvm_host_prepare_stage2(void *pgt_pool_base);
int kvm_guest_prepare_stage2(struct pkvm_hyp_vm *vm, void *pgd);
void kvm_guest_destroy_stage2(struct pkvm_hyp_vm *vm);

1
arch/arm64/kvm/hyp/include/nvhe/mm.h
View File

@ -29,6 +29,7 @@ int __pkvm_create_private_mapping(phys_addr_t phys, size_t size,
enum kvm_pgtable_prot prot,
unsigned long *haddr);
int pkvm_create_stack(phys_addr_t phys, unsigned long *haddr);
int pkvm_check_host_ownership(void);
int pkvm_alloc_private_va_range(size_t size, unsigned long *haddr);
#endif /* __KVM_HYP_MM_H */

9
arch/arm64/kvm/hyp/include/nvhe/pkvm.h
View File

@ -12,6 +12,15 @@
#include <nvhe/gfp.h>
#include <nvhe/spinlock.h>
/*
* HCR_EL2 bits EL2 takes from the host on each entry, per VM type. The rest
* are EL2's own and nothing the host sets there reaches the guest.
*/
#define PKVM_HCR_EL2_HOST_PVM (HCR_EL2_TWI | HCR_EL2_TWE | HCR_EL2_VSE)
#define PKVM_HCR_EL2_HOST_NPVM (PKVM_HCR_EL2_HOST_PVM | HCR_EL2_VI | HCR_EL2_VF | \
HCR_EL2_TVM | HCR_EL2_TID2 | HCR_EL2_TID4 | \
HCR_EL2_TID5 | HCR_EL2_TTLBOS)
/*
* Holds the relevant data for maintaining the vcpu state completely at hyp.
*/

7
arch/arm64/kvm/hyp/nvhe/hyp-main.c
View File

@ -216,6 +216,7 @@ static void sync_debug_state(struct pkvm_hyp_vcpu *hyp_vcpu)
static void flush_hyp_vcpu(struct pkvm_hyp_vcpu *hyp_vcpu)
{
struct kvm_vcpu *host_vcpu = hyp_vcpu->host_vcpu;
u64 host_hcr_mask = PKVM_HCR_EL2_HOST_PVM;
fpsimd_sve_flush();
flush_debug_state(hyp_vcpu);
@ -228,6 +229,7 @@ static void flush_hyp_vcpu(struct pkvm_hyp_vcpu *hyp_vcpu)
if (!pkvm_hyp_vcpu_is_protected(hyp_vcpu)) {
if (vcpu_get_flag(host_vcpu, PKVM_HOST_STATE_DIRTY))
flush_hyp_vcpu_state(hyp_vcpu);
host_hcr_mask = PKVM_HCR_EL2_HOST_NPVM;
} else {
hyp_vcpu->vcpu.arch.ctxt = host_vcpu->arch.ctxt;
}
@ -241,9 +243,8 @@ static void flush_hyp_vcpu(struct pkvm_hyp_vcpu *hyp_vcpu)
* trap-control bit, so it must flow to the hyp vCPU alongside TWI/TWE
* for the vSError to be delivered. sync_hyp_vcpu() reflects it back.
*/
hyp_vcpu->vcpu.arch.hcr_el2 &= ~(HCR_TWI | HCR_TWE | HCR_VSE);
hyp_vcpu->vcpu.arch.hcr_el2 |= READ_ONCE(host_vcpu->arch.hcr_el2) &
(HCR_TWI | HCR_TWE | HCR_VSE);
hyp_vcpu->vcpu.arch.hcr_el2 &= ~host_hcr_mask;
hyp_vcpu->vcpu.arch.hcr_el2 |= READ_ONCE(host_vcpu->arch.hcr_el2) & host_hcr_mask;
hyp_vcpu->vcpu.arch.iflags = host_vcpu->arch.iflags;

20
arch/arm64/kvm/hyp/nvhe/mem_protect.c
View File

@ -450,6 +450,14 @@ bool addr_is_memory(phys_addr_t phys)
return !!find_mem_range(phys, &range);
}
bool addr_is_hyp_text(phys_addr_t phys)
{
phys_addr_t start = ALIGN_DOWN(__hyp_pa(__hyp_text_start), PAGE_SIZE);
phys_addr_t end = PAGE_ALIGN(__hyp_pa(__hyp_text_end));
return phys >= start && phys < end;
}
static bool is_in_mem_range(u64 addr, struct kvm_mem_range *range)
{
return range->start <= addr && addr < range->end;
@ -633,6 +641,18 @@ int host_stage2_set_owner_locked(phys_addr_t addr, u64 size, u8 owner_id)
return ret;
}
bool host_stage2_pte_is_hyp_owned(kvm_pte_t pte)
{
if (kvm_pte_valid(pte))
return false;
if (FIELD_GET(KVM_INVALID_PTE_TYPE_MASK, pte) !=
KVM_HOST_INVALID_PTE_TYPE_DONATION)
return false;
return FIELD_GET(KVM_HOST_DONATION_PTE_OWNER_MASK, pte) == PKVM_ID_HYP;
}
#define KVM_HOST_PTE_OWNER_GUEST_HANDLE_MASK GENMASK(15, 0)
/* We need 40 bits for the GFN to cover a 52-bit IPA with 4k pages and LPA2 */
#define KVM_HOST_PTE_OWNER_GUEST_GFN_MASK GENMASK(55, 16)

79
arch/arm64/kvm/hyp/nvhe/mm.c
View File

@ -25,6 +25,7 @@ struct memblock_region hyp_memory[HYP_MEMBLOCK_REGIONS];
unsigned int hyp_memblock_nr;
static u64 __io_map_base;
static u64 __io_map_next;
struct hyp_fixmap_slot {
u64 addr;
@ -50,7 +51,7 @@ static int __pkvm_alloc_private_va_range(unsigned long start, size_t size)
hyp_assert_lock_held(&pkvm_pgd_lock);
if (!start || start < __io_map_base)
if (!start || start < __io_map_next)
return -EINVAL;
/* The allocated size is always a multiple of PAGE_SIZE */
@ -60,7 +61,7 @@ static int __pkvm_alloc_private_va_range(unsigned long start, size_t size)
if (cur > __hyp_vmemmap)
return -ENOMEM;
__io_map_base = cur;
__io_map_next = cur;
return 0;
}
@ -70,7 +71,7 @@ static int __pkvm_alloc_private_va_range(unsigned long start, size_t size)
* @size: The size of the VA range to reserve.
* @haddr: The hypervisor virtual start address of the allocation.
*
* The private virtual address (VA) range is allocated above __io_map_base
* The private virtual address (VA) range is allocated above __io_map_next
* and aligned based on the order of @size.
*
* Return: 0 on success or negative error code on failure.
@ -81,7 +82,7 @@ int pkvm_alloc_private_va_range(size_t size, unsigned long *haddr)
int ret;
hyp_spin_lock(&pkvm_pgd_lock);
addr = __io_map_base;
addr = __io_map_next;
ret = __pkvm_alloc_private_va_range(addr, size);
hyp_spin_unlock(&pkvm_pgd_lock);
@ -341,7 +342,7 @@ static int create_fixblock(void)
return -EINVAL;
hyp_spin_lock(&pkvm_pgd_lock);
addr = ALIGN(__io_map_base, PMD_SIZE);
addr = ALIGN(__io_map_next, PMD_SIZE);
ret = __pkvm_alloc_private_va_range(addr, PMD_SIZE);
if (ret)
goto unlock;
@ -426,6 +427,7 @@ int hyp_create_idmap(u32 hyp_va_bits)
*/
__io_map_base = start & BIT(hyp_va_bits - 2);
__io_map_base ^= BIT(hyp_va_bits - 2);
__io_map_next = __io_map_base;
__hyp_vmemmap = __io_map_base | BIT(hyp_va_bits - 3);
return __pkvm_create_mappings(start, end - start, start, PAGE_HYP_EXEC);
@ -433,19 +435,19 @@ int hyp_create_idmap(u32 hyp_va_bits)
int pkvm_create_stack(phys_addr_t phys, unsigned long *haddr)
{
unsigned long addr, prev_base;
unsigned long addr, prev_next;
size_t size;
int ret;
hyp_spin_lock(&pkvm_pgd_lock);
prev_base = __io_map_base;
prev_next = __io_map_next;
/*
* Efficient stack verification using the NVHE_STACK_SHIFT bit implies
* an alignment of our allocation on the order of the size.
*/
size = NVHE_STACK_SIZE * 2;
addr = ALIGN(__io_map_base, size);
addr = ALIGN(__io_map_next, size);
ret = __pkvm_alloc_private_va_range(addr, size);
if (!ret) {
@ -461,7 +463,7 @@ int pkvm_create_stack(phys_addr_t phys, unsigned long *haddr)
ret = kvm_pgtable_hyp_map(&pkvm_pgtable, addr + NVHE_STACK_SIZE,
NVHE_STACK_SIZE, phys, PAGE_HYP);
if (ret)
__io_map_base = prev_base;
__io_map_next = prev_next;
}
hyp_spin_unlock(&pkvm_pgd_lock);
@ -470,6 +472,65 @@ int pkvm_create_stack(phys_addr_t phys, unsigned long *haddr)
return ret;
}
static int check_page_ownership(phys_addr_t phys)
{
kvm_pte_t pte;
bool host_ok;
int ret;
if (addr_is_memory(phys)) {
struct hyp_page *page = hyp_phys_to_page(phys);
if (get_hyp_state(page) != PKVM_PAGE_OWNED ||
get_host_state(page) != PKVM_NOPAGE)
return -EPERM;
}
ret = kvm_pgtable_get_leaf(&host_mmu.pgt, phys, &pte, NULL);
if (ret)
return ret;
/* Hyp text may stay host-readable, see fix_host_ownership_walker(). */
if (kvm_pte_valid(pte) && addr_is_hyp_text(phys))
host_ok = !(kvm_pgtable_stage2_pte_prot(pte) & KVM_PGTABLE_PROT_W);
else
host_ok = host_stage2_pte_is_hyp_owned(pte);
return host_ok ? 0 : -EPERM;
}
static int check_host_ownership_walker(const struct kvm_pgtable_visit_ctx *ctx,
enum kvm_pgtable_walk_flags visit)
{
phys_addr_t phys, end;
int ret;
if (!kvm_pte_valid(ctx->old))
return 0;
phys = kvm_pte_to_phys(ctx->old);
end = phys + kvm_granule_size(ctx->level);
for (; phys < end; phys += PAGE_SIZE) {
ret = check_page_ownership(phys);
if (ret)
return ret;
}
return 0;
}
int pkvm_check_host_ownership(void)
{
struct kvm_pgtable_walker walker = {
.cb = check_host_ownership_walker,
.flags = KVM_PGTABLE_WALK_LEAF,
};
/* The private range and the vmemmap share one quarter of the VA space. */
return kvm_pgtable_walk(&pkvm_pgtable, __io_map_base,
BIT(pkvm_pgtable.ia_bits - 2), &walker);
}
static void *admit_host_page(void *arg)
{
struct kvm_hyp_memcache *host_mc = arg;

54
arch/arm64/kvm/hyp/nvhe/pkvm.c
View File

@ -30,6 +30,7 @@ unsigned int kvm_host_sve_max_vl;
*/
static DEFINE_PER_CPU(struct pkvm_hyp_vcpu *, loaded_hyp_vcpu);
/* The PKVM_HCR_EL2_HOST_{PVM,NPVM} bits of this value come from the host on each entry. */
static void pkvm_vcpu_reset_hcr(struct kvm_vcpu *vcpu)
{
vcpu->arch.hcr_el2 = HCR_GUEST_FLAGS;
@ -47,18 +48,17 @@ static void pkvm_vcpu_reset_hcr(struct kvm_vcpu *vcpu)
if (cpus_have_final_cap(ARM64_HAS_STAGE2_FWB))
vcpu->arch.hcr_el2 |= HCR_FWB;
if (cpus_have_final_cap(ARM64_HAS_EVT) &&
!cpus_have_final_cap(ARM64_MISMATCHED_CACHE_TYPE) &&
kvm_read_vm_id_reg(vcpu->kvm, SYS_CTR_EL0) == read_cpuid(CTR_EL0))
vcpu->arch.hcr_el2 |= HCR_TID4;
else
vcpu->arch.hcr_el2 |= HCR_TID2;
/*
* Without AArch32 EL1, leave RW set and let the entry fail with an
* illegal exception return: the *32_EL2 registers EL2 would otherwise
* switch are UNDEFINED there.
*/
if (vcpu_has_feature(vcpu, KVM_ARM_VCPU_EL1_32BIT) &&
cpus_have_final_cap(ARM64_HAS_32BIT_EL1))
vcpu->arch.hcr_el2 &= ~HCR_EL2_RW;
if (vcpu_has_ptrauth(vcpu))
vcpu->arch.hcr_el2 |= (HCR_API | HCR_APK);
if (kvm_has_mte(vcpu->kvm))
vcpu->arch.hcr_el2 |= HCR_ATA;
}
static void pvm_init_traps_hcr(struct kvm_vcpu *vcpu)
@ -76,6 +76,13 @@ static void pvm_init_traps_hcr(struct kvm_vcpu *vcpu)
*/
val |= HCR_TACR | HCR_TIDCP | HCR_TID3 | HCR_TID1;
if (cpus_have_final_cap(ARM64_HAS_EVT) &&
!cpus_have_final_cap(ARM64_MISMATCHED_CACHE_TYPE) &&
kvm_read_vm_id_reg(kvm, SYS_CTR_EL0) == read_cpuid(CTR_EL0))
val |= HCR_EL2_TID4;
else
val |= HCR_EL2_TID2;
if (!kvm_has_feat(kvm, ID_AA64PFR0_EL1, RAS, IMP)) {
val |= HCR_TERR | HCR_TEA;
val &= ~(HCR_FIEN);
@ -360,7 +367,7 @@ static void pkvm_init_features_from_host(struct pkvm_hyp_vm *hyp_vm, const struc
if (test_bit(KVM_ARCH_FLAG_WRITABLE_IMP_ID_REGS, &host_arch_flags))
hyp_vm->kvm.arch.midr_el1 = host_kvm->arch.midr_el1;
return;
goto out;
}
if (kvm_pkvm_ext_allowed(kvm, KVM_CAP_ARM_MTE))
@ -379,13 +386,14 @@ static void pkvm_init_features_from_host(struct pkvm_hyp_vm *hyp_vm, const struc
if (kvm_pkvm_ext_allowed(kvm, KVM_CAP_ARM_PTRAUTH_GENERIC))
set_bit(KVM_ARM_VCPU_PTRAUTH_GENERIC, allowed_features);
if (kvm_pkvm_ext_allowed(kvm, KVM_CAP_ARM_SVE)) {
if (kvm_pkvm_ext_allowed(kvm, KVM_CAP_ARM_SVE))
set_bit(KVM_ARM_VCPU_SVE, allowed_features);
kvm->arch.flags |= host_arch_flags & BIT(KVM_ARCH_FLAG_GUEST_HAS_SVE);
}
bitmap_and(kvm->arch.vcpu_features, host_kvm->arch.vcpu_features,
allowed_features, KVM_VCPU_MAX_FEATURES);
out:
__assign_bit(KVM_ARCH_FLAG_GUEST_HAS_SVE, &kvm->arch.flags,
kvm_vcpu_has_feature(kvm, KVM_ARM_VCPU_SVE));
}
static void unpin_host_vcpu(struct kvm_vcpu *host_vcpu)
@ -398,10 +406,10 @@ static void unpin_host_sve_state(struct pkvm_hyp_vcpu *hyp_vcpu)
{
void *sve_state;
if (!vcpu_has_feature(&hyp_vcpu->vcpu, KVM_ARM_VCPU_SVE))
sve_state = hyp_vcpu->vcpu.arch.sve_state;
if (!sve_state)
return;
sve_state = hyp_vcpu->vcpu.arch.sve_state;
hyp_unpin_shared_mem(sve_state,
sve_state + vcpu_sve_state_size(&hyp_vcpu->vcpu));
}
@ -450,7 +458,7 @@ static int pkvm_vcpu_init_sve(struct pkvm_hyp_vcpu *hyp_vcpu, struct kvm_vcpu *h
unsigned int sve_max_vl;
size_t sve_state_size;
void *sve_state;
int ret = 0;
int ret;
if (!vcpu_has_feature(vcpu, KVM_ARM_VCPU_SVE)) {
vcpu_clear_flag(vcpu, VCPU_SVE_FINALIZED);
@ -459,25 +467,21 @@ static int pkvm_vcpu_init_sve(struct pkvm_hyp_vcpu *hyp_vcpu, struct kvm_vcpu *h
/* Limit guest vector length to the maximum supported by the host. */
sve_max_vl = min(READ_ONCE(host_vcpu->arch.sve_max_vl), kvm_host_sve_max_vl);
sve_state_size = sve_state_size_from_vl(sve_max_vl);
sve_state = kern_hyp_va(READ_ONCE(host_vcpu->arch.sve_state));
if (!sve_state || !sve_state_size) {
ret = -EINVAL;
goto err;
}
if (!sve_vl_valid(sve_max_vl) || !sve_state)
return -EINVAL;
sve_state_size = sve_state_size_from_vl(sve_max_vl);
ret = hyp_pin_shared_mem(sve_state, sve_state + sve_state_size);
if (ret)
goto err;
return ret;
vcpu->arch.sve_state = sve_state;
vcpu->arch.sve_max_vl = sve_max_vl;
return 0;
err:
clear_bit(KVM_ARM_VCPU_SVE, vcpu->kvm->arch.vcpu_features);
return ret;
}
static int vm_copy_id_regs(struct pkvm_hyp_vcpu *hyp_vcpu)

16
arch/arm64/kvm/hyp/nvhe/setup.c
View File

@ -217,7 +217,7 @@ static int fix_host_ownership_walker(const struct kvm_pgtable_visit_ctx *ctx,
case PKVM_PAGE_OWNED:
set_hyp_state(page, PKVM_PAGE_OWNED);
/* hyp text is RO in the host stage-2 to be inspected on panic. */
if (prot == PAGE_HYP_EXEC) {
if (addr_is_hyp_text(phys)) {
set_host_state(page, PKVM_NOPAGE);
return host_stage2_idmap_locked(phys, PAGE_SIZE, KVM_PGTABLE_PROT_R);
} else {
@ -269,6 +269,16 @@ static int fix_host_ownership(void)
return ret;
}
/* The stacks sit in the private VA range, not the linear map. */
for (i = 0; i < hyp_nr_cpus; i++) {
struct kvm_nvhe_init_params *params = per_cpu_ptr(&kvm_init_params, i);
u64 start = params->stack_hyp_va - NVHE_STACK_SIZE;
ret = kvm_pgtable_walk(&pkvm_pgtable, start, NVHE_STACK_SIZE, &walker);
if (ret)
return ret;
}
return 0;
}
@ -324,6 +334,10 @@ void __noreturn __pkvm_init_finalise(void)
if (ret)
goto out;
ret = pkvm_check_host_ownership();
if (ret)
goto out;
ret = hyp_ffa_init(ffa_proxy_pages);
if (ret)
goto out;

3
arch/arm64/kvm/hypercalls.c
View File

@ -185,7 +185,8 @@ static int kvm_smccc_set_filter(struct kvm *kvm, struct kvm_smccc_filter __user
start = filter.base;
end = start + filter.nr_functions - 1;
if (end < start || filter.action >= NR_SMCCC_FILTER_ACTIONS)
if (!filter.nr_functions || end < start ||
filter.action >= NR_SMCCC_FILTER_ACTIONS)
return -EINVAL;
mutex_lock(&kvm->arch.config_lock);

62
arch/arm64/kvm/mmu.c
View File

@ -59,27 +59,36 @@ static phys_addr_t stage2_range_addr_end(phys_addr_t addr, phys_addr_t end)
* long will also starve other vCPUs. We have to also make sure that the page
* tables are not freed while we released the lock.
*/
static int stage2_apply_range(struct kvm_s2_mmu *mmu, phys_addr_t addr,
static int stage2_apply_range(struct kvm_s2_mmu *mmu, phys_addr_t start,
phys_addr_t end,
int (*fn)(struct kvm_pgtable *, u64, u64),
bool resched)
{
struct kvm *kvm = kvm_s2_mmu_to_kvm(mmu);
bool lock_dropped = false;
phys_addr_t addr = start;
int ret;
u64 next;
do {
struct kvm_pgtable *pgt = mmu->pgt;
/*
* We may be raced on PGT teardown when we release the
* kvm->mmu_lock. That's fine as the PGT is legitimately no
* longer present.
*/
if (!pgt)
return -EINVAL;
return lock_dropped ? 0 : -EINVAL;
next = stage2_range_addr_end(addr, end);
ret = fn(pgt, addr, next - addr);
if (ret)
break;
if (resched && next != end)
if (resched && next != end) {
cond_resched_rwlock_write(&kvm->mmu_lock);
lock_dropped = true;
}
} while (addr = next, addr != end);
return ret;
@ -1459,32 +1468,11 @@ transparent_hugepage_adjust(struct kvm *kvm, struct kvm_memory_slot *memslot,
return PAGE_SIZE;
}
static int get_vma_page_shift(struct vm_area_struct *vma, unsigned long hva)
static int get_vma_page_shift(struct vm_area_struct *vma)
{
unsigned long pa;
if (is_vm_hugetlb_page(vma) && !(vma->vm_flags & VM_PFNMAP))
if (is_vm_hugetlb_page(vma))
return huge_page_shift(hstate_vma(vma));
if (!(vma->vm_flags & VM_PFNMAP))
return PAGE_SHIFT;
VM_BUG_ON(is_vm_hugetlb_page(vma));
pa = (vma->vm_pgoff << PAGE_SHIFT) + (hva - vma->vm_start);
#ifndef __PAGETABLE_PMD_FOLDED
if ((hva & (PUD_SIZE - 1)) == (pa & (PUD_SIZE - 1)) &&
ALIGN_DOWN(hva, PUD_SIZE) >= vma->vm_start &&
ALIGN(hva, PUD_SIZE) <= vma->vm_end)
return PUD_SHIFT;
#endif
if ((hva & (PMD_SIZE - 1)) == (pa & (PMD_SIZE - 1)) &&
ALIGN_DOWN(hva, PMD_SIZE) >= vma->vm_start &&
ALIGN(hva, PMD_SIZE) <= vma->vm_end)
return PMD_SHIFT;
return PAGE_SHIFT;
}
@ -1785,7 +1773,7 @@ static short kvm_s2_resolve_vma_size(const struct kvm_s2_fault_desc *s2fd,
vma_shift = PAGE_SHIFT;
} else {
s2vi->max_map_size = PUD_SIZE;
vma_shift = get_vma_page_shift(vma, s2fd->hva);
vma_shift = get_vma_page_shift(vma);
}
switch (vma_shift) {
@ -1943,16 +1931,6 @@ static int kvm_s2_fault_pin_pfn(const struct kvm_s2_fault_desc *s2fd,
return -EFAULT;
}
} else {
/*
* If the page was identified as device early by looking at
* the VMA flags, vma_pagesize is already representing the
* largest quantity we can map. If instead it was mapped
* via __kvm_faultin_pfn(), vma_pagesize is set to PAGE_SIZE
* and must not be upgraded.
*
* In both cases, we don't let transparent_hugepage_adjust()
* change things at the last minute.
*/
s2vi->map_non_cacheable = true;
}
@ -2042,10 +2020,10 @@ static int kvm_s2_fault_map(const struct kvm_s2_fault_desc *s2fd,
/*
* If we are not forced to use page mapping, check if we are
* backed by a THP and thus use block mapping if possible.
* backed by a huge stage-1 mapping and thus use block mapping if
* possible.
*/
if (mapping_size == PAGE_SIZE &&
!(s2vi->max_map_size == PAGE_SIZE || s2vi->map_non_cacheable)) {
if (mapping_size == PAGE_SIZE && s2vi->max_map_size != PAGE_SIZE) {
if (perm_fault_granule > PAGE_SIZE) {
mapping_size = perm_fault_granule;
} else {
@ -2126,10 +2104,6 @@ static int user_mem_abort(const struct kvm_s2_fault_desc *s2fd)
return ret;
}
/*
* Let's check if we will get back a huge page backed by hugetlbfs, or
* get block mapping for device MMIO region.
*/
ret = kvm_s2_fault_pin_pfn(s2fd, &s2vi);
if (ret != 1)
return ret;

115
arch/arm64/kvm/nested.c
View File

@ -45,11 +45,24 @@ struct vncr_tlb {
*/
#define S2_MMU_PER_VCPU 2
void kvm_init_nested(struct kvm *kvm)
int kvm_init_nested(struct kvm *kvm)
{
kvm->arch.nested_mmus = NULL;
kvm->arch.nested_mmus = kvmalloc_objs(struct kvm_s2_mmu *,
KVM_MAX_VCPUS * S2_MMU_PER_VCPU,
GFP_KERNEL_ACCOUNT);
kvm->arch.nested_mmus_size = 0;
atomic_set(&kvm->arch.vncr_tlb_count, 0);
return kvm->arch.nested_mmus ? 0 : -ENOMEM;
}
void kvm_destroy_nested(struct kvm *kvm)
{
for (int i = 0; i < kvm->arch.nested_mmus_size; i+= S2_MMU_PER_VCPU)
kvfree(kvm->arch.nested_mmus[i]);
kvm->arch.nested_mmus_size = 0;
kvfree(kvm->arch.nested_mmus);
}
static int init_nested_s2_mmu(struct kvm *kvm, struct kvm_s2_mmu *mmu)
@ -70,8 +83,9 @@ static int init_nested_s2_mmu(struct kvm *kvm, struct kvm_s2_mmu *mmu)
int kvm_vcpu_init_nested(struct kvm_vcpu *vcpu)
{
struct kvm *kvm = vcpu->kvm;
struct kvm_s2_mmu *tmp;
int num_mmus, ret = 0;
int num_mmus;
lockdep_assert_held(&kvm->arch.config_lock);
if (test_bit(KVM_ARM_VCPU_HAS_EL2_E2H0, kvm->arch.vcpu_features) &&
!cpus_have_final_cap(ARM64_HAS_HCR_NV1))
@ -84,51 +98,40 @@ int kvm_vcpu_init_nested(struct kvm_vcpu *vcpu)
if (!vcpu->arch.ctxt.vncr_array)
return -ENOMEM;
/*
* Let's treat memory allocation failures as benign: If we fail to
* allocate anything, return an error and keep the allocated array
* alive. Userspace may try to recover by initializing the vcpu
* again, and there is no reason to affect the whole VM for this.
*/
num_mmus = atomic_read(&kvm->online_vcpus) * S2_MMU_PER_VCPU;
if (num_mmus > kvm->arch.nested_mmus_size) {
tmp = kvzalloc_objs(*tmp, num_mmus, GFP_KERNEL_ACCOUNT);
struct kvm_s2_mmu *tmp;
int i, ret = 0;
tmp = kvzalloc_objs(*tmp, S2_MMU_PER_VCPU, GFP_KERNEL_ACCOUNT);
if (!tmp)
return -ENOMEM;
ret = -ENOMEM;
write_lock(&kvm->mmu_lock);
if (kvm->arch.nested_mmus_size) {
memcpy(tmp, kvm->arch.nested_mmus,
size_mul(sizeof(*tmp), kvm->arch.nested_mmus_size));
for (int i = 0; i < kvm->arch.nested_mmus_size; i++)
tmp[i].pgt->mmu = &tmp[i];
for (i = 0; !ret && i < S2_MMU_PER_VCPU; i++) {
ret = init_nested_s2_mmu(kvm, &tmp[i]);
if (ret)
break;
}
swap(kvm->arch.nested_mmus, tmp);
if (ret) {
while (--i >= 0)
kvm_free_stage2_pgd(&tmp[i]);
write_unlock(&kvm->mmu_lock);
kvfree(tmp);
free_page((unsigned long)vcpu->arch.ctxt.vncr_array);
vcpu->arch.ctxt.vncr_array = NULL;
return ret;
}
kvfree(tmp);
guard(write_lock)(&kvm->mmu_lock);
for (i = 0; i < S2_MMU_PER_VCPU; i++)
kvm->arch.nested_mmus[i + kvm->arch.nested_mmus_size] = &tmp[i];
kvm->arch.nested_mmus_size += S2_MMU_PER_VCPU;
}
for (int i = kvm->arch.nested_mmus_size; !ret && i < num_mmus; i++)
ret = init_nested_s2_mmu(kvm, &kvm->arch.nested_mmus[i]);
if (ret) {
for (int i = kvm->arch.nested_mmus_size; i < num_mmus; i++)
kvm_free_stage2_pgd(&kvm->arch.nested_mmus[i]);
free_page((unsigned long)vcpu->arch.ctxt.vncr_array);
vcpu->arch.ctxt.vncr_array = NULL;
return ret;
}
kvm->arch.nested_mmus_size = num_mmus;
return 0;
}
@ -742,7 +745,7 @@ void kvm_s2_mmu_iterate_by_vmid(struct kvm *kvm, u16 vmid,
write_lock(&kvm->mmu_lock);
for (int i = 0; i < kvm->arch.nested_mmus_size; i++) {
struct kvm_s2_mmu *mmu = &kvm->arch.nested_mmus[i];
struct kvm_s2_mmu *mmu = kvm->arch.nested_mmus[i];
if (!kvm_s2_mmu_valid(mmu))
continue;
@ -784,7 +787,7 @@ struct kvm_s2_mmu *lookup_s2_mmu(struct kvm_vcpu *vcpu)
* if S2 translation is disabled.
*/
for (int i = 0; i < kvm->arch.nested_mmus_size; i++) {
struct kvm_s2_mmu *mmu = &kvm->arch.nested_mmus[i];
struct kvm_s2_mmu *mmu = kvm->arch.nested_mmus[i];
if (!kvm_s2_mmu_valid(mmu))
continue;
@ -823,7 +826,7 @@ static struct kvm_s2_mmu *get_s2_mmu_nested(struct kvm_vcpu *vcpu)
for (i = kvm->arch.nested_mmus_next;
i < (kvm->arch.nested_mmus_size + kvm->arch.nested_mmus_next);
i++) {
s2_mmu = &kvm->arch.nested_mmus[i % kvm->arch.nested_mmus_size];
s2_mmu = kvm->arch.nested_mmus[i % kvm->arch.nested_mmus_size];
if (atomic_read(&s2_mmu->refcnt) == 0)
break;
@ -1260,6 +1263,17 @@ void kvm_handle_s1e2_tlbi(struct kvm_vcpu *vcpu, u32 inst, u64 val)
invalidate_vncr_va(vcpu->kvm, &scope);
}
static void kvm_invalidate_vncr_ipa_all(struct kvm *kvm)
{
struct kvm_pgtable *pgt = kvm->arch.mmu.pgt;
lockdep_assert_held_write(&kvm->mmu_lock);
/* if the mmu lock was dropped, pgt teardown may have raced. */
if (pgt)
kvm_invalidate_vncr_ipa(kvm, 0, BIT(pgt->ia_bits));
}
void kvm_nested_s2_wp(struct kvm *kvm)
{
int i;
@ -1270,13 +1284,13 @@ void kvm_nested_s2_wp(struct kvm *kvm)
return;
for (i = 0; i < kvm->arch.nested_mmus_size; i++) {
struct kvm_s2_mmu *mmu = &kvm->arch.nested_mmus[i];
struct kvm_s2_mmu *mmu = kvm->arch.nested_mmus[i];
if (kvm_s2_mmu_valid(mmu))
kvm_stage2_wp_range(mmu, 0, kvm_phys_size(mmu));
}
kvm_invalidate_vncr_ipa(kvm, 0, BIT(kvm->arch.mmu.pgt->ia_bits));
kvm_invalidate_vncr_ipa_all(kvm);
}
void kvm_nested_s2_unmap(struct kvm *kvm, bool may_block)
@ -1289,13 +1303,13 @@ void kvm_nested_s2_unmap(struct kvm *kvm, bool may_block)
return;
for (i = 0; i < kvm->arch.nested_mmus_size; i++) {
struct kvm_s2_mmu *mmu = &kvm->arch.nested_mmus[i];
struct kvm_s2_mmu *mmu = kvm->arch.nested_mmus[i];
if (kvm_s2_mmu_valid(mmu))
kvm_stage2_unmap_range(mmu, 0, kvm_phys_size(mmu), may_block);
}
kvm_invalidate_vncr_ipa(kvm, 0, BIT(kvm->arch.mmu.pgt->ia_bits));
kvm_invalidate_vncr_ipa_all(kvm);
}
void kvm_nested_s2_flush(struct kvm *kvm)
@ -1308,7 +1322,7 @@ void kvm_nested_s2_flush(struct kvm *kvm)
return;
for (i = 0; i < kvm->arch.nested_mmus_size; i++) {
struct kvm_s2_mmu *mmu = &kvm->arch.nested_mmus[i];
struct kvm_s2_mmu *mmu = kvm->arch.nested_mmus[i];
if (kvm_s2_mmu_valid(mmu))
kvm_stage2_flush_range(mmu, 0, kvm_phys_size(mmu));
@ -1317,17 +1331,12 @@ void kvm_nested_s2_flush(struct kvm *kvm)
void kvm_arch_flush_shadow_all(struct kvm *kvm)
{
int i;
for (i = 0; i < kvm->arch.nested_mmus_size; i++) {
struct kvm_s2_mmu *mmu = &kvm->arch.nested_mmus[i];
for (int i = 0; i < kvm->arch.nested_mmus_size; i++) {
struct kvm_s2_mmu *mmu = kvm->arch.nested_mmus[i];
if (!WARN_ON(atomic_read(&mmu->refcnt)))
kvm_free_stage2_pgd(mmu);
}
kvfree(kvm->arch.nested_mmus);
kvm->arch.nested_mmus = NULL;
kvm->arch.nested_mmus_size = 0;
kvm_uninit_stage2_mmu(kvm);
}

1
arch/arm64/kvm/sys_regs.c
View File

@ -926,6 +926,7 @@ static u64 *demux_wb_reg(struct kvm_vcpu *vcpu, const struct sys_reg_desc *rd)
struct kvm_guest_debug_arch *dbg = &vcpu->arch.vcpu_debug_state;
switch (rd->Op2) {
case 0b001:
case 0b100:
return &dbg->dbg_bvr[rd->CRm];
case 0b101:

12
arch/arm64/kvm/vgic/vgic-init.c
View File

@ -97,6 +97,9 @@ int kvm_vgic_create(struct kvm *kvm, u32 type)
/*
* - Acquiring the vCPU mutex for every *online* vCPU to prevent
* concurrent vCPU ioctls for vCPUs already visible to userspace.
* This also ensures KVM isn't in the middle of creating a vCPU,
* i.e. that there are no vCPUs that have been created but aren't
* yet fully online.
*/
ret = -EBUSY;
if (kvm_trylock_all_vcpus(kvm))
@ -105,18 +108,11 @@ int kvm_vgic_create(struct kvm *kvm, u32 type)
/*
* - Taking the config_lock which protects VGIC data structures such
* as the per-vCPU arrays of private IRQs (SGIs, PPIs).
*/
mutex_lock(&kvm->arch.config_lock);
/*
* - Bailing on the entire thing if a vCPU is in the middle of creation,
* dropped the kvm->lock, but hasn't reached kvm_arch_vcpu_create().
*
* The whole combination of this guarantees that no vCPU can get into
* KVM with a VGIC configuration inconsistent with the VM's VGIC.
*/
if (kvm->created_vcpus != atomic_read(&kvm->online_vcpus))
goto out_unlock;
mutex_lock(&kvm->arch.config_lock);
if (irqchip_in_kernel(kvm)) {
ret = -EEXIST;

58
arch/arm64/kvm/vgic/vgic-its.c
View File

@ -319,12 +319,16 @@ static int update_lpi_config(struct kvm *kvm, struct vgic_irq *irq,
return ret;
}
static int update_affinity(struct vgic_irq *irq, struct kvm_vcpu *vcpu)
static int update_affinity(struct vgic_irq *irq,
struct kvm_vcpu *from_vcpu, struct kvm_vcpu *vcpu)
{
struct its_vlpi_map map;
int ret;
guard(raw_spinlock_irqsave)(&irq->irq_lock);
if (from_vcpu && irq->target_vcpu != from_vcpu)
return 0;
irq->target_vcpu = vcpu;
if (!irq->hw)
@ -362,7 +366,7 @@ static void update_affinity_ite(struct kvm *kvm, struct its_ite *ite)
return;
vcpu = collection_to_vcpu(kvm, ite->collection);
update_affinity(ite->irq, vcpu);
update_affinity(ite->irq, NULL, vcpu);
}
/*
@ -856,7 +860,7 @@ static int vgic_its_cmd_handle_movi(struct kvm *kvm, struct vgic_its *its,
vgic_its_invalidate_cache(its);
return update_affinity(ite->irq, vcpu);
return update_affinity(ite->irq, NULL, vcpu);
}
static bool __is_visible_gfn_locked(struct vgic_its *its, gpa_t gpa)
@ -1383,7 +1387,7 @@ static int vgic_its_cmd_handle_movall(struct kvm *kvm, struct vgic_its *its,
if (!irq)
continue;
update_affinity(irq, vcpu2);
update_affinity(irq, vcpu1, vcpu2);
vgic_put_irq(kvm, irq);
}
@ -1658,7 +1662,7 @@ static void vgic_mmio_write_its_baser(struct kvm *kvm,
unsigned long val)
{
const struct vgic_its_abi *abi = vgic_its_get_abi(its);
u64 entry_size, table_type;
u64 old, entry_size, table_type;
u64 reg, *regptr, clearbits = 0;
/* When GITS_CTLR.Enable is 1, we ignore write accesses. */
@ -1681,7 +1685,9 @@ static void vgic_mmio_write_its_baser(struct kvm *kvm,
return;
}
reg = update_64bit_reg(*regptr, addr & 7, len, val);
old = *regptr;
reg = update_64bit_reg(old, addr & 7, len, val);
reg &= ~GITS_BASER_RO_MASK;
reg &= ~clearbits;
@ -1691,7 +1697,8 @@ static void vgic_mmio_write_its_baser(struct kvm *kvm,
*regptr = reg;
if (!(reg & GITS_BASER_VALID)) {
/* The ITS driver rewrites an unchanged GITS_BASER<n> on resume. */
if (reg != old) {
/* Take the its_lock to prevent a race with a save/restore */
mutex_lock(&its->its_lock);
switch (table_type) {
@ -1702,6 +1709,8 @@ static void vgic_mmio_write_its_baser(struct kvm *kvm,
vgic_its_free_collection_list(kvm, its);
break;
}
/* A concurrent injection may have cached a translation. */
vgic_its_invalidate_cache(its);
mutex_unlock(&its->its_lock);
}
}
@ -2019,18 +2028,22 @@ static int vgic_its_attr_regs_access(struct kvm_device *dev,
return ret;
}
static u32 compute_next_devid_offset(struct list_head *h,
static u32 compute_next_devid_offset(struct vgic_its *its, u64 baser,
struct its_device *dev)
{
struct its_device *next;
u32 next_offset;
struct its_device *next = dev;
if (list_is_last(&dev->dev_list, h))
return 0;
next = list_next_entry(dev, dev_list);
next_offset = next->device_id - dev->device_id;
/*
* Point at the next device vgic_its_save_device_tables() saves. It
* sorts device_list first, so the subtraction cannot underflow.
*/
list_for_each_entry_continue(next, &its->device_list, dev_list) {
if (vgic_its_check_id(its, baser, next->device_id, NULL))
return min_t(u32, next->device_id - dev->device_id,
VITS_DTE_MAX_DEVID_OFFSET);
}
return min_t(u32, next_offset, VITS_DTE_MAX_DEVID_OFFSET);
return 0;
}
static u32 compute_next_eventid_offset(struct list_head *h, struct its_ite *ite)
@ -2271,17 +2284,18 @@ static int vgic_its_restore_itt(struct vgic_its *its, struct its_device *dev)
* vgic_its_save_dte - Save a device table entry at a given GPA
*
* @its: ITS handle
* @baser: GITS_BASER<dev> the caller is saving against
* @dev: ITS device
* @ptr: GPA
*/
static int vgic_its_save_dte(struct vgic_its *its, struct its_device *dev,
gpa_t ptr)
static int vgic_its_save_dte(struct vgic_its *its, u64 baser,
struct its_device *dev, gpa_t ptr)
{
u64 val, itt_addr_field;
u32 next_offset;
itt_addr_field = dev->itt_addr >> 8;
next_offset = compute_next_devid_offset(&its->device_list, dev);
next_offset = compute_next_devid_offset(its, baser, dev);
val = (1ULL << KVM_ITS_DTE_VALID_SHIFT |
((u64)next_offset << KVM_ITS_DTE_NEXT_SHIFT) |
(itt_addr_field << KVM_ITS_DTE_ITTADDR_SHIFT) |
@ -2380,15 +2394,16 @@ static int vgic_its_save_device_tables(struct vgic_its *its)
int ret;
gpa_t eaddr;
/* Don't fail a save that userspace must be able to issue. */
if (!vgic_its_check_id(its, baser,
dev->device_id, &eaddr))
return -EINVAL;
continue;
ret = vgic_its_save_itt(its, dev);
if (ret)
return ret;
ret = vgic_its_save_dte(its, dev, eaddr);
ret = vgic_its_save_dte(its, baser, dev, eaddr);
if (ret)
return ret;
}
@ -2541,9 +2556,6 @@ static int vgic_its_save_collection_table(struct vgic_its *its)
max_size = GITS_BASER_NR_PAGES(baser) * SZ_64K;
list_for_each_entry(collection, &its->collection_list, coll_list) {
if (!vgic_its_check_id(its, baser, collection->collection_id, NULL))
return -EINVAL;
ret = vgic_its_save_cte(its, collection, gpa);
if (ret)
return ret;

6
arch/arm64/kvm/vgic/vgic-v2.c
View File

@ -122,6 +122,10 @@ void vgic_v2_fold_lr_state(struct kvm_vcpu *vcpu)
for (int lr = 0; lr < vgic_cpu->vgic_v2.used_lrs; lr++)
vgic_v2_fold_lr(vcpu, cpuif->vgic_lr[lr]);
cpuif->used_lrs = 0;
if (!irq)
return;
/* See the GICv3 equivalent for the EOIcount handling rationale */
list_for_each_entry_continue(irq, &vgic_cpu->ap_list_head, ap_list) {
u32 lr;
@ -144,8 +148,6 @@ void vgic_v2_fold_lr_state(struct kvm_vcpu *vcpu)
vgic_v2_fold_lr(vcpu, lr);
eoicount--;
}
cpuif->used_lrs = 0;
}
void vgic_v2_deactivate(struct kvm_vcpu *vcpu, u32 val)

6
arch/arm64/kvm/vgic/vgic-v3.c
View File

@ -155,6 +155,10 @@ void vgic_v3_fold_lr_state(struct kvm_vcpu *vcpu)
for (int lr = 0; lr < cpuif->used_lrs; lr++)
vgic_v3_fold_lr(vcpu, cpuif->vgic_lr[lr]);
cpuif->used_lrs = 0;
if (!irq)
return;
/*
* EOIMode=0: use EOIcount to emulate deactivation. We are
* guaranteed to deactivate in reverse order of the activation, so
@ -188,8 +192,6 @@ void vgic_v3_fold_lr_state(struct kvm_vcpu *vcpu)
vgic_v3_fold_lr(vcpu, lr);
eoicount--;
}
cpuif->used_lrs = 0;
}
void vgic_v3_deactivate(struct kvm_vcpu *vcpu, u64 val)

21
arch/arm64/kvm/vgic/vgic.c
View File

@ -853,6 +853,17 @@ static void vgic_prune_ap_list(struct kvm_vcpu *vcpu)
goto retry;
}
/*
* Fix the last_lr_irq refcount which was obtained while
* populating the LRs. This can also result in the LPI being
* deleted.
*/
irq = *host_data_ptr(last_lr_irq);
if (irq) {
deleted_lpis |= vgic_put_irq_norelease(vcpu->kvm, irq);
*host_data_ptr(last_lr_irq) = NULL;
}
raw_spin_unlock(&vgic_cpu->ap_list_lock);
if (unlikely(deleted_lpis))
@ -866,9 +877,6 @@ static void vgic_fold_state(struct kvm_vcpu *vcpu)
return;
}
if (!*host_data_ptr(last_lr_irq))
return;
if (kvm_vgic_global_state.type == VGIC_V2)
vgic_v2_fold_lr_state(vcpu);
else
@ -1021,11 +1029,14 @@ static void vgic_flush_lr_state(struct kvm_vcpu *vcpu)
scoped_guard(raw_spinlock, &irq->irq_lock) {
if (likely(vgic_target_oracle(irq) == vcpu)) {
vgic_populate_lr(vcpu, irq, count++);
*host_data_ptr(last_lr_irq) = irq;
if (count == kvm_vgic_global_state.nr_lr) {
vgic_get_irq_ref(irq);
*host_data_ptr(last_lr_irq) = irq;
}
}
}
if (count == kvm_vgic_global_state.nr_lr)
if (*host_data_ptr(last_lr_irq))
break;
}

37
arch/arm64/net/bpf_jit_comp.c
View File

@ -2508,10 +2508,11 @@ bool bpf_jit_supports_large_stack(void)
return true;
}
static void invoke_bpf_prog(struct jit_ctx *ctx, struct bpf_tramp_node *node,
int bargs_off, int retval_off, int run_ctx_off,
bool save_ret)
static void invoke_bpf_prog(struct jit_ctx *ctx, struct bpf_tramp_image *im,
struct bpf_tramp_node *node, int bargs_off,
int retval_off, int run_ctx_off, bool save_ret)
{
void *skip;
__le32 *branch;
u64 enter_prog;
u64 exit_prog;
@ -2521,6 +2522,10 @@ static void invoke_bpf_prog(struct jit_ctx *ctx, struct bpf_tramp_node *node,
enter_prog = (u64)bpf_trampoline_enter(p);
exit_prog = (u64)bpf_trampoline_exit(p);
/* nop, patched to skip this prog when it is detached */
skip = ctx->ro_image + ctx->idx;
emit(A64_NOP, ctx);
if (node->cookie == 0) {
/* if cookie is zero, one instruction is enough to store it */
emit(A64_STR64I(A64_ZR, A64_SP, run_ctx_off + cookie_off), ctx);
@ -2573,11 +2578,13 @@ static void invoke_bpf_prog(struct jit_ctx *ctx, struct bpf_tramp_node *node,
emit(A64_ADD_I(1, A64_R(2), A64_SP, run_ctx_off), ctx);
emit_call(exit_prog, ctx);
bpf_tramp_image_add_skip(im, p, skip, ctx->ro_image + ctx->idx);
}
static void invoke_bpf_mod_ret(struct jit_ctx *ctx, struct bpf_tramp_nodes *tn,
int bargs_off, int retval_off, int run_ctx_off,
__le32 **branches)
static void invoke_bpf_mod_ret(struct jit_ctx *ctx, struct bpf_tramp_image *im,
struct bpf_tramp_nodes *tn, int bargs_off,
int retval_off, int run_ctx_off, __le32 **branches)
{
int i;
@ -2586,7 +2593,7 @@ static void invoke_bpf_mod_ret(struct jit_ctx *ctx, struct bpf_tramp_nodes *tn,
*/
emit(A64_STR64I(A64_ZR, A64_SP, retval_off), ctx);
for (i = 0; i < tn->nr_nodes; i++) {
invoke_bpf_prog(ctx, tn->nodes[i], bargs_off, retval_off,
invoke_bpf_prog(ctx, im, tn->nodes[i], bargs_off, retval_off,
run_ctx_off, true);
/* if (*(u64 *)(sp + retval_off) != 0)
* goto do_fexit;
@ -2986,7 +2993,7 @@ static int prepare_trampoline(struct jit_ctx *ctx, struct bpf_tramp_image *im,
store_func_meta(ctx, meta, func_meta_off);
cookie_bargs_off--;
}
invoke_bpf_prog(ctx, fentry->nodes[i], bargs_off,
invoke_bpf_prog(ctx, im, fentry->nodes[i], bargs_off,
retval_off, run_ctx_off,
flags & BPF_TRAMP_F_RET_FENTRY_RET);
}
@ -2997,7 +3004,7 @@ static int prepare_trampoline(struct jit_ctx *ctx, struct bpf_tramp_image *im,
if (!branches)
return -ENOMEM;
invoke_bpf_mod_ret(ctx, fmod_ret, bargs_off, retval_off,
invoke_bpf_mod_ret(ctx, im, fmod_ret, bargs_off, retval_off,
run_ctx_off, branches);
}
@ -3010,9 +3017,6 @@ static int prepare_trampoline(struct jit_ctx *ctx, struct bpf_tramp_image *im,
emit(A64_RET(A64_R(10)), ctx);
/* store return value */
emit(A64_STR64I(A64_R(0), A64_SP, retval_off), ctx);
/* reserve a nop for bpf_tramp_image_put */
im->ip_after_call = ctx->ro_image + ctx->idx;
emit(A64_NOP, ctx);
}
/* update the branches saved in invoke_bpf_mod_ret with cbnz */
@ -3034,12 +3038,11 @@ static int prepare_trampoline(struct jit_ctx *ctx, struct bpf_tramp_image *im,
store_func_meta(ctx, meta, func_meta_off);
cookie_bargs_off--;
}
invoke_bpf_prog(ctx, fexit->nodes[i], bargs_off, retval_off,
invoke_bpf_prog(ctx, im, fexit->nodes[i], bargs_off, retval_off,
run_ctx_off, false);
}
if (flags & BPF_TRAMP_F_CALL_ORIG) {
im->ip_epilogue = ctx->ro_image + ctx->idx;
/* for the first pass, assume the worst case */
if (!ctx->image)
ctx->idx += 4;
@ -3098,7 +3101,7 @@ int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags,
.image = NULL,
.idx = 0,
};
struct bpf_tramp_image im;
struct bpf_tramp_image im = {};
struct arg_aux aaux;
int ret;
@ -3385,6 +3388,10 @@ int bpf_arch_text_poke(void *ip, enum bpf_text_poke_type old_t,
* longer reachable, since bpf_tramp_image_put() function already
* uses percpu_ref and task-based rcu to do the sync, no need to call
* the sync version here, see bpf_tramp_image_put() for details.
*
* 3. when a detached prog is patched out of a trampoline, a CPU that
* still executes the old nop calls the prog before it went through
* a quiescent state, and the prog is freed after grace periods.
*/
ret = aarch64_insn_patch_text_nosync(ip, new_insn);
out:

60
arch/loongarch/net/bpf_jit.c
View File

@ -1578,6 +1578,24 @@ void *bpf_arch_text_copy(void *dst, void *src, size_t len)
return ret ? ERR_PTR(-EINVAL) : dst;
}
int arch_bpf_trampoline_skip(void *nop, void *target)
{
u32 old_insn = INSN_NOP;
u32 new_insn = larch_insn_gen_b((unsigned long)nop, (unsigned long)target);
int ret;
if (memcmp(nop, &old_insn, LOONGARCH_INSN_SIZE))
return -EFAULT;
cpus_read_lock();
mutex_lock(&text_mutex);
ret = larch_insn_text_copy(nop, &new_insn, LOONGARCH_INSN_SIZE);
mutex_unlock(&text_mutex);
cpus_read_unlock();
return ret;
}
int bpf_arch_text_poke(void *ip, enum bpf_text_poke_type old_t,
enum bpf_text_poke_type new_t, void *old_addr,
void *new_addr)
@ -1696,13 +1714,18 @@ static void restore_stk_args(struct jit_ctx *ctx, int nr_stk_args, int args_off,
}
}
static int invoke_bpf_prog(struct jit_ctx *ctx, struct bpf_tramp_node *n,
int args_off, int retval_off, int run_ctx_off, bool save_ret)
static int invoke_bpf_prog(struct jit_ctx *ctx, struct bpf_tramp_image *im,
struct bpf_tramp_node *n, int args_off, int retval_off,
int run_ctx_off, bool save_ret)
{
int ret;
u32 *branch;
struct bpf_prog *p = n->link->prog;
int cookie_off = offsetof(struct bpf_tramp_run_ctx, bpf_cookie);
void *skip = ctx->ro_image + ctx->idx;
/* nop, patched to a b over this prog when it is detached */
emit_insn(ctx, nop);
if (n->cookie)
emit_store_stack_imm64(ctx, LOONGARCH_GPR_T1,
@ -1755,13 +1778,17 @@ static int invoke_bpf_prog(struct jit_ctx *ctx, struct bpf_tramp_node *n,
/* arg3: &run_ctx */
emit_insn(ctx, addid, LOONGARCH_GPR_A2, LOONGARCH_GPR_FP, -run_ctx_off);
ret = emit_call(ctx, (const u64)bpf_trampoline_exit(p));
if (ret)
return ret;
return ret;
bpf_tramp_image_add_skip(im, p, skip, ctx->ro_image + ctx->idx);
return 0;
}
static int invoke_bpf(struct jit_ctx *ctx, struct bpf_tramp_nodes *tn,
int args_off, int retval_off, int run_ctx_off,
int func_meta_off, bool save_ret, u64 func_meta, int cookie_off)
static int invoke_bpf(struct jit_ctx *ctx, struct bpf_tramp_image *im,
struct bpf_tramp_nodes *tn, int args_off, int retval_off,
int run_ctx_off, int func_meta_off, bool save_ret,
u64 func_meta, int cookie_off)
{
int i, cur_cookie = (cookie_off - args_off) / 8;
@ -1774,7 +1801,8 @@ static int invoke_bpf(struct jit_ctx *ctx, struct bpf_tramp_nodes *tn,
emit_store_stack_imm64(ctx, LOONGARCH_GPR_T1, -func_meta_off, meta);
cur_cookie--;
}
err = invoke_bpf_prog(ctx, tn->nodes[i], args_off, retval_off, run_ctx_off, save_ret);
err = invoke_bpf_prog(ctx, im, tn->nodes[i], args_off, retval_off,
run_ctx_off, save_ret);
if (err)
return err;
}
@ -2017,7 +2045,7 @@ static int __arch_prepare_bpf_trampoline(struct jit_ctx *ctx, struct bpf_tramp_i
}
if (fentry->nr_nodes) {
ret = invoke_bpf(ctx, fentry, args_off, retval_off, run_ctx_off, func_meta_off,
ret = invoke_bpf(ctx, im, fentry, args_off, retval_off, run_ctx_off, func_meta_off,
flags & BPF_TRAMP_F_RET_FENTRY_RET, func_meta, cookie_off);
if (ret)
return ret;
@ -2029,7 +2057,7 @@ static int __arch_prepare_bpf_trampoline(struct jit_ctx *ctx, struct bpf_tramp_i
emit_insn(ctx, std, LOONGARCH_GPR_ZERO, LOONGARCH_GPR_FP, -retval_off);
for (i = 0; i < fmod_ret->nr_nodes; i++) {
ret = invoke_bpf_prog(ctx, fmod_ret->nodes[i],
ret = invoke_bpf_prog(ctx, im, fmod_ret->nodes[i],
args_off, retval_off, run_ctx_off, true);
if (ret)
goto out;
@ -2051,10 +2079,6 @@ static int __arch_prepare_bpf_trampoline(struct jit_ctx *ctx, struct bpf_tramp_i
goto out;
emit_insn(ctx, std, LOONGARCH_GPR_A0, LOONGARCH_GPR_FP, -retval_off);
emit_insn(ctx, std, regmap[BPF_REG_0], LOONGARCH_GPR_FP, -(retval_off - 8));
im->ip_after_call = ctx->ro_image + ctx->idx;
/* Reserve space for the move_imm + jirl instruction */
for (i = 0; i < LOONGARCH_LONG_JUMP_NINSNS; i++)
emit_insn(ctx, nop);
}
for (i = 0; ctx->image && i < fmod_ret->nr_nodes; i++) {
@ -2068,14 +2092,13 @@ static int __arch_prepare_bpf_trampoline(struct jit_ctx *ctx, struct bpf_tramp_i
emit_store_stack_imm64(ctx, LOONGARCH_GPR_T1, -func_meta_off, func_meta);
if (fexit->nr_nodes) {
ret = invoke_bpf(ctx, fexit, args_off, retval_off, run_ctx_off,
ret = invoke_bpf(ctx, im, fexit, args_off, retval_off, run_ctx_off,
func_meta_off, false, func_meta, cookie_off);
if (ret)
goto out;
}
if (flags & BPF_TRAMP_F_CALL_ORIG) {
im->ip_epilogue = ctx->ro_image + ctx->idx;
move_addr(ctx, LOONGARCH_GPR_A0, (const u64)im);
ret = emit_call(ctx, (const u64)__bpf_tramp_exit);
if (ret)
@ -2177,11 +2200,8 @@ int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags,
struct bpf_tramp_nodes *tnodes, void *func_addr)
{
int ret;
struct jit_ctx ctx;
struct bpf_tramp_image im;
ctx.image = NULL;
ctx.idx = 0;
struct jit_ctx ctx = {};
struct bpf_tramp_image im = {};
ret = __arch_prepare_bpf_trampoline(&ctx, &im, m, tnodes, func_addr, flags);

1
arch/loongarch/vdso/Makefile
View File

@ -16,6 +16,7 @@ ccflags-vdso := \
$(filter -m64,$(KBUILD_CFLAGS)) \
$(filter -march=%,$(KBUILD_CFLAGS)) \
$(filter -m%-float,$(KBUILD_CFLAGS)) \
$(CONFIG_CC_OPT_INLINE_MEMSET) \
$(CLANG_FLAGS) \
-D__VDSO__

1
arch/powerpc/kernel/vdso/Makefile
View File

@ -43,6 +43,7 @@ ccflags-y := -fno-common -fno-builtin -DBUILD_VDSO
ccflags-y += $(DISABLE_LATENT_ENTROPY_PLUGIN)
ccflags-y += $(call cc-option, -fno-stack-protector)
ccflags-y += -DDISABLE_BRANCH_PROFILING
ccflags-y += $(CONFIG_CC_OPT_INLINE_MEMSET)
ccflags-y += -ffreestanding -fasynchronous-unwind-tables
ccflags-remove-y := $(CC_FLAGS_FTRACE)
ldflags-y := -Wl,--hash-style=both -nostdlib -shared -z noexecstack $(CLANG_FLAGS)

2
arch/powerpc/kvm/book3s_hv.c
View File

@ -3058,7 +3058,6 @@ static int kvmppc_core_vcpu_create_hv(struct kvm_vcpu *vcpu)
init_waitqueue_head(&vcpu->arch.cpu_run);
mutex_lock(&kvm->lock);
vcore = NULL;
err = -EINVAL;
if (cpu_has_feature(CPU_FTR_ARCH_300)) {
@ -3091,7 +3090,6 @@ static int kvmppc_core_vcpu_create_hv(struct kvm_vcpu *vcpu)
mutex_unlock(&kvm->arch.mmu_setup_lock);
}
}
mutex_unlock(&kvm->lock);
if (!vcore)
return err;

45
arch/powerpc/net/bpf_jit_comp.c
View File

@ -602,14 +602,18 @@ int arch_protect_bpf_trampoline(void *image, unsigned int size)
}
static int invoke_bpf_prog(u32 *image, u32 *ro_image, struct codegen_context *ctx,
struct bpf_tramp_node *n, int regs_off, int retval_off,
int run_ctx_off, bool save_ret)
struct bpf_tramp_image *im, struct bpf_tramp_node *n,
int regs_off, int retval_off, int run_ctx_off, bool save_ret)
{
struct bpf_prog *p = n->link->prog;
ppc_inst_t branch_insn;
u32 jmp_idx;
u32 jmp_idx, skip_idx;
int ret = 0;
/* nop, patched to skip this prog when it is detached */
skip_idx = ctx->idx;
EMIT(PPC_RAW_NOP());
/* Save cookie */
if (IS_ENABLED(CONFIG_PPC64)) {
PPC_LI64(_R3, n->cookie);
@ -679,13 +683,17 @@ static int invoke_bpf_prog(u32 *image, u32 *ro_image, struct codegen_context *ct
EMIT(PPC_RAW_ADDI(_R5, _R1, run_ctx_off));
ret = bpf_jit_emit_func_call_rel(image, ro_image, ctx,
(unsigned long)bpf_trampoline_exit(p));
if (ret)
return ret;
return ret;
if (ro_image) /* image is NULL for dummy pass */
bpf_tramp_image_add_skip(im, p, &ro_image[skip_idx], &ro_image[ctx->idx]);
return 0;
}
static int invoke_bpf_mod_ret(u32 *image, u32 *ro_image, struct codegen_context *ctx,
struct bpf_tramp_nodes *tn, int regs_off, int retval_off,
int run_ctx_off, u32 *branches)
struct bpf_tramp_image *im, struct bpf_tramp_nodes *tn,
int regs_off, int retval_off, int run_ctx_off, u32 *branches)
{
int i;
@ -696,8 +704,8 @@ static int invoke_bpf_mod_ret(u32 *image, u32 *ro_image, struct codegen_context
EMIT(PPC_RAW_LI(_R3, 0));
EMIT(PPC_RAW_STL(_R3, _R1, retval_off));
for (i = 0; i < tn->nr_nodes; i++) {
if (invoke_bpf_prog(image, ro_image, ctx, tn->nodes[i], regs_off, retval_off,
run_ctx_off, true))
if (invoke_bpf_prog(image, ro_image, ctx, im, tn->nodes[i], regs_off,
retval_off, run_ctx_off, true))
return -EINVAL;
/*
@ -1043,8 +1051,8 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im
cookie_ctx_off--;
}
if (invoke_bpf_prog(image, ro_image, ctx, fentry->nodes[i], regs_off, retval_off,
run_ctx_off, flags & BPF_TRAMP_F_RET_FENTRY_RET))
if (invoke_bpf_prog(image, ro_image, ctx, im, fentry->nodes[i], regs_off,
retval_off, run_ctx_off, flags & BPF_TRAMP_F_RET_FENTRY_RET))
return -EINVAL;
}
@ -1053,7 +1061,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im
if (!branches)
return -ENOMEM;
if (invoke_bpf_mod_ret(image, ro_image, ctx, fmod_ret, regs_off, retval_off,
if (invoke_bpf_mod_ret(image, ro_image, ctx, im, fmod_ret, regs_off, retval_off,
run_ctx_off, branches)) {
ret = -EINVAL;
goto cleanup;
@ -1090,11 +1098,6 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im
/* Restore updated tail_call_cnt */
if (flags & BPF_TRAMP_F_TAIL_CALL_CTX)
bpf_trampoline_restore_tail_call_cnt(image, ctx, bpf_frame_size, r4_off);
/* Reserve space to patch branch instruction to skip fexit progs */
if (ro_image) /* image is NULL for dummy pass */
im->ip_after_call = &((u32 *)ro_image)[ctx->idx];
EMIT(PPC_RAW_NOP());
}
/* Update branches saved in invoke_bpf_mod_ret with address of do_fexit */
@ -1123,16 +1126,14 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im
cookie_ctx_off--;
}
if (invoke_bpf_prog(image, ro_image, ctx, fexit->nodes[i], regs_off, retval_off,
run_ctx_off, false)) {
if (invoke_bpf_prog(image, ro_image, ctx, im, fexit->nodes[i], regs_off,
retval_off, run_ctx_off, false)) {
ret = -EINVAL;
goto cleanup;
}
}
if (flags & BPF_TRAMP_F_CALL_ORIG) {
if (ro_image) /* image is NULL for dummy pass */
im->ip_epilogue = &((u32 *)ro_image)[ctx->idx];
PPC_LI_ADDR(_R3, im);
ret = bpf_jit_emit_func_call_rel(image, ro_image, ctx,
(unsigned long)__bpf_tramp_exit);
@ -1192,7 +1193,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im
int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags,
struct bpf_tramp_nodes *tnodes, void *func_addr)
{
struct bpf_tramp_image im;
struct bpf_tramp_image im = {};
int ret;
ret = __arch_prepare_bpf_trampoline(&im, NULL, NULL, NULL, m, flags, tnodes, func_addr);
@ -1320,7 +1321,7 @@ int bpf_arch_text_poke(void *ip, enum bpf_text_poke_type old_t,
/*
* If we are not poking at bpf prog entry, then we are simply patching in/out
* an unconditional branch instruction at im->ip_after_call
* an unconditional branch instruction in a trampoline image
*/
if (offset) {
if (old_t == BPF_MOD_CALL || new_t == BPF_MOD_CALL) {

20
arch/riscv/include/asm/csr.h
View File

@ -188,12 +188,24 @@
#define HGATP_MODE_SHIFT HGATP32_MODE_SHIFT
#endif
/* VSIP & HVIP relation */
/*
* VSIP & HVIP relation
*
* The bit positions are same between VSIP and HVIP for interrupt
* numbers 13-63, where there's a shift for the SSI, STI and SEI.
*/
#define VSIP_TO_HVIP_SHIFT (IRQ_VS_SOFT - IRQ_S_SOFT)
#define VSIP_VALID_MASK ((_AC(1, UL) << IRQ_S_SOFT) | \
#define VSIP_BIAS_MASK ((_AC(1, UL) << IRQ_S_SOFT) | \
(_AC(1, UL) << IRQ_S_TIMER) | \
(_AC(1, UL) << IRQ_S_EXT) | \
(_AC(1, UL) << IRQ_PMU_OVF))
(_AC(1, UL) << IRQ_S_EXT))
#define VSIP_NO_BIAS_MASK (_AC(1, UL) << IRQ_PMU_OVF)
#define VSIP_VALID_MASK (VSIP_BIAS_MASK | VSIP_NO_BIAS_MASK)
#define vsip_to_hvip(_vsip) ((((_vsip) & VSIP_BIAS_MASK) << \
VSIP_TO_HVIP_SHIFT) | \
((_vsip) & VSIP_NO_BIAS_MASK))
#define hvip_to_vsip(_hvip) ((((_hvip) >> VSIP_TO_HVIP_SHIFT) & \
VSIP_BIAS_MASK) | \
((_hvip) & VSIP_NO_BIAS_MASK))
/* AIA CSR bits */
#define TOPI_IID_SHIFT 16

1
arch/riscv/kernel/vdso/Makefile
View File

@ -36,6 +36,7 @@ endif
ccflags-y := -fno-stack-protector
ccflags-y += -DDISABLE_BRANCH_PROFILING
ccflags-y += -fno-builtin
ccflags-y += $(CONFIG_CC_OPT_INLINE_MEMSET)
ccflags-y += $(KBUILD_BASE_ISA)$(CFI_MARCH)
ccflags-y += $(CFI_FULL)
asflags-y += $(KBUILD_BASE_ISA)$(CFI_MARCH)

2
arch/riscv/kvm/aia_device.c
View File

@ -237,7 +237,7 @@ static int aia_init(struct kvm *kvm)
return -EBUSY;
/* We might be in the middle of creating a VCPU? */
if (kvm->created_vcpus != atomic_read(&kvm->online_vcpus))
if (kvm_is_vcpu_creation_in_progress(kvm))
return -EBUSY;
/* Number of sources should be less than or equals number of IDs */

11
arch/riscv/kvm/aia_imsic.c
View File

@ -965,9 +965,14 @@ int kvm_riscv_aia_imsic_rw_attr(struct kvm *kvm, unsigned long type,
if (!vcpu)
return -ENODEV;
if (mutex_lock_killable(&vcpu->mutex))
return -EINTR;
imsic = vcpu->arch.aia_context.imsic_state;
if (!imsic)
return -ENODEV;
if (!imsic) {
rc = -ENODEV;
goto out_unlock;
}
isel = KVM_DEV_RISCV_AIA_IMSIC_GET_ISEL(type);
read_lock_irqsave(&imsic->vsfile_lock, flags);
@ -991,6 +996,8 @@ int kvm_riscv_aia_imsic_rw_attr(struct kvm *kvm, unsigned long type,
rc = imsic_vsfile_rw(vsfile_hgei, vsfile_cpu, imsic->nr_eix,
isel, write, val);
out_unlock:
mutex_unlock(&vcpu->mutex);
return rc;
}

10
arch/riscv/kvm/mmu.c
View File

@ -627,6 +627,7 @@ int kvm_riscv_mmu_map(struct kvm_vcpu *vcpu, struct kvm_memory_slot *memslot,
int ret;
kvm_pfn_t hfn;
bool is_hugetlb;
bool unused = false;
bool writable;
unsigned int vma_pageshift;
gfn_t gfn = gpa >> PAGE_SHIFT;
@ -707,6 +708,8 @@ int kvm_riscv_mmu_map(struct kvm_vcpu *vcpu, struct kvm_memory_slot *memslot,
vma_pageshift, current);
return 0;
}
if (is_sigpending_pfn(hfn))
return -EINTR;
if (is_error_noslot_pfn(hfn))
return -EFAULT;
@ -719,8 +722,10 @@ int kvm_riscv_mmu_map(struct kvm_vcpu *vcpu, struct kvm_memory_slot *memslot,
write_lock(&kvm->mmu_lock);
if (mmu_invalidate_retry(kvm, mmu_seq))
if (mmu_invalidate_retry(kvm, mmu_seq)) {
unused = true;
goto out_unlock;
}
/*
* Check if we are backed by a THP and thus use block mapping if
@ -743,7 +748,8 @@ int kvm_riscv_mmu_map(struct kvm_vcpu *vcpu, struct kvm_memory_slot *memslot,
kvm_err("Failed to map in G-stage\n");
out_unlock:
kvm_release_faultin_page(kvm, page, ret && ret != -EEXIST, writable);
kvm_release_faultin_page(kvm, page,
unused || (ret && ret != -EEXIST), writable);
write_unlock(&kvm->mmu_lock);
return ret;
}

20
arch/riscv/kvm/nacl.c
View File

@ -42,12 +42,24 @@ void __kvm_riscv_nacl_hfence(void *shmem,
}
}
entp = shmem + SBI_NACL_SHMEM_HFENCE_ENTRY_CONFIG(i);
*entp = cpu_to_lelong(control);
/*
* Per SBI v3.0 section 15.1.2, the Page_Number and Page_Count
* words must be updated before the Config word with its Pending
* bit set. WRITE_ONCE() stops the compiler from reordering the
* stores and smp_wmb() makes the parameter words globally
* visible to the SBI implementation (or NACL hardware) before
* the Pending bit is set.
*/
entp = shmem + SBI_NACL_SHMEM_HFENCE_ENTRY_PNUM(i);
*entp = cpu_to_lelong(page_num);
WRITE_ONCE(*entp, cpu_to_lelong(page_num));
entp = shmem + SBI_NACL_SHMEM_HFENCE_ENTRY_PCOUNT(i);
*entp = cpu_to_lelong(page_count);
WRITE_ONCE(*entp, cpu_to_lelong(page_count));
/* Ensure the parameter words are visible before the Pending bit */
smp_wmb();
entp = shmem + SBI_NACL_SHMEM_HFENCE_ENTRY_CONFIG(i);
WRITE_ONCE(*entp, cpu_to_lelong(control));
}
int kvm_riscv_nacl_enable(void)

3
arch/riscv/kvm/vcpu.c
View File

@ -491,8 +491,7 @@ bool kvm_riscv_vcpu_has_interrupts(struct kvm_vcpu *vcpu, u64 mask)
bool ret;
raw_spin_lock_irqsave(&vcpu->arch.irqs_pending_lock, flags);
ie = ((vcpu->arch.guest_csr.vsie & VSIP_VALID_MASK)
<< VSIP_TO_HVIP_SHIFT) & (unsigned long)mask;
ie = vsip_to_hvip(vcpu->arch.guest_csr.vsie) & (unsigned long)mask;
ie |= vcpu->arch.guest_csr.vsie & ~IRQ_LOCAL_MASK &
(unsigned long)mask;
ret = vcpu->arch.irqs_pending[0] & ie;

2
arch/riscv/kvm/vcpu_exit.c
View File

@ -283,7 +283,7 @@ int kvm_riscv_vcpu_exit(struct kvm_vcpu *vcpu, struct kvm_run *run,
}
/* Print details in-case of error */
if (ret < 0) {
if (ret < 0 && ret != -EINTR) {
kvm_err("VCPU exit error %d\n", ret);
kvm_err("SEPC=0x%lx SSTATUS=0x%lx HSTATUS=0x%lx\n",
vcpu->arch.guest_context.sepc,

8
arch/riscv/kvm/vcpu_onereg.c
View File

@ -272,7 +272,7 @@ static int kvm_riscv_vcpu_general_get_csr(struct kvm_vcpu *vcpu,
if (reg_num == KVM_REG_RISCV_CSR_REG(sip)) {
kvm_riscv_vcpu_flush_interrupts(vcpu);
*out_val = (csr->hvip >> VSIP_TO_HVIP_SHIFT) & VSIP_VALID_MASK;
*out_val = hvip_to_vsip(csr->hvip);
*out_val |= csr->hvip & ~IRQ_LOCAL_MASK;
} else
*out_val = ((unsigned long *)csr)[reg_num];
@ -293,10 +293,8 @@ static int kvm_riscv_vcpu_general_set_csr(struct kvm_vcpu *vcpu,
reg_num = array_index_nospec(reg_num, regs_max);
if (reg_num == KVM_REG_RISCV_CSR_REG(sip)) {
reg_val &= VSIP_VALID_MASK;
reg_val <<= VSIP_TO_HVIP_SHIFT;
}
if (reg_num == KVM_REG_RISCV_CSR_REG(sip))
reg_val = vsip_to_hvip(reg_val);
((unsigned long *)csr)[reg_num] = reg_val;

41
arch/riscv/kvm/vcpu_pmu.c
View File

@ -270,12 +270,13 @@ static int pmu_ctr_read(struct kvm_vcpu *vcpu, unsigned long cidx,
return -EINVAL;
pmc->counter_val = kvpmu->fw_event[fevent_code].value;
*out_val = pmc->counter_val;
} else if (pmc->perf_event) {
pmc->counter_val += perf_event_read_value(pmc->perf_event, &enabled, &running);
*out_val = pmc->counter_val +
perf_event_read_value(pmc->perf_event, &enabled, &running);
} else {
return -EINVAL;
}
*out_val = pmc->counter_val;
return 0;
}
@ -454,6 +455,14 @@ int kvm_riscv_vcpu_pmu_snapshot_set_shmem(struct kvm_vcpu *vcpu, unsigned long s
}
}
/*
* Clear any previously installed snapshot area to avoid leaking
* the old sdata and to keep sdata/snapshot_addr consistent if
* the re-install fails below.
*/
if (kvpmu->snapshot_addr != INVALID_GPA)
kvm_pmu_clear_snapshot_area(vcpu);
kvpmu->sdata = kzalloc(snapshot_area_size, GFP_ATOMIC | __GFP_ACCOUNT);
if (!kvpmu->sdata) {
sbiret = SBI_ERR_FAILURE;
@ -645,7 +654,6 @@ int kvm_riscv_vcpu_pmu_ctr_stop(struct kvm_vcpu *vcpu, unsigned long ctr_base,
{
struct kvm_pmu *kvpmu = vcpu_to_pmu(vcpu);
int i, pmc_index, sbiret = 0;
u64 enabled, running;
struct kvm_pmc *pmc;
int fevent_code;
bool snap_flag_set = flags & SBI_PMU_STOP_FLAG_TAKE_SNAPSHOT;
@ -675,14 +683,19 @@ int kvm_riscv_vcpu_pmu_ctr_stop(struct kvm_vcpu *vcpu, unsigned long ctr_base,
goto out;
}
if (!kvpmu->fw_event[fevent_code].started)
if (!kvpmu->fw_event[fevent_code].started) {
sbiret = SBI_ERR_ALREADY_STOPPED;
kvpmu->fw_event[fevent_code].started = false;
} else {
kvpmu->fw_event[fevent_code].started = false;
pmc->counter_val = kvpmu->fw_event[fevent_code].value;
}
} else if (pmc->perf_event) {
if (pmc->started) {
/* Stop counting the counter */
perf_event_disable(pmc->perf_event);
/*
* Stop the counter and fold the live count into counter_val.
* Reset the event value to avoid redundant accumulation.
*/
pmc->counter_val += perf_event_pause(pmc->perf_event, true);
pmc->started = false;
} else {
sbiret = SBI_ERR_ALREADY_STOPPED;
@ -696,11 +709,6 @@ int kvm_riscv_vcpu_pmu_ctr_stop(struct kvm_vcpu *vcpu, unsigned long ctr_base,
}
if (snap_flag_set && !sbiret) {
if (pmc->cinfo.type == SBI_PMU_CTR_TYPE_FW)
pmc->counter_val = kvpmu->fw_event[fevent_code].value;
else if (pmc->perf_event)
pmc->counter_val += perf_event_read_value(pmc->perf_event,
&enabled, &running);
/*
* The counter and overflow indices in the snapshot region are w.r.to
* cbase. Modify the set bit in the counter mask instead of the pmc_index
@ -727,9 +735,10 @@ int kvm_riscv_vcpu_pmu_ctr_stop(struct kvm_vcpu *vcpu, unsigned long ctr_base,
}
}
if (shmem_needs_update)
kvm_vcpu_write_guest(vcpu, kvpmu->snapshot_addr, kvpmu->sdata,
sizeof(struct riscv_pmu_snapshot_data));
if (shmem_needs_update &&
kvm_vcpu_write_guest(vcpu, kvpmu->snapshot_addr, kvpmu->sdata,
sizeof(struct riscv_pmu_snapshot_data)))
sbiret = SBI_ERR_FAILURE;
out:
retdata->err_val = sbiret;

4
arch/riscv/kvm/vcpu_sbi_hsm.c
View File

@ -95,9 +95,9 @@ static int kvm_sbi_ext_hsm_handler(struct kvm_vcpu *vcpu, struct kvm_run *run,
ret = kvm_sbi_hsm_vcpu_get_status(vcpu);
if (ret >= 0) {
retdata->out_val = ret;
retdata->err_val = 0;
ret = 0;
}
return 0;
break;
case SBI_EXT_HSM_HART_SUSPEND:
switch (lower_32_bits(cp->a0)) {
case SBI_HSM_SUSPEND_RET_DEFAULT:

5
arch/riscv/kvm/vcpu_timer.c
View File

@ -61,10 +61,13 @@ static enum hrtimer_restart kvm_riscv_vcpu_hrtimer_expired(struct hrtimer *h)
static int kvm_riscv_vcpu_timer_cancel(struct kvm_vcpu_timer *t)
{
if (!t->init_done || !t->next_set)
if (!t->init_done)
return -EINVAL;
hrtimer_cancel(&t->hrt);
if (!t->next_set)
return -EINVAL;
t->next_set = false;
return 0;

56
arch/riscv/net/bpf_jit_comp64.c
View File

@ -860,6 +860,24 @@ static int gen_jump_or_nops(void *target, void *ip, u32 *insns, bool is_call)
return emit_jump_and_link(is_call ? RV_REG_T0 : RV_REG_ZERO, rvoff, false, &ctx);
}
int arch_bpf_trampoline_skip(void *nop, void *target)
{
u32 old_insn = rv_nop();
u32 new_insn = rv_jal(RV_REG_ZERO, (target - nop) >> 1);
int ret;
if (memcmp(nop, &old_insn, sizeof(old_insn)))
return -EFAULT;
cpus_read_lock();
mutex_lock(&text_mutex);
ret = patch_text(nop, &new_insn, sizeof(new_insn));
mutex_unlock(&text_mutex);
cpus_read_unlock();
return ret;
}
int bpf_arch_text_poke(void *ip, enum bpf_text_poke_type old_t,
enum bpf_text_poke_type new_t, void *old_addr,
void *new_addr)
@ -942,12 +960,17 @@ static void emit_store_stack_imm64(u8 reg, int stack_off, u64 imm64,
emit_sd(RV_REG_FP, stack_off, reg, ctx);
}
static int invoke_bpf_prog(struct bpf_tramp_node *node, int args_off, int retval_off,
int run_ctx_off, bool save_ret, struct rv_jit_context *ctx)
static int invoke_bpf_prog(struct bpf_tramp_image *im, struct bpf_tramp_node *node,
int args_off, int retval_off, int run_ctx_off, bool save_ret,
struct rv_jit_context *ctx)
{
int ret, branch_off;
struct bpf_prog *p = node->link->prog;
int cookie_off = offsetof(struct bpf_tramp_run_ctx, bpf_cookie);
void *skip = ctx->ro_insns + ctx->ninsns;
/* nop, patched to a jal over this prog when it is detached */
emit(rv_nop(), ctx);
if (node->cookie)
emit_store_stack_imm64(RV_REG_T1, -run_ctx_off + cookie_off, node->cookie, ctx);
@ -1000,13 +1023,17 @@ static int invoke_bpf_prog(struct bpf_tramp_node *node, int args_off, int retval
/* arg3: &run_ctx */
emit_addi(RV_REG_A2, RV_REG_FP, -run_ctx_off, ctx);
ret = emit_call((const u64)bpf_trampoline_exit(p), true, ctx);
if (ret)
return ret;
return ret;
bpf_tramp_image_add_skip(im, p, skip, ctx->ro_insns + ctx->ninsns);
return 0;
}
static int invoke_bpf(struct bpf_tramp_nodes *tn, int args_off, int retval_off,
int run_ctx_off, int func_meta_off, bool save_ret, u64 func_meta,
int cookie_off, struct rv_jit_context *ctx)
static int invoke_bpf(struct bpf_tramp_image *im, struct bpf_tramp_nodes *tn,
int args_off, int retval_off, int run_ctx_off, int func_meta_off,
bool save_ret, u64 func_meta, int cookie_off,
struct rv_jit_context *ctx)
{
int i, cur_cookie = (cookie_off - args_off) / 8;
@ -1019,8 +1046,8 @@ static int invoke_bpf(struct bpf_tramp_nodes *tn, int args_off, int retval_off,
emit_store_stack_imm64(RV_REG_T1, -func_meta_off, meta, ctx);
cur_cookie--;
}
err = invoke_bpf_prog(tn->nodes[i], args_off, retval_off, run_ctx_off,
save_ret, ctx);
err = invoke_bpf_prog(im, tn->nodes[i], args_off, retval_off,
run_ctx_off, save_ret, ctx);
if (err)
return err;
}
@ -1208,7 +1235,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im,
}
if (fentry->nr_nodes) {
ret = invoke_bpf(fentry, args_off, retval_off, run_ctx_off, func_meta_off,
ret = invoke_bpf(im, fentry, args_off, retval_off, run_ctx_off, func_meta_off,
flags & BPF_TRAMP_F_RET_FENTRY_RET, func_meta, cookie_off, ctx);
if (ret)
return ret;
@ -1222,7 +1249,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im,
/* cleanup to avoid garbage return value confusion */
emit_sd(RV_REG_FP, -retval_off, RV_REG_ZERO, ctx);
for (i = 0; i < fmod_ret->nr_nodes; i++) {
ret = invoke_bpf_prog(fmod_ret->nodes[i], args_off, retval_off,
ret = invoke_bpf_prog(im, fmod_ret->nodes[i], args_off, retval_off,
run_ctx_off, true, ctx);
if (ret)
goto out;
@ -1249,10 +1276,6 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im,
emit_sd(RV_REG_FP, -tcc_off, RV_REG_TCC, ctx);
emit_sd(RV_REG_FP, -retval_off, RV_REG_A0, ctx);
emit_sd(RV_REG_FP, -(retval_off - 8), regmap[BPF_REG_0], ctx);
im->ip_after_call = ctx->ro_insns + ctx->ninsns;
/* 2 nops reserved for auipc+jalr pair */
emit(rv_nop(), ctx);
emit(rv_nop(), ctx);
}
/* update branches saved in invoke_bpf_mod_ret with bnez */
@ -1268,14 +1291,13 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im,
emit_store_stack_imm64(RV_REG_T1, -func_meta_off, func_meta, ctx);
if (fexit->nr_nodes) {
ret = invoke_bpf(fexit, args_off, retval_off, run_ctx_off, func_meta_off,
ret = invoke_bpf(im, fexit, args_off, retval_off, run_ctx_off, func_meta_off,
false, func_meta, cookie_off, ctx);
if (ret)
goto out;
}
if (flags & BPF_TRAMP_F_CALL_ORIG) {
im->ip_epilogue = ctx->ro_insns + ctx->ninsns;
emit_imm(RV_REG_A0, ctx->insns ? (const s64)im : RV_MAX_COUNT_IMM, ctx);
ret = emit_call((const u64)__bpf_tramp_exit, true, ctx);
if (ret)
@ -1337,7 +1359,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im,
int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags,
struct bpf_tramp_nodes *tnodes, void *func_addr)
{
struct bpf_tramp_image im;
struct bpf_tramp_image im = {};
struct rv_jit_context ctx;
int ret;

8
arch/s390/include/asm/debug.h
View File

@ -460,7 +460,11 @@ static int VNAME(var, active_entries)[EARLY_AREAS] __initdata
#define __REGISTER_STATIC_DEBUG_INFO(var, name, pages, areas, view) \
static int __init VNAME(var, reg)(void) \
{ \
debug_register_static(&var, (pages), (areas)); \
int rc; \
\
rc = debug_register_static(&var, (pages), (areas)); \
if (rc) \
return rc; \
debug_register_view(&var, (view)); \
return 0; \
} \
@ -493,7 +497,7 @@ static debug_info_t __refdata var = \
static debug_info_t __used __section(".s390dbf_info") *VNAME(var, info) = &var; \
__REGISTER_STATIC_DEBUG_INFO(var, name, pages, nr_areas, view)
void debug_register_static(debug_info_t *id, int pages_per_area, int nr_areas);
int debug_register_static(debug_info_t *id, int pages_per_area, int nr_areas);
#endif /* MODULE */

18
arch/s390/kernel/debug.c
View File

@ -434,7 +434,8 @@ static debug_info_t *debug_info_copy(debug_info_t *in, int mode)
debug_info_free(rc);
} while (1);
if (mode == NO_AREAS)
/* debug_register_static() failure leaves areas NULL, bounds intact */
if (mode == NO_AREAS || !in->areas)
goto out;
for (i = 0; i < in->nr_areas; i++) {
@ -823,6 +824,9 @@ ssize_t debug_dump(debug_info_t *id, struct debug_view *view,
file_private_info_t *p_info;
size_t size, offset = 0;
if (!id)
return -EINVAL;
/* Need space for '\0' byte */
if (buf_size < 1)
return 0;
@ -950,8 +954,12 @@ EXPORT_SYMBOL(debug_register);
*
* Note: This function is called automatically via an initcall generated by
* DEFINE_STATIC_DEBUG_INFO.
*
* Return:
* - 0 on success
* - negative error code on failure
*/
void debug_register_static(debug_info_t *id, int pages_per_area, int nr_areas)
int debug_register_static(debug_info_t *id, int pages_per_area, int nr_areas)
{
unsigned long flags;
debug_info_t *copy;
@ -959,7 +967,7 @@ void debug_register_static(debug_info_t *id, int pages_per_area, int nr_areas)
if (!initialized) {
pr_err("Tried to register debug feature %s too early\n",
id->name);
return;
return -EINVAL;
}
debug_get_param(id->name, &id->level, &pages_per_area, false);
@ -975,7 +983,7 @@ void debug_register_static(debug_info_t *id, int pages_per_area, int nr_areas)
id->active_entries = NULL;
raw_spin_unlock_irqrestore(&id->lock, flags);
return;
return -ENOMEM;
}
/* Replace static trace area with dynamic copy. */
@ -993,6 +1001,8 @@ void debug_register_static(debug_info_t *id, int pages_per_area, int nr_areas)
mutex_lock(&debug_mutex);
_debug_register(id);
mutex_unlock(&debug_mutex);
return 0;
}
/* Remove debugfs entries. */

8
arch/s390/kernel/uv.c
View File

@ -825,7 +825,7 @@ static int find_secret_in_page(const u8 secret_id[UV_SECRET_ID_LEN],
{
u16 i;
for (i = 0; i < list->total_num_secrets; i++) {
for (i = 0; i < list->num_secr_stored; i++) {
if (memcmp(secret_id, list->secrets[i].id, UV_SECRET_ID_LEN) == 0) {
*secret = list->secrets[i].hdr;
return 0;
@ -846,11 +846,14 @@ int uv_find_secret(const u8 secret_id[UV_SECRET_ID_LEN],
struct uv_secret_list *list,
struct uv_secret_list_item_hdr *secret)
{
u16 start_idx = 0;
u16 start_idx;
u16 list_rc;
int ret;
list->next_secret_idx = 0;
do {
start_idx = list->next_secret_idx;
uv_list_secrets(list, start_idx, &list_rc, NULL);
if (list_rc != UVC_RC_EXECUTED && list_rc != UVC_RC_MORE_DATA) {
if (list_rc == UVC_RC_INV_CMD)
@ -861,7 +864,6 @@ int uv_find_secret(const u8 secret_id[UV_SECRET_ID_LEN],
ret = find_secret_in_page(secret_id, list, secret);
if (ret == 0)
return ret;
start_idx = list->next_secret_idx;
} while (list_rc == UVC_RC_MORE_DATA && start_idx < list->next_secret_idx);
return -ENOENT;

1
arch/s390/kernel/vdso/Makefile
View File

@ -29,6 +29,7 @@ KBUILD_CFLAGS_VDSO := $(filter-out -munaligned-symbols,$(KBUILD_CFLAGS_VDSO))
KBUILD_CFLAGS_VDSO := $(filter-out -fno-asynchronous-unwind-tables,$(KBUILD_CFLAGS_VDSO))
KBUILD_CFLAGS_VDSO += -fPIC -fno-common -fno-builtin -fasynchronous-unwind-tables
KBUILD_CFLAGS_VDSO += -fno-stack-protector $(DISABLE_KSTACK_ERASE)
KBUILD_CFLAGS_VDSO += $(CONFIG_CC_OPT_INLINE_MEMSET)
ldflags-y := -shared -soname=linux-vdso.so.1 \
--hash-style=both --build-id=sha1 \
$(call ld-option, --eh-frame-hdr) -T

65
arch/s390/kvm/gmap/dat.c
View File

@ -621,17 +621,20 @@ int dat_get_storage_key(union asce asce, gfn_t gfn, union skey *skey)
union pte *ptep;
int rc;
again:
skey->skey = 0;
rc = dat_entry_walk(NULL, gfn, asce, DAT_WALK_ANY, TABLE_TYPE_PAGE_TABLE, &crstep, &ptep);
if (rc)
return rc;
if (!ptep) {
union crste crste;
union crste crste = READ_ONCE(*crstep);
crste = READ_ONCE(*crstep);
if (!crste.h.fc || !crste.s.fc1.pr)
if (!crste_leaf(crste) && !crste.h.i)
goto again;
if (!crste.s.fc1.pr)
return 0;
skey->skey = page_get_storage_key(large_crste_to_phys(crste, gfn));
return 0;
}
@ -662,13 +665,20 @@ int dat_set_storage_key(struct kvm_s390_mmu_cache *mc, union asce asce, gfn_t gf
union pte *ptep;
int rc;
again:
rc = dat_entry_walk(mc, gfn, asce, DAT_WALK_LEAF_ALLOC, TABLE_TYPE_PAGE_TABLE,
&crstep, &ptep);
if (rc)
return rc;
if (!ptep) {
page_set_storage_key(large_crste_to_phys(*crstep, gfn), skey.skey, !nq);
union crste crste = READ_ONCE(*crstep);
/* A large page has been split concurrently, try again */
if (!crste_leaf(crste))
goto again;
page_set_storage_key(large_crste_to_phys(crste, gfn), skey.skey, !nq);
return 0;
}
@ -718,15 +728,22 @@ int dat_cond_set_storage_key(struct kvm_s390_mmu_cache *mmc, union asce asce, gf
union pte *ptep;
int rc;
again:
rc = dat_entry_walk(mmc, gfn, asce, DAT_WALK_LEAF_ALLOC, TABLE_TYPE_PAGE_TABLE,
&crstep, &ptep);
if (rc)
return rc;
if (!ptep) {
union crste crste = READ_ONCE(*crstep);
/* A large page has been split concurrently, try again */
if (!crste_leaf(crste))
goto again;
if (!oldkey)
oldkey = &prev;
return page_cond_set_storage_key(large_crste_to_phys(*crstep, gfn), skey, oldkey,
return page_cond_set_storage_key(large_crste_to_phys(crste, gfn), skey, oldkey,
nq, mr, mc);
}
@ -768,7 +785,7 @@ int dat_reset_reference_bit(union asce asce, gfn_t gfn, union skey *skey)
int rc;
skey->skey = 0;
again:
rc = dat_entry_walk(NULL, gfn, asce, DAT_WALK_ANY, TABLE_TYPE_PAGE_TABLE, &crstep, &ptep);
if (rc)
return rc;
@ -776,9 +793,12 @@ int dat_reset_reference_bit(union asce asce, gfn_t gfn, union skey *skey)
if (!ptep) {
union crste crste = READ_ONCE(*crstep);
if (!crste.h.fc || !crste.s.fc1.pr)
/* A large page has been split concurrently, try again */
if (!crste_leaf(crste) && !crste.h.i)
goto again;
if (!crste.s.fc1.pr)
return 0;
skey->skey = page_reset_referenced(large_crste_to_phys(*crstep, gfn)) << 1;
skey->skey = page_reset_referenced(large_crste_to_phys(crste, gfn)) << 1;
return 0;
}
old = pgste_get_lock(ptep);
@ -846,19 +866,12 @@ long dat_reset_skeys(union asce asce, gfn_t start)
}
#endif /* KVM_S390_MANAGES_S390_GUEST */
struct slot_priv {
unsigned long token;
struct kvm_s390_mmu_cache *mc;
};
static long _dat_slot_pte(union pte *ptep, gfn_t gfn, gfn_t next, struct dat_walk *walk)
{
struct slot_priv *p = walk->priv;
union crste dummy = { .val = p->token };
union pte new_pte, pte = READ_ONCE(*ptep);
union pgste pgste;
new_pte = _PTE_TOK(dummy.tok.type, dummy.tok.par);
new_pte = walk->priv ? _PTE_EMPTY : _PTE_TOK(_DAT_TOKEN_PIC, PGM_ADDRESSING);
/* Table entry already in the desired state. */
if (pte.val == new_pte.val)
@ -875,10 +888,9 @@ static long _dat_slot_pte(union pte *ptep, gfn_t gfn, gfn_t next, struct dat_wal
static long _dat_slot_crste(union crste *crstep, gfn_t gfn, gfn_t next, struct dat_walk *walk)
{
union crste new_crste, crste = READ_ONCE(*crstep);
struct slot_priv *p = walk->priv;
struct kvm_s390_mmu_cache *mc = walk->priv;
new_crste.val = p->token;
new_crste.h.tt = crste.h.tt;
new_crste = mc ? _CRSTE_EMPTY(crste.h.tt) : _CRSTE_HOLE(crste.h.tt);
/* Table entry already in the desired state. */
if (crste.val == new_crste.val)
@ -902,7 +914,10 @@ static long _dat_slot_crste(union crste *crstep, gfn_t gfn, gfn_t next, struct d
if (!crste.h.fc && !crste.h.i)
return 0;
/* Split (install a lower level table), and handle things there. */
return dat_split_crste(p->mc, crstep, gfn, walk->asce, false);
if (mc)
return dat_split_crste(mc, crstep, gfn, walk->asce, false);
/* A large page should never cross memslots boundaries */
return -EINVAL;
}
static const struct dat_walk_ops dat_slot_ops = {
@ -910,16 +925,10 @@ static const struct dat_walk_ops dat_slot_ops = {
.crste_ops = { _dat_slot_crste, _dat_slot_crste, _dat_slot_crste, _dat_slot_crste, },
};
int dat_set_slot(struct kvm_s390_mmu_cache *mc, union asce asce, gfn_t start, gfn_t end,
u16 type, u16 param)
int dat_set_slot(struct kvm_s390_mmu_cache *mc, union asce asce, gfn_t start, gfn_t end)
{
struct slot_priv priv = {
.token = _CRSTE_TOK(0, type, param).val,
.mc = mc,
};
return _dat_walk_gfn_range(start, end, asce, &dat_slot_ops,
DAT_WALK_IGN_HOLES | DAT_WALK_ANY, &priv);
DAT_WALK_IGN_HOLES | DAT_WALK_ANY, mc);
}
static void pgste_set_unlock_multiple(union pte *first, int n, union pgste *pgstes)

10
arch/s390/kvm/gmap/dat.h
View File

@ -547,8 +547,7 @@ long dat_reset_skeys(union asce asce, gfn_t start);
unsigned long dat_get_ptval(struct page_table *table, struct ptval_param param);
void dat_set_ptval(struct page_table *table, struct ptval_param param, unsigned long val);
int dat_set_slot(struct kvm_s390_mmu_cache *mc, union asce asce, gfn_t start, gfn_t end,
u16 type, u16 param);
int dat_set_slot(struct kvm_s390_mmu_cache *mc, union asce asce, gfn_t start, gfn_t end);
#if KVM_S390_MANAGES_S390_GUEST
int dat_set_prefix_notif_bit(union asce asce, gfn_t gfn);
@ -973,16 +972,15 @@ static inline int get_level(union crste *crstep, union pte *ptep)
return ptep ? TABLE_TYPE_PAGE_TABLE : crstep->h.tt;
}
static inline int dat_delete_slot(struct kvm_s390_mmu_cache *mc, union asce asce, gfn_t start,
unsigned long npages)
static inline int dat_delete_slot(union asce asce, gfn_t start, unsigned long npages)
{
return dat_set_slot(mc, asce, start, start + npages, _DAT_TOKEN_PIC, PGM_ADDRESSING);
return dat_set_slot(NULL, asce, start, start + npages);
}
static inline int dat_create_slot(struct kvm_s390_mmu_cache *mc, union asce asce, gfn_t start,
unsigned long npages)
{
return dat_set_slot(mc, asce, start, start + npages, _DAT_TOKEN_NONE, 0);
return dat_set_slot(mc, asce, start, start + npages);
}
static inline bool crste_is_ucas(union crste crste)

6
arch/s390/kvm/gmap/gmap.c
View File

@ -994,11 +994,13 @@ static long _destroy_pages_pte(union pte *ptep, gfn_t gfn, gfn_t next, struct da
static long _destroy_pages_crste(union crste *crstep, gfn_t gfn, gfn_t next, struct dat_walk *walk)
{
phys_addr_t origin, cur, end;
union crste crste;
if (!crstep->h.fc || !crstep->s.fc1.pr)
crste = READ_ONCE(*crstep);
if (!crste.h.fc || !crste.s.fc1.pr)
return 0;
origin = crste_origin_large(*crstep);
origin = crste_origin_large(crste);
cur = ((max(gfn, walk->start) - gfn) << PAGE_SHIFT) + origin;
end = ((min(next, walk->end) - gfn) << PAGE_SHIFT) + origin;
for ( ; cur < end; cur += PAGE_SIZE)

91
arch/s390/kvm/gmap/kvm_mmu.c
View File

@ -47,6 +47,9 @@ int s390_kvm_mmu_prepare_memory_region(struct kvm *kvm,
struct kvm_memory_slot *new,
enum kvm_mr_change change)
{
struct kvm_s390_mmu_cache *mc __free(kvm_s390_mmu_cache) = NULL;
int rc = 0;
if (kvm_is_ucontrol(kvm) && new && new->id < KVM_USER_MEM_SLOTS)
return -EINVAL;
@ -61,6 +64,10 @@ int s390_kvm_mmu_prepare_memory_region(struct kvm *kvm,
* and munmap() stuff in this slot after doing this call at any
* time.
*/
if (change != KVM_MR_MOVE && change != KVM_MR_CREATE) {
WARN(1, "Unknown KVM MR CHANGE: %d\n", change);
return -EINVAL;
}
if (new->userspace_addr & ~PAGE_MASK)
return -EINVAL;
if ((new->base_gfn + new->npages) * PAGE_SIZE > kvm->arch.mem_limit)
@ -69,65 +76,41 @@ int s390_kvm_mmu_prepare_memory_region(struct kvm *kvm,
return -EINVAL;
}
if (!kvm_s390_is_migration_mode(kvm))
return 0;
/*
* Turn off migration mode when:
* - userspace creates a new memslot with dirty logging off,
* - userspace modifies an existing memslot (MOVE or FLAGS_ONLY) and
* dirty logging is turned off.
* Migration mode expects dirty page logging being enabled to store
* its dirty bitmap.
*/
if (change != KVM_MR_DELETE &&
!(new->flags & KVM_MEM_LOG_DIRTY_PAGES))
WARN(kvm_s390_vm_stop_migration(kvm),
"Failed to stop migration mode");
return 0;
}
void s390_kvm_mmu_commit_memory_region(struct kvm *kvm,
struct kvm_memory_slot *old,
const struct kvm_memory_slot *new,
enum kvm_mr_change change)
{
struct kvm_s390_mmu_cache *mc __free(kvm_s390_mmu_cache) = NULL;
int rc = 0;
guard(mutex)(&kvm->slots_arch_lock);
if (kvm->arch.migration_mode) {
/*
* Turn off migration mode when:
* - userspace creates a new memslot with dirty logging off,
* - userspace modifies an existing memslot (MOVE or FLAGS_ONLY)
* and dirty logging is turned off.
* Migration mode expects dirty page logging being enabled to
* store its dirty bitmap.
*/
if (change != KVM_MR_DELETE &&
!(new->flags & KVM_MEM_LOG_DIRTY_PAGES))
WARN(kvm_s390_vm_stop_migration(kvm),
"Failed to stop migration mode");
}
if (change == KVM_MR_FLAGS_ONLY)
return;
mc = kvm_s390_new_mmu_cache();
if (!mc) {
rc = -ENOMEM;
goto out;
return 0;
if (change != KVM_MR_DELETE) {
/* Enough capacity to add a new memslot */
mc = kvm_s390_new_mmu_cache();
if (!mc)
return -ENOMEM;
}
scoped_guard(write_lock, &kvm->mmu_lock) {
kvm_s390_update_cmma_dirty(kvm, old);
switch (change) {
case KVM_MR_DELETE:
rc = dat_delete_slot(mc, kvm->arch.gmap->asce, old->base_gfn, old->npages);
break;
case KVM_MR_MOVE:
rc = dat_delete_slot(mc, kvm->arch.gmap->asce, old->base_gfn, old->npages);
if (rc)
break;
fallthrough;
case KVM_MR_CREATE:
if (change == KVM_MR_DELETE || change == KVM_MR_MOVE)
rc = dat_delete_slot(kvm->arch.gmap->asce, old->base_gfn, old->npages);
if (!rc && (change == KVM_MR_MOVE || change == KVM_MR_CREATE))
rc = dat_create_slot(mc, kvm->arch.gmap->asce, new->base_gfn, new->npages);
break;
case KVM_MR_FLAGS_ONLY:
break;
default:
WARN(1, "Unknown KVM MR CHANGE: %d\n", change);
}
}
out:
if (rc)
pr_warn("failed to commit memory region\n");
/*
* Can only be triggered if dat_{create,delete}_slot() found an
* internal inconsistency or if the mmu cache ran out of memory;
* both should be impossible.
*/
KVM_BUG_ON(rc, kvm);
return rc;
}

4
arch/s390/kvm/gmap/kvm_mmu.h
View File

@ -10,9 +10,5 @@ int s390_kvm_mmu_prepare_memory_region(struct kvm *kvm,
const struct kvm_memory_slot *old,
struct kvm_memory_slot *new,
enum kvm_mr_change change);
void s390_kvm_mmu_commit_memory_region(struct kvm *kvm,
struct kvm_memory_slot *old,
const struct kvm_memory_slot *new,
enum kvm_mr_change change);
#endif /* ARCH_KVM_GMAP_KVM_MMU_H */

13
arch/s390/kvm/s390/gaccess.c
View File

@ -1589,12 +1589,25 @@ static inline int ___gaccess_shadow_fault(struct kvm_vcpu *vcpu, struct gmap *sg
parent = READ_ONCE(sg->parent);
if (!parent)
return -EAGAIN;
retry:
scoped_guard(spinlock, &parent->children_lock) {
if (READ_ONCE(sg->parent) != parent)
return -EAGAIN;
sg->invalidated = false;
rc = _gaccess_do_shadow(vcpu->arch.mc, sg, saddr, walk);
}
if (rc == -ENOENT) {
struct kvm_memory_slot *slot;
struct guest_fault *entries;
entries = get_entries(walk);
slot = kvm_vcpu_gfn_to_memslot(vcpu, entries[LEVEL_MEM].gfn);
if (!slot)
return PGM_ADDRESSING;
rc = gmap_link(vcpu->arch.mc, parent, entries + LEVEL_MEM, slot);
if (!rc)
goto retry;
}
if (!rc)
kvm_s390_release_faultin_array(vcpu->kvm, walk->raw_entries, false);
return rc;

81
arch/s390/kvm/s390/interrupt.c
View File

@ -2984,61 +2984,58 @@ static int adapter_indicators_set(struct kvm *kvm,
struct s390_io_adapter *adapter,
struct kvm_s390_adapter_int *adapter_int)
{
unsigned long bit;
int summary_set, idx;
struct s390_map_info *ind_info, *summary_info;
void *map;
struct page *ind_page, *summary_page;
unsigned long flags;
unsigned long bit;
int summary_set;
void *map;
ind_page = NULL;
spin_lock_irqsave(&adapter->maps_lock, flags);
ind_info = get_map_info(adapter, adapter_int->ind_addr);
scoped_guard(spinlock_irqsave, &adapter->maps_lock) {
ind_info = get_map_info(adapter, adapter_int->ind_addr);
if (ind_info) {
map = page_address(ind_info->page);
bit = get_ind_bit(ind_info->addr, adapter_int->ind_offset, adapter->swap);
set_bit(bit, map);
}
}
if (!ind_info) {
spin_unlock_irqrestore(&adapter->maps_lock, flags);
ind_page = pin_map_page(kvm, adapter_int->ind_addr, 0);
if (!ind_page)
return -1;
idx = srcu_read_lock(&kvm->srcu);
map = page_address(ind_page);
bit = get_ind_bit(adapter_int->ind_addr,
adapter_int->ind_offset, adapter->swap);
set_bit(bit, map);
mark_page_dirty(kvm, adapter_int->ind_gaddr >> PAGE_SHIFT);
set_page_dirty_lock(ind_page);
srcu_read_unlock(&kvm->srcu, idx);
unpin_user_page(ind_page);
} else {
map = page_address(ind_info->page);
bit = get_ind_bit(ind_info->addr, adapter_int->ind_offset, adapter->swap);
set_bit(bit, map);
spin_unlock_irqrestore(&adapter->maps_lock, flags);
}
scoped_guard(srcu, &kvm->srcu)
mark_page_dirty(kvm, gpa_to_gfn(adapter_int->ind_gaddr));
spin_lock_irqsave(&adapter->maps_lock, flags);
summary_info = get_map_info(adapter, adapter_int->summary_addr);
scoped_guard(spinlock_irqsave, &adapter->maps_lock) {
summary_info = get_map_info(adapter, adapter_int->summary_addr);
if (summary_info) {
map = page_address(summary_info->page);
bit = get_ind_bit(summary_info->addr, adapter_int->summary_offset,
adapter->swap);
summary_set = test_and_set_bit(bit, map);
}
}
if (!summary_info) {
spin_unlock_irqrestore(&adapter->maps_lock, flags);
summary_page = pin_map_page(kvm, adapter_int->summary_addr, 0);
if (!summary_page)
return -1;
idx = srcu_read_lock(&kvm->srcu);
map = page_address(summary_page);
bit = get_ind_bit(adapter_int->summary_addr,
adapter_int->summary_offset, adapter->swap);
summary_set = test_and_set_bit(bit, map);
mark_page_dirty(kvm, adapter_int->summary_gaddr >> PAGE_SHIFT);
set_page_dirty_lock(summary_page);
srcu_read_unlock(&kvm->srcu, idx);
unpin_user_page(summary_page);
} else {
map = page_address(summary_info->page);
bit = get_ind_bit(summary_info->addr, adapter_int->summary_offset,
adapter->swap);
summary_set = test_and_set_bit(bit, map);
spin_unlock_irqrestore(&adapter->maps_lock, flags);
}
scoped_guard(srcu, &kvm->srcu)
mark_page_dirty(kvm, gpa_to_gfn(adapter_int->summary_gaddr));
return summary_set ? 0 : 1;
}
@ -3048,26 +3045,29 @@ static int adapter_indicators_set_fast(struct kvm *kvm,
struct kvm_s390_adapter_int *adapter_int,
int setbit)
{
struct s390_map_info *ind_info, *summary_info;
unsigned long bit;
int summary_set;
struct s390_map_info *ind_info, *summary_info;
void *map;
spin_lock(&adapter->maps_lock);
guard(srcu)(&kvm->srcu);
guard(spinlock)(&adapter->maps_lock);
ind_info = get_map_info(adapter, adapter_int->ind_addr);
if (!ind_info) {
spin_unlock(&adapter->maps_lock);
if (!ind_info)
return -EWOULDBLOCK;
}
map = page_address(ind_info->page);
bit = get_ind_bit(ind_info->addr, adapter_int->ind_offset, adapter->swap);
if (setbit)
if (setbit) {
set_bit(bit, map);
summary_info = get_map_info(adapter, adapter_int->summary_addr);
if (!summary_info) {
spin_unlock(&adapter->maps_lock);
return -EWOULDBLOCK;
mark_page_dirty(kvm, gpa_to_gfn(adapter_int->ind_gaddr));
}
summary_info = get_map_info(adapter, adapter_int->summary_addr);
if (!summary_info)
return -EWOULDBLOCK;
map = page_address(summary_info->page);
bit = get_ind_bit(summary_info->addr, adapter_int->summary_offset,
adapter->swap);
@ -3077,7 +3077,8 @@ static int adapter_indicators_set_fast(struct kvm *kvm,
summary_set = test_and_set_bit(bit, map);
else
summary_set = test_and_clear_bit(bit, map);
spin_unlock(&adapter->maps_lock);
mark_page_dirty(kvm, gpa_to_gfn(adapter_int->summary_gaddr));
return summary_set ? 0 : 1;
}
@ -3228,9 +3229,9 @@ int kvm_s390_set_irq_state(struct kvm_vcpu *vcpu, void __user *irqstate, int len
break;
}
}
if (storestatus) {
n = kvm_s390_store_status_unloaded(vcpu, KVM_S390_STORE_STATUS_NOADDR);
scoped_guard(srcu, &vcpu->kvm->srcu)
n = kvm_s390_store_status_unloaded(vcpu, KVM_S390_STORE_STATUS_NOADDR);
return r ? r : n;
}

8
arch/s390/kvm/s390/s390.c
View File

@ -3579,12 +3579,11 @@ void kvm_arch_vcpu_put(struct kvm_vcpu *vcpu)
void kvm_arch_vcpu_postcreate(struct kvm_vcpu *vcpu)
{
mutex_lock(&vcpu->kvm->lock);
preempt_disable();
vcpu->arch.sie_block->epoch = vcpu->kvm->arch.epoch;
vcpu->arch.sie_block->epdx = vcpu->kvm->arch.epdx;
preempt_enable();
mutex_unlock(&vcpu->kvm->lock);
if (!kvm_is_ucontrol(vcpu->kvm)) {
vcpu->arch.gmap = vcpu->kvm->arch.gmap;
sca_add_vcpu(vcpu);
@ -3757,13 +3756,11 @@ static int kvm_s390_vcpu_setup(struct kvm_vcpu *vcpu)
kvm_s390_vcpu_pci_setup(vcpu);
mutex_lock(&vcpu->kvm->lock);
if (kvm_s390_pv_is_protected(vcpu->kvm)) {
rc = kvm_s390_pv_create_cpu(vcpu, &uvrc, &uvrrc);
if (rc)
kvm_s390_vcpu_unsetup_cmma(vcpu);
}
mutex_unlock(&vcpu->kvm->lock);
return rc;
}
@ -5766,7 +5763,7 @@ static long cmma_d_count_pte(union pte *ptep, gfn_t gfn, gfn_t next, struct dat_
return 0;
}
void kvm_s390_update_cmma_dirty(struct kvm *kvm, struct kvm_memory_slot *old)
void kvm_s390_update_cmma_dirty(struct kvm *kvm, const struct kvm_memory_slot *old)
{
const struct dat_walk_ops ops = { .pte_entry = cmma_d_count_pte, };
@ -5781,7 +5778,6 @@ void kvm_arch_commit_memory_region(struct kvm *kvm, struct kvm_memory_slot *old,
const struct kvm_memory_slot *new,
enum kvm_mr_change change)
{
s390_kvm_mmu_commit_memory_region(kvm, old, new, change);
}
/**

2
arch/s390/kvm/s390/s390.h
View File

@ -472,7 +472,7 @@ int __kvm_s390_mprotect_many(struct gmap *gmap, gpa_t gpa, u8 npages, unsigned i
unsigned long bits);
bool kvm_arch_setup_async_pf(struct kvm_vcpu *vcpu);
void kvm_s390_update_cmma_dirty(struct kvm *kvm, struct kvm_memory_slot *old);
void kvm_s390_update_cmma_dirty(struct kvm *kvm, const struct kvm_memory_slot *old);
int kvm_s390_vm_stop_migration(struct kvm *kvm);

46
arch/s390/net/bpf_jit_comp.c
View File

@ -2566,6 +2566,8 @@ struct bpf_tramp_jit {
int r14_off; /* Offset of saved %r14, has to be at the
* bottom */
int do_fexit; /* do_fexit: label */
int skip[BPF_MAX_TRAMP_LINKS]; /* skip: labels after each prog */
int nr_progs;
};
static void load_imm64(struct bpf_jit *jit, int dst_reg, u64 val)
@ -2584,6 +2586,7 @@ static void emit_store_stack_imm64(struct bpf_jit *jit, int tmp_reg, int stack_o
}
static int invoke_bpf_prog(struct bpf_tramp_jit *tjit,
struct bpf_tramp_image *im,
const struct btf_func_model *m,
struct bpf_tramp_node *node, bool save_ret)
{
@ -2591,8 +2594,20 @@ static int invoke_bpf_prog(struct bpf_tramp_jit *tjit,
int cookie_off = tjit->run_ctx_off +
offsetof(struct bpf_tramp_run_ctx, bpf_cookie);
struct bpf_prog *p = node->link->prog;
void *skip = jit->prg_buf + jit->prg;
int idx = tjit->nr_progs++;
int patch;
if (idx >= ARRAY_SIZE(tjit->skip))
return -E2BIG;
/*
* nop, patched to skip this prog when it is detached
*/
/* brcl 0,skip */
EMIT6_PCREL_RILC(0xc0040000, 0, tjit->skip[idx]);
/*
* run_ctx.cookie = node->cookie;
*/
@ -2652,10 +2667,15 @@ static int invoke_bpf_prog(struct bpf_tramp_jit *tjit,
/* brasl %r14,__bpf_prog_exit */
EMIT6_PCREL_RILB_PTR(0xc0050000, REG_14, bpf_trampoline_exit(p));
/* skip: */
tjit->skip[idx] = jit->prg;
bpf_tramp_image_add_skip(im, p, skip, jit->prg_buf + jit->prg);
return 0;
}
static int invoke_bpf(struct bpf_tramp_jit *tjit,
struct bpf_tramp_image *im,
const struct btf_func_model *m,
struct bpf_tramp_nodes *tn, bool save_ret,
u64 func_meta, int cookie_off)
@ -2670,7 +2690,7 @@ static int invoke_bpf(struct bpf_tramp_jit *tjit,
emit_store_stack_imm64(jit, REG_0, tjit->func_meta_off, meta);
cur_cookie--;
}
if (invoke_bpf_prog(tjit, m, tn->nodes[i], save_ret))
if (invoke_bpf_prog(tjit, im, m, tn->nodes[i], save_ret))
return -EINVAL;
}
@ -2712,6 +2732,11 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im,
u64 func_meta;
int i, j;
/* The skip labels are taken from the previous pass. */
tjit->nr_progs = 0;
if (im)
im->nr_skips = 0;
/* Support as many stack arguments as "mvc" instruction can handle. */
nr_reg_args = min_t(int, m->nr_args, MAX_NR_REG_ARGS);
nr_stack_args = m->nr_args - nr_reg_args;
@ -2875,7 +2900,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im,
emit_store_stack_imm64(jit, REG_0, tjit->retval_off, 0);
}
if (invoke_bpf(tjit, m, fentry, flags & BPF_TRAMP_F_RET_FENTRY_RET,
if (invoke_bpf(tjit, im, m, fentry, flags & BPF_TRAMP_F_RET_FENTRY_RET,
func_meta, cookie_off))
return -EINVAL;
@ -2889,7 +2914,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im,
0xf000 | tjit->retval_off);
for (i = 0; i < fmod_ret->nr_nodes; i++) {
if (invoke_bpf_prog(tjit, m, fmod_ret->nodes[i], true))
if (invoke_bpf_prog(tjit, im, m, fmod_ret->nodes[i], true))
return -EINVAL;
/*
@ -2943,15 +2968,6 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im,
/* mvc tccnt_off(%r15),tail_call_cnt(4,%r15) */
_EMIT6(0xd203f000 | tjit->tccnt_off,
0xf000 | offsetof(struct prog_frame, tail_call_cnt));
im->ip_after_call = jit->prg_buf + jit->prg;
/*
* The following nop will be patched by bpf_tramp_image_put().
*/
/* brcl 0,im->ip_epilogue */
EMIT6_PCREL_RILC(0xc0040000, 0, (u64)im->ip_epilogue);
}
/* Set the "is_return" flag for fsession. */
@ -2962,12 +2978,10 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im,
/* do_fexit: */
tjit->do_fexit = jit->prg;
if (invoke_bpf(tjit, m, fexit, false, func_meta, cookie_off))
if (invoke_bpf(tjit, im, m, fexit, false, func_meta, cookie_off))
return -EINVAL;
if (flags & BPF_TRAMP_F_CALL_ORIG) {
im->ip_epilogue = jit->prg_buf + jit->prg;
/*
* __bpf_tramp_exit(im);
*/
@ -3016,7 +3030,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im,
int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags,
struct bpf_tramp_nodes *tnodes, void *orig_call)
{
struct bpf_tramp_image im;
struct bpf_tramp_image im = {};
struct bpf_tramp_jit tjit;
int ret;

7
arch/s390/pci/pci_event.c
View File

@ -226,6 +226,7 @@ static pci_ers_result_t zpci_event_attempt_error_recovery(struct pci_dev *pdev,
device_lock(&pdev->dev);
if (pdev->error_state == pci_channel_io_perm_failure) {
ers_res = PCI_ERS_RESULT_DISCONNECT;
status_str = "skipped (permanent failure)";
goto out_unlock;
}
pdev->error_state = pci_channel_io_frozen;
@ -297,8 +298,8 @@ static pci_ers_result_t zpci_event_attempt_error_recovery(struct pci_dev *pdev,
driver->err_handler->resume(pdev);
pci_uevent_ers(pdev, PCI_ERS_RESULT_RECOVERED);
out_unlock:
zpci_report_status(zdev, pdev, "recovery", status_str);
device_unlock(&pdev->dev);
zpci_report_status(zdev, "recovery", status_str);
return ers_res;
}
@ -361,8 +362,10 @@ static void __zpci_event_error(struct zpci_ccdf_err *ccdf)
__zpci_event_print_error(pdev, ccdf);
if (!pdev)
if (!pdev) {
zpci_report_status(zdev, NULL, "error event", "no pdev bound");
goto no_pdev;
}
switch (ccdf->pec) {
case 0x002a: /* Error event concerns FMB */

32
arch/s390/pci/pci_report.c
View File

@ -87,9 +87,23 @@ static struct debug_view debug_log_view = {
NULL
};
static ssize_t zpci_report_pdev(struct pci_dev *pdev, char *buf, size_t size)
{
struct pci_driver *driver;
const char *start = buf;
char *end = buf + size;
device_lock_assert(&pdev->dev);
buf += scnprintf(buf, end - buf, "state: %s\n", zpci_state_str(pdev->error_state));
driver = to_pci_driver(pdev->dev.driver);
buf += scnprintf(buf, end - buf, "driver: %s\n", (driver) ? driver->name : "n/a");
return buf - start;
}
/**
* zpci_report_status - Report the status of operations on a PCI device
* @zdev: The PCI device for which to report status
* @zdev: The zPCI device for which to report status
* @pdev: The PCI device associated with the zdev if any, NULL otherwise
* @operation: A string representing the operation reported
* @status: A string representing the status of the operation
*
@ -103,15 +117,14 @@ static struct debug_view debug_log_view = {
*
* Return: 0 on success an error code < 0 otherwise.
*/
int zpci_report_status(struct zpci_dev *zdev, const char *operation, const char *status)
int zpci_report_status(struct zpci_dev *zdev, struct pci_dev *pdev,
const char *operation, const char *status)
{
struct zpci_report_error *report;
struct pci_driver *driver = NULL;
struct pci_dev *pdev = NULL;
char *buf, *end;
int ret;
if (!zdev || !zdev->zbus)
if (!zdev)
return -ENODEV;
/* Protected virtualization hosts get nothing from us */
@ -121,18 +134,13 @@ int zpci_report_status(struct zpci_dev *zdev, const char *operation, const char
report = (void *)get_zeroed_page(GFP_KERNEL);
if (!report)
return -ENOMEM;
if (zdev->zbus->bus)
pdev = pci_get_slot(zdev->zbus->bus, zdev->devfn);
if (pdev)
driver = to_pci_driver(pdev->dev.driver);
buf = report->data.log_data;
end = report->data.log_data + ZPCI_REPORT_DATA_SIZE;
buf += scnprintf(buf, end - buf, "report: %s\n", operation);
buf += scnprintf(buf, end - buf, "status: %s\n", status);
buf += scnprintf(buf, end - buf, "state: %s\n",
(pdev) ? zpci_state_str(pdev->error_state) : "n/a");
buf += scnprintf(buf, end - buf, "driver: %s\n", (driver) ? driver->name : "n/a");
if (pdev)
buf += zpci_report_pdev(pdev, buf, end - buf);
ret = debug_dump(pci_debug_msg_id, &debug_log_view, buf, end - buf, true);
if (ret < 0)
pr_err("Reading PCI debug messages failed with code %d\n", ret);

4
arch/s390/pci/pci_report.h
View File

@ -8,9 +8,11 @@
*/
#ifndef __S390_PCI_REPORT_H
#define __S390_PCI_REPORT_H
#include <linux/pci.h>
struct zpci_dev;
int zpci_report_status(struct zpci_dev *zdev, const char *operation, const char *status);
int zpci_report_status(struct zpci_dev *zdev, struct pci_dev *pdev,
const char *operation, const char *status);
#endif /* __S390_PCI_REPORT_H */

10
arch/x86/coco/sev/svsm.c
View File

@ -74,6 +74,14 @@ int svsm_perform_call_protocol(struct svsm_call *call)
flags = native_local_irq_save();
/*
* 'caa' is a per-CPU variable. To avoid using a stale or incorrect
* 'caa' if the task is preempted or migrated to another CPU after it
* is fetched, always fetch 'caa' and then issue the SVSM call with
* interrupts disabled. This ensures the correct 'caa' is used.
*/
call->caa = svsm_get_caa();
ghcb = __sev_get_ghcb(&state);
do {
@ -321,7 +329,6 @@ int snp_svsm_vtpm_send_command(u8 *buffer)
{
struct svsm_call call = {};
call.caa = svsm_get_caa();
call.rax = SVSM_VTPM_CALL(SVSM_VTPM_CMD);
call.rcx = __pa(buffer);
@ -345,7 +352,6 @@ bool snp_svsm_vtpm_probe(void)
if (!snp_vmpl)
return false;
call.caa = svsm_get_caa();
call.rax = SVSM_VTPM_CALL(SVSM_VTPM_QUERY);
if (svsm_perform_call_protocol(&call))

2
arch/x86/crypto/aesni-intel_glue.c
View File

@ -1293,6 +1293,7 @@ static void gcm_process_assoc(const struct aes_gcm_key *key, u8 ghash_acc[16],
unsigned int len;
const u8 *src = walk.addr;
assoclen -= orig_len_this_step;
if (unlikely(pos)) {
len = min(len_this_step, 16 - pos);
memcpy(&buf[pos], src, len);
@ -1320,7 +1321,6 @@ static void gcm_process_assoc(const struct aes_gcm_key *key, u8 ghash_acc[16],
kernel_fpu_end();
kernel_fpu_begin();
}
assoclen -= orig_len_this_step;
}
if (unlikely(pos))
aes_gcm_aad_update(key, ghash_acc, buf, pos, flags);

2
arch/x86/entry/vdso/vdso64/Makefile
View File

@ -14,7 +14,7 @@ vobjs-$(CONFIG_X86_SGX) += vsgx.o
vobjs-$(CONFIG_FUTEX_ROBUST_UNLOCK) += vfutex.o
# Compilation flags
flags-y := -DBUILD_VDSO64 -m64 -mcmodel=small
flags-y := -DBUILD_VDSO64 -m64 -mcmodel=small $(CONFIG_CC_OPT_INLINE_MEMSET)
# The location of this include matters!
include $(src)/../common/Makefile.include

9
arch/x86/events/amd/brs.c
View File

@ -343,11 +343,10 @@ void amd_brs_drain(void)
if (!amd_brs_match_plm(event, from, to))
continue;
perf_clear_branch_entry_bitfields(br+nr);
br[nr].from = from;
br[nr].to = to;
br[nr] = (struct perf_branch_entry){
.from = from,
.to = to,
};
nr++;
}
empty:

16
arch/x86/events/amd/lbr.c
View File

@ -184,13 +184,6 @@ void amd_pmu_lbr_read(void)
entry.to.split.reserved)
continue;
perf_clear_branch_entry_bitfields(br + out);
br[out].from = sign_ext_branch_ip(entry.from.split.ip);
br[out].to = sign_ext_branch_ip(entry.to.split.ip);
br[out].mispred = entry.from.split.mispredict;
br[out].predicted = !br[out].mispred;
/*
* Set branch speculation information using the status of
* the valid and spec bits.
@ -208,7 +201,14 @@ void amd_pmu_lbr_read(void)
* speculative and took the correct path
*/
idx = (entry.to.split.valid << 1) | entry.to.split.spec;
br[out].spec = lbr_spec_map[idx];
br[out] = (struct perf_branch_entry){
.from = sign_ext_branch_ip(entry.from.split.ip),
.to = sign_ext_branch_ip(entry.to.split.ip),
.mispred = entry.from.split.mispredict,
.predicted = !entry.from.split.mispredict,
.spec = lbr_spec_map[idx],
};
out++;
}

131
arch/x86/events/intel/core.c
View File

@ -506,6 +506,8 @@ static struct event_constraint intel_pnc_event_constraints[] = {
INTEL_EVENT_CONSTRAINT(0xce, 0x1),
INTEL_UEVENT_CONSTRAINT(0x01b1, 0x8),
INTEL_UEVENT_CONSTRAINT(0x01b2, 0xf),
INTEL_UEVENT_CONSTRAINT(0x02b2, 0xf),
INTEL_UEVENT_CONSTRAINT(0x0847, 0xf),
INTEL_UEVENT_CONSTRAINT(0x0446, 0xf),
INTEL_UEVENT_CONSTRAINT(0x0846, 0xf),
@ -520,6 +522,14 @@ static struct extra_reg intel_pnc_extra_regs[] __read_mostly = {
INTEL_UEVENT_EXTRA_REG(0x022a, MSR_OMR_1, 0x40ffffff0000ffffull, OMR_1),
INTEL_UEVENT_EXTRA_REG(0x042a, MSR_OMR_2, 0x40ffffff0000ffffull, OMR_2),
INTEL_UEVENT_EXTRA_REG(0x082a, MSR_OMR_3, 0x40ffffff0000ffffull, OMR_3),
INTEL_UEVENT_EXTRA_REG(0x014f, MSR_OMR_0, 0x40ffffff0000ffffull, OMR_0),
INTEL_UEVENT_EXTRA_REG(0x024f, MSR_OMR_1, 0x40ffffff0000ffffull, OMR_1),
INTEL_UEVENT_EXTRA_REG(0x044f, MSR_OMR_2, 0x40ffffff0000ffffull, OMR_2),
INTEL_UEVENT_EXTRA_REG(0x084f, MSR_OMR_3, 0x40ffffff0000ffffull, OMR_3),
INTEL_UEVENT_EXTRA_REG(0x01d6, MSR_OMR_0, 0x40ffffff0000ffffull, OMR_0),
INTEL_UEVENT_EXTRA_REG(0x02d6, MSR_OMR_1, 0x40ffffff0000ffffull, OMR_1),
INTEL_UEVENT_EXTRA_REG(0x04d6, MSR_OMR_2, 0x40ffffff0000ffffull, OMR_2),
INTEL_UEVENT_EXTRA_REG(0x08d6, MSR_OMR_3, 0x40ffffff0000ffffull, OMR_3),
INTEL_UEVENT_PEBS_LDLAT_EXTRA_REG(0x01cd),
INTEL_UEVENT_EXTRA_REG(0x02c6, MSR_PEBS_FRONTEND, 0x9, FE),
INTEL_UEVENT_EXTRA_REG(0x03c6, MSR_PEBS_FRONTEND, 0x7fff1f, FE),
@ -5292,12 +5302,15 @@ static struct perf_guest_switch_msr *intel_guest_get_msrs(int *nr, void *data)
struct kvm_pmu *kvm_pmu = (struct kvm_pmu *)data;
u64 intel_ctrl = hybrid(cpuc->pmu, intel_ctrl);
u64 pebs_mask = cpuc->pebs_enabled & x86_pmu.pebs_capable;
int global_ctrl, pebs_enable;
u64 guest_pebs_mask;
int global_ctrl;
/*
* In addition to obeying exclude_guest/exclude_host, remove bits being
* used for PEBS when running a guest, because PEBS writes to virtual
* addresses (not physical addresses).
* addresses (not physical addresses). If the guest wants to utilize
* PEBS, and PEBS can be safely enabled in the guest, bits for the guest's
* PEBS-enabled counters will be OR'd back in as appropriate.
*/
*nr = 0;
global_ctrl = (*nr)++;
@ -5327,41 +5340,68 @@ static struct perf_guest_switch_msr *intel_guest_get_msrs(int *nr, void *data)
return arr;
}
if (!kvm_pmu || !x86_pmu.pebs_ept)
/*
* If the CPU doesn't support PEBS in the guest, then there's nothing
* more to do as disabling PMCs via PERF_GLOBAL_CTRL is sufficient on
* CPUs with guest/host isolation.
*/
if (!x86_pmu.pebs_ept)
return arr;
/*
* Restrict guest PEBS events to counters that (a) perf supports, (b)
* the guest wants to use for PEBS, (c) are not excluded from counting
* in the guest, and (d) _are_ excluded from counting in the host.
*/
guest_pebs_mask = pebs_mask & intel_ctrl & kvm_pmu->pebs_enable &
~cpuc->intel_ctrl_host_mask &
cpuc->intel_ctrl_guest_mask;
/*
* Disable counters where the guest PMC is different than the host PMC
* being used on behalf of the guest, as the PEBS record includes
* PERF_GLOBAL_STATUS, i.e. the guest will see overflow status for the
* wrong counter(s).
*/
guest_pebs_mask &= ~kvm_pmu->host_cross_mapped_mask;
/*
* FIXME: Allow guest and host usage of PEBS events to co-exist instead
* of disabling guest PEBS entirely if the host is using PEBS.
* What exactly goes wrong if guest and host are using PEBS is
* unknown.
*/
if (pebs_mask & ~cpuc->intel_ctrl_guest_mask)
guest_pebs_mask = 0;
/*
* Context switch DS_AREA and PEBS_DATA_CFG if and only if PEBS will be
* active in the guest; if no records will be generated while the guest
* is running, then simply keep the host values resident in hardware.
*/
arr[(*nr)++] = (struct perf_guest_switch_msr){
.msr = MSR_IA32_DS_AREA,
.host = (unsigned long)cpuc->ds,
.guest = kvm_pmu->ds_area,
.guest = guest_pebs_mask ? kvm_pmu->ds_area : (unsigned long)cpuc->ds,
};
if (x86_pmu.intel_cap.pebs_baseline) {
arr[(*nr)++] = (struct perf_guest_switch_msr){
.msr = MSR_PEBS_DATA_CFG,
.host = cpuc->active_pebs_data_cfg,
.guest = kvm_pmu->pebs_data_cfg,
.guest = guest_pebs_mask ? kvm_pmu->pebs_data_cfg :
cpuc->active_pebs_data_cfg,
};
}
pebs_enable = (*nr)++;
arr[pebs_enable] = (struct perf_guest_switch_msr){
.msr = MSR_IA32_PEBS_ENABLE,
.host = cpuc->pebs_enabled & ~cpuc->intel_ctrl_guest_mask,
.guest = pebs_mask & ~cpuc->intel_ctrl_host_mask & kvm_pmu->pebs_enable,
};
if (arr[pebs_enable].host) {
/* Disable guest PEBS if host PEBS is enabled. */
arr[pebs_enable].guest = 0;
} else {
/* Disable guest PEBS thoroughly for cross-mapped PEBS counters. */
arr[pebs_enable].guest &= ~kvm_pmu->host_cross_mapped_mask;
arr[global_ctrl].guest &= ~kvm_pmu->host_cross_mapped_mask;
/* Set hw GLOBAL_CTRL bits for PEBS counter when it runs for guest */
arr[global_ctrl].guest |= arr[pebs_enable].guest;
}
/*
* Do NOT mess with PEBS_ENABLED. As above, disabling counters via
* PERF_GLOBAL_CTRL is sufficient, and loading a stale PEBS_ENABLED,
* e.g. on VM-Exit, can put the system in a bad state. Simply enable
* counters in PERF_GLOBAL_CTRL, as perf load PEBS_ENABLED with the
* full value, i.e. perf *also* relies on PERF_GLOBAL_CTRL.
*/
arr[global_ctrl].guest |= guest_pebs_mask;
return arr;
}
@ -6561,6 +6601,8 @@ static void intel_pmu_filter(struct pmu *pmu, int cpu, bool *ret)
PMU_FORMAT_ATTR(offcore_rsp, "config1:0-63");
PMU_FORMAT_ATTR(offmodule_rsp, "config1:0-63");
PMU_FORMAT_ATTR(ldlat, "config1:0-15");
PMU_FORMAT_ATTR(frontend, "config1:0-23");
@ -6609,6 +6651,20 @@ static struct attribute *skl_format_attr[] = {
NULL,
};
static struct attribute *pnc_format_attr_rtm[] = {
&format_attr_in_tx.attr,
&format_attr_in_tx_cp.attr,
&format_attr_offmodule_rsp.attr,
&format_attr_ldlat.attr,
NULL
};
static struct attribute *pnc_format_attr[] = {
&format_attr_offmodule_rsp.attr,
&format_attr_ldlat.attr,
NULL
};
static __initconst const struct x86_pmu core_pmu = {
.name = "core",
.handle_irq = x86_pmu_handle_irq,
@ -7474,6 +7530,7 @@ static struct attribute *adl_hybrid_tsx_attrs[] = {
FORMAT_ATTR_HYBRID(in_tx, hybrid_big);
FORMAT_ATTR_HYBRID(in_tx_cp, hybrid_big);
FORMAT_ATTR_HYBRID(offcore_rsp, hybrid_big_small_tiny);
FORMAT_ATTR_HYBRID(offmodule_rsp, hybrid_big_small_tiny);
FORMAT_ATTR_HYBRID(ldlat, hybrid_big_small_tiny);
FORMAT_ATTR_HYBRID(frontend, hybrid_big);
@ -7512,6 +7569,23 @@ static struct attribute *mtl_hybrid_extra_attr[] = {
NULL
};
static struct attribute *nvl_hybrid_extra_attr_rtm[] = {
ADL_HYBRID_RTM_FORMAT_ATTR,
FORMAT_HYBRID_PTR(offmodule_rsp),
FORMAT_HYBRID_PTR(ldlat),
FORMAT_HYBRID_PTR(frontend),
FORMAT_HYBRID_PTR(snoop_rsp),
NULL
};
static struct attribute *nvl_hybrid_extra_attr[] = {
FORMAT_HYBRID_PTR(offmodule_rsp),
FORMAT_HYBRID_PTR(ldlat),
FORMAT_HYBRID_PTR(frontend),
FORMAT_HYBRID_PTR(snoop_rsp),
NULL
};
static bool is_attr_for_this_pmu(struct kobject *kobj, struct attribute *attr)
{
struct device *dev = kobj_to_dev(kobj);
@ -8549,6 +8623,8 @@ __init int intel_pmu_init(void)
case INTEL_DIAMONDRAPIDS_X:
intel_pmu_init_pnc(NULL);
x86_pmu.pebs_latency_data = pnc_latency_data;
extra_attr = boot_cpu_has(X86_FEATURE_RTM) ?
pnc_format_attr_rtm : pnc_format_attr;
pr_cont("Panthercove events, ");
name = "panthercove";
@ -8557,13 +8633,12 @@ __init int intel_pmu_init(void)
glc_common:
intel_pmu_init_glc(NULL);
intel_pmu_pebs_data_source_skl(true);
extra_attr = boot_cpu_has(X86_FEATURE_RTM) ?
hsw_format_attr : nhm_format_attr;
glc_base:
x86_pmu.pebs_ept = 1;
x86_pmu.hw_config = hsw_hw_config;
x86_pmu.get_event_constraints = glc_get_event_constraints;
extra_attr = boot_cpu_has(X86_FEATURE_RTM) ?
hsw_format_attr : nhm_format_attr;
extra_skl_attr = skl_format_attr;
mem_attr = glc_events_attrs;
td_attr = glc_td_events_attrs;
@ -8785,7 +8860,7 @@ __init int intel_pmu_init(void)
mem_attr = mtl_hybrid_mem_attrs;
tsx_attr = adl_hybrid_tsx_attrs;
extra_attr = boot_cpu_has(X86_FEATURE_RTM) ?
mtl_hybrid_extra_attr_rtm : mtl_hybrid_extra_attr;
nvl_hybrid_extra_attr_rtm : nvl_hybrid_extra_attr;
/* Initialize big core specific PerfMon capabilities.*/
pmu = &x86_pmu.hybrid_pmu[X86_HYBRID_PMU_CORE_IDX];
@ -8794,8 +8869,6 @@ __init int intel_pmu_init(void)
/* Initialize Atom core specific PerfMon capabilities.*/
pmu = &x86_pmu.hybrid_pmu[X86_HYBRID_PMU_ATOM_IDX];
intel_pmu_init_arw(&pmu->pmu);
intel_pmu_pebs_data_source_lnl();
break;
default:

74
arch/x86/events/intel/ds.c
View File

@ -277,8 +277,8 @@ static u64 pnc_pebs_l2_hit_data_source[PNC_PEBS_DATA_SOURCE_MAX] = {
0, /* 0x06: Reserved */
OP_LH | P(LVL, L2) | LEVEL(L2) | P(SNOOP, HIT), /* 0x07: L2 Hit Snoop HIT */
OP_LH | P(LVL, L2) | LEVEL(L2) | P(SNOOP, HITM), /* 0x08: L2 Hit Snoop Hit Modified */
OP_LH | P(LVL, L2) | LEVEL(L2) | P(SNOOP, MISS), /* 0x09: Prefetch Promotion */
OP_LH | P(LVL, L2) | LEVEL(L2) | P(SNOOP, MISS), /* 0x0a: Cross Core Prefetch Promotion */
OP_LH | P(LVL, L2) | LEVEL(L2) | P(SNOOP, NONE), /* 0x09: Prefetch Promotion */
OP_LH | P(LVL, L2) | LEVEL(L2) | P(SNOOP, NONE), /* 0x0a: Cross Core Prefetch Promotion */
0, /* 0x0b: Reserved */
0, /* 0x0c: Reserved */
0, /* 0x0d: Reserved */
@ -455,6 +455,7 @@ static inline void pebs_set_tlb_lock(u64 *val, bool tlb, bool lock)
static u64 __grt_latency_data(struct perf_event *event, u64 status,
u8 dse, bool tlb, bool lock, bool blk)
{
union perf_mem_data_src src;
u64 val;
WARN_ON_ONCE(is_hybrid() &&
@ -470,7 +471,16 @@ static u64 __grt_latency_data(struct perf_event *event, u64 status,
else
val |= P(BLK, NA);
return val;
src.val = val;
if (event->hw.flags &
(PERF_X86_EVENT_PEBS_LDLAT | PERF_X86_EVENT_PEBS_LD_HSW))
src.mem_op = P(OP, LOAD);
if (event->hw.flags &
(PERF_X86_EVENT_PEBS_STLAT | PERF_X86_EVENT_PEBS_ST_HSW))
src.mem_op = P(OP, STORE);
return src.val;
}
u64 grt_latency_data(struct perf_event *event, u64 status)
@ -528,7 +538,11 @@ static u64 arw_latency_data(struct perf_event *event, u64 status)
val |= P(BLK, NA);
src.val = val;
if (event->hw.flags & PERF_X86_EVENT_PEBS_ST_HSW)
if (event->hw.flags &
(PERF_X86_EVENT_PEBS_LDLAT | PERF_X86_EVENT_PEBS_LD_HSW))
src.mem_op = P(OP, LOAD);
if (event->hw.flags &
(PERF_X86_EVENT_PEBS_STLAT | PERF_X86_EVENT_PEBS_ST_HSW))
src.mem_op = P(OP, STORE);
return src.val;
@ -563,7 +577,11 @@ static u64 lnc_latency_data(struct perf_event *event, u64 status)
val |= P(BLK, NA);
src.val = val;
if (event->hw.flags & PERF_X86_EVENT_PEBS_ST_HSW)
if (event->hw.flags &
(PERF_X86_EVENT_PEBS_LDLAT | PERF_X86_EVENT_PEBS_LD_HSW))
src.mem_op = P(OP, LOAD);
if (event->hw.flags &
(PERF_X86_EVENT_PEBS_STLAT | PERF_X86_EVENT_PEBS_ST_HSW))
src.mem_op = P(OP, STORE);
return src.val;
@ -621,7 +639,11 @@ u64 pnc_latency_data(struct perf_event *event, u64 status)
val |= P(BLK, NA);
src.val = val;
if (event->hw.flags & PERF_X86_EVENT_PEBS_ST_HSW)
if (event->hw.flags &
(PERF_X86_EVENT_PEBS_LDLAT | PERF_X86_EVENT_PEBS_LD_HSW))
src.mem_op = P(OP, LOAD);
if (event->hw.flags &
(PERF_X86_EVENT_PEBS_STLAT | PERF_X86_EVENT_PEBS_ST_HSW))
src.mem_op = P(OP, STORE);
return src.val;
@ -1291,22 +1313,22 @@ struct event_constraint intel_glm_pebs_event_constraints[] = {
struct event_constraint intel_grt_pebs_event_constraints[] = {
/* Allow all events as PEBS with no flags */
INTEL_HYBRID_LAT_CONSTRAINT(0x5d0, 0x3),
INTEL_HYBRID_LAT_CONSTRAINT(0x6d0, 0x3f),
INTEL_HYBRID_LDLAT_CONSTRAINT(0x5d0, 0x3),
INTEL_HYBRID_STLAT_CONSTRAINT(0x6d0, 0x3f),
EVENT_CONSTRAINT_END
};
struct event_constraint intel_cmt_pebs_event_constraints[] = {
/* Allow all events as PEBS with no flags */
INTEL_HYBRID_LAT_CONSTRAINT(0x5d0, 0x3),
INTEL_HYBRID_LAT_CONSTRAINT(0x6d0, 0xff),
INTEL_HYBRID_LDLAT_CONSTRAINT(0x5d0, 0x3),
INTEL_HYBRID_STLAT_CONSTRAINT(0x6d0, 0xff),
EVENT_CONSTRAINT_END
};
struct event_constraint intel_dkt_pebs_event_constraints[] = {
/* Allow all events as PEBS with no flags */
INTEL_HYBRID_LAT_CONSTRAINT(0x5d0, 0xff),
INTEL_HYBRID_LAT_CONSTRAINT(0x6d0, 0xff),
INTEL_HYBRID_LDLAT_CONSTRAINT(0x5d0, 0xff),
INTEL_HYBRID_STLAT_CONSTRAINT(0x6d0, 0xff),
EVENT_CONSTRAINT_END
};
@ -1496,24 +1518,8 @@ struct event_constraint intel_lnc_pebs_event_constraints[] = {
INTEL_FLAGS_UEVENT_CONSTRAINT(0x012a, 0x1), /* OCR.* events */
INTEL_FLAGS_UEVENT_CONSTRAINT(0x012b, 0x1), /* OCR.* events */
INTEL_FLAGS_UEVENT_CONSTRAINT(0x04a4, 0x1), /* TOPDOWN.BAD_SPEC_SLOTS */
INTEL_FLAGS_UEVENT_CONSTRAINT(0x08a4, 0x1), /* TOPDOWN.BR_MISPREDICT_SLOTS */
INTEL_FLAGS_UEVENT_CONSTRAINT(0x10a4, 0x8), /* TOPDOWN.MEMORY_BOUND_SLOTS */
INTEL_HYBRID_LDLAT_CONSTRAINT(0x1cd, 0x3fc),
INTEL_HYBRID_STLAT_CONSTRAINT(0x2cd, 0x3),
INTEL_FLAGS_UEVENT_CONSTRAINT_DATALA_LD(0x11d0, 0xf), /* MEM_INST_RETIRED.STLB_MISS_LOADS */
INTEL_FLAGS_UEVENT_CONSTRAINT_DATALA_ST(0x12d0, 0xf), /* MEM_INST_RETIRED.STLB_MISS_STORES */
INTEL_FLAGS_UEVENT_CONSTRAINT_DATALA_LD(0x21d0, 0xf), /* MEM_INST_RETIRED.LOCK_LOADS */
INTEL_FLAGS_UEVENT_CONSTRAINT_DATALA_LD(0x41d0, 0xf), /* MEM_INST_RETIRED.SPLIT_LOADS */
INTEL_FLAGS_UEVENT_CONSTRAINT_DATALA_ST(0x42d0, 0xf), /* MEM_INST_RETIRED.SPLIT_STORES */
INTEL_FLAGS_UEVENT_CONSTRAINT_DATALA_LD(0x81d0, 0xf), /* MEM_INST_RETIRED.ALL_LOADS */
INTEL_FLAGS_UEVENT_CONSTRAINT_DATALA_ST(0x82d0, 0xf), /* MEM_INST_RETIRED.ALL_STORES */
INTEL_FLAGS_UEVENT_CONSTRAINT(0x87d0, 0x3ff), /* MEM_INST_RETIRED.ANY */
INTEL_FLAGS_EVENT_CONSTRAINT_DATALA_LD_RANGE(0xd1, 0xd4, 0xf),
INTEL_FLAGS_EVENT_CONSTRAINT(0xd0, 0xf),
/*
* Everything else is handled by PMU_FL_PEBS_ALL, because we
@ -1526,18 +1532,6 @@ struct event_constraint intel_lnc_pebs_event_constraints[] = {
struct event_constraint intel_pnc_pebs_event_constraints[] = {
INTEL_HYBRID_LDLAT_CONSTRAINT(0x1cd, 0xfc),
INTEL_HYBRID_STLAT_CONSTRAINT(0x2cd, 0x3),
INTEL_FLAGS_UEVENT_CONSTRAINT_DATALA_LD(0x11d0, 0xf), /* MEM_INST_RETIRED.STLB_MISS_LOADS */
INTEL_FLAGS_UEVENT_CONSTRAINT_DATALA_ST(0x12d0, 0xf), /* MEM_INST_RETIRED.STLB_MISS_STORES */
INTEL_FLAGS_UEVENT_CONSTRAINT_DATALA_LD(0x21d0, 0xf), /* MEM_INST_RETIRED.LOCK_LOADS */
INTEL_FLAGS_UEVENT_CONSTRAINT_DATALA_LD(0x41d0, 0xf), /* MEM_INST_RETIRED.SPLIT_LOADS */
INTEL_FLAGS_UEVENT_CONSTRAINT_DATALA_ST(0x42d0, 0xf), /* MEM_INST_RETIRED.SPLIT_STORES */
INTEL_FLAGS_UEVENT_CONSTRAINT_DATALA_LD(0x81d0, 0xf), /* MEM_INST_RETIRED.ALL_LOADS */
INTEL_FLAGS_UEVENT_CONSTRAINT_DATALA_ST(0x82d0, 0xf), /* MEM_INST_RETIRED.ALL_STORES */
INTEL_FLAGS_EVENT_CONSTRAINT_DATALA_LD_RANGE(0xd1, 0xd4, 0xf),
INTEL_FLAGS_EVENT_CONSTRAINT(0xd0, 0xf),
INTEL_FLAGS_EVENT_CONSTRAINT(0xd6, 0xf),
/*
* Everything else is handled by PMU_FL_PEBS_ALL, because we

63
arch/x86/events/intel/lbr.c
View File

@ -756,10 +756,10 @@ void intel_pmu_lbr_read_32(struct cpu_hw_events *cpuc)
rdmsrq(x86_pmu.lbr_from + lbr_idx, msr_lastbranch.lbr);
perf_clear_branch_entry_bitfields(br);
br->from = msr_lastbranch.from;
br->to = msr_lastbranch.to;
*br = (struct perf_branch_entry){
.from = msr_lastbranch.from,
.to = msr_lastbranch.to,
};
br++;
}
cpuc->lbr_stack.nr = i;
@ -847,14 +847,15 @@ void intel_pmu_lbr_read_64(struct cpu_hw_events *cpuc)
if (abort && x86_pmu.lbr_double_abort && out > 0)
out--;
perf_clear_branch_entry_bitfields(br+out);
br[out].from = from;
br[out].to = to;
br[out].mispred = mis;
br[out].predicted = pred;
br[out].in_tx = in_tx;
br[out].abort = abort;
br[out].cycles = cycles;
br[out] = (struct perf_branch_entry){
.from = from,
.to = to,
.mispred = mis,
.predicted = pred,
.in_tx = in_tx,
.abort = abort,
.cycles = cycles,
};
out++;
}
cpuc->lbr_stack.nr = out;
@ -905,6 +906,7 @@ static void intel_pmu_store_lbr(struct cpu_hw_events *cpuc,
struct perf_branch_entry *e;
struct lbr_entry *lbr;
u64 from, to, info;
bool mispred;
int i;
for (i = 0; i < x86_pmu.lbr_nr; i++) {
@ -921,24 +923,27 @@ static void intel_pmu_store_lbr(struct cpu_hw_events *cpuc,
to = rdlbr_to(i, lbr);
info = rdlbr_info(i, lbr);
perf_clear_branch_entry_bitfields(e);
mispred = get_lbr_mispred(info);
e->from = from;
e->to = to;
e->mispred = get_lbr_mispred(info);
e->predicted = !e->mispred;
e->in_tx = !!(info & LBR_INFO_IN_TX);
e->abort = !!(info & LBR_INFO_ABORT);
e->cycles = get_lbr_cycles(info);
e->type = get_lbr_br_type(info);
/*
* Leverage the reserved field of cpuc->lbr_entries[i] to
* temporarily store the branch counters information.
* The later code will decide what content can be disclosed
* to the perf tool. Pleae see intel_pmu_lbr_counters_reorder().
*/
e->reserved = (info >> LBR_INFO_BR_CNTR_OFFSET) & LBR_INFO_BR_CNTR_FULL_MASK;
*e = (struct perf_branch_entry){
.from = from,
.to = to,
.mispred = mispred,
.predicted = !mispred,
.in_tx = !!(info & LBR_INFO_IN_TX),
.abort = !!(info & LBR_INFO_ABORT),
.cycles = get_lbr_cycles(info),
.type = get_lbr_br_type(info),
/*
* Leverage the reserved field of
* cpuc->lbr_entries[i] to temporarily store the
* branch counters information. The later code will
* decide what content can be disclosed to the perf
* tool. Pleae see intel_pmu_lbr_counters_reorder().
*/
.reserved = (info >> LBR_INFO_BR_CNTR_OFFSET) &
LBR_INFO_BR_CNTR_FULL_MASK,
};
}
cpuc->lbr_stack.nr = i;

4
arch/x86/events/perf_event.h
View File

@ -517,10 +517,6 @@ struct cpu_hw_events {
__EVENT_CONSTRAINT(c, n, INTEL_ARCH_EVENT_MASK|X86_ALL_EVENT_FLAGS, \
HWEIGHT(n), 0, PERF_X86_EVENT_PEBS_ST)
#define INTEL_HYBRID_LAT_CONSTRAINT(c, n) \
__EVENT_CONSTRAINT(c, n, INTEL_ARCH_EVENT_MASK|X86_ALL_EVENT_FLAGS, \
HWEIGHT(n), 0, PERF_X86_EVENT_PEBS_LAT_HYBRID)
#define INTEL_HYBRID_LDLAT_CONSTRAINT(c, n) \
__EVENT_CONSTRAINT(c, n, INTEL_ARCH_EVENT_MASK|X86_ALL_EVENT_FLAGS, \
HWEIGHT(n), 0, PERF_X86_EVENT_PEBS_LAT_HYBRID|PERF_X86_EVENT_PEBS_LD_HSW)

27
arch/x86/kernel/cpu/mce/core.c
View File

@ -2108,6 +2108,9 @@ bool filter_mce(struct mce *m)
static __always_inline void exc_machine_check_kernel(struct pt_regs *regs)
{
irqentry_state_t irq_state;
unsigned long dr7;
dr7 = local_db_save();
WARN_ON_ONCE(user_mode(regs));
@ -2116,20 +2119,26 @@ static __always_inline void exc_machine_check_kernel(struct pt_regs *regs)
* mce_check_crashing_cpu() for details.
*/
if (mca_cfg.initialized && mce_check_crashing_cpu())
return;
goto out;
irq_state = irqentry_nmi_enter(regs);
do_machine_check(regs);
irqentry_nmi_exit(regs, irq_state);
out:
local_db_restore(dr7);
}
static __always_inline void exc_machine_check_user(struct pt_regs *regs)
{
unsigned long dr7;
irqentry_enter_from_user_mode(regs);
dr7 = local_db_save();
do_machine_check(regs);
local_db_restore(dr7);
irqentry_exit_to_user_mode(regs);
}
@ -2138,21 +2147,13 @@ static __always_inline void exc_machine_check_user(struct pt_regs *regs)
/* MCE hit kernel mode */
DEFINE_IDTENTRY_MCE(exc_machine_check)
{
unsigned long dr7;
dr7 = local_db_save();
exc_machine_check_kernel(regs);
local_db_restore(dr7);
}
/* The user mode variant. */
DEFINE_IDTENTRY_MCE_USER(exc_machine_check)
{
unsigned long dr7;
dr7 = local_db_save();
exc_machine_check_user(regs);
local_db_restore(dr7);
}
#ifdef CONFIG_X86_FRED
@ -2169,28 +2170,20 @@ DEFINE_IDTENTRY_MCE_USER(exc_machine_check)
*/
DEFINE_FREDENTRY_MCE(exc_machine_check)
{
unsigned long dr7;
dr7 = local_db_save();
if (user_mode(regs))
exc_machine_check_user(regs);
else
exc_machine_check_kernel(regs);
local_db_restore(dr7);
}
#endif
#else
/* 32bit unified entry point */
DEFINE_IDTENTRY_RAW(exc_machine_check)
{
unsigned long dr7;
dr7 = local_db_save();
if (user_mode(regs))
exc_machine_check_user(regs);
else
exc_machine_check_kernel(regs);
local_db_restore(dr7);
}
#endif

8
arch/x86/kvm/mmu/mmu.c
View File

@ -2512,6 +2512,12 @@ static void shadow_walk_init_using_root(struct kvm_shadow_walk_iterator *iterato
iterator->addr = addr;
iterator->shadow_addr = root;
if (WARN_ON_ONCE(!VALID_PAGE(root)) || kvm_mmu_is_dummy_root(root)) {
iterator->level = 0;
return;
}
iterator->level = vcpu->arch.mmu->root_role.level;
if (iterator->level >= PT64_ROOT_4LEVEL &&
@ -5058,7 +5064,7 @@ static int kvm_tdp_page_prefault(struct kvm_vcpu *vcpu, gpa_t gpa,
if (signal_pending(current))
return -EINTR;
if (kvm_check_request(KVM_REQ_VM_DEAD, vcpu))
if (kvm_test_request(KVM_REQ_VM_DEAD, vcpu))
return -EIO;
cond_resched();

8
arch/x86/kvm/pmu.c
View File

@ -823,14 +823,6 @@ void kvm_pmu_deliver_pmi(struct kvm_vcpu *vcpu)
bool kvm_pmu_is_valid_msr(struct kvm_vcpu *vcpu, u32 msr)
{
switch (msr) {
case MSR_CORE_PERF_GLOBAL_STATUS:
case MSR_CORE_PERF_GLOBAL_CTRL:
case MSR_CORE_PERF_GLOBAL_OVF_CTRL:
return kvm_pmu_has_perf_global_ctrl(vcpu_to_pmu(vcpu));
default:
break;
}
return kvm_pmu_call(msr_idx_to_pmc)(vcpu, msr) ||
kvm_pmu_call(is_valid_msr)(vcpu, msr);
}

7
arch/x86/kvm/svm/nested.c
View File

@ -2125,13 +2125,8 @@ static bool svm_get_nested_state_pages(struct kvm_vcpu *vcpu)
return false;
}
if (!nested_svm_merge_msrpm(vcpu)) {
vcpu->run->exit_reason = KVM_EXIT_INTERNAL_ERROR;
vcpu->run->internal.suberror =
KVM_INTERNAL_ERROR_EMULATION;
vcpu->run->internal.ndata = 0;
if (!nested_svm_merge_msrpm(vcpu))
return false;
}
if (kvm_hv_verify_vp_assist(vcpu))
return false;

61
arch/x86/kvm/svm/sev.c
View File

@ -488,6 +488,20 @@ static void snp_guest_req_cleanup(struct kvm *kvm)
sev->guest_resp_buf = NULL;
}
static int sev_alloc_have_run_cpus(struct kvm_sev_info *sev)
{
if (!zalloc_cpumask_var(&sev->have_run_cpus, GFP_KERNEL_ACCOUNT))
return -ENOMEM;
return 0;
}
static void sev_free_have_run_cpus(struct kvm_sev_info *sev)
{
free_cpumask_var(sev->have_run_cpus);
memset(&sev->have_run_cpus, 0, sizeof(sev->have_run_cpus));
}
static int __sev_guest_init(struct kvm *kvm, struct kvm_sev_cmd *argp,
struct kvm_sev_init *data,
unsigned long vm_type)
@ -545,10 +559,9 @@ static int __sev_guest_init(struct kvm *kvm, struct kvm_sev_cmd *argp,
if (ret)
goto e_free_asid;
if (!zalloc_cpumask_var(&sev->have_run_cpus, GFP_KERNEL_ACCOUNT)) {
ret = -ENOMEM;
ret = sev_alloc_have_run_cpus(sev);
if (ret)
goto e_free_asid;
}
/* This needs to happen after SEV/SNP firmware initialization. */
if (snp_active) {
@ -566,7 +579,7 @@ static int __sev_guest_init(struct kvm *kvm, struct kvm_sev_cmd *argp,
return 0;
e_free:
free_cpumask_var(sev->have_run_cpus);
sev_free_have_run_cpus(sev);
e_free_asid:
argp->error = init_args.error;
sev_asid_free(sev);
@ -1125,9 +1138,6 @@ static int sev_launch_update_vmsa(struct kvm *kvm, struct kvm_sev_cmd *argp)
if (!sev_es_guest(kvm))
return -ENOTTY;
if (kvm_is_vcpu_creation_in_progress(kvm))
return -EBUSY;
ret = kvm_lock_all_vcpus(kvm);
if (ret)
return ret;
@ -2035,6 +2045,13 @@ static void sev_migrate_from(struct kvm *dst_kvm, struct kvm *src_kvm)
struct kvm_sev_info *mirror;
unsigned long i;
/*
* Do cache maintenance on the source VM *before* clearing "SEV active",
* as memory reclaim flows won't trigger cache maintenance on the VM
* once it's no longer an SEV VM.
*/
sev_writeback_caches(src_kvm);
dst->active = true;
dst->asid = src->asid;
dst->handle = src->handle;
@ -2115,10 +2132,6 @@ static int sev_check_source_vcpus(struct kvm *dst, struct kvm *src)
struct kvm_vcpu *src_vcpu;
unsigned long i;
if (kvm_is_vcpu_creation_in_progress(src) ||
kvm_is_vcpu_creation_in_progress(dst))
return -EBUSY;
if (!sev_es_guest(src))
return 0;
@ -2187,11 +2200,14 @@ int sev_vm_move_enc_context_from(struct kvm *kvm, unsigned int source_fd)
* the set of CPUs from the source. If a CPU was used to run a vCPU in
* the source VM but is never used for the destination VM, then the CPU
* can only have cached memory that was accessible to the source VM.
* Furthermore, KVM *must* perform cache maintenance on the source VM,
* as the source VM may have access to memory that the destination VM
* does not, i.e. KVM could skip flushes if memory is reclaimed from
* the old VM but not the new VM.
*/
if (!zalloc_cpumask_var(&dst_sev->have_run_cpus, GFP_KERNEL_ACCOUNT)) {
ret = -ENOMEM;
ret = sev_alloc_have_run_cpus(dst_sev);
if (ret)
goto out_source_vcpu;
}
sev_migrate_from(kvm, source_kvm);
kvm_vm_dead(source_kvm);
@ -2510,9 +2526,6 @@ static int snp_launch_update_vmsa(struct kvm *kvm, struct kvm_sev_cmd *argp)
unsigned long i;
int ret;
if (kvm_is_vcpu_creation_in_progress(kvm))
return -EBUSY;
ret = kvm_lock_all_vcpus(kvm);
if (ret)
return ret;
@ -2888,10 +2901,9 @@ int sev_vm_copy_enc_context_from(struct kvm *kvm, unsigned int source_fd)
}
mirror_sev = to_kvm_sev_info(kvm);
if (!zalloc_cpumask_var(&mirror_sev->have_run_cpus, GFP_KERNEL_ACCOUNT)) {
ret = -ENOMEM;
ret = sev_alloc_have_run_cpus(mirror_sev);
if (ret)
goto e_unlock;
}
/*
* The mirror kvm holds an enc_context_owner ref so its asid can't
@ -2980,13 +2992,17 @@ void sev_vm_destroy(struct kvm *kvm)
struct list_head *head = &sev->regions_list;
struct list_head *pos, *q;
/*
* Free the mask even if the VM is not *currently* an SEV VM, as it may
* have been an SEV VM prior to intra-host migration.
*/
sev_free_have_run_cpus(sev);
if (!sev_guest(kvm))
return;
WARN_ON(!list_empty(&sev->mirror_vms));
free_cpumask_var(sev->have_run_cpus);
/*
* If this is a mirror VM, remove it from the owner's list of a mirrors
* and skip ASID cleanup (the ASID is tied to the lifetime of the owner).
@ -3616,7 +3632,6 @@ int pre_sev_run(struct vcpu_svm *svm, int cpu)
sd->sev_vmcbs[asid] = svm->vmcb;
svm->vmcb->control.tlb_ctl = TLB_CONTROL_FLUSH_ASID;
vmcb_mark_dirty(svm->vmcb, VMCB_ASID);
return 0;
}

43
arch/x86/kvm/svm/svm.c
View File

@ -673,16 +673,7 @@ static void clr_dr_intercepts(struct vcpu_svm *svm)
static bool msr_write_intercepted(struct vcpu_svm *svm, u32 msr)
{
/*
* For non-nested case:
* If the L01 MSR bitmap does not intercept the MSR, then we need to
* save it.
*
* For nested case:
* If the L02 MSR bitmap does not intercept the MSR, then we need to
* save it.
*/
void *msrpm = is_guest_mode(&svm->vcpu) ? svm->nested.msrpm : svm->msrpm;
void *msrpm = __va(__sme_clr(svm->vmcb->control.msrpm_base_pa));
return svm_test_msr_bitmap_write(msrpm, msr);
}
@ -1902,8 +1893,7 @@ static void new_asid(struct vcpu_svm *svm, struct svm_cpu_data *sd)
if (sd->next_asid > sd->max_asid) {
++sd->asid_generation;
sd->next_asid = sd->min_asid;
svm->vmcb->control.tlb_ctl = TLB_CONTROL_FLUSH_ALL_ASID;
vmcb_mark_dirty(svm->vmcb, VMCB_ASID);
sd->flush_all_asids = true;
}
svm->current_vmcb->asid_generation = sd->asid_generation;
@ -4528,6 +4518,9 @@ static __no_kcsan fastpath_t svm_vcpu_run(struct kvm_vcpu *vcpu, u64 run_flags)
svm->vmcb->control.asid = svm->asid;
vmcb_mark_dirty(svm->vmcb, VMCB_ASID);
}
if (this_cpu_ptr(&svm_data)->flush_all_asids)
svm->vmcb->control.tlb_ctl = TLB_CONTROL_FLUSH_ALL_ASID;
svm->vmcb->save.cr2 = vcpu->arch.cr2;
if (guest_cpu_cap_has(vcpu, X86_FEATURE_ERAPS) &&
@ -4618,16 +4611,23 @@ static __no_kcsan fastpath_t svm_vcpu_run(struct kvm_vcpu *vcpu, u64 run_flags)
vcpu->arch.nested_run_pending = 0;
}
svm->vmcb->control.tlb_ctl = TLB_CONTROL_DO_NOTHING;
if (!svm_is_vmrun_failure(svm->vmcb->control.exit_code)) {
this_cpu_ptr(&svm_data)->flush_all_asids = false;
svm->vmcb->control.tlb_ctl = TLB_CONTROL_DO_NOTHING;
/*
* Unconditionally mask off the CLEAR_RAP bit, the AND is just as cheap
* as the TEST+Jcc to avoid it.
*/
if (cpu_feature_enabled(X86_FEATURE_ERAPS))
svm->vmcb->control.erap_ctl &= ~ERAP_CONTROL_CLEAR_RAP;
/*
* Unconditionally mask off the CLEAR_RAP bit, the AND is just
* as cheap as the TEST+Jcc to avoid it.
*/
if (cpu_feature_enabled(X86_FEATURE_ERAPS))
svm->vmcb->control.erap_ctl &= ~ERAP_CONTROL_CLEAR_RAP;
vmcb_mark_all_clean(svm->vmcb);
vmcb_mark_all_clean(svm->vmcb);
if (!msr_write_intercepted(svm, MSR_AMD64_PERF_CNTR_GLOBAL_CTL))
rdmsrq(MSR_AMD64_PERF_CNTR_GLOBAL_CTL,
vcpu_to_pmu(vcpu)->global_ctrl);
}
/* if exit due to PF check for async PF */
if (svm->vmcb->control.exit_code == SVM_EXIT_EXCP_BASE + PF_VECTOR)
@ -4636,9 +4636,6 @@ static __no_kcsan fastpath_t svm_vcpu_run(struct kvm_vcpu *vcpu, u64 run_flags)
kvm_clear_available_registers(vcpu, SVM_REGS_LAZY_LOAD_SET);
if (!msr_write_intercepted(svm, MSR_AMD64_PERF_CNTR_GLOBAL_CTL))
rdmsrq(MSR_AMD64_PERF_CNTR_GLOBAL_CTL, vcpu_to_pmu(vcpu)->global_ctrl);
trace_kvm_exit(vcpu, KVM_ISA_SVM);
svm_complete_interrupts(vcpu);

1
arch/x86/kvm/svm/svm.h
View File

@ -376,6 +376,7 @@ struct svm_cpu_data {
u32 next_asid;
u32 min_asid;
bool flush_all_asids;
bool bp_spec_reduce_set;
struct vmcb *save_area;

21
arch/x86/kvm/vmx/nested.c
View File

@ -1754,6 +1754,9 @@ static void copy_vmcs12_to_shadow(struct vcpu_vmx *vmx)
static void copy_enlightened_to_vmcs12(struct vcpu_vmx *vmx, u32 hv_clean_fields)
{
#ifdef CONFIG_KVM_HYPERV
const u64 runtime_controls = HV_VMX_ENLIGHTENED_CLEAN_FIELD_CONTROL_GRP1 |
HV_VMX_ENLIGHTENED_CLEAN_FIELD_CONTROL_GRP2 |
HV_VMX_ENLIGHTENED_CLEAN_FIELD_CONTROL_PROC;
struct vmcs12 *vmcs12 = vmx->nested.cached_vmcs12;
struct hv_enlightened_vmcs *evmcs = nested_vmx_evmcs(vmx);
struct kvm_vcpu_hv *hv_vcpu = to_hv_vcpu(&vmx->vcpu);
@ -1762,6 +1765,9 @@ static void copy_enlightened_to_vmcs12(struct vcpu_vmx *vmx, u32 hv_clean_fields
vmcs12->tpr_threshold = evmcs->tpr_threshold;
vmcs12->guest_rip = evmcs->guest_rip;
if ((hv_clean_fields & runtime_controls) != runtime_controls)
vmx->nested.force_msr_bitmap_recalc = true;
if (unlikely(!(hv_clean_fields &
HV_VMX_ENLIGHTENED_CLEAN_FIELD_ENLIGHTENMENTSCONTROL))) {
hv_vcpu->nested.pa_page_gpa = evmcs->partition_assist_page;
@ -3465,10 +3471,6 @@ static bool nested_get_vmcs12_pages(struct kvm_vcpu *vcpu)
} else {
pr_debug_ratelimited("%s: no backing for APIC-access address in vmcs12\n",
__func__);
vcpu->run->exit_reason = KVM_EXIT_INTERNAL_ERROR;
vcpu->run->internal.suberror =
KVM_INTERNAL_ERROR_EMULATION;
vcpu->run->internal.ndata = 0;
return false;
}
}
@ -3539,11 +3541,6 @@ static bool vmx_get_nested_state_pages(struct kvm_vcpu *vcpu)
if (!nested_get_evmcs_page(vcpu)) {
pr_debug_ratelimited("%s: enlightened vmptrld failed\n",
__func__);
vcpu->run->exit_reason = KVM_EXIT_INTERNAL_ERROR;
vcpu->run->internal.suberror =
KVM_INTERNAL_ERROR_EMULATION;
vcpu->run->internal.ndata = 0;
return false;
}
#endif
@ -3915,8 +3912,12 @@ static int nested_vmx_run(struct kvm_vcpu *vcpu, bool launch)
vmentry_failed:
vcpu->arch.nested_run_pending = 0;
if (status == NVMX_VMENTRY_KVM_INTERNAL_ERROR)
if (status == NVMX_VMENTRY_KVM_INTERNAL_ERROR) {
vcpu->run->exit_reason = KVM_EXIT_INTERNAL_ERROR;
vcpu->run->internal.suberror = KVM_INTERNAL_ERROR_EMULATION;
vcpu->run->internal.ndata = 0;
return 0;
}
if (status == NVMX_VMENTRY_VMEXIT)
return 1;
WARN_ON_ONCE(status != NVMX_VMENTRY_VMFAIL);

3
arch/x86/kvm/vmx/pmu_intel.c
View File

@ -187,6 +187,9 @@ static bool intel_is_valid_msr(struct kvm_vcpu *vcpu, u32 msr)
int ret;
switch (msr) {
case MSR_CORE_PERF_GLOBAL_STATUS:
case MSR_CORE_PERF_GLOBAL_CTRL:
case MSR_CORE_PERF_GLOBAL_OVF_CTRL:
case MSR_CORE_PERF_FIXED_CTR_CTRL:
return kvm_pmu_has_perf_global_ctrl(pmu);
case MSR_IA32_PEBS_ENABLE:

7
arch/x86/kvm/vmx/tdx.c
View File

@ -1995,7 +1995,7 @@ static int tdx_handle_ept_violation(struct kvm_vcpu *vcpu)
if (kvm_vcpu_has_events(vcpu) || signal_pending(current))
break;
if (kvm_check_request(KVM_REQ_VM_DEAD, vcpu)) {
if (kvm_test_request(KVM_REQ_VM_DEAD, vcpu)) {
ret = -EIO;
break;
}
@ -2728,11 +2728,6 @@ static tdx_vm_state_guard_t tdx_acquire_vm_state_locks(struct kvm *kvm)
mutex_lock(&kvm->lock);
if (kvm->created_vcpus != atomic_read(&kvm->online_vcpus)) {
r = -EBUSY;
goto out_err;
}
r = kvm_lock_all_vcpus(kvm);
if (r)
goto out_err;

6
arch/x86/kvm/x86.c
View File

@ -8060,7 +8060,7 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)
bool req_immediate_exit = false;
if (kvm_request_pending(vcpu)) {
if (kvm_check_request(KVM_REQ_VM_DEAD, vcpu)) {
if (kvm_test_request(KVM_REQ_VM_DEAD, vcpu)) {
r = -EIO;
goto out;
}
@ -8072,6 +8072,10 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)
if (kvm_check_request(KVM_REQ_GET_NESTED_STATE_PAGES, vcpu)) {
if (unlikely(!kvm_nested_call(get_nested_state_pages)(vcpu))) {
vcpu->run->exit_reason = KVM_EXIT_INTERNAL_ERROR;
vcpu->run->internal.suberror = KVM_INTERNAL_ERROR_EMULATION;
vcpu->run->internal.ndata = 0;
kvm_make_request(KVM_REQ_GET_NESTED_STATE_PAGES, vcpu);
r = 0;
goto out;
}

26
arch/x86/net/bpf_jit_comp.c
View File

@ -3558,16 +3558,21 @@ static void restore_regs(const struct btf_func_model *m, u8 **prog,
}
static int invoke_bpf_prog(const struct btf_func_model *m, u8 **pprog,
struct bpf_tramp_image *im,
struct bpf_tramp_node *node, int stack_size,
int run_ctx_off, bool save_ret,
void *image, void *rw_image)
{
u8 *prog = *pprog;
u8 *jmp_insn;
u8 *jmp_insn, *skip;
int ctx_cookie_off = offsetof(struct bpf_tramp_run_ctx, bpf_cookie);
struct bpf_prog *p = node->link->prog;
u64 cookie = node->cookie;
/* nop, patched to skip this prog when it is detached */
skip = image + (prog - (u8 *)rw_image);
emit_nops(&prog, X86_PATCH_SIZE);
/* mov rdi, cookie */
emit_mov_imm64(&prog, BPF_REG_1, (long) cookie >> 32, (u32) (long) cookie);
@ -3642,6 +3647,8 @@ static int invoke_bpf_prog(const struct btf_func_model *m, u8 **pprog,
if (emit_rsb_call(&prog, bpf_trampoline_exit(p), image + (prog - (u8 *)rw_image)))
return -EINVAL;
bpf_tramp_image_add_skip(im, p, skip, image + (prog - (u8 *)rw_image));
*pprog = prog;
return 0;
}
@ -3673,6 +3680,7 @@ static int emit_cond_near_jump(u8 **pprog, void *func, void *ip, u8 jmp_cond)
}
static int invoke_bpf(const struct btf_func_model *m, u8 **pprog,
struct bpf_tramp_image *im,
struct bpf_tramp_nodes *tl, int stack_size,
int run_ctx_off, int func_meta_off, bool save_ret,
void *image, void *rw_image, u64 func_meta,
@ -3687,7 +3695,7 @@ static int invoke_bpf(const struct btf_func_model *m, u8 **pprog,
func_meta | (cur_cookie << BPF_TRAMP_COOKIE_INDEX_SHIFT));
cur_cookie--;
}
if (invoke_bpf_prog(m, &prog, tl->nodes[i], stack_size,
if (invoke_bpf_prog(m, &prog, im, tl->nodes[i], stack_size,
run_ctx_off, save_ret, image, rw_image))
return -EINVAL;
}
@ -3696,6 +3704,7 @@ static int invoke_bpf(const struct btf_func_model *m, u8 **pprog,
}
static int invoke_bpf_mod_ret(const struct btf_func_model *m, u8 **pprog,
struct bpf_tramp_image *im,
struct bpf_tramp_nodes *tl, int stack_size,
int run_ctx_off, u8 **branches,
void *image, void *rw_image)
@ -3709,7 +3718,7 @@ static int invoke_bpf_mod_ret(const struct btf_func_model *m, u8 **pprog,
emit_mov_imm32(&prog, false, BPF_REG_0, 0);
emit_stx(&prog, BPF_DW, BPF_REG_FP, BPF_REG_0, -8);
for (i = 0; i < tl->nr_nodes; i++) {
if (invoke_bpf_prog(m, &prog, tl->nodes[i], stack_size, run_ctx_off, true,
if (invoke_bpf_prog(m, &prog, im, tl->nodes[i], stack_size, run_ctx_off, true,
image, rw_image))
return -EINVAL;
@ -3981,7 +3990,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im
}
if (fentry->nr_nodes) {
if (invoke_bpf(m, &prog, fentry, regs_off, run_ctx_off, func_meta_off,
if (invoke_bpf(m, &prog, im, fentry, regs_off, run_ctx_off, func_meta_off,
flags & BPF_TRAMP_F_RET_FENTRY_RET, image, rw_image,
func_meta, cookie_off))
return -EINVAL;
@ -3993,7 +4002,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im
if (!branches)
return -ENOMEM;
if (invoke_bpf_mod_ret(m, &prog, fmod_ret, regs_off,
if (invoke_bpf_mod_ret(m, &prog, im, fmod_ret, regs_off,
run_ctx_off, branches, image, rw_image)) {
ret = -EINVAL;
goto cleanup;
@ -4023,8 +4032,6 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im
}
/* remember return value in a stack for bpf prog to access */
emit_stx(&prog, BPF_DW, BPF_REG_FP, BPF_REG_0, -8);
im->ip_after_call = image + (prog - (u8 *)rw_image);
emit_nops(&prog, X86_PATCH_SIZE);
}
if (fmod_ret->nr_nodes) {
@ -4049,7 +4056,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im
emit_store_stack_imm64(&prog, BPF_REG_0, -func_meta_off, func_meta);
if (fexit->nr_nodes) {
if (invoke_bpf(m, &prog, fexit, regs_off, run_ctx_off, func_meta_off,
if (invoke_bpf(m, &prog, im, fexit, regs_off, run_ctx_off, func_meta_off,
false, image, rw_image, func_meta, cookie_off)) {
ret = -EINVAL;
goto cleanup;
@ -4064,7 +4071,6 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im
* restored to R0.
*/
if (flags & BPF_TRAMP_F_CALL_ORIG) {
im->ip_epilogue = image + (prog - (u8 *)rw_image);
/* arg1: mov rdi, im */
emit_mov_imm64(&prog, BPF_REG_1, (long) im >> 32, (u32) (long) im);
if (emit_rsb_call(&prog, __bpf_tramp_exit, image + (prog - (u8 *)rw_image))) {
@ -4152,7 +4158,7 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, void *i
int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags,
struct bpf_tramp_nodes *tnodes, void *func_addr)
{
struct bpf_tramp_image im;
struct bpf_tramp_image im = {};
if (!trampoline_size_image)
return -ENOMEM;

99
arch/x86/pci/fixup.c
View File

@ -886,6 +886,105 @@ static void quirk_clear_strap_no_soft_reset_dev2_f0(struct pci_dev *dev)
}
}
DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_AMD, 0x15b8, quirk_clear_strap_no_soft_reset_dev2_f0);
/*
* Enhanced atomic operations can cause corruption with 64-bit DMA
* on these devices.
*/
#define RX_ENH_ATOMIC_EN BIT(8)
static const u32 nbio_7_7_pcie_smn_addrs[] = {
0x111401d0,
0x111411d0,
0x111421d0,
0x111431d0,
0x111441d0,
0x112401d0,
0x112411d0,
0x112421d0,
0x112431d0,
0x112441d0,
0x112451d0,
0x113401d0,
0x114401d0,
};
static const u32 nbio_7_11_pcie_smn_addrs[] = {
0x112401d0,
0x112411d0,
0x112421d0,
0x112431d0,
0x112441d0,
0x112451d0,
0x113401d0,
0x113411d0,
0x113421d0,
0x113431d0,
0x113441d0,
0x113451d0,
};
static void quirk_amd_nbio_enhanced_atomic(struct pci_dev *host_bridge,
const u32 *smn_addrs,
size_t nr_smn_addrs)
{
bool changed = false;
size_t i;
u32 data;
int ret;
for (i = 0; i < nr_smn_addrs; i++) {
ret = amd_smn_read(0, smn_addrs[i], &data);
if (ret)
continue;
if (!(data & RX_ENH_ATOMIC_EN))
continue;
data = data & ~RX_ENH_ATOMIC_EN;
ret = amd_smn_write(0, smn_addrs[i], data);
if (ret)
continue;
if (changed)
continue;
ret = amd_smn_read(0, smn_addrs[i], &data);
if (ret)
continue;
if (data & RX_ENH_ATOMIC_EN)
continue;
changed = true;
}
if (changed)
pci_info(host_bridge, "enhanced atomics disabled\n");
}
static void quirk_amd_nbio_7_7_disable_enhanced_atomic(struct pci_dev *dev)
{
quirk_amd_nbio_enhanced_atomic(dev, nbio_7_7_pcie_smn_addrs,
ARRAY_SIZE(nbio_7_7_pcie_smn_addrs));
}
static void quirk_amd_nbio_7_11_disable_enhanced_atomic(struct pci_dev *dev)
{
quirk_amd_nbio_enhanced_atomic(dev, nbio_7_11_pcie_smn_addrs,
ARRAY_SIZE(nbio_7_11_pcie_smn_addrs));
}
/* Phoenix, Hawk Point (NBIO 7.7) */
DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_AMD, 0x14E8,
quirk_amd_nbio_7_7_disable_enhanced_atomic);
DECLARE_PCI_FIXUP_RESUME(PCI_VENDOR_ID_AMD, 0x14E8,
quirk_amd_nbio_7_7_disable_enhanced_atomic);
/* Strix, Krackan, Strix Halo (NBIO 7.11) */
DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_AMD, 0x1507,
quirk_amd_nbio_7_11_disable_enhanced_atomic);
DECLARE_PCI_FIXUP_RESUME(PCI_VENDOR_ID_AMD, 0x1507,
quirk_amd_nbio_7_11_disable_enhanced_atomic);
DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_AMD, 0x1122,
quirk_amd_nbio_7_11_disable_enhanced_atomic);
DECLARE_PCI_FIXUP_RESUME(PCI_VENDOR_ID_AMD, 0x1122,
quirk_amd_nbio_7_11_disable_enhanced_atomic);
#endif
/*

5
block/blk-cgroup.c
View File

@ -2091,6 +2091,7 @@ static inline struct blkcg_gq *blkg_tryget_closest(struct bio *bio,
struct request_queue *q = bio->bi_bdev->bd_queue;
struct blkcg *blkcg = css_to_blkcg(css);
struct blkcg_gq *blkg;
unsigned long flags;
rcu_read_lock();
blkg = blkg_lookup(blkcg, q);
@ -2105,11 +2106,11 @@ static inline struct blkcg_gq *blkg_tryget_closest(struct bio *bio,
* Fast path failed, we're probably issuing IO in this cgroup the first
* time, hold lock to create new blkg.
*/
spin_lock_irq(&q->queue_lock);
spin_lock_irqsave(&q->queue_lock, flags);
blkg = blkg_lookup_create(blkcg, bio->bi_bdev->bd_disk);
if (blkg)
blkg = blkg_lookup_tryget(blkg);
spin_unlock_irq(&q->queue_lock);
spin_unlock_irqrestore(&q->queue_lock, flags);
return blkg;
}

63
block/blk-mq.c
View File

@ -447,16 +447,6 @@ static struct request *blk_mq_rq_ctx_init(struct blk_mq_alloc_data *data,
WRITE_ONCE(rq->deadline, 0);
req_ref_set(rq, 1);
if (rq->rq_flags & RQF_USE_SCHED) {
struct elevator_queue *e = data->q->elevator;
INIT_HLIST_NODE(&rq->hash);
RB_CLEAR_NODE(&rq->rb_node);
if (e->type->ops.prepare_request)
e->type->ops.prepare_request(rq);
}
return rq;
}
@ -498,6 +488,12 @@ __blk_mq_alloc_requests_batch(struct blk_mq_alloc_data *data)
return rq_list_pop(data->cached_rqs);
}
static bool blk_op_bypass_sched(blk_opf_t opf)
{
return (opf & REQ_OP_MASK) == REQ_OP_FLUSH ||
blk_op_is_passthrough(opf);
}
static void blk_mq_limit_depth(struct blk_mq_alloc_data *data)
{
struct elevator_mq_ops *ops;
@ -518,12 +514,10 @@ static void blk_mq_limit_depth(struct blk_mq_alloc_data *data)
* Flush/passthrough requests are special and go directly to the
* dispatch list, they are not subject to the async_depth limit.
*/
if ((data->cmd_flags & REQ_OP_MASK) == REQ_OP_FLUSH ||
blk_op_is_passthrough(data->cmd_flags))
if (blk_op_bypass_sched(data->cmd_flags))
return;
WARN_ON_ONCE(data->flags & BLK_MQ_REQ_RESERVED);
data->rq_flags |= RQF_USE_SCHED;
/*
* By default, sync requests have no limit, and async requests are
@ -534,6 +528,29 @@ static void blk_mq_limit_depth(struct blk_mq_alloc_data *data)
ops->limit_depth(data->cmd_flags, data);
}
/*
* Finish initializing a request once it has been claimed for an operation.
* Cached requests are allocated before that operation is known.
*/
static void blk_mq_rq_late_init(struct request *rq, u64 alloc_time_ns)
{
struct elevator_queue *e;
blk_mq_rq_time_init(rq, alloc_time_ns);
if (!(rq->rq_flags & RQF_SCHED_TAGS) || (rq->rq_flags & RQF_RESV) ||
blk_op_bypass_sched(rq->cmd_flags))
return;
rq->rq_flags |= RQF_USE_SCHED;
INIT_HLIST_NODE(&rq->hash);
RB_CLEAR_NODE(&rq->rb_node);
e = rq->q->elevator;
if (e->type->ops.prepare_request)
e->type->ops.prepare_request(rq);
}
static struct request *__blk_mq_alloc_requests(struct blk_mq_alloc_data *data)
{
struct request_queue *q = data->q;
@ -562,7 +579,7 @@ static struct request *__blk_mq_alloc_requests(struct blk_mq_alloc_data *data)
if (data->nr_tags > 1) {
rq = __blk_mq_alloc_requests_batch(data);
if (rq) {
blk_mq_rq_time_init(rq, alloc_time_ns);
blk_mq_rq_late_init(rq, alloc_time_ns);
return rq;
}
data->nr_tags = 1;
@ -590,7 +607,7 @@ static struct request *__blk_mq_alloc_requests(struct blk_mq_alloc_data *data)
if (!(data->rq_flags & RQF_SCHED_TAGS))
blk_mq_inc_active_requests(data->hctx);
rq = blk_mq_rq_ctx_init(data, blk_mq_tags_from_data(data), tag);
blk_mq_rq_time_init(rq, alloc_time_ns);
blk_mq_rq_late_init(rq, alloc_time_ns);
return rq;
}
@ -637,8 +654,6 @@ static struct request *blk_mq_alloc_cached_request(struct request_queue *q,
if (plug->nr_ios == 1)
return NULL;
rq = blk_mq_rq_cache_fill(q, plug, opf, flags);
if (!rq)
return NULL;
} else {
rq = rq_list_peek(&plug->cached_rqs);
if (!rq || rq->q != q)
@ -646,15 +661,13 @@ static struct request *blk_mq_alloc_cached_request(struct request_queue *q,
if (blk_mq_get_hctx_type(opf) != rq->mq_hctx->type)
return NULL;
if (op_is_flush(rq->cmd_flags) != op_is_flush(opf))
return NULL;
rq_list_pop(&plug->cached_rqs);
blk_mq_rq_time_init(rq, blk_time_get_ns());
rq->cmd_flags = opf;
INIT_LIST_HEAD(&rq->queuelist);
blk_mq_rq_late_init(rq, blk_time_get_ns());
}
rq->cmd_flags = opf;
INIT_LIST_HEAD(&rq->queuelist);
return rq;
}
@ -766,7 +779,7 @@ struct request *blk_mq_alloc_request_hctx(struct request_queue *q,
if (!(data.rq_flags & RQF_SCHED_TAGS))
blk_mq_inc_active_requests(data.hctx);
rq = blk_mq_rq_ctx_init(&data, blk_mq_tags_from_data(&data), tag);
blk_mq_rq_time_init(rq, alloc_time_ns);
blk_mq_rq_late_init(rq, alloc_time_ns);
rq->__data_len = 0;
rq->phys_gap_bit = 0;
rq->__sector = (sector_t) -1;
@ -3060,8 +3073,6 @@ static struct request *blk_mq_get_cached_request(struct blk_plug *plug,
if (type != rq->mq_hctx->type &&
(type != HCTX_TYPE_READ || rq->mq_hctx->type != HCTX_TYPE_DEFAULT))
return NULL;
if (op_is_flush(rq->cmd_flags) != op_is_flush(opf))
return NULL;
rq_list_pop(&plug->cached_rqs);
return rq;
}
@ -3163,9 +3174,9 @@ void blk_mq_submit_bio(struct bio *bio)
new_request:
if (rq) {
rq_qos_throttle(rq->q, bio);
blk_mq_rq_time_init(rq, blk_time_get_ns());
rq->cmd_flags = bio->bi_opf;
INIT_LIST_HEAD(&rq->queuelist);
blk_mq_rq_late_init(rq, blk_time_get_ns());
} else {
rq = blk_mq_get_new_requests(q, plug, bio);
if (unlikely(!rq)) {

15
block/blk-zoned.c
View File

@ -2018,12 +2018,17 @@ static int disk_revalidate_zone_resources(struct gendisk *disk,
struct blk_revalidate_zone_args *args)
{
struct queue_limits *lim = &disk->queue->limits;
unsigned long long nr_zones;
unsigned int pool_size;
int ret = 0;
args->disk = disk;
args->nr_zones =
DIV_ROUND_UP_ULL(get_capacity(disk), lim->chunk_sectors);
nr_zones = DIV_ROUND_UP_ULL(get_capacity(disk), lim->chunk_sectors);
if (nr_zones > UINT_MAX) {
pr_warn("%s: Too many zones (%llu)\n", disk->disk_name, nr_zones);
return -EINVAL;
}
args->nr_zones = nr_zones;
/* Cached zone conditions: 1 byte per zone */
args->zones_cond = kzalloc(args->nr_zones, GFP_NOIO);
@ -2131,6 +2136,12 @@ static int blk_revalidate_zone_cond(struct blk_zone *zone, unsigned int idx,
{
enum blk_zone_cond cond = zone->cond;
if (idx >= args->nr_zones) {
pr_warn("%s: Zone report index %u exceeds zone count %u\n",
args->disk->disk_name, idx, args->nr_zones);
return -EINVAL;
}
/* Check that the zone condition is consistent with the zone type. */
switch (cond) {
case BLK_ZONE_COND_NOT_WP:

5
block/fops.c
View File

@ -402,6 +402,11 @@ static ssize_t blkdev_direct_IO(struct kiocb *iocb, struct iov_iter *iter)
if (blkdev_dio_invalid(bdev, iocb, iter))
return -EINVAL;
/* HIPRI needs private as bio; HAS_METADATA keeps it as uio_meta */
if ((iocb->ki_flags & IOCB_HIPRI) &&
(iocb->ki_flags & IOCB_HAS_METADATA))
return -EOPNOTSUPP;
if (iov_iter_rw(iter) == WRITE) {
u16 max_write_streams = bdev_max_write_streams(bdev);

2
block/mq-deadline.c
View File

@ -685,7 +685,7 @@ static void dd_insert_requests(struct blk_mq_hw_ctx *hctx,
blk_mq_free_requests(&free);
}
/* Callback from inside blk_mq_rq_ctx_init(). */
/* Callback from inside blk_mq_rq_late_init(). */
static void dd_prepare_request(struct request *rq)
{
rq->elv.priv[0] = NULL;

51
crypto/aes.c
View File

@ -637,7 +637,18 @@ static struct skcipher_alg skcipher_algs[] = {
.decrypt = crypto_aes_cbc_decrypt,
},
#endif
#if IS_ENABLED(CONFIG_CRYPTO_CTS)
/*
* Don't register library-based "cts(cbc(aes))" on architectures where
* it might block a "better" implementation from being instantiated via
* the "cts" template. These exclusions are temporary and will go away
* as the arch-optimized AES code is migrated into the library.
*/
#if IS_ENABLED(CONFIG_CRYPTO_CTS) && \
!(IS_ENABLED(CONFIG_ARM) || \
IS_ENABLED(CONFIG_ARM64) || \
IS_ENABLED(CONFIG_PPC) || \
IS_ENABLED(CONFIG_S390) || \
IS_ENABLED(CONFIG_SPARC))
{
.base.cra_name = "cts(cbc(aes))",
.base.cra_driver_name = "cts-cbc-aes-lib",
@ -687,7 +698,13 @@ static struct skcipher_alg skcipher_algs[] = {
.decrypt = crypto_aes_xctr_crypt,
},
#endif
#if IS_ENABLED(CONFIG_CRYPTO_XTS)
/*
* Don't register library-based "xts(aes)" on architectures where it
* might block a "better" implementation from being instantiated via the
* "xts" template. This exclusion is temporary and will go away when
* the library AES-XTS is optimized for SPARC.
*/
#if IS_ENABLED(CONFIG_CRYPTO_XTS) && !IS_ENABLED(CONFIG_SPARC)
{
.base.cra_name = "xts(aes)",
.base.cra_driver_name = "xts-aes-lib",
@ -980,7 +997,20 @@ static __maybe_unused int crypto_aes_ccm_decrypt(struct aead_request *req)
}
static struct aead_alg aead_algs[] = {
#if IS_ENABLED(CONFIG_CRYPTO_GCM)
/*
* Don't register library-based "gcm(aes)" and "rfc4106(gcm(aes))" on
* architectures where they might block a "better" implementation from
* being instantiated via the "gcm" and "rfc4106" templates. These
* exclusions are temporary and will go away as the arch-optimized AES
* code is migrated into the library.
*/
#if IS_ENABLED(CONFIG_CRYPTO_GCM) && \
!(IS_ENABLED(CONFIG_ARM) || \
IS_ENABLED(CONFIG_ARM64) || \
IS_ENABLED(CONFIG_PPC) || \
IS_ENABLED(CONFIG_RISCV) || \
IS_ENABLED(CONFIG_S390) || \
IS_ENABLED(CONFIG_SPARC))
{
.base.cra_name = "gcm(aes)",
.base.cra_driver_name = "gcm-aes-lib",
@ -1012,7 +1042,20 @@ static struct aead_alg aead_algs[] = {
.chunksize = AES_BLOCK_SIZE,
},
#endif /* CONFIG_CRYPTO_GCM */
#if IS_ENABLED(CONFIG_CRYPTO_CCM)
/*
* Don't register library-based "ccm(aes)" on architectures where it
* might block a "better" implementation from being instantiated via the
* "ccm" template. These exclusions are temporary and will go away as
* the arch-optimized AES code is migrated into the library.
*/
#if IS_ENABLED(CONFIG_CRYPTO_CCM) && \
!(IS_ENABLED(CONFIG_ARM) || \
IS_ENABLED(CONFIG_ARM64) || \
IS_ENABLED(CONFIG_PPC) || \
IS_ENABLED(CONFIG_RISCV) || \
IS_ENABLED(CONFIG_S390) || \
IS_ENABLED(CONFIG_SPARC) || \
IS_ENABLED(CONFIG_X86))
{
.base.cra_name = "ccm(aes)",
.base.cra_driver_name = "ccm-aes-lib",

Some files were not shown because too many files have changed in this diff Show More