memmove() copies byte wise for the backward copy case, when the mvc
instruction cannot be used. This is quite slow, but can be optimized
with the mvcrl instruction, which is available since z15.
Some numbers (measured on a shared z16 LPAR) show that the new
implementation is nearly always faster, except for the non realistic
one and two byte cases:
size old new
1 2ns 3ns
2 4ns 5ns
4 5ns 5ns
8 8ns 5ns
16 12ns 6ns
32 8ns 7ns
64 15ns 7ns
128 31ns 9ns
256 64ns 10ns
512 129ns 18ns
1024 250ns 19ns
2048 498ns 38ns
Reviewed-by: Juergen Christ <jchrist@linux.ibm.com>
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
Signed-off-by: Alexander Gordeev <agordeev@linux.ibm.com>
Convert memset(16|32|64)() from assembler to C, which should make it
easier to read and change, if required. And it allows the compiler to
optimize the code, and use different instructions, except for the used
inline assemblies.
Reviewed-by: Juergen Christ <jchrist@linux.ibm.com>
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
Signed-off-by: Alexander Gordeev <agordeev@linux.ibm.com>
Convert memcpy() from assembler to C, which should make it easier to
read and change, if required. And it allows the compiler to optimize
the code, and use different instructions, except for the used inline
assemblies.
Reviewed-by: Juergen Christ <jchrist@linux.ibm.com>
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
Signed-off-by: Alexander Gordeev <agordeev@linux.ibm.com>
Convert memset() from assembler to C, which should make it easier to
read and change, if required. And it allows the compiler to optimize
the code, and use different instructions, except for the used inline
assemblies.
Reviewed-by: Juergen Christ <jchrist@linux.ibm.com>
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
Signed-off-by: Alexander Gordeev <agordeev@linux.ibm.com>
Convert memmove() from assembler to C, which should make it easier to
read and change, if required. And it allows the compiler to optimize
the code, and use different instructions, except for the used inline
assemblies.
Reviewed-by: Juergen Christ <jchrist@linux.ibm.com>
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
Signed-off-by: Alexander Gordeev <agordeev@linux.ibm.com>
Use -ffreestanding for string.o to avoid that the compiler generates
calls into themselves for standard library functions like memset().
Reviewed-by: Juergen Christ <jchrist@linux.ibm.com>
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
Signed-off-by: Alexander Gordeev <agordeev@linux.ibm.com>
Upcoming changes will result in a .noinstr.text section within the
boot and purgatory string.o binary. Explicitly add the new section to
avoid orphaned warnings from the linker.
Reviewed-by: Juergen Christ <jchrist@linux.ibm.com>
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
Signed-off-by: Alexander Gordeev <agordeev@linux.ibm.com>
The purgatory code is compiled without the -march option. This means the
default architecture level of the compiler is used. This can cause
problems, e.g. if instructions used in inline assemblies are for a higher
architecture level than the default architecture level of the compiler.
Use z10 as minimum architecture level, similar to the boot code, to enforce
a defined architecture level set.
Reviewed-by: Juergen Christ <jchrist@linux.ibm.com>
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
Signed-off-by: Alexander Gordeev <agordeev@linux.ibm.com>
Jan Polensky says:
===================
Rust support on s390 requires a small set of architecture-specific pieces
before the generic Rust kernel infrastructure can be used.
The series wires up s390 as a Rust-capable 64-bit architecture, adds the
missing assembly interfaces needed by Rust for WARN/BUG reporting and for
static branches, adjusts bindgen parameters to avoid repr layout conflicts
caused by packed and aligned s390 structures, and fixes issues discovered
during testing.
s390 currently requires rustc with support for -Zpacked-stack, and the
minimum tool version gating is adjusted accordingly.
Link: https://github.com/Rust-for-Linux/linux/issues/2
Tested against: rustc 1.96.0 (ac68faa20 2026-05-25)
===================
Signed-off-by: Alexander Gordeev <agordeev@linux.ibm.com>
Enable building Rust code on s390 by wiring the architecture into the
kernel Rust infrastructure.
Add s390 to the Rust arch support documentation, provide the s390 Rust
target and required compiler flags, and set the bindgen target for
arch/s390. Adjust the Rust target generation and minimum rustc version
gating so the s390 setup is handled explicitly.
The Rust toolchain uses the "s390x" triple naming for the 64 bit target.
Rust support is currently incompatible with CONFIG_EXPOLINE, which
relies on compiler support for the -mindirect-branch= and
-mfunction_return= options. Therefore, select HAVE_RUST only when
EXPOLINE is disabled.
Acked-by: Miguel Ojeda <ojeda@kernel.org>
Acked-by: Gary Guo <gary@garyguo.net>
Acked-by: Heiko Carstens <hca@linux.ibm.com>
Signed-off-by: Jan Polensky <japo@linux.ibm.com>
Signed-off-by: Alexander Gordeev <agordeev@linux.ibm.com>
The __arch_cmpxchg1, __arch_cmpxchg2, __arch_xchg1, and __arch_xchg2
functions emulate 1-byte and 2-byte atomic operations using 4-byte
cmpxchg instructions, since s390 lacks native 1/2-byte cmpxchg support.
When KASAN is enabled, the READ_ONCE() operations in these functions
trigger stack-out-of-bounds warnings because they perform 4-byte reads
when only 1 or 2 bytes should be accessed.
Mark these functions as __no_sanitize_or_inline to prevent KASAN
instrumentation while maintaining correct functionality.
This resolves the following KASAN error during rust_atomics KUnit tests:
BUG: KASAN: stack-out-of-bounds in rust_helper_atomic_i8_xchg+0xb2/0xc0
Read of size 4 at addr 001bff7ffdbefcf0 by task kunit_try_catch/142
Reported-by: Miguel Ojeda <miguel.ojeda.sandonis@gmail.com>
Link: https://lore.kernel.org/rust-for-linux/CANiq72m4GVWFYqnxNtCHTPu7XcGewHB5LNwOoayTfnXs9pPbNg@mail.gmail.com/
Suggested-by: Gary Guo <gary@garyguo.net>
Link: https://lore.kernel.org/rust-for-linux/DITFTAVVHTNQ.380OHUHGTOI6M@garyguo.net/
Acked-by: Heiko Carstens <hca@linux.ibm.com>
Acked-by: Gary Guo <gary@garyguo.net>
Signed-off-by: Jan Polensky <japo@linux.ibm.com>
Signed-off-by: Alexander Gordeev <agordeev@linux.ibm.com>
Add a dedicated string helper file with a memchr wrapper that uses the
kernel's instrumented memchr() function to ensure KASAN and FORTIFY_SOURCE
protections are preserved for Rust code.
Reported-by: Miguel Ojeda <miguel.ojeda.sandonis@gmail.com>
Link: https://lore.kernel.org/rust-for-linux/CANiq72mXAZc0sNM7ShX8VDVs_7zJddawP-e=wt+ERr1YUCcWUw@mail.gmail.com/
Signed-off-by: Jan Polensky <japo@linux.ibm.com>
Acked-by: Heiko Carstens <hca@linux.ibm.com>
Acked-by: Gary Guo <gary@garyguo.net>
Signed-off-by: Alexander Gordeev <agordeev@linux.ibm.com>
Bindgen attempts to generate Rust layouts for a number of s390 structs
that are packed but contain, or transitively contain, aligned fields.
Rust rejects such layouts with E0588 ("packed type cannot transitively
contain a #[repr(align)] type").
Add the affected s390 types to the opaque type list so bindgen emits
opaque blob types instead of full representations. This matches existing
workarounds for x86 types such as alt_instr and x86_msi_data.
Link: https://lore.kernel.org/all/e5c7aa10-590d-0d20-dd3b-385bee2377e7@intel.com/
Acked-by: Miguel Ojeda <ojeda@kernel.org>
Reviewed-by: Gary Guo <gary@garyguo.net>
Acked-by: Heiko Carstens <hca@linux.ibm.com>
Signed-off-by: Jan Polensky <japo@linux.ibm.com>
Signed-off-by: Alexander Gordeev <agordeev@linux.ibm.com>
Rust static branch support needs the s390 jump label instruction sequence
and __jump_table emission in a reusable form. The current implementation
embeds the sequence directly in the C asm goto blocks, which cannot be
shared with Rust.
Introduce ARCH_STATIC_BRANCH_ASM and ARCH_STATIC_BRANCH_JUMP_ASM to
describe the brcl sequences for the likely-false and likely-true cases
and to emit the same __jump_table entries as before. Switch the existing
C helpers to use the new macros to avoid duplication without changing
the generated code.
Acked-by: Gary Guo <gary@garyguo.net>
Acked-by: Miguel Ojeda <ojeda@kernel.org>
Acked-by: Heiko Carstens <hca@linux.ibm.com>
Reviewed-by: Alice Ryhl <aliceryhl@google.com>
Signed-off-by: Jan Polensky <japo@linux.ibm.com>
Signed-off-by: Alexander Gordeev <agordeev@linux.ibm.com>
Rust WARN and BUG support relies on ARCH_WARN_ASM to emit __bug_table
entries. On s390 the macro is missing, so Rust code cannot generate
proper WARN/BUG metadata for the kernel's bug reporting infrastructure.
Define ARCH_WARN_ASM to produce the same assembly sequence and
__bug_table entry format as the existing s390 BUG handling, including
the monitor call. Define ARCH_WARN_REACHABLE as empty since s390 does
not provide reachability analysis for warning paths.
Acked-by: Gary Guo <gary@garyguo.net>
Acked-by: Miguel Ojeda <ojeda@kernel.org>
Acked-by: Heiko Carstens <hca@linux.ibm.com>
Signed-off-by: Jan Polensky <japo@linux.ibm.com>
Signed-off-by: Alexander Gordeev <agordeev@linux.ibm.com>
Revisit and reorganize the locking and lock coverage of the
ap->lock spinlock as used in the two sysfs functions
se_bind_store() and se_associate_store().
A kernel run reported a possible deadlock situation, caused by
holding the spinlock (ap->lock) while triggering a uevent.
The fix rearranges the code protected by the spinlock by excluding
the uevent invocation, which does not require protection.
Additionally, the start of the protected region is moved earlier
to cover more lines, ensuring a consistent view of the AP queue
state between reading and updating its struct fields.
=====================================================
WARNING: SOFTIRQ-safe -> SOFTIRQ-unsafe lock order detected
7.1.0-20260601.rc6.git12.516b5dbd4d4a.300.fc44.s390x+debug #1 Not tainted
-----------------------------------------------------
setupseguest.sh/11034 [HC0[0]:SC0[2]:HE1:SE0] is trying to acquire:
000001c991f498e8 (fs_reclaim){+.+.}-{0:0}, at: __kmalloc_cache_noprof+0x5a/0x6d0
and this task is already holding:
000000c4a1a12378 (&aq->lock){+.-.}-{2:2}, at: se_bind_store+0x96/0x3a0
which would create a new lock dependency:
(&aq->lock){+.-.}-{2:2} -> (fs_reclaim){+.+.}-{0:0}
but this new dependency connects a SOFTIRQ-irq-safe lock:
(&aq->lock){+.-.}-{2:2}
... which became SOFTIRQ-irq-safe at:
__lock_acquire+0x5ae/0x15a0
lock_acquire+0x14c/0x400
_raw_spin_lock_bh+0x58/0xb0
ap_tasklet_fn+0x72/0xd0
tasklet_action_common+0x174/0x1b0
handle_softirqs+0x180/0x5c0
irq_exit_rcu+0x196/0x200
do_ext_irq+0x12a/0x4d0
ext_int_handler+0xc6/0xf0
folio_zero_user+0x1c6/0x240
folio_zero_user+0x182/0x240
vma_alloc_anon_folio_pmd+0xa0/0x1d0
__do_huge_pmd_anonymous_page+0x3a/0x200
__handle_mm_fault+0x56c/0x590
handle_mm_fault+0xa2/0x370
do_exception+0x292/0x590
__do_pgm_check+0x136/0x3e0
pgm_check_handler+0x114/0x160
to a SOFTIRQ-irq-unsafe lock:
(fs_reclaim){+.+.}-{0:0}
... which became SOFTIRQ-irq-unsafe at:
...
__lock_acquire+0x5ae/0x15a0
lock_acquire+0x14c/0x400
__fs_reclaim_acquire+0x44/0x50
fs_reclaim_acquire+0xbe/0x100
fs_reclaim_correct_nesting+0x20/0x70
dotest+0x5e/0x148
locking_selftest+0x2854/0x2a88
start_kernel+0x3b2/0x4f0
startup_continue+0x2e/0x40
other info that might help us debug this:
Possible interrupt unsafe locking scenario:
CPU0 CPU1
---- ----
lock(fs_reclaim);
local_irq_disable();
lock(&aq->lock);
lock(fs_reclaim);
<Interrupt>
lock(&aq->lock);
*** DEADLOCK ***
4 locks held by setupseguest.sh/11034:
#0: 000000c485d01440 (sb_writers#4){.+.+}-{0:0}, at: vfs_write+0x2fc/0x380
#1: 000000c4d2283288 (&of->mutex#2){+.+.}-{3:3}, at: kernfs_fop_write_iter+0x12a0x270
#2: 000000c4a1830e48 (kn->active#172){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x1e/0x270
#3: 000000c4a1a12378 (&aq->lock){+.-.}-{2:2}, at: se_bind_store+0x96/0x3a0
the dependencies between SOFTIRQ-irq-safe lock and the holding lock:
-> (&aq->lock){+.-.}-{2:2} {
HARDIRQ-ON-W at:
__lock_acquire+0x5ae/0x15a0
lock_acquire+0x14c/0x400
_raw_spin_lock_bh+0x58/0xb0
ap_queue_init_state+0x2e/0x50
ap_scan_domains+0x5d6/0x620
ap_scan_adapter+0x4c0/0x810
ap_scan_bus+0x70/0x350
ap_scan_bus_wq_callback+0x56/0x80
process_one_work+0x2ba/0x820
worker_thread+0x21a/0x400
kthread+0x164/0x190
__ret_from_fork+0x4c/0x340
ret_from_fork+0xa/0x30
IN-SOFTIRQ-W at:
__lock_acquire+0x5ae/0x15a0
lock_acquire+0x14c/0x400
_raw_spin_lock_bh+0x58/0xb0
ap_tasklet_fn+0x72/0xd0
tasklet_action_common+0x174/0x1b0
handle_softirqs+0x180/0x5c0
irq_exit_rcu+0x196/0x200
do_ext_irq+0x12a/0x4d0
ext_int_handler+0xc6/0xf0
folio_zero_user+0x1c6/0x240
folio_zero_user+0x182/0x240
vma_alloc_anon_folio_pmd+0xa0/0x1d0
__do_huge_pmd_anonymous_page+0x3a/0x200
__handle_mm_fault+0x56c/0x590
handle_mm_fault+0xa2/0x370
do_exception+0x292/0x590
__do_pgm_check+0x136/0x3e0
pgm_check_handler+0x114/0x160
INITIAL USE at:
__lock_acquire+0x5ae/0x15a0
lock_acquire+0x14c/0x400
_raw_spin_lock_bh+0x58/0xb0
ap_queue_init_state+0x2e/0x50
ap_scan_domains+0x5d6/0x620
ap_scan_adapter+0x4c0/0x810
ap_scan_bus+0x70/0x350
ap_scan_bus_wq_callback+0x56/0x80
process_one_work+0x2ba/0x820
worker_thread+0x21a/0x400
kthread+0x164/0x190
__ret_from_fork+0x4c/0x340
ret_from_fork+0xa/0x30
}
... key at: [<000001c9936e8aa0>] __key.7+0x0/0x10
the dependencies between the lock to be acquired
and SOFTIRQ-irq-unsafe lock:
-> (fs_reclaim){+.+.}-{0:0} {
HARDIRQ-ON-W at:
__lock_acquire+0x5ae/0x15a0
lock_acquire+0x14c/0x400
__fs_reclaim_acquire+0x44/0x50
fs_reclaim_acquire+0xbe/0x100
fs_reclaim_correct_nesting+0x20/0x70
dotest+0x5e/0x148
locking_selftest+0x2854/0x2a88
start_kernel+0x3b2/0x4f0
startup_continue+0x2e/0x40
SOFTIRQ-ON-W at:
__lock_acquire+0x5ae/0x15a0
lock_acquire+0x14c/0x400
__fs_reclaim_acquire+0x44/0x50
fs_reclaim_acquire+0xbe/0x100
fs_reclaim_correct_nesting+0x20/0x70
dotest+0x5e/0x148
locking_selftest+0x2854/0x2a88
start_kernel+0x3b2/0x4f0
startup_continue+0x2e/0x40
INITIAL USE at:
__lock_acquire+0x5ae/0x15a0
lock_acquire+0x14c/0x400
__fs_reclaim_acquire+0x44/0x50
fs_reclaim_acquire+0xbe/0x100
fs_reclaim_correct_nesting+0x20/0x70
dotest+0x5e/0x148
locking_selftest+0x2854/0x2a88
start_kernel+0x3b2/0x4f0
startup_continue+0x2e/0x40
}
... key at: [<000001c991f498e8>] __fs_reclaim_map+0x0/0x30
... acquired at:
check_prev_add+0x178/0xf40
__lock_acquire+0x12aa/0x15a0
lock_acquire+0x14c/0x400
__fs_reclaim_acquire+0x44/0x50
fs_reclaim_acquire+0xbe/0x100
__kmalloc_cache_noprof+0x5a/0x6d0
kobject_uevent_env+0xd4/0x420
ap_send_se_bind_uevent+0x48/0x70
se_bind_store+0x146/0x3a0
kernfs_fop_write_iter+0x18c/0x270
vfs_write+0x23c/0x380
ksys_write+0x88/0x120
__do_syscall+0x170/0x750
system_call+0x72/0x90
stack backtrace:
CPU: 6 UID: 0 PID: 11034 Comm: setupseguest.sh Not tainted 7.1.0-20260601.rc6.git2.516b5dbd4d4a.300.fc44.s390x+debug #1 PREEMPT
Hardware name: IBM 9175 ME1 701 (KVM/Linux)
Call Trace:
[<000001c98ffa0a7e>] dump_stack_lvl+0xae/0x108
[<000001c9900a6d7a>] print_bad_irq_dependency+0x47a/0x480
[<000001c9900a7184>] check_irq_usage+0x404/0x4c0
[<000001c9900a73b8>] check_prev_add+0x178/0xf40
[<000001c9900aaf1a>] __lock_acquire+0x12aa/0x15a0
[<000001c9900ab35c>] lock_acquire+0x14c/0x400
[<000001c9903be454>] __fs_reclaim_acquire+0x44/0x50
[<000001c9903be51e>] fs_reclaim_acquire+0xbe/0x100
[<000001c9903cf4ca>] __kmalloc_cache_noprof+0x5a/0x6d0
[<000001c9910ca9d4>] kobject_uevent_env+0xd4/0x420
[<000001c990d84098>] ap_send_se_bind_uevent+0x48/0x70
[<000001c990d87416>] se_bind_store+0x146/0x3a0
[<000001c99057da7c>] kernfs_fop_write_iter+0x18c/0x270
[<000001c99047712c>] vfs_write+0x23c/0x380
[<000001c990477438>] ksys_write+0x88/0x120
[<000001c9910f64e0>] __do_syscall+0x170/0x750
[<000001c99110a412>] system_call+0x72/0x90
INFO: lockdep is turned off.
Fixes: 4179c39842 ("s390/ap: Implement SE bind and associate uevents")
Reported-by: Ingo Franzki <ifranzki@linux.ibm.com>
Suggested-by: Finn Callies <fcallies@linux.ibm.com>
Reviewed-by: Finn Callies <fcallies@linux.ibm.com>
Signed-off-by: Harald Freudenberger <freude@linux.ibm.com>
Signed-off-by: Alexander Gordeev <agordeev@linux.ibm.com>
Provide an s390 specific implementation of arch_this_cpu_write()
instead of the generic variant. The generic variant uses a quite
expensive raw_local_irq_save() / raw_local_irq_restore() pair.
Get rid of this by providing an own variant which makes use of the new
percpu code section infrastructure.
With this the text size of the kernel image is reduced by ~1k (defconfig).
Acked-by: Alexander Gordeev <agordeev@linux.ibm.com>
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
Signed-off-by: Alexander Gordeev <agordeev@linux.ibm.com>
Provide an s390 specific implementation of arch_this_cpu_read() instead
of the generic variant. The generic variant uses preempt_disable() /
preempt_enable() pair and READ_ONCE().
Get rid of the preempt_disable() / preempt_enable() pairs by providing an
own variant which makes use of the new percpu code section infrastructure.
With this the text size of the kernel image is reduced by ~1k
(defconfig). Also 87 generated preempt_schedule_notrace() function
calls within the kernel image (modules not counted) are removed.
Acked-by: Alexander Gordeev <agordeev@linux.ibm.com>
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
Signed-off-by: Alexander Gordeev <agordeev@linux.ibm.com>
Convert arch_this_cpu_[and|or]() to make use of the new percpu code
section infrastructure.
There is no user of this_cpu_and() and only one user of this_cpu_or()
within the kernel. Therefore this conversion has hardly any effect,
and also removes only preempt_schedule_notrace() function call.
Acked-by: Alexander Gordeev <agordeev@linux.ibm.com>
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
Signed-off-by: Alexander Gordeev <agordeev@linux.ibm.com>
Convert arch_this_cpu_add_return() to make use of the new percpu code
section infrastructure.
With this the text size of the kernel image is reduced by ~4k
(defconfig). Also 66 generated preempt_schedule_notrace() function
calls within the kernel image (modules not counted) are removed.
Acked-by: Alexander Gordeev <agordeev@linux.ibm.com>
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
Signed-off-by: Alexander Gordeev <agordeev@linux.ibm.com>
Add missing do { } while (0) constructs in order to avoid potential
build failures.
Reported-by: Sashiko <sashiko-bot@kernel.org>
Closes: https://sashiko.dev/#/patchset/20260319120503.4046659-1-hca%40linux.ibm.com
Reviewed-by: Alexander Gordeev <agordeev@linux.ibm.com>
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
Signed-off-by: Alexander Gordeev <agordeev@linux.ibm.com>
With the intended removal of PREEMPT_NONE this_cpu operations based on
atomic instructions, guarded with preempt_disable()/preempt_enable() pairs
become more expensive: the preempt_disable() / preempt_enable() pairs are
not optimized away anymore during compile time.
In particular the conditional call to preempt_schedule_notrace() after
preempt_enable() adds additional code and register pressure.
E.g. this simple C code sequence
DEFINE_PER_CPU(long, foo);
long bar(long a) { return this_cpu_add_return(foo, a); }
generates this code:
11a976: eb af f0 68 00 24 stmg %r10,%r15,104(%r15)
11a97c: b9 04 00 ef lgr %r14,%r15
11a980: b9 04 00 b2 lgr %r11,%r2
11a984: e3 f0 ff c8 ff 71 lay %r15,-56(%r15)
11a98a: e3 e0 f0 98 00 24 stg %r14,152(%r15)
11a990: eb 01 03 a8 00 6a asi 936,1 <- __preempt_count_add(1)
11a996: c0 10 00 d2 ac b5 larl %r1,1b70300 <- address of percpu var
11a9a0: e3 10 23 b8 00 08 ag %r1,952 <- add percpu offset
11a9a6: eb ab 10 00 00 e8 laag %r10,%r11,0(%r1) <- atomic op
11a9ac: eb ff 03 a8 00 6e alsi 936,-1 <- __preempt_count_dec_and_test()
11a9b2: a7 54 00 05 jnhe 11a9bc <bar+0x4c>
11a9b6: c0 e5 00 76 d1 bd brasl %r14,ff4d30 <preempt_schedule_notrace>
11a9bc: b9 e8 b0 2a agrk %r2,%r10,%r11
11a9c0: eb af f0 a0 00 04 lmg %r10,%r15,160(%r15)
11a9c6 07 fe br %r14
Even though the above example is more or less the worst case, since the
branch to preempt_schedule_notrace() requires a stackframe, which
otherwise wouldn't be necessary, there is also the conditional jnhe branch
instruction.
Get rid of the conditional branch with the following code sequence:
11a8e6: c0 30 00 d0 c5 0d larl %r3,1b33300
11a8ec: b9 04 00 43 lgr %r4,%r3
11a8f0: eb 00 43 c0 00 52 mviy 960,4
11a8f6: e3 40 03 b8 00 08 ag %r4,952
11a8fc: eb 52 40 00 00 e8 laag %r5,%r2,0(%r4)
11a902: eb 00 03 c0 00 52 mviy 960,0
11a908: b9 08 00 25 agr %r2,%r5
11a90c 07 fe br %r14
The general idea is that this_cpu operations based on atomic instructions
are guarded with mviy instructions:
- The first mviy instruction writes the register number, which contains
the percpu address variable to lowcore. This also indicates that a
percpu code section is executed.
- The first instruction following the mviy instruction must be the ag
instruction which adds the percpu offset to the percpu address register.
- Afterwards the atomic percpu operation follows.
- Then a second mviy instruction writes a zero to lowcore, which indicates
the end of the percpu code section.
- In case of an interrupt/exception/nmi the register number which was
written to lowcore is copied to the exception frame (pt_regs), and a zero
is written to lowcore.
- On return to the previous context it is checked if a percpu code section
was executed (saved register number not zero), and if the process was
migrated to a different cpu. If the percpu offset was already added to
the percpu address register (instruction address does _not_ point to the
ag instruction) the content of the percpu address register is adjusted so
it points to percpu variable of the new cpu.
Reviewed-by: Alexander Gordeev <agordeev@linux.ibm.com>
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
Signed-off-by: Alexander Gordeev <agordeev@linux.ibm.com>
zcrypt_rng_device_add() allocates a buffer for the software random
number generator data cache.
This buffer can be allocated with kmalloc() as there's nothing special
about it to go directly to the page allocator.
kmalloc() provides a better API that does not require ugly casts and
kfree() does not need to know the size of the freed object.
Performance difference between kmalloc() and __get_free_pages() is not
measurable as both allocators take an object/page from a per-CPU list for
fast path allocations.
For the slow path the performance is anyway determined by the amount of
reclaim involved rather than by what allocator is used.
Replace use of get_zeroed_page() with kzalloc() and free_page() with
kfree().
Link: https://lore.kernel.org/all/635405e4-9423-4a25-a6e7-e03c8ea0bcbe@redhat.com
Reviewed-by: Harald Freudenberger <freude@linux.ibm.com>
Reviewed-by: Heiko Carstens <hca@linux.ibm.com>
Signed-off-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Signed-off-by: Alexander Gordeev <agordeev@linux.ibm.com>
trng_read() allocates a temporary staging buffer for CPACF TRNG
random data before copying it to userspace.
This buffer can be allocated with kmalloc() as there's nothing special
about it to go directly to the page allocator.
kmalloc() provides a better API that does not require ugly casts and
kfree() does not need to know the size of the freed object.
Performance difference between kmalloc() and __get_free_pages() is not
measurable as both allocators take an object/page from a per-CPU list for
fast path allocations.
For the slow path the performance is anyway determined by the amount of
reclaim involved rather than by what allocator is used.
Replace use of __get_free_page() with kmalloc() and free_page() with
kfree().
Link: https://lore.kernel.org/all/635405e4-9423-4a25-a6e7-e03c8ea0bcbe@redhat.com
Reviewed-by: Heiko Carstens <hca@linux.ibm.com>
Signed-off-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Signed-off-by: Alexander Gordeev <agordeev@linux.ibm.com>
qeth_get_trap_id() allocates a temporary buffer for STSI system
information queries used to build trap identification strings.
This buffer can be allocated with kmalloc() as there's nothing special
about it to go directly to the page allocator.
kmalloc() provides a better API that does not require ugly casts and
kfree() does not need to know the size of the freed object.
Performance difference between kmalloc() and __get_free_pages() is not
measurable as both allocators take an object/page from a per-CPU list for
fast path allocations.
For the slow path the performance is anyway determined by the amount of
reclaim involved rather than by what allocator is used.
Replace use of get_zeroed_page() with kzalloc() and free_page() with
kfree().
Link: https://lore.kernel.org/all/635405e4-9423-4a25-a6e7-e03c8ea0bcbe@redhat.com
Acked-by: Alexandra Winter <wintera@linux.ibm.com>
Signed-off-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Reviewed-by: Heiko Carstens <hca@linux.ibm.com>
Signed-off-by: Alexander Gordeev <agordeev@linux.ibm.com>
hvc_iucv_alloc() allocates a send staging buffer for accumulating
outbound terminal characters before they are copied into a separate
IUCV message buffer for transmission to the hypervisor. The staging
buffer itself is never passed to any IUCV function.
This buffer can be allocated with kmalloc() as there's nothing special
about it to go directly to the page allocator.
kmalloc() provides a better API that does not require ugly casts and
kfree() does not need to know the size of the freed object.
Performance difference between kmalloc() and __get_free_pages() is not
measurable as both allocators take an object/page from a per-CPU list for
fast path allocations.
For the slow path the performance is anyway determined by the amount of
reclaim involved rather than by what allocator is used.
Replace use of get_zeroed_page() with kzalloc() and free_page() with
kfree().
Link: https://lore.kernel.org/all/635405e4-9423-4a25-a6e7-e03c8ea0bcbe@redhat.com
Reviewed-by: Heiko Carstens <hca@linux.ibm.com>
Signed-off-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Signed-off-by: Alexander Gordeev <agordeev@linux.ibm.com>
DASD driver uses get_zeroed_page() to allocate pages for the Extended Error
Reporting software ring buffer and for a scratch buffer for formatting
sense dump diagnostic text.
These buffers can be allocated with kmalloc() as there's nothing special
about it to go directly to the page allocator.
kmalloc() provides a better API that does not require ugly casts and
kfree() does not need to know the size of the freed object.
Performance difference between kmalloc() and __get_free_pages() is not
measurable as both allocators take an object/page from a per-CPU list for
fast path allocations.
For the slow path the performance is anyway determined by the amount of
reclaim involved rather than by what allocator is used.
Replace use of get_zeroed_page() with kzalloc() and free_page() with
kfree().
Link: https://lore.kernel.org/all/635405e4-9423-4a25-a6e7-e03c8ea0bcbe@redhat.com
Signed-off-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Reviewed-by: Heiko Carstens <hca@linux.ibm.com>
Signed-off-by: Alexander Gordeev <agordeev@linux.ibm.com>
con3270_alloc_view() allocates a staging buffer used to assemble
3270 datastream content before it is copied into channel program
requests.
This buffer can be allocated with kmalloc() as there's nothing special
about it to go directly to the page allocator.
kmalloc() provides a better API that does not require ugly casts and
kfree() does not need to know the size of the freed object.
Performance difference between kmalloc() and __get_free_pages() is not
measurable as both allocators take an object/page from a per-CPU list for
fast path allocations.
For the slow path the performance is anyway determined by the amount of
reclaim involved rather than by what allocator is used.
Replace use of __get_free_page() with kmalloc() and free_page() with
kfree().
Link: https://lore.kernel.org/all/635405e4-9423-4a25-a6e7-e03c8ea0bcbe@redhat.com
Reviewed-by: Heiko Carstens <hca@linux.ibm.com>
Signed-off-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Signed-off-by: Alexander Gordeev <agordeev@linux.ibm.com>
Move GR_NUM / VX_NUM macros to separate insn-common-asm.h header file
so they can be reused for non-fpu insn constructs.
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
Reviewed-by: Steffen Eiden <seiden@linux.ibm.com>
Signed-off-by: Alexander Gordeev <agordeev@linux.ibm.com>
Use the ".irp" directive to get rid of all the repeated ".ifc" usages
in fpu-insn-asm.h.
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
Reviewed-by: Steffen Eiden <seiden@linux.ibm.com>
Signed-off-by: Alexander Gordeev <agordeev@linux.ibm.com>
Rearrange some fields within AP and zcrypt structs to reduce
memory consumption and unused holes with the help of pahole
analysis of the code.
Signed-off-by: Harald Freudenberger <freude@linux.ibm.com>
Reviewed-by: Finn Callies <fcallies@linux.ibm.com>
Reviewed-by: Holger Dengler <dengler@linux.ibm.com>
Signed-off-by: Alexander Gordeev <agordeev@linux.ibm.com>
In various code paths, page_table_check_pte_clear() is called
before converting a secure page, while in others it is called
after. Make this consistent and always perform the conversion
after the PTC hook has been called. Also make all conversion‑
eligibility condition checks look the same, and rework the one
in ptep_get_and_clear_full() slightly.
Acked-by: Claudio Imbrenda <imbrenda@linux.ibm.com>
Signed-off-by: Alexander Gordeev <agordeev@linux.ibm.com>
strlcat() shouldn't be used anymore (see fortify-string.h), and will be
deprecated / removed sooner or later [1].
Therefore remove the s390 implementation of strlcat() in favor of the
generic variant.
[1] https://lore.kernel.org/all/20260514160719.105084-3-manuelebner@mailbox.org/
Reviewed-by: Alexander Gordeev <agordeev@linux.ibm.com>
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
Signed-off-by: Alexander Gordeev <agordeev@linux.ibm.com>
In order to allow comparing trace timestamps between different
systems or virtual machines on s390, add a s390-tod trace clock.
This clock just uses the returned TOD clock value from stcke
directly.
Signed-off-by: Sven Schnelle <svens@linux.ibm.com>
Reviewed-by: Ilya Leoshkevich <iii@linux.ibm.com>
Reviewed-by: Heiko Carstens <hca@linux.ibm.com>
Signed-off-by: Alexander Gordeev <agordeev@linux.ibm.com>
Remove unconditional 'n' defaults from def_tristate statements,
as they override the later 'KUNIT_ALL_TESTS' default, rendering
it dead Kconfig code.
This dead code was identified by kconfirm, a static analysis tool
for Kconfig.
Also include S390_KPROBES_SANITY_TEST in KUNIT_ALL_TESTS.
Signed-off-by: Julian Braha <julianbraha@gmail.com>
Signed-off-by: Alexander Gordeev <agordeev@linux.ibm.com>
Replace symbolic permissions with octal permissions, which are preferred.
Reviewed-by: Christian Borntraeger <borntraeger@linux.ibm.com>
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
Signed-off-by: Alexander Gordeev <agordeev@linux.ibm.com>
allmodconfig with clang W=1 points out unused global variables:
drivers/s390/char/zcore.c:49:23: error: variable
'zcore_reipl_file' set but not used [-Werror,-Wunused-but-set-global]
drivers/s390/char/zcore.c:50:23: error: variable
'zcore_hsa_file' set but not used [-Werror,-Wunused-but-set-global]
Remove both of them, since there is no point in keeping them.
Reviewed-by: Christian Borntraeger <borntraeger@linux.ibm.com>
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
Signed-off-by: Alexander Gordeev <agordeev@linux.ibm.com>
allmodconfig with clang W=1 points out an unused global variable:
drivers/s390/char/sclp_vt220.c:85:12: error: variable
'sclp_vt220_buffered_chars' set but not used [-Werror,-Wunused-but-set-global]
Just remove it.
Reviewed-by: Christian Borntraeger <borntraeger@linux.ibm.com>
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
Signed-off-by: Alexander Gordeev <agordeev@linux.ibm.com>
allmodconfig with clang W=1 points out an unused global variable:
arch/s390/kernel/debug.c:1237:33: error: variable
's390dbf_sysctl_header' set but not used [-Werror,-Wunused-but-set-global]
Just remove the variable. There is no point in adding error handling for a
failing register_sysctl() call.
Reviewed-by: Christian Borntraeger <borntraeger@linux.ibm.com>
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
Signed-off-by: Alexander Gordeev <agordeev@linux.ibm.com>
allmodconfig with clang W=1 points out an unused global variable:
arch/s390/appldata/appldata_base.c:54:33: error: variable
'appldata_sysctl_header' set but not used [-Werror,-Wunused-but-set-global]
Just remove the variable. There is no point in adding error handling for a
failing register_sysctl() call.
Acked-by: Christian Borntraeger <borntraeger@linux.ibm.com>
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
Signed-off-by: Alexander Gordeev <agordeev@linux.ibm.com>
There are many loops in the form of
while (READ_ONCE(*somelocation))
cpu_relax();
Strictly speaking the architecture requires serialization instead of only a
compiler barrier in the loop so the READ_ONCE() will see an updated value.
However real hardware does not require this (see IBM z Systems Processor
Optimization Primer - FAQ [1]), but it is still recommended to add
serialization. Given that cpu_relax() is doing nothing useful, it does
not hurt to add the single and fast instruction which makes sure that
serialization happens, and such loops may be left a bit faster.
[1] https://community.ibm.com/community/user/viewdocument/microprocessor-optimization-primer
Reviewed-by: Alexander Gordeev <agordeev@linux.ibm.com>
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
Signed-off-by: Alexander Gordeev <agordeev@linux.ibm.com>
cpu_relax() is defined identically at two different locations.
Just like most other architectures remove the implementation at
asm/processor.h and only use the one at asm/vdso/processor.h,
avoiding code duplication.
Reviewed-by: Alexander Gordeev <agordeev@linux.ibm.com>
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
Signed-off-by: Alexander Gordeev <agordeev@linux.ibm.com>
Use an alternative to implement bcr_serialize() and use alternative
patching to select between serialization and fast-serialization
depending on the corresponding facility bit.
Reviewed-by: Jan Polensky <japo@linux.ibm.com>
Reviewed-by: Alexander Gordeev <agordeev@linux.ibm.com>
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
Signed-off-by: Alexander Gordeev <agordeev@linux.ibm.com>
On arm64 the empty zero page is going to be mapped read-only [1].
Do the same for s390 with an explicit set_memory_ro() call.
[1] https://lore.kernel.org/all/20260427153416.2103979-19-ardb+git@google.com/
Suggested-by: Christian Borntraeger <borntraeger@linux.ibm.com>
Acked-by: Alexander Gordeev <agordeev@linux.ibm.com>
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
Signed-off-by: Alexander Gordeev <agordeev@linux.ibm.com>
Problem determination using s390dbf logging sometimes requires changing
the default logging level or log area size. While this is possible
using sysfs interfaces, there is no easy way to adjust these parameters
for early boot code that emits logs before userspace is available.
Add an s390dbf kernel parameter to address this shortcoming. The
parameter can be used to specify log level and area size (in units of
pages). A level of '-' turns logging off for an area. Logs can be
identified by name or a shell-style pattern.
Parameter format:
s390dbf=<name|pattern>:[<level>|-]:[<pages>][,...]
Example:
s390dbf=cio*:6:128,sclp_err::2
Specified parameters are applied immediately during debug area
registration for regular log areas. For early, static debug areas,
log levels are changed during early_param() parsing, while size
changes are applied at arch_initcall-time.
Signed-off-by: Peter Oberparleiter <oberpar@linux.ibm.com>
Tested-by: Vineeth Vijayan <vneethv@linux.ibm.com>
Reviewed-by: Heiko Carstens <hca@linux.ibm.com>
Signed-off-by: Alexander Gordeev <agordeev@linux.ibm.com>
Notify userspace about two important events on AP queues
when run within Secure Execution (SE) environment:
- Send AP CHANGE uevent with "SE_BIND=1" on successful bind
operation on this AP queue device.
- Send AP CHANGE uevent with "SE_ASSOC=<association_index>"
on successful association operation with the secret of the
reported index on this AP queue device.
Note there is no SE unbind/unassociate event. Unbind/unassociate
can have different triggers and technically there is no signaling
done which the AP code could catch. A user space application can,
if this information is crucial, query the sysfs attribute se_bind
on the AP queue which runs a synchronous TAPQ. If the attribute
returns with "unbound" a reset took place and SE bind and associate
states are unbound and unassociated.
Suggested-by: Marc Hartmayer mhartmay@linux.ibm.com
Signed-off-by: Harald Freudenberger <freude@linux.ibm.com>
Reviewed-by: Holger Dengler <dengler@linux.ibm.com>
Signed-off-by: Alexander Gordeev <agordeev@linux.ibm.com>
The new SCLP action qualifier 4 is used by user-space code to provide
NVMe SMART log data to the platform.
Signed-off-by: Niklas Schnelle <schnelle@linux.ibm.com>
Acked-by: Heiko Carstens <hca@linux.ibm.com>
Reviewed-by: Peter Oberparleiter <oberpar@linux.ibm.com>
Signed-off-by: Alexander Gordeev <agordeev@linux.ibm.com>
Ensure that all devices currently offline are purged correctly.
Previously, purging logic relied on the internal FSM state to
determine whether a device was offline. However, devices with a
target state of offline could be skipped if CIO internal
processing was still ongoing during the purge operation.
Update the purge decision logic to rely on the online variable
in the cdev structure instead of the internal FSM state,
providing a more reliable indication of actual device
availability.
Signed-off-by: Vineeth Vijayan <vneethv@linux.ibm.com>
Reviewed-by: Peter Oberparleiter <oberpar@linux.ibm.com>
Signed-off-by: Alexander Gordeev <agordeev@linux.ibm.com>
There is no need to call task_stack_page(),
because try_get_task_stack() already takes care of that.
Signed-off-by: Maninder Singh <maninder1.s@samsung.com>
Reviewed-by: Heiko Carstens <hca@linux.ibm.com>
Signed-off-by: Alexander Gordeev <agordeev@linux.ibm.com>
