- Expose btf_ids to trace events
In order to allow BPF programs to attach to system call trace events (which
are actually pseudo trace events built on top of raw_syscall events),
expose the BTF ID of the events. This will allow BPF programs better
precision in attaching to events.
- Use "u64" to assign to hist_field->type
Instead of using kstrdup("u64", GFP_KERNEL) to assign the hist_field->type,
just point it to "u64" instead. The hist_field->type is freed via
kfree_const().
- Replace kmalloc()/strcpy() with kstrdup() for trace_printk
Instead of having two calls to copy the module format string, just use
kstrdup().
- Use __free() in trace event histograms and triggres where possible
- Use seq_buf in trace event code instead of strcat()
Instead of calculating the size of the buffer to use and filling it with
strcat(), use the seq_buf infrastructure that takes care of making sure
not to overflow the string size.
- Reject invalid preemptirq_delay_test CPU affinity
The preempt_delay_test module can take an invalid CPU affinity mask and
create confusing output. Simply have the module reject invalid affinity
masks.
- Prevent division by zero in ftrace_ops sample module code
If the ftrace_ops sample module code receives the module parameter
nr_function_calls set to zero, it can cause a division by zero error.
- Warn when an event dereferences a parameter in TP_printk()
On boot up and module load, the trace event TP_printk() is scanned for
possible bugs. As the TP_printk() code is executed when the user reads the
"trace" file and processes the data written when the trace_event executed,
the data it reads can be literally days old. The scan currently checks for
dereferencing printk formats like "%pI6". But it does not check if the
parameters themselves have a dereference like:
TP_printk("offset %08x: value %08x",
(u32)(__entry->addr - __entry->edma->membase), __entry->value)
__entry represents the pointer to the event on the ring buffer. The
__entry->edma->membase is dereferencing a pointer on the ring buffer to
find membase, but the __entry->edma may no longer be a valid pointer.
Warn on this case too.
- Replace some strcpy() with strscpy()
- Clean up mmiotrace events to use assign_type() macro
The assign_type() macro makes sure the event type is indeed the type that
is being parsed. The mmiotrace trace was written before that macro was
created so it just simply typecasted the pointer.
Replace the typecasting with the macro.
- Have the ENUM processing to numbers only process what is added
The code that converts ENUMs to their numbers in the trace events scanned
all events to do the processing. This was true when a module was loaded
too. That is, instead of processing just the events for the module, it
processed *all* events. Even the builtin ones that were processed at boot
up.
Add a check for the event->module matching mod if it is a module before
processing it.
-----BEGIN PGP SIGNATURE-----
iIoEABYKADIWIQRRSw7ePDh/lE+zeZMp5XQQmuv6qgUCan+WhRQccm9zdGVkdEBn
b29kbWlzLm9yZwAKCRAp5XQQmuv6qk1sAQDIvuux39ii379T9hTm7hgXF9Dr8aH7
RgxILUmr3c4y6QD+Iv9QQto+NfauqjkEMdPuVdOyUpDCYPTRw9i/YCty7Qk=
=vya7
-----END PGP SIGNATURE-----
Merge tag 'trace-v7.3' of git://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace
Pull tracing updates from Steven Rostedt:
- Expose btf_ids to trace events
In order to allow BPF programs to attach to system call trace events
(which are actually pseudo trace events built on top of raw_syscall
events), expose the BTF ID of the events. This will allow BPF
programs better precision in attaching to events.
- Use "u64" to assign to hist_field->type
Instead of using kstrdup("u64", GFP_KERNEL) to assign the
hist_field->type, just point it to "u64" instead. The
hist_field->type is freed via kfree_const().
- Replace kmalloc()/strcpy() with kstrdup() for trace_printk
Instead of having two calls to copy the module format string, just
use kstrdup().
- Use __free() in trace event histograms and triggres where possible
- Use seq_buf in trace event code instead of strcat()
Instead of calculating the size of the buffer to use and filling it
with strcat(), use the seq_buf infrastructure that takes care of
making sure not to overflow the string size.
- Reject invalid preemptirq_delay_test CPU affinity
The preempt_delay_test module can take an invalid CPU affinity mask
and create confusing output. Simply have the module reject invalid
affinity masks.
- Prevent division by zero in ftrace_ops sample module code
If the ftrace_ops sample module code receives the module parameter
nr_function_calls set to zero, it can cause a division by zero error.
- Warn when an event dereferences a parameter in TP_printk()
On boot up and module load, the trace event TP_printk() is scanned
for possible bugs. As the TP_printk() code is executed when the user
reads the "trace" file and processes the data written when the
trace_event executed, the data it reads can be literally days old.
The scan currently checks for dereferencing printk formats like
"%pI6". But it does not check if the parameters themselves have a
dereference like:
TP_printk("offset %08x: value %08x",
(u32)(__entry->addr - __entry->edma->membase), __entry->value)
__entry represents the pointer to the event on the ring buffer. The
__entry->edma->membase is dereferencing a pointer on the ring buffer
to find membase, but the __entry->edma may no longer be a valid
pointer.
Warn on this case too.
- Replace some strcpy() with strscpy()
- Clean up mmiotrace events to use assign_type() macro
The assign_type() macro makes sure the event type is indeed the type
that is being parsed. The mmiotrace trace was written before that
macro was created so it just simply typecasted the pointer.
Replace the typecasting with the macro.
- Have the ENUM processing to numbers only process what is added
The code that converts ENUMs to their numbers in the trace events
scanned all events to do the processing. This was true when a module
was loaded too. That is, instead of processing just the events for
the module, it processed *all* events. Even the builtin ones that
were processed at boot up.
Add a check for the event->module matching mod if it is a module
before processing it.
* tag 'trace-v7.3' of git://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace: (21 commits)
tracing: Have trace_event_update_all() only handle module that is loading
tracing: Cleanup event_enable_trigger_parse() by using __free()
tracing: Report every TP_printk double dereference
tracing/mmiotrace: Use trace_assign_type() in mmio_print_mark()
tracing: Make per-template BTF id lists file-local
tracing: Use seq_buf for string concatenation
tracing: Use strscpy() instead of strcpy() in trace_sched_switch
tracing: Warn when an event dereferences a pointer in TP_printk()
samples/ftrace: Prevent division by zero when nr_function_calls is zero
tracing: Reject invalid preemptirq_delay_test CPU affinity
fgraph: Use trace_seq_putc() in print_graph_return()
tracing/user_events: Replace a seq_printf() call by seq_puts() in user_seq_show()
tracing/user_events: Use seq_putc() in two functions
tracing: Bound histogram expression strings with seq_buf
tracing: Return ERR_PTR() from expr_str()
tracing: Use __free() for expr_str() buffer
kernel/trace/trace_printk: Use kstrdup() instead of kmalloc() and strcpy()
tracing: Point constant hist field type to string literal
selftests/bpf: Add test for tracepoint btf_ids tracefs file
tracing: Expose tracepoint BTF ids via tracefs
...
- Deprecrate ftrace_enabled in disabling ftrace
The file /proc/sys/kernel/ftrace_enabled was created when ftrace was first
introduced back in 2008. It was to be a "kill switch" if something was to
go wrong. It was also used as a way to turn off function tracing for the
latency tracers that would have it on by default. But in 2013 (Linux 3.10)
the option "function-trace" was introduced to disable function tracing for
the latency tracers as the "ftrace_enabled" file was considered too big of
a hammer and caused too many side effects.
When live kernel patching came along, disabling ftrace via the
ftrace_enabled file would put the system into an unstable state if a live
kernel patch was installed. This created the need to mark some function
hooks as "PERMANENT".
Now there's a need for BPF usage marked as PERMANENT for the same reasons.
The file "ftrace_enabled" usage is no longer viable. It doesn't do what it
says it does and there is no reason to use it.
Make writing '0' to it a nop and print a message saying its usage is
deprecated. The return value of writing '0' is -EOPNOTSUPP so that user
space will error on that write (hopefully to inform any developer that it
no longer works).
Eventually the file should be removed completely, but for now just making
it not do anything is the path forward to that.
- Update the livepatch tests to handle ftrace_enabled being disabled
Because in the past, livepatch was broken by ftrace_enabled being turned
off, there's a test case that checks to make sure it still doesn't break.
But having the write of '0' return an error caused that test to break.
Updated the test to handle the new change.
-----BEGIN PGP SIGNATURE-----
iIoEABYKADIWIQRRSw7ePDh/lE+zeZMp5XQQmuv6qgUCan9buBQccm9zdGVkdEBn
b29kbWlzLm9yZwAKCRAp5XQQmuv6qkrAAP9lStpPUUVGUVprdRttG8tv4LlSMqX3
+A5WdksIKD+XMAD+ODdXKQkjynRpTf9YnIV3PXNg6Adi3ItOXG1+UsbWGQY=
=QLhM
-----END PGP SIGNATURE-----
Merge tag 'ftrace-v7.3' of git://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace
Pull ftrace updates from Steven Rostedt:
- Deprecrate ftrace_enabled in disabling ftrace
The file /proc/sys/kernel/ftrace_enabled was created when ftrace was
first introduced back in 2008. It was to be a "kill switch" if
something was to go wrong. It was also used as a way to turn off
function tracing for the latency tracers that would have it on by
default. But in 2013 (Linux 3.10) the option "function-trace" was
introduced to disable function tracing for the latency tracers as the
"ftrace_enabled" file was considered too big of a hammer and caused
too many side effects.
When live kernel patching came along, disabling ftrace via the
ftrace_enabled file would put the system into an unstable state if a
live kernel patch was installed. This created the need to mark some
function hooks as "PERMANENT".
Now there's a need for BPF usage marked as PERMANENT for the same
reasons.
The file "ftrace_enabled" usage is no longer viable. It doesn't do
what it says it does and there is no reason to use it.
Make writing '0' to it a nop and print a message saying its usage is
deprecated. The return value of writing '0' is -EOPNOTSUPP so that
user space will error on that write (hopefully to inform any
developer that it no longer works).
Eventually the file should be removed completely, but for now just
making it not do anything is the path forward to that.
- Update the livepatch tests to handle ftrace_enabled being disabled
Because in the past, livepatch was broken by ftrace_enabled being
turned off, there's a test case that checks to make sure it still
doesn't break. But having the write of '0' return an error caused
that test to break. Updated the test to handle the new change.
* tag 'ftrace-v7.3' of git://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace:
selftests/livepatch: update test-ftrace.sh for deprecated ftrace_enabled
ftrace: deprecate disabling via ftrace_enabled sysctl
- Switch LTL and DOT parsers to Lark in code generation tool
The rvgen code generation tool originally parsed DOT files and LTL
specifications using custom string parsing and Ply, which is no longer
maintained. The DOT parser was fragile and prone to failure on minor
format variations. Both LTL and DOT parsers have been rewritten to use
the Lark parsing library.
- Simplify Hybrid Automata clock variables
The clock variables in hybrid automata monitors now use a single
representation of the elapsed time since the clock was reset, rather
than converting between invariant and guard representations.
This allows simpler code generation for the newly refactored parser.
- Generate cleanup hook for per-obj monitor
The code generation scripts now adds a cleanup function to per-obj
monitors for the user to wire to the appropriate event (e.g.
sched_process_exit for tasks).
- Reduce read_lock scope during per-task cleanup
Take the tasklist_lock only when necessary, that is when iterating
over for_each_process_thread().
- Simplify task monitor slot management
Only rely on the slot array for per-task slot management to avoid
inconsistency with the unused counter.
- Improve rvgen code robustness and templates
Use pathlib in rvgen and improve kernel path discovery. Also improve
consistency across templates when generating code (e.g. author
placeholder and monitor struct name).
- Update rtapp sleep monitor
Simplify the sleep monitor by excluding kernel threads and
updating the nanosleep check to focus only on CLOCK_REALTIME. Also
switch to use the sched_exit tracepoint to run in the context of the
offending (wakee) task.
- Add wakeup monitor
Add the new rtapp/wakeup monitor to detect when lower-priority tasks
wake up higher-priority ones, complementing the existing sleep monitor
by running in the waker context and capturing its stack trace.
- Fix tools/rv exit status on failure
Ensure the rv tool returns a failure exit code when a monitor fails to
start because it was already running.
- Add automated selftests for tools/rv and rvgen
Introduced automated bash selftests to validate rv monitor listing and
execution under different configurations. Added tests for the rvgen code
generator, validating generated files against expected output (golden).
Tests are reachable via make check.
- Add KUnit test coverage for verification monitors
Added comprehensive KUnit tests to validate the functionality of
deterministic, hybrid, and LTL monitors by emulating event sequences
and timing in a mock environment without affecting the running kernel
while expecting mock reactions to fire. Ensure real RV monitors cannot
run during KUnit tests to avoid state corruption.
- Mock current in rv monitors
Mock the call to current in rv monitors when the KUnit tests are built
to allow them to run the test on dummy tasks. No overhead is expected
when KUnit tests aren't running.
- Introduce rvgen kunit subcommand
Added a new 'kunit' subcommand to rvgen to automatically patch an already
generated monitor with KUnit integration templates by parsing its event
handlers and creating the required mock structures and initializations.
- Refine kernel verification selftests
Added new selftests for the deadline and stall monitors and rearranged
the existing wwnr_printk test to resolve flakiness.
Additionally, fixed an issue in the selftests framework where negative
assertion failures were not correctly propagated due to shell rules.
- Fix 32-bit build of nomiss KUnit test
A previous commit introduced a division between an u64 and a constant
value and that doesn't build on 32-bit systems. Use div_u64() instead.
- Document changes in sleep monitor
The sleep monitor introduced some changes in the past like allowing
epoll_wait() as a valid sleep and a task going to runnable before
scheduling as a valid wakeup. Document both.
-----BEGIN PGP SIGNATURE-----
iIoEABYKADIWIQRRSw7ePDh/lE+zeZMp5XQQmuv6qgUCan+BsRQccm9zdGVkdEBn
b29kbWlzLm9yZwAKCRAp5XQQmuv6qicKAQD5aLUnn4zrZGuT+k5qmNmmbOsAXriS
g+TfUXShvRoL6QD+KrsUyNf9KzLd+IqJG1N3G3wbhOiVd60Y5srLZP87UgA=
=iqRx
-----END PGP SIGNATURE-----
Merge tag 'trace-rv-v7.3' of git://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace
Pull Real-time Verifier updates from Steven Rostedt:
- Switch LTL and DOT parsers to Lark in code generation tool
The rvgen code generation tool originally parsed DOT files and LTL
specifications using custom string parsing and Ply, which is no
longer maintained. The DOT parser was fragile and prone to failure on
minor format variations. Both LTL and DOT parsers have been rewritten
to use the Lark parsing library.
- Simplify Hybrid Automata clock variables
The clock variables in hybrid automata monitors now use a single
representation of the elapsed time since the clock was reset, rather
than converting between invariant and guard representations. This
allows simpler code generation for the newly refactored parser.
- Generate cleanup hook for per-obj monitor
The code generation scripts now adds a cleanup function to per-obj
monitors for the user to wire to the appropriate event (e.g.
sched_process_exit for tasks).
- Reduce read_lock scope during per-task cleanup
Take the tasklist_lock only when necessary, that is when iterating
over for_each_process_thread().
- Simplify task monitor slot management
Only rely on the slot array for per-task slot management to avoid
inconsistency with the unused counter.
- Improve rvgen code robustness and templates
Use pathlib in rvgen and improve kernel path discovery. Also improve
consistency across templates when generating code (e.g. author
placeholder and monitor struct name).
- Update rtapp sleep monitor
Simplify the sleep monitor by excluding kernel threads and updating
the nanosleep check to focus only on CLOCK_REALTIME. Also switch to
use the sched_exit tracepoint to run in the context of the offending
(wakee) task.
- Add wakeup monitor
Add the new rtapp/wakeup monitor to detect when lower-priority tasks
wake up higher-priority ones, complementing the existing sleep
monitor by running in the waker context and capturing its stack
trace.
- Fix tools/rv exit status on failure
Ensure the rv tool returns a failure exit code when a monitor fails
to start because it was already running.
- Add automated selftests for tools/rv and rvgen
Introduced automated bash selftests to validate rv monitor listing
and execution under different configurations. Added tests for the
rvgen code generator, validating generated files against expected
output (golden). Tests are reachable via make check.
- Add KUnit test coverage for verification monitors
Added comprehensive KUnit tests to validate the functionality of
deterministic, hybrid, and LTL monitors by emulating event sequences
and timing in a mock environment without affecting the running kernel
while expecting mock reactions to fire. Ensure real RV monitors
cannot run during KUnit tests to avoid state corruption.
- Mock current in rv monitors
Mock the call to current in rv monitors when the KUnit tests are
built to allow them to run the test on dummy tasks. No overhead is
expected when KUnit tests aren't running.
- Introduce rvgen kunit subcommand
Added a new 'kunit' subcommand to rvgen to automatically patch an
already generated monitor with KUnit integration templates by parsing
its event handlers and creating the required mock structures and
initializations.
- Refine kernel verification selftests
Added new selftests for the deadline and stall monitors and
rearranged the existing wwnr_printk test to resolve flakiness.
Additionally, fixed an issue in the selftests framework where
negative assertion failures were not correctly propagated due to
shell rules.
- Fix 32-bit build of nomiss KUnit test
A previous commit introduced a division between an u64 and a constant
value and that doesn't build on 32-bit systems. Use div_u64()
instead.
- Document changes in sleep monitor
The sleep monitor introduced some changes in the past like allowing
epoll_wait() as a valid sleep and a task going to runnable before
scheduling as a valid wakeup. Document both.
* tag 'trace-rv-v7.3' of git://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace: (40 commits)
Documentation/rv: Explain epoll and aborted sleeps
rv: Fix 32-bit build of nomiss KUnit test
selftests/verification: Add selftests for deadline and stall monitors
selftests/verification: Rearrange the wwnr_printk test
selftests/verification: Fix wrong errexit assumption
rv: Add KUnit tests for some LTL monitors
rv: Add KUnit mock for current
rv: Add KUnit tests for some DA/HA monitors
rv: Export task monitor slot and react symbols
verification/rvgen: Add selftests for rvgen kunit
verification/rvgen: Add the rvgen kunit subcommand
verification/rvgen: Add selftests
verification/rvgen: Add golden and spec folders for tests
tools/rv: Add selftests
verification/rvgen: Improve consistency in template files
verification/rvgen: Use pathlib instead of os.path
verification/rvgen: Improve rv_dir discovery in RVGenerator
tools/rv: Fix exit status when monitor execution fails
rv: Use generic rv_this for the rv_monitor variable in LTL
rv/rtapp: Add wakeup monitor
...
bh_pool_kick_normal() and bh_pool_kick_highpri() are registered via
init_irq_work() without the IRQ_WORK_HARD_IRQ flag. On PREEMPT_RT, such
irq_work items are processed by the per-CPU irq_workd kthread in preemptible
task context with IRQs enabled. However, raise_softirq_irqoff() requires
IRQs to be disabled. Calling it from irq_workd trips the lockdep assertion
in __raise_softirq_irqoff() and the non-atomic update of the softirq pending
mask can lose bits raised by an interrupt on the same CPU. Replace
raise_softirq_irqoff() with raise_softirq() in the irq_work handlers.
Fixes: 2f34d7337d ("workqueue: Fix queue_work_on() with BH workqueues")
Cc: stable@vger.kernel.org # v6.9+
Signed-off-by: Zqiang <qiang.zhang@linux.dev>
Signed-off-by: Tejun Heo <tj@kernel.org>
- Consolidate the VDSO datastore further and provide support for
mlock_all() and prefaulting.
- Provide 32-bit legacy time related functionality only if
CONFIG_COMPAT_32BIT_TIME is enabled. The config switch exists, but
architecture code still exposes the legacy functionality even disabled.
Clean this up by adding the missing guards and validating at build time
that the VDSO is legacy free if disabled.
- Consolidate the VDSO related config options in core and drivers, which
removes some non-sensical dependencies and quite an amount of #ifdeffery.
- Clean up the PAGE_SIZE definition maze
-----BEGIN PGP SIGNATURE-----
iQJEBAABCgAuFiEEQp8+kY+LLUocC4bMphj1TA10mKEFAmqCxfkQHHRnbHhAa2Vy
bmVsLm9yZwAKCRCmGPVMDXSYoSQkEADEOmcrm3vwMbaElrAfQRp4MOQMnDpKPCDa
0EwdkXKJyqfOGx1PNKbcWZe10qShpXqyz+q3MUbHiOVh7KxkaGj0GOA/+xD+Tgs5
RAFXCq+CY4Ccjix7vlyOBLvGgomuznAtRDCFprVq5OTXyQfVvS++5SHzaU/0xXiY
ZZEInrFP61BEJ8LJycpoN5431EBcUX7XpSkr9eHtuSOslOBzbTEtBbDp0AjDGse0
5qM/oZDufZDQQr31kVQiL0dN9bBWwdGNtBwZBjsf6KioY9M00LO0E9twSCSrI39F
kgD2N97RDjTewu/zEe0fvCDVypqel2mBvjfiYU4c8g6pprdJ4Hq1qMcDAtMBUDBZ
9jR8B09eYZwHLdbM5PFjiMf9d6p25P7X+twFI9Hylcm1YZtM4OTnamDhQOQDp1Fd
QdrFevXg1zvYVi1rrDYarcPfg/tGsetOTujMQ1m50/AJZOoEdcX9jZR/wfdW8ti6
X9TIn3SDzhauXkmeR86tHxtOmHDirVfBWCCCEyVT2YzGIIje25cf266fCPdcGRYl
kzhtycg8LqpTRMWjhlaf56NtD9qQlbvDY8L23tQLJUYhDp8xOxqyuRPRdpJVM0b3
4vp/Z8Sgz8tpEHO2ARsXfuu2CzilTBH0B2KL3f+a8CwpzIEadjIfc+6EEYqwgOAn
+/RhckFp/A==
=jHl1
-----END PGP SIGNATURE-----
Merge tag 'timers-vdso-2026-08-17' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
Pull VDSO updates from Thomas Gleixner:
- Consolidate the VDSO datastore further and provide support for
mlock_all() and prefaulting.
- Provide 32-bit legacy time related functionality only if
CONFIG_COMPAT_32BIT_TIME is enabled. The config switch exists,
but architecture code still exposes the legacy functionality even
disabled.
Clean this up by adding the missing guards and validating at build
time that the VDSO is legacy free if disabled.
- Consolidate the VDSO related config options in core and drivers,
which removes some non-sensical dependencies and quite an amount of
#ifdeffery.
- Clean up the PAGE_SIZE definition maze
* tag 'timers-vdso-2026-08-17' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip: (30 commits)
random: vDSO: Drop custom PAGE_SIZE definitions
LoongArch: Remove CONFIG_GENERIC_GETTIMEOFDAY ifdeffery
clocksource/drivers/timer-riscv: Remove CONFIG_GENERIC_GETTIMEOFDAY ifdeffery
clocksource/drivers/arm_arch_timer: Remove CONFIG_GENERIC_GETTIMEOFDAY ifdeffery
clocksource/drivers/mips-gic-timer: Remove CONFIG_GENERIC_GETTIMEOFDAY ifdeffery
MIPS: csrc-r4k: Remove CONFIG_GENERIC_GETTIMEOFDAY ifdeffery
vDSO: Make clockmode constants available without CONFIG_GENERIC_GETTIMEOFDAY
kbuild: Support generated asm-headers in subdirectories
vdso: Rename HAVE_GENERIC_VDSO to VDSO_DATASTORE
vdso: Drop HAVE_GENERIC_VDSO from architecture kconfig files
vdso: Automatically select HAVE_GENERIC_VDSO if necessary
MIPS: vdso: Stop using CONFIG_HAVE_GENERIC_VDSO
vdso: Remove the dependency on HAVE_GENERIC_VDSO from ARCH_HAS_VDSO_ARCH_DATA
futex: Remove dependency on HAVE_GENERIC_VDSO from FUTEX_ROBUST_UNLOCK
vdso/gettimeofday: Verify COMPAT_32BIT_TIME interactions
sparc: vdso: Respect COMPAT_32BIT_TIME
MIPS: VDSO: Respect COMPAT_32BIT_TIME
powerpc/vdso: Respect COMPAT_32BIT_TIME
ARM: VDSO: Respect COMPAT_32BIT_TIME
arm64: vdso32: Respect COMPAT_32BIT_TIME
...
- Fix a subtly inconsistency in the timekeeping code, which fails to
account for the monotonicity adjustment in ntp_error. For small changes
of the clocksource multiplicator (+/-1) which are typically used by the
NTP PLL this is hardly to observe. But for larger adjustments,
e.g. caused by a direct frequency setting through adjtimex() the
one-time uncompensated offset is significant.
Cure this by adjusting ntp_error with the resulting offset so that the
discrepancy is smoothed away over time
- Make tick length calculations correct in NTP. The timekeeping core
takes the quantisation of the clocksource into account when calculating
the tick length to compensate for the deviation of the nominal
NTP_INTERVAL_LENGTH. While timekeeping gets this right, NTP is not
aware of that, which means it operates on the nominal value and not on
the actual value which is determined by the clock source frequency. The
rounding of a coarse clocksource like the ACPI PM timer results in a
+127 PPM deviation.
Cure this by exposing the deviation to the NTP code so that it can
operate on the same data as the timekeeping core. This is purely kernel
internal. User space still sees the nominal tick lenght via adjtimex().
- The accuracy of the NTP adjustments is fairly approximate as the code
assumes that the invocations are precisely in NTP interval frequency
ticks and the final adjustment can over and under-run.
Cure this by adjusting ntp_error by the intended skew on each tick to
achieve the desired rate.
- Handle the two competing skews of time offset and time adjustment
correctly by calculating the conflict portion between the skews and
adjusting both accordingly.
- A set of updates and improvements for the selftests
- The usual small fixes and improvements all over the place
-----BEGIN PGP SIGNATURE-----
iQJEBAABCgAuFiEEQp8+kY+LLUocC4bMphj1TA10mKEFAmqCwxwQHHRnbHhAa2Vy
bmVsLm9yZwAKCRCmGPVMDXSYoaOwD/45XBW2s5/L+GSy52r/APrzYOqgjYQdn2UQ
XI85iTjDqg9sfZQNFbr7toJ84gY/wQ0ZxykLMi+LlOSZVe5azSepoUHMg9XGk8S0
4kI+GLznmnDLFypTezpzk3DUgNgu8rhOYvPsgwQEgx0DuzWHRZgNoNDk4AfJ8HP6
8cRgPIJ7CpIbP7/p0Kc5IjW3ybo849wBirl7ZDxNRRao+Dkc56ArZvSfQgnQu5Kw
D3Ei+AFcxpBQ2GEJT4Z/92yNf1x2AXjkhfijgO6hIFzPAR+c049fG5sgneo3GoLQ
riduEPQHvoiJO3gVGzoICF2DAuVOwMHwIBlGtNEDd+FcMAvMU4V3lQi8lJV+9CpH
SJwa5R6KJHg2aTuoPup8lmfJN7BqmthrkZwcYVrv2Ikt5yWGYuoW0/e00So9IfUO
R7UG/VlruckK+T0GCuKBAAfm/+zm6Sbir+4liPDZwDz4dW2lkvkZnAb6xHPCev8S
ioYL/Jqr3GmIMgK7UelUkQ/gy5YCek89JAszdlD+blLAjUMcDeYsZaAN9rSdjzSu
dbEA4Hze6Q8HpIW35bHkytyyjNNhvL6xobjKBNRuOPUHTFMe+s5itMApRlhsu32J
6EL/D1TuwkiHlqaSek7ZqUAiitZoNWxa1NyreM9EuZ7qWPjUWOzlsDXkP2v298+t
p27xt+/8sA==
=prJt
-----END PGP SIGNATURE-----
Merge tag 'timers-core-2026-08-17' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
Pull timer and timekeeping core updates from Thomas Gleixner:
- Fix a subtly inconsistency in the timekeeping code, which fails to
account for the monotonicity adjustment in ntp_error.
For small changes of the clocksource multiplicator (+/-1) which are
typically used by the NTP PLL this is hard to observe. But for larger
adjustments, e.g. caused by a direct frequency setting through
adjtimex() the one-time uncompensated offset is significant.
Cure this by adjusting ntp_error with the resulting offset so that
the discrepancy is smoothed away over time
- Make tick length calculations correct in NTP.
The timekeeping core takes the quantisation of the clocksource into
account when calculating the tick length to compensate for the
deviation of the nominal NTP_INTERVAL_LENGTH.
While timekeeping gets this right, NTP is not aware of that, which
means it operates on the nominal value and not on the actual value
which is determined by the clock source frequency. The rounding of a
coarse clocksource like the ACPI PM timer results in a +127 PPM
deviation.
Cure this by exposing the deviation to the NTP code so that it can
operate on the same data as the timekeeping core. This is purely
kernel internal. User space still sees the nominal tick lenght via
adjtimex().
- The accuracy of the NTP adjustments is fairly approximate as the code
assumes that the invocations are precisely in NTP interval frequency
ticks and the final adjustment can over and under-run.
Cure this by adjusting ntp_error by the intended skew on each tick to
achieve the desired rate.
- Handle the two competing skews of time offset and time adjustment
correctly by calculating the conflict portion between the skews and
adjusting both accordingly.
- A set of updates and improvements for the selftests
- The usual small fixes and improvements all over the place
* tag 'timers-core-2026-08-17' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip: (58 commits)
selftests: timers: nsleep-lat: Check all calls to clock_nanosleep() and clock_gettime()
selftests: timers: nsleep-lat: Reuse kselftest error numbers
selftests: timers: nsleep-lat: Explicitly list the tested clocks
selftests: timers: nsleep-lat: Use NSEC_PER_MSEC define for unreasonable latency
selftests: timers: nanosleep: Report each test separately
selftests: timers: nanosleep: Explicitly handle timer_delete() failure
selftests: timers: nanosleep: Move all single clock tests out of the loop in main()
selftests: timers: nanosleep: Reuse kselftest error numbers
selftests: timers: nanosleep: Explicitly list the tested clocks
selftests: timers: nanosleep: Drop output alignment
selftests: timers: Use clock_name() and constants from clock-helpers.h
selftests: Add clock-helpers.h
timer_list: Use ktime_t over nanoseconds
timer_list: Use standard 'long long' format placeholders
hrtimer: Add a lockdep assertion to hrtimer_update_base()
timekeeping: Use u32 for clock_was_set_seq
timekeeping: Rename clockid_aux_valid() to clockid_is_aux_clock()
hrtimer: Account nr_retries on recovered interrupt retries
timers/itimer: Zero-init old itimerval before copy to userspace
nohz: Replace dead select with choice default
...
- Reduce the preemption disabled sections in smp_call_function*().
The various smp call functions keep preemption disabled accross the
full operation which includes the wait for completion. Especially the
latter can take some time when one of the target CPUs is not
immediately responding to the IPI, which can result in large latency
spikes.
To improve this provide a per task CPU mask to track the CPUs to wait
for. That makes the information required for the wait task local and
therefore allows to reenable preemption before the wait. While this
comes with moderate extra memory cost this reduces SMP function call
induced latency measured in a fleet for high priority tasks from ~17ms
to ~1.5ms (~90%).
- Reduce the overhead of the CSD debug code by replacing the heavy memory
barriers with smp_store_release()/acquire().
- Remove obsolute unused hotplug states
-----BEGIN PGP SIGNATURE-----
iQJEBAABCgAuFiEEQp8+kY+LLUocC4bMphj1TA10mKEFAmqC7pcQHHRnbHhAa2Vy
bmVsLm9yZwAKCRCmGPVMDXSYoTL1D/9dgn7nuKCC4iwnihq2gZnM39780kNgsRCB
9lAGAEEr1u2iciYIA+oXMqVLBGqrOkpCNMcChUqnyrH5H0coOgDFICTWTM4qLZxi
ujPRcbwRDFqoHrd7PcHb7z6z9OIyiUQBRbQAJjk6+QnfA6IDbqcYB9RI2FrEexfG
Wojl2VX1vITd4q55bhSdbUHHYm+QOP5cpBfUSkS1QjkH577kCJ0kkLhYg/8fuakr
w1RaqU35I0W6z0AOh709vLxvchb60rMMX5/UkDHrRJx11gceimpQ+MTaz3PvZZmK
djzCip9mLcZryP5bgdfJTVTGtEaQj0ggnDMmi2MiAuJhcvqokjaZpYfH/+wiNDuh
vaNK+oZx9E3A3AKXs7MeXm9vqSh+nEHN4UcXosdT+gZqGrHTh4Mj4MT0gDwDzXXY
A8WB3K95PJkucaqtXU+6bZZeVj7Pzc8tXySr7GUlxRipW/6ZJsz99s+mIG4zvMjo
Y11pJIH7PLhnuAffMdgpmuQfiq6DKaGCjvWertB4TZPm+l/aBMvoA1QZbfNrUUPp
Gf/IMHqcb3fmurgChn1bSlm9HDktjPzlmLWhHaVDJFrYZLYmbIzXzUNQCrBAuPQW
Ui8BBOE2c5sn8bO411rxFOK3JETH/BBCNxmY4YQZAlU0na2/rVJ7ofKhT3zKMijd
3M6BaQAxsw==
=0b4v
-----END PGP SIGNATURE-----
Merge tag 'smp-core-2026-08-17' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
Pull SMP core updates from Thomas Gleixner:
- Reduce the preemption disabled sections in smp_call_function*().
The various smp call functions keep preemption disabled accross the
full operation which includes the wait for completion. Especially the
latter can take some time when one of the target CPUs is not
immediately responding to the IPI, which can result in large latency
spikes.
To improve this provide a per task CPU mask to track the CPUs to wait
for. That makes the information required for the wait task local and
therefore allows to reenable preemption before the wait.
While this comes with moderate extra memory cost this reduces SMP
function call induced latency measured in a fleet for high priority
tasks from ~17ms to ~1.5ms (~90%).
- Reduce the overhead of the CSD debug code by replacing the heavy
memory barriers with smp_store_release()/acquire()
- Remove obsolute unused hotplug states
* tag 'smp-core-2026-08-17' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip:
scftorture: Remove preempt_disable() in scftorture_invoke_one()
smp: Remove preempt_disable() from on_each_cpu_cond_mask()
smp: Remove preempt_disable() from smp_call_function()
smp: Enable preemption early in smp_call_function_many_cond()
smp: Alloc percpu csd data in smpcfd_prepare_cpu() only once
smp: Use task-local IPI cpumask in smp_call_function_many_cond()
smp: Refactor remote CPU selection in smp_call_function_any()
smp: Enable preemption early in smp_call_function_single()
smp: Disable preemption explicitly in __csd_lock_wait()
cpu/hotplug: Remove CPUHP_AP_ARM_CORESIGHT_CTI_STARTING
smp: Use release stores for csd_lock_record() state
- A set of improvements to various futex self tests
- Conversion to the selftest harness
- Provide and use thread creation and synchronization helpers to reduce
the dependency on delays, which tend to fail on loaded test systems
- New tests for validating owner exit scenarios for robust and PI futexes
- Runtime detect supported features and skip the tests if the kernel
has no support
- A few minor fixes
-----BEGIN PGP SIGNATURE-----
iQJEBAABCgAuFiEEQp8+kY+LLUocC4bMphj1TA10mKEFAmqC2iUQHHRnbHhAa2Vy
bmVsLm9yZwAKCRCmGPVMDXSYoTPBD/4sMc+o2TiWn4bf7vcoln3XlVA8oWkiJCIB
UUIW1n77CTK9jF50/xb77uvDvSHYr7Zx/Gm1QOvz/xdMvbgqUtPCExBoQXEz/1VV
y+s7x3K4pwI7BLzEPNsvC2kN0+cVisMXdCi5C5ot7xo3AilfcxxSIwauGyEdQuqZ
ArvVJjaBCXjRE1EtaVTeiGSINeR4kWPohOkH7dou/OISSd+EcA1kOsTrWlgVa2AA
o7Vy4u5VsYK7m+UVZUYmwf7+pU91iDj0FZ9fMYYskOw5F+Xax4seNO4bozre80LJ
3LE2aB8Rg5qGYCTL3D54O1c/2wczJueqGPHy+bF9SOOdlzo0N4Ro8wYUa3MmQuY5
8+JyfFOsFcqflCYFK/ketzu3JrlZdu8DV9pJ67+RIKpHIXG5VRZHBDDrkL60PF7x
TQoaZ57sOP7e5GJ+ByxxkltPuXGsIILcPytZoTtm6NwQIh2eSOtPSEPh9VprSChn
vhY50WJuZ6L//Z7YQfXPQ9B7Xe+39OIWgY1GnUKF09BNVx6V4/As7Nc8llH+7cR7
eyiVqs0uRDlnpe5zQkAj0hdYR9WK/XVAQ26rsHZDJnSnoalJbl2AjdH1FIg8xm7n
0PPKqr3hZC4lilpwoBXXPQLIOBKNAUXZpVWxEopGSFIZ5tdrcnNcxgYFMuKiN5cQ
glxSVQVl0g==
=6VmJ
-----END PGP SIGNATURE-----
Merge tag 'locking-futex-2026-08-17' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
Pull futex updates from Thomas Gleixner:
- Improvements to various futex self tests:
- Conversion to the selftest harness
- Provide and use thread creation and synchronization helpers to
reduce the dependency on delays, which tend to fail on loaded test
systems
- New tests for validating owner exit scenarios for robust and PI
futexes
- Runtime detect supported features and skip the tests if the kernel
has no support
- A few minor fixes
* tag 'locking-futex-2026-08-17' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip:
selftests/futex: Give circular-list nodes static storage
selftests/futex: Use thread synchronization helpers instead of usleep()
selftests/futex: Provide thread creation and synchronization helpers
selftests/futex: Dynamically skip unsupported tests
selftests/futex: Add FUTEX_LOCK_PI owner-exiting coverage
selftests/futex: Migrate robust_list to harness
selftests/futex: Migrate futex_priv_hash to harness
selftests/futex: Migrate futex_numa_mpol to harness
selftests/futex: Migrate futex_requeue_pi_signal_restart to harness
selftests/futex: Migrate futex_requeue_pi_mismatched_ops to harness
selftests/futex: Migrate futex_requeue_pi to harness
selftests/futex: Migrate futex_requeue to harness
selftests/futex: Migrate futex_wait_uninitialized_heap to harness
selftests/futex: Migrate futex_wait_private_mapped_file to harness
selftests/futex: Migrate futex_wait to harness
selftests/futex: Correct validation logic in waitv
selftests/futex: Migrate functional tests to harness
selftests/futex: Remove static keyword from 'head'
futex: Remove unnecessary NULL check before kvfree()
selftests/rseq: Replace glibc-specific __GNUC_PREREQ with portable check
- Remove pointless NULL checks of the kstats_irqs field. That's a
historical left over and not longer required.
- Add Radu Rendec as reviewer. Radu thankfully stepped up to help
reviewing the interrupt core and the related drivers code.
- The usual small improvements and fixes
-----BEGIN PGP SIGNATURE-----
iQJEBAABCgAuFiEEQp8+kY+LLUocC4bMphj1TA10mKEFAmqCtM8QHHRnbHhAa2Vy
bmVsLm9yZwAKCRCmGPVMDXSYoV4SD/465axAMEli+6LbBcxWVmX1SCUasn+VM/2g
VUUPONEDuf/Ucig+OC1QlvfZ/5d9pLiIwh3TakWPbkFtZpyfi60KiEQw5OW67TYB
5TYgJI8BquIAQGzjUCKIzZmEvQhbuJzRjz07CIZnCtQBK3b/ftSDCwm6ynhhYRQC
5yX/ay84pHPCyWVupkXV8jwjFm6v28U17j8TVd2VwEIOrEBa8dbOxOqUlRzrjbwE
2dmo7r3BPSpyI/VpYiOW3wEXYq03+sR3MTNXQ9VWqLVcaUJUdDnbTnXqMbaXpGcO
oV4vnuPv2Ah2TkB8xF9KTcyCKJbS2BKb5BtUXEJBQ/kPnB1wD712N2oZNuaN3omF
SM4QoWojDpNPrPYWG99T4sNbemxfiyhIJzPSE/SgE6g0cWxh3d3wh5VfOnkp4ige
gZKHVz7EcpjcyVz4xgYhaZk+7FmzAb6FMi4UhLZ9KHbAhdH87LSDDxvcUua2X3Js
v1RdV3pkH1M3tBcLfO0D1neGL4VCL9zs8Ldf0jJBvcOy8kmjcs4uCll54hEbCQwI
N7Qt7JF/r1LwxJRMC4OzsXp2upCbT5gaCOeG2HJuQUSy//0RL72NzjQe4cJSw5ya
5Xnv5d/6L+hnwkOo7LPp9JYORXZ673D/vQgBg9Nf64Iz0i9SWnNB/fsmiSoTjw/c
ZmqKGgEyHA==
=6wX8
-----END PGP SIGNATURE-----
Merge tag 'irq-core-2026-08-17' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
Pull generic interrupt subsystem updates from Thomas Gleixner:
- Remove pointless NULL checks of the kstats_irqs field. That's a
historical left over and not longer required.
- Add Radu Rendec as reviewer. Radu thankfully stepped up to help
reviewing the interrupt core and the related drivers code.
- The usual small improvements and fixes
* tag 'irq-core-2026-08-17' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip:
MAINTAINERS: Add Radu Rendec as reviewer for the interrupt subsystem
genirq/msi: Move misplaced EXPORT_SYMBOL_GPL for msi_domain_free_irqs_all()
parisc: Remove unnecessary NULL check of the kstat_irqs field
genirq: Remove unnecessary NULL check of the kstat_irqs field
irqdomain: Remove unnedded NULL check in __irq_domain_[de]activate_irq()
genirq/manage: Use irqd_get_parent_data() helper in __irq_get_irqchip_state()
irqdomain: Plug leak in irq_domain_alloc_irqs_locked() error path
- Make syscall user dispatching configurable
Not all architectures can makes use of syscall user dispatching. Allow
them to disable the feature completely.
- Consolidate stack randomization for the generic entry code and the
architectures using it.
Stack randomization on syscall entry was sprinkled throughout the
architecture specific low level entry code and in some cases at the
wrong points, e.g. before establishing state, which violates the
non-instrumentable constraints of that code.
Clean this up by integrating stack randomization into the generic entry
code helpers so that it is invoked at the earliest possible point right
after establishing state and converting all generic entry code using
architecture over.
- Clean up the syscall number handling in the generic entry code. It
works correctly for architectures which have a separate return value
storage in pt_regs, but fails to distinguish the case where user space
handed in -1 as syscall number from the case where the entry code
rejects it by returning -1 to the callers. Aside of that the return
value functionality of those interfaces is not really intuitive.
Fix this by separating the decision to reject a syscall (user dispatch,
ptrace, seccomp ...) from the potential modification of the syscall
number through these mechanisms.
This solves most of the problems for architectures which do not have a
separate return value storage in pt_regs except for the case where a
tracepoint has a BPF script or a probe attached which overwrite both
the syscall number and the return value. But that's a problem which
cannot be solved in the generic code, that only can be addressed by
separating the storage model in the affected architectures.
-----BEGIN PGP SIGNATURE-----
iQJEBAABCgAuFiEEQp8+kY+LLUocC4bMphj1TA10mKEFAmqCs10QHHRnbHhAa2Vy
bmVsLm9yZwAKCRCmGPVMDXSYoaf6D/0ZBG1Yb0/C/6lrI185qPu38aGOROuAcxP+
RV1O1x6C83w2hCLBH8LeswY2x4/iGbdftne/hfmvu8eNCE5MzBfYvXhLL4If75Tc
IJ6C8uummnDmrT1TFuWHryTAfjyF28gt0+GGq0Zy5Hyz9b4CTJqOMx5u6KV4cZuJ
odoNQpE/GlWo40wCSTYP/Tt5xONrogk2pMQtFyV8JEoaXkdYSj/V815yojEmofYU
fmgPPO5/vOnZzE4b29gZyndXnU1Boah7r1l5fg7c9za376yCEEzh/ApPhovHyY0A
t8zjnrtooZ27IUKbcsyycrAM14asfcmViDNDgaCj8ttBioQaCnxO1BpKWjVxEZhE
AbM6q3Q66ER4Df6GNhZjPqT5Lr7E7+vLLarhXLWztsGQklIx4AFbrsa73hA20UC9
1PSeMd45JSxH3yA8vMauXAGHFK1tD1V8Lgofu69+2Z3jtKB+aU0fqWeL1jesSEM0
oCGhUb3hIC1pz3KVA0MGmNTm0yyQJYTGZL7wADYNV5NbxJVqXgo37qa/0n94Gf/4
TG3OwY4Sb/H/sve7v/eY4IvxVh+xs3dLZP8ZoqMlPCp9JIxc6iNoe6VHqPI7PFnM
fXwDtsy+bRF/SKnB/32qxnR7UJqmdNH3XIjd+lXWliKt6UYoC79/MEKN5DmJcO9P
CykZUWa72A==
=XUd9
-----END PGP SIGNATURE-----
Merge tag 'core-entry-2026-08-17' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
Pull generic entry code updates from Thomas Gleixner:
- Make syscall user dispatching configurable
Not all architectures can makes use of syscall user dispatching.
Allow them to disable the feature completely.
- Consolidate stack randomization for the generic entry code and the
architectures using it.
Stack randomization on syscall entry was sprinkled throughout the
architecture specific low level entry code and in some cases at the
wrong points, e.g. before establishing state, which violates the
non-instrumentable constraints of that code.
Clean this up by integrating stack randomization into the generic
entry code helpers so that it is invoked at the earliest possible
point right after establishing state and converting all generic entry
code using architecture over.
- Clean up the syscall number handling in the generic entry code. It
works correctly for architectures which have a separate return value
storage in pt_regs, but fails to distinguish the case where user
space handed in -1 as syscall number from the case where the entry
code rejects it by returning -1 to the callers. Aside of that the
return value functionality of those interfaces is not really
intuitive.
Fix this by separating the decision to reject a syscall (user
dispatch, ptrace, seccomp ...) from the potential modification of the
syscall number through these mechanisms.
This solves most of the problems for architectures which do not have
a separate return value storage in pt_regs except for the case where
a tracepoint has a BPF script or a probe attached which overwrite
both the syscall number and the return value. But that's a problem
which cannot be solved in the generic code, that only can be
addressed by separating the storage model in the affected
architectures.
* tag 'core-entry-2026-08-17' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip: (23 commits)
entry, treewide: Make syscall_enter_from_user_mode[_work]() indicate syscall execution
entry: Make return type of syscall_trace_enter() bool
entry: Rework trace_syscall_enter()
entry: Rework syscall_audit_enter()
syscall_user_dispatch: Introduce ARCH_SUPPORTS_SYSCALL_USER_DISPATCH
entry: Fix seccomp bypass after ptrace with TSYNC
x86/entry: Simplify the syscall number logic
x86/entry: Get rid of the sys_ni_syscall() indirection
x86/entry: Make syscall functions static
ptrace, treewide: Rename ptrace_report_syscall_entry() to ptrace_report_syscall_permit_entry()
seccomp, treewide: Rename and convert __secure_computing() to return boolean
entry: Use syscall number instead of rereading it
entry: Remove syscall_enter_from_user_mode()
x86/syscall: Use [syscall_]enter_from_user_mode_randomize_stack()
s390/syscall: Use enter_from_user_mode_randomize_stack()
riscv/syscall: Use syscall_enter_from_user_mode_randomize_stack()
powerpc/syscall: Use syscall_enter_from_user_mode_randomize_stack()
loongarch/syscall: Use syscall_enter_from_user_mode_randomize_stack()
entry: Provide [syscall_]enter_from_user_mode_randomize_stack()
randomize_kstack: Provide add_random_kstack_offset_irqsoff()
...
Load-balancing updates:
- Series to 'flatten the pick': improve cgroup scheduling, which has
always been problematic and painful, which has caused various
scheduling misbehavior such as the mishandling of reniced tasks et al.
Add various cgroup weight distribution methods via cgroup_mode:
'up', 'max', 'concur' and 'tasks' - with the default being 'concur'
which is the most precise yet also most expensive version.
Finally, change cgroup scheduling to a single runqueue.
(Peter Zijlstra)
- Series to improve the scheduling latency of short slice tasks
(Vincent Guittot)
- Series to fix cluster scheduling in the presence of asymmetric capacity
(Ricardo Neri)
- Prefer fully idle cores for NOHZ balancing (Andrea Righi)
- Don't trigger active load-balancing if src_rq->curr is not on_rq
(Xin Zhao)
PSI updates:
- Skip irqtime accounting when no new irq time has elapsed
(Usama Arif)
Scheduler debugging updates:
- Remove unused schedstats (Shrikanth Hegde)
- Defer freeing of cpumask memblock memory to initcall
(Waiman Long)
Misc fixes and updates by Yu C Chen, K Prateek Nayak, Peter Zijlstra,
Vincent Guittot, Xin Zhao, Yury Norov, Zhan Xusheng.
Signed-off-by: Ingo Molnar <mingo@kernel.org>
-----BEGIN PGP SIGNATURE-----
iQJFBAABCgAvFiEEBpT5eoXrXCwVQwEKEnMQ0APhK1gFAmqC3WARHG1pbmdvQGtl
cm5lbC5vcmcACgkQEnMQ0APhK1hawRAAuVAreXtsp7Ok6aRHmiOMR+Va2EsHIL4B
3MY5do2oUfiJSCDtIn3BXqjM3djOXdbQLQwm6r9/NkyJ6hsOsRgZaP1L/w07FiY0
33ilIhZIH9QZ2RzvEjZcFXVwnWH6zP8Ag/gTsch3g8TIX2JJcZBzcJjQM8ZIdcBL
oxwWNe9z3Mh0/x2N+N/s1xstaZf95bWQKeBVBGjiOAakGRWMEaQmMtOUujJNmx/p
YFtNcJL6pLGfD+ASOtLxYWts4z1JyDmwvje8SZmRsgjZdhoxRlnEIgU6jrNjmPqu
NfR5w6R193Pa+FapawYi0ttuawapHvvSDDUxjl+eIccPMw0mBdjzn1XuNIt6pclC
Q4VDUQEDBYvbeznNfLuvvsLVBq/QIjim391lg9DuAGreuRx00Ij0hXOCPk6TM4H8
tfUZ8N25vIAbYrFajSDyOYdgAa2Z2VnM1Q1ko29LPb3WFfZsEqgIg7+ukoXCZ5n8
67pUrOYLuSlLXJGlJY9WDz1nbeUh3e0f2390fif3VBcQ7roPEmoL8CYZhh8Vd2rq
ug8kOpL/91/PDgSPbxRRIEYhe9X1kzWZxqpjvO93tZ+GhXUxtLFG4PjnbQhINy1A
BMNlxcq4ktVHk9gwLaG9wcySjINnN/wK91o3dQ9YjwTuCAxFi/AqUVjY+OOmguAh
TO0mkEq99M0=
=HTPk
-----END PGP SIGNATURE-----
Merge tag 'sched-core-2026-08-17' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
Pull scheduler updates from Ingo Molnar:
"Load-balancing updates:
- 'flatten the pick': improve cgroup scheduling, which has always
been problematic and painful, which has caused various scheduling
misbehavior such as the mishandling of reniced tasks et al.
Add various cgroup weight distribution methods via cgroup_mode:
'up', 'max', 'concur' and 'tasks' - with the default being 'concur'
which is the most precise yet also most expensive version.
Finally, change cgroup scheduling to a single runqueue (Peter
Zijlstra)
- Series to improve the scheduling latency of short slice tasks
(Vincent Guittot)
- Series to fix cluster scheduling in the presence of asymmetric
capacity (Ricardo Neri)
- Prefer fully idle cores for NOHZ balancing (Andrea Righi)
- Don't trigger active load-balancing if src_rq->curr is not on_rq
(Xin Zhao)
PSI updates:
- Skip irqtime accounting when no new irq time has elapsed (Usama
Arif)
Scheduler debugging updates:
- Remove unused schedstats (Shrikanth Hegde)
- Defer freeing of cpumask memblock memory to initcall (Waiman Long)
Misc fixes and updates by Yu C Chen, K Prateek Nayak, Peter Zijlstra,
Vincent Guittot, Xin Zhao, Yury Norov, Zhan Xusheng"
* tag 'sched-core-2026-08-17' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip: (33 commits)
sched/fair: Fix flat hierarchy
sched/isolation: Defer freeing of cpumask memblock memory to initcall
sched/topology: Restore SD_PREFER_SIBLING in domains with asymmetric capacity
sched/fair: Allow load balancing between CPUs of identical capacity
sched/fair: Skip misfit load accounting when the destination CPU cannot help
sched/fair: Check CPU capacity before comparing group types during load balance
sched/fair: Also gate overloaded status update for SD_ASYM_CPUCAPACITY
sched/fair: Do not skip CPUs of similar capacity with busy SMT siblings
sched/fair: Prefer fully idle cores for NOHZ balancing
stop_machine: Make stop_one_cpu_nowait() return void
sched/eevdf: Delayed dequeue task can't preempt
sched/fair: Fix stale comments referring to removed CFS concepts
sched/debug: Remove unused schedstats
sched/psi: skip irqtime accounting when no new irq time has elapsed
sched/fair: Reflow sched_balance_rq()
sched/fair: Simplify balance_interval reset logic in sched_balance_rq()
sched/fair: Don't trigger active lb if src_rq->curr is not on_rq
sched/eevdf: Speedup short slice task scheduling
sched/eevdf: Always update slice protection
sched/eevdf: Cancel slice protection if short slice task is eligible
...
Futexes:
- Use runtime constants for futex_hash computation
(K Prateek Nayak, Peter Zijlstra)
- Optimise the size check get_futex_key() (Sebastian Andrzej Siewior)
- Avoid private hash use-after-free on final put (Felix Hoffmann)
- Tell kmemleak we're not leaking __futex_queues (Peter Zijlstra)
Rust integration updates:
- Implement refcounted interrupt disable and SpinLockIrq for Rust
(Boqun Feng, Heiko Carstens, Joel Fernandes, Lyude Paul)
- Rust sync: add helpers for mb, dma_mb and friends;
add generic memory barriers and use LKMM atomics
instead of Rust atomics in the revocable code (Gary Guo)
- Add abstraction and integrate synchronize_rcu() (Philipp Stanner)
Lock debugging:
- Add qspinlock contended_release tracepoint
(Dmitry Ilvokhin, Peter Zijlstra)
- Enable the printing of held locks of remote running tasks and print
task CPU (Ingo Molnar)
- percpu-rwsem: Annotate intentional data race in readers_active_check()
(Sun Shaojie)
Misc fixes and updates by Boqun Feng, Peter Zijlstra, Fangrui Song,
Naveen Kumar Chaudhary and Thomas Huth.
Signed-off-by: Ingo Molnar <mingo@kernel.org>
-----BEGIN PGP SIGNATURE-----
iQJFBAABCgAvFiEEBpT5eoXrXCwVQwEKEnMQ0APhK1gFAmqC2KMRHG1pbmdvQGtl
cm5lbC5vcmcACgkQEnMQ0APhK1gNwg//awvTQONfhPanAyTgl7CLDSlMSHdqmlyh
Ue0/Q8Ef1Cy4jwXY2FE2A0b1VcM6cGpDPoryVdg/wMdUXRNwinzAEXmxIkRy9kve
4LybrZwDShgLxJ7pJ6KKhgjgDiat8EdYmOwCBEE3LnP7AYhkAb8BFetA3YZJvzPa
KfA2BRYCgvBTid6yOAuXWm55Ev92AczOBamBzTxCadcaDGtNGXtQO6LfnqiQDOav
X5tVoANBeaQtSs1+LxE41WdNOiRoBuy0IFFvXtZRal6PZYuGGmZ5tbQvscD099em
haVwQyzDHQrqzglv71M0KRTXvYzdGveMRg/Au1SQnuLO3V6Vd5rMQ1g7I2M9Ln0f
Pg+tlRvQ77mLoqcgrtl0W/u0fRR4eDkiJ1pmG+98oniPwau23RdbFhC0vKFz3ikF
WHMgk3/9TcULylgF1Tj6QLmNrBY3Vx8LBdsFjhflEw7bG4cW42D91npmXIiEDE6K
tJc9CcaVdyE75o59z2Dtjj+qQVBlNPlfKQFXFL7p3jU/gFw2SzYuqon66X3kGmr0
mKJ9UNJdkLdiCjxS/QiMcDeYhwJksJqxFBkH50z3Kzmo84JsSpUFkoa6GM4aSiGn
HEwgC0Q7oOXVNIKUBYk5QaRW0HSk55hbsX2TWkvpeBYkE1zXshVZCCmgpaTSJgb5
oFmiwrGfUjo=
=slqC
-----END PGP SIGNATURE-----
Merge tag 'locking-core-2026-08-17' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
Pull locking updates from Ingo Molnar:
"Futexes:
- Use runtime constants for futex_hash computation (K Prateek Nayak,
Peter Zijlstra)
- Optimise the size check get_futex_key() (Sebastian Andrzej Siewior)
- Avoid private hash use-after-free on final put (Felix Hoffmann)
- Tell kmemleak we're not leaking __futex_queues (Peter Zijlstra)
Rust integration updates:
- Implement refcounted interrupt disable and SpinLockIrq for Rust
(Boqun Feng, Heiko Carstens, Joel Fernandes, Lyude Paul)
- Rust sync: add helpers for mb, dma_mb and friends; add generic
memory barriers and use LKMM atomics instead of Rust atomics in the
revocable code (Gary Guo)
- Add abstraction and integrate synchronize_rcu() (Philipp Stanner)
Lock debugging:
- Add qspinlock contended_release tracepoint (Dmitry Ilvokhin, Peter
Zijlstra)
- Enable the printing of held locks of remote running tasks and print
task CPU (Ingo Molnar)
- percpu-rwsem: Annotate intentional data race in readers_active_check()
(Sun Shaojie)
Misc fixes and updates by Boqun Feng, Peter Zijlstra, Fangrui Song,
Naveen Kumar Chaudhary and Thomas Huth"
* tag 'locking-core-2026-08-17' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip: (44 commits)
rust: sync: Introduce SpinLockIrq::lock_with() and friends
rust: sync: Add SpinLockIrq
rust: sync: Use super::* in spinlock.rs
rust: helper: Add spin_{un,}lock_irq_{enable,disable}() helpers
rust: Introduce interrupt module
s390/preempt: Enable HAS_SEPARATE_PREEMPT_RESCHED_BITS
arm64: sched/preempt: Enable HAS_SEPARATE_PREEMPT_RESCHED_BITS
preempt: Introduce HAS_SEPARATE_PREEMPT_RESCHED_BITS
sched: Avoid signed comparison of preempt_count() in __cant_migrate()
sched: Remove the unused preempt_offset parameter of __cant_sleep()
locking: Switch to _irq_{disable,enable}() variants in cleanup guards
irq: Add KUnit test for refcounted interrupt enable/disable
irq,spin_lock: Add counted interrupt disabling/enabling
openrisc: Include <linux/cpumask.h> in smp.h
preempt: Introduce __preempt_count_{sub,add}_return()
preempt: Introduce HARDIRQ_DISABLE_BITS
preempt: Track NMI nesting to separate per-CPU counter
futex: Tell kmemleak we're not leaking __futex_queues
x86/paravirt: Trace contended_release on unlock
tracing/lock: Use TRACE_EVENT_FN() for contended_release
...
uprobes updates:
- Patch series to fix a category of bugs with optimized uprobes that
can clobber the redzone area with call instruction storing return
address on stack where user code may keep temporary data without
adjusting RSP. Fix this by moving the optimized uprobes on top of
10-bytes NOP instruction, so we can squeeze another instruction to
escape the redzone area before doing the call.
(Jiri Olsa, Andrii Nakryiko)
- Switch uretprobes_srcu to SRCU-fast-updown, to improve performance
(Puranjay Mohan)
Intel CPU PMU driver updates:
- Optimize ACR handling in match_prev_assignment() (Dapeng Mi)
- Patch series to fix various PMU driver bugs and data leaks (Dapeng Mi)
- Fix Intel PT stop/start with no update (Adrian Hunter)
Intel uncore PMU driver updates:
- Series to fix various uncore PMU setup robustness bugs (Zide Chen)
AMD uncore PMU driver updates:
- Add group validation (Sandipan Das)
Misc fixes and updates by Dapeng Mi, Randy Dunlap and Zide Chen.
Signed-off-by: Ingo Molnar <mingo@kernel.org>
-----BEGIN PGP SIGNATURE-----
iQJFBAABCgAvFiEEBpT5eoXrXCwVQwEKEnMQ0APhK1gFAmqC1G0RHG1pbmdvQGtl
cm5lbC5vcmcACgkQEnMQ0APhK1gMehAAh/p3A42YKWgcEjnlmCTJ+RTGTrLbL/OB
Q2uj4H9D32lBV8+ZyBjt3iVT5u8Z3pliUMAq4yaLua6SiWO4gbb/XLp7K9tnHSj9
J/9zvOdxC74ZmnuKYzcjmSAFzYL0VmfHoHUMXNKEqjP/cfZnlp7Zhs7n0s+aEt9h
IzkgzUt3maY7xjXnBXiHHmIvOrGt5i2avXFJxUeE6YPnRjUK6+8b8jzJTsvL7GrM
lVTG7D0ptRELVKpgvbHWAK4738Oxls0tB6KmEHO+lMwM5i2OFy/Xs3PxX7Z6ruII
3+RUAk9K8MNpxG+eIUzBs+Qu3a3cS/bAMS4LiD+oKSg0OxAbnvRey1l0uQOZLevb
UUG3mNMpOJ0rfQIBMT0FhJZ7Rk6V0XRCfjyXnrZhfjCa879g+GPxrHoSAVjMa0PG
S1y51mj5QNeXwMi7U9Pwl88GF2TLsxj1kNlpgMrFfZywjXD5wp2Oqd6s+fVuFk/X
5FuUcY7evchxXjlfrYM84gtBBklVeOmd4yrqAGmhM5LUoFBCTt6HPzzn8U9Z8BIm
qZvtSsZ4/FGQPNrAbGXXQ1tWxxP+g0HeRYmyxoj2n6Xpope8vDdWzs/cA1NeG9Dl
4sbwkiOmWUCBniR05oTTCFtSejnr3UIq/H/ulUJRSRHocGaveFVhuUJYqQRsfUSX
d223r/dlwj0=
=V8n9
-----END PGP SIGNATURE-----
Merge tag 'perf-core-2026-08-17' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
Pull performance events updates from Ingo Molnar:
"uprobes updates:
- Fix a category of bugs with optimized uprobes that can clobber the
redzone area with call instruction storing return address on stack
where user code may keep temporary data without adjusting RSP.
Fix this by moving the optimized uprobes on top of 10-bytes NOP
instruction, so we can squeeze another instruction to escape the
redzone area before doing the call (Jiri Olsa, Andrii Nakryiko)
- Switch uretprobes_srcu to SRCU-fast-updown, to improve performance
(Puranjay Mohan)
Intel CPU PMU driver updates:
- Optimize ACR handling in match_prev_assignment() (Dapeng Mi)
- Fix various PMU driver bugs and data leaks (Dapeng Mi)
- Fix Intel PT stop/start with no update (Adrian Hunter)
Intel uncore PMU driver updates:
- Fix various uncore PMU setup robustness bugs (Zide Chen)
AMD uncore PMU driver updates:
- Add group validation (Sandipan Das)
.. and misc fixes and updates by Dapeng Mi, Randy Dunlap and Zide Chen"
* tag 'perf-core-2026-08-17' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip: (42 commits)
perf/x86: Optimize ACR handling in match_prev_assignment()
perf/x86/intel: Fix intel_cap handling on hybrid PMUs
perf/x86: Remove stale fixed counter helper and fix hybrid PMU access
perf/x86/intel: Unwind cpuc state if PEBS buffer setup fails
perf/x86: Guard intel_pmu_cpu_dead() against invalid hybrid PMU casts
perf/x86: Free hybrid state on PMU init failure
perf/x86: Unregister PMI handler on PMU init failure
perf/x86/intel/pt: Fix stop/start with no update
perf/x86/intel/pt: Use bitwise access for PERF_HES_STOPPED
perf/x86/intel/pt: Factor out pt_config_enable()
uprobes: Switch uretprobes_srcu to SRCU-fast-updown
srcu: Add lock guard for srcu_fast_updown flavor
perf/x86/intel/pt: Drop kernel-doc for deleted struct members
perf/x86/amd/uncore: Add group validation
selftests/bpf: Add tests for forked/cloned optimized uprobes
selftests/bpf: Add tests for uprobe nop10 red zone clobbering
selftests/bpf: Add reattach tests for uprobe syscall
selftests/bpf: Change uprobe/usdt trigger bench code to use nop10
selftests/bpf: Change uprobe syscall tests to use nop10
selftests/bpf: Emit nop,nop10 instructions combo for x86_64 arch
...
On PREEMPT_RT, bh_worker() wraps work item execution in pool->cb_lock to
provide a handshake for canceling BH work items. When a CPU goes down,
drain_dead_softirq_workfn() runs the dead pool's bh_worker() nested inside
the local pool's bh_worker(), acquiring the cb_locks of two different pools
without a nesting annotation. lockdep reports possible recursive locking:
============================================
WARNING: possible recursive locking detected
--------------------------------------------
ktimers/0/16 is trying to acquire lock:
ffff8880b873a990 (&pool->cb_lock){+...}-{3:3}, at: bh_worker+0x7d/0x880
but task is already holding lock:
ffff8880b863a990 (&pool->cb_lock){+...}-{3:3}, at: bh_worker+0x7d/0x880
Call Trace:
bh_worker+0x7d/0x880 kernel/workqueue.c:3688
drain_dead_softirq_workfn+0x95/0x220 kernel/workqueue.c:3763
process_scheduled_works+0xa8e/0x14e0 kernel/workqueue.c:3405
bh_worker+0x46a/0x880 kernel/workqueue.c:3708
tasklet_action+0xc/0x70 kernel/softirq.c:965
The nesting can't deadlock. A pool's bh_worker() runs nested only while the
pool's CPU is dead, entered from a live pool's bh_worker() on the draining
CPU, so the ordering is always live to dead. CPU hotplug operations are
serialized and the drain is synchronous, so the nesting depth never exceeds
two. Annotate the inner acquisition with SINGLE_DEPTH_NESTING.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reported-by: syzbot+1bd20115328f8254ed62@syzkaller.appspotmail.com
Closes: https://syzkaller.appspot.com/bug?extid=1bd20115328f8254ed62
Fixes: ad7c7f4b9c ("workqueue: Provide a handshake for canceling BH workers")
Cc: stable@vger.kernel.org # v6.18+
ops.cgroup_set_bandwidth() is delivered from scx_group_set_bandwidth(),
which runs from the cpu.max cgroup interface write path (tg_set_bandwidth())
in process context. scx_group_set_bandwidth() holds
percpu_down_read(&scx_cgroup_ops_rwsem), whose read side may sleep.
The call site is therefore sleepable, like ops.cgroup_init().
bpf_scx_check_member() rejects a sleepable program on any member not on its
allow-list, so a BPF scheduler cannot allocate -- which is sleepable -- when
a cgroup gains a cpu.max limit at runtime; it must instead pre-reserve memory
for a callback that cannot allocate. Add cgroup_set_bandwidth() to the
allow-list so the callback can allocate on demand, and document that it may
block.
A scheduler must decide at load time whether to mark the callback sleepable,
but the allow-list entry is a verifier property with no symbol to probe. Add
a compatibility marker whose presence in the kernel's BTF lets userspace detect
this support: DEFINE_SCX_COMPAT_MARKER() emits an empty, callerless function,
here scx_compat_marker_cgroup_set_bandwidth_may_sleep(). It is __used
__retain so neither the compiler nor the linker (under
CONFIG_LD_DEAD_CODE_DATA_ELIMINATION) drops it. The markers share the
scx_compat_marker_ prefix and are collected near the end of ext.c so more
can be added as further capabilities appear.
Signed-off-by: Changwoo Min <changwoo@igalia.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
KHO:
* Fix size calculation in kho_preserved_memory_reserve() for preservations
larger than 2 GiB
LUO:
* move liveupdate selftest utilities into a library so that selftests of
subsystems participating in liveupdate, e.g. PCI and VFIO, can use them and
drop direct ioctl calls from the tests
* add end to end liveupdate test infrastructure that allows running the
tests across a kexec in QEMU
* remove redundant INIT_LIST_HEAD in luo_session_alloc()
* remember the error status of an FLB retrieve() and return it on subsequent
attempts rather than retrying retrieve() with an FLB in an unexpected
state
* reference count the outgoing FLB so that it cannot be freed while a caller
is using it, the same way it's done for the incoming FLB
* reject nonzero reserved field in LIVEUPDATE_SESSION_FINISH so that it can
be reused by a future extension
-----BEGIN PGP SIGNATURE-----
iQEzBAABCgAdFiEEeOVYVaWZL5900a/pOQOGJssO/ZEFAmp9oz8ACgkQOQOGJssO
/ZHtBwf+LyzW59fGyPd8hHASkqcGzlvRGe9E1ML1XjbR7vMPPxCwt/WbJ6V5dbIb
Se/jNVrcnaI0NE1vudeA3uNqdboQwOQ8+KiPdLp/SjGsPptlQgpORX86gMf5ggYR
rVl7FFCqQcZmVHbAvLWhuSdVfhdADBjeR50JfqVcLUch1L48ykkHLJG+n7cEsClg
2YGsANNyM+l72JPBHzjRsy57O4fxCr5z9AQLJTHJYkT7N3j3aXprCzY00duFfiZw
4xOlvgUAkcxEOs5J27Ra8X+Wy6EmNwL72vZQLC7JuLCaGnfQ0EAtHuOB/E5CQX9G
o9fwgR8mY4/UPM2ao7+MjkV2cv23yA==
=p+yU
-----END PGP SIGNATURE-----
Merge tag 'liveupdate-v7.3-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/liveupdate/linux
Pull liveupdate updates from Mike Rapoport:
"Kexec Handover:
- Fix size calculation in kho_preserved_memory_reserve() for
preservations larger than 2 GiB
Live Update Orchestrator:
- move liveupdate selftest utilities into a library so that selftests
of subsystems participating in liveupdate, e.g. PCI and VFIO, can
use them and drop direct ioctl calls from the tests
- add end to end liveupdate test infrastructure that allows running
the tests across a kexec in QEMU
- remove redundant INIT_LIST_HEAD in luo_session_alloc()
- remember the error status of an FLB retrieve() and return it on
subsequent attempts rather than retrying retrieve() with an FLB in
an unexpected state
- reference count the outgoing FLB so that it cannot be freed while a
caller is using it, the same way it's done for the incoming FLB
- reject nonzero reserved field in LIVEUPDATE_SESSION_FINISH so that
it can be reused by a future extension"
* tag 'liveupdate-v7.3-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/liveupdate/linux:
kho: fix size calculation in kho_preserved_memory_reserve()
selftests/liveupdate: Move luo_test_utils.* into a reusable library
selftests/liveupdate: Use luo_test_utils.c for liveupdate ioctl APIs
liveupdate: Remember FLB retrieve() status
liveupdate: Reference count outgoing FLB data
liveupdate: reject nonzero reserved value for SESSION_FINISH
liveupdate: Remove redundant INIT_LIST_HEAD in luo_session_alloc
selftests/liveupdate: add end to end test infrastructure and scripts
* Deduplicate crash memory allocation and the exclusion of reserved crash
kernel regions from architecture specific code into a generic
crash_prepare_headers() and enable crashkernel CMA reservation on arm64 and
riscv reservation on arm64 and riscv.
* Skip purgatory checksum verification when the kexec segments cannot be
corrupted by DMA, which saves about 250ms on kexec.
* Replace __ASSEMBLY__ with the compiler provided __ASSEMBLER__ in
include/linux/kexec.h.
* Fix a keyring refcount imbalance in the kdump kernel's dm-crypt key restore
path, which over-dropped the user keyring reference when more than one key
was restored.
-----BEGIN PGP SIGNATURE-----
iQEzBAABCgAdFiEEeOVYVaWZL5900a/pOQOGJssO/ZEFAmp9ouoACgkQOQOGJssO
/ZEH/Af/b1qmnw8uUe2wzd1zcAEEsk6YwF8lmS5o4k7skiVKlHWQuDzPrwrxFTvg
d6obJrLllMN2ULyLQ8ghMKbsdo8RCGBIpXF+bPEXmIF8bA/PhtDaxvi3scsxvie6
zU3geZbExlzg0Ik525I0DyPoEZc2JAg5ietp8bF6ZN5HNkDtBoNvi2ROc4mR9kzO
agrczVnIEuIE3Tk+IfD6081iS3Wz/PjRC+yPBU1yYFobA6/bL7kFma7kED8PnxvI
2+oENcd1FtC/WBlmluZE0BC4SjNcsT1geyzAFlGlRvNzbHdtj13j6mTBSsbRpJMW
WdewVpIpFGUqRbUIIdvOISLOcf62lg==
=PjFs
-----END PGP SIGNATURE-----
Merge tag 'kexec-v7.3-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/liveupdate/linux
Pull kexec updates from Mike Rapoport:
- Deduplicate crash memory allocation and the exclusion of reserved
crash kernel regions from architecture specific code into a generic
crash_prepare_headers() and enable crashkernel CMA reservation on
arm64 and riscv reservation on arm64 and riscv.
- Skip purgatory checksum verification when the kexec segments cannot
be corrupted by DMA, which saves about 250ms on kexec.
- Replace __ASSEMBLY__ with the compiler provided __ASSEMBLER__ in
include/linux/kexec.h.
- Fix a keyring refcount imbalance in the kdump kernel's dm-crypt key
restore path, which over-dropped the user keyring reference when
more than one key was restored.
* tag 'kexec-v7.3-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/liveupdate/linux:
crash_dump: release keyring reference at the correct time
kexec: Replace __ASSEMBLY__ with __ASSEMBLER__ in header file
kexec_file: skip checksum verification when safe
riscv: kexec_file: Add support for crashkernel CMA reservation
arm64: kexec_file: Add support for crashkernel CMA reservation
powerpc/kexec_file: Use crash_exclude_core_ranges() helper
LoongArch: kexec_file: Use crash_prepare_headers() helper to simplify code
riscv: kexec_file: Use crash_prepare_headers() helper to simplify code
x86/crash: Use crash_prepare_headers() helper to simplify code
arm64: kexec_file: Use crash_prepare_headers() helper to simplify code
crash: Add crash_prepare_headers() to exclude crash kernel memory
powerpc/crash: sort crash memory ranges before preparing elfcorehdr
riscv: kexec_file: Fix crashk_low_res not exclude bug
- Minor fixes and cleanups in assorted cpufreq drivers (Dan Carpenter,
Guru Das Srinagesh, Haoxiang Li, Karl Mehltretter, Sasha Finkelstein,
and Pan Chuang)
- Fix cpufreq table creation and bios_limits() callback in the Rust
bindings (Priya Bala Govindasamy)
- Add IPQ5210 support to qcom-nvmem driver (Varadarajan Narayanan)
- Adjust the .adjust_perf() cpufreq driver callback to allow the
maximum performance value to be passed to drivers and update the
intel_pstate driver to use it (Rafael Wysocki)
- Set policy->cur to the actual requested frequency in the intel_pstate
driver when the performance policy is used (Rafael Wysocki)
- Simplify HWP handling on Broadwell processors in intel_pstate (Rafael
Wysocki)
- Fix setting minimum P-state at init time in intel_pstate (Rafael
Wysocki)
- Consolidate frequency values computation in intel_pstate and clean up
code in that driver (Rafael Wysocki)
- Add missing kernel-doc descriptions for structure and union members
in the amd-pstate driver (David Vernet)
- Handle missing policy in dynamic EPP callbacks in the amd-pstate
driver (EDAMAMEX)
- Introduce EXPORT_SYMBOL_FOR_PSTATE_UT() to export amd-pstate driver
symbols to the amd-pstate-ut subdriver (K Prateek Nayak)
- Add dynamic EPP as an "energy_performance_preference" mode in
amd-pstate, remove the "amd_dynamic_epp" kernel command line option
and the "dynamic_epp" sysfs attribute, and update the dynamic_epp
documentation accordingly (K Prateek Nayak)
- Add unit tests for CPPC Performance Priority and the "dynamic" EPP
mode in the amd-pstate driver (K Prateek Nayak)
- Set min_limit_freq based on bios_min_perf in amd-pstate and remove
the defensive check for bios_min_perf from it (K Prateek Nayak)
- Fix EPP return type and handle errors in amd-pstate during
initialization, toggle auto_sel in active mode on shared memory
systems, and cache the firmware programmed EPP value (Marco Scardovi)
- Skip tests in amd-pstate-ut if the amd-pstate driver is not in active
use (Qianheng Peng)
- Replace sprintf() with sysfs_emit() in sysfs show in the cpufreq
schedutil governor and fix a self-contradictory comment in
sugov_iowait_apply() (Zhongqiu Han)
- Fix the usage example for the sampling_rate tunable of the ondemand
cpufreq governor in admin-guide (wangxiaodong)
- Avoid using deep idle states during initialization in the intel_idle
driver to work around device handling issues (Rafael Wysocki)
- Fix and refactor the ACPI processor driver code related to ACPI _LPI
support and add ACPI _LPI support to intel_idle based on that ACPI
processor driver update (Rafael Wysocki)
- Backup and restore governor for cpufreq sptests (Yiwei Lin)
- Remove unnecessary sudo from quick_shuffle() and remove unused local
variables from switch_show_governor() in cpufreq selftests (Jinseok
Kim)
- Rename the PM core module parameter prefix to "pm" and allow the PM
transition (DPM) watchdog to be disabled by default (Tzung-Bi Shih)
- Fix off-by-one in wakelocks number limit check in the system sleep
sysfs interface (Haowen Tu)
- Remove kernel-doc markings from helper descriptions in the core
hibernation code (Adi Nata)
- Use %pe to print error pointer values in the hibernation core (Ronan
Marchal)
- Fix memory leak in snapshot_write_next() error path (Malaya Kumar
Rout)
- Delay allocating and linking the next swap_map_page in the
hibernation image saving code until another image page actually
needs to be recorded (Haesung Kim)
- Fix cleanup ordering around scope-based pointers in OPP (Gregor
Herburger).
- Use clk_get_optional() for optional clocks in OPP (Praveen Talari).
- Stop setting runtime_error on runtime resume callback failures to
allow drivers to recover from resume issues (Praveen Talari)
- Handle PMU registration failure during probe in the intel_rapl_tpmi
driver (Sumeet Pawnikar)
- Avoid optional imports in intel_pstate_tracer unless they are really
needed (Yousef Alhouseen)
- Add generic CPPC performance display to the cpupower utility, build
and call CPPC information on non-AMD processors, make cpupower print
kernel and hardware frequency information, and add libm to cpupower
for generic CPPC view (Jeremy Linton)
- Remove conditional return with no effect from cpupower (Sang-Heon
Jeon)
-----BEGIN PGP SIGNATURE-----
iQFGBAABCAAwFiEEcM8Aw/RY0dgsiRUR7l+9nS/U47UFAmp+DXsSHHJqd0Byand5
c29ja2kubmV0AAoJEO5fvZ0v1OO10Z4IAIYwM9RKwbwCqPMKGvWExIb+1kpEy1aw
wyFnYSiQd09XCt8/YjERFR9gB7CfGduisTxEOqOdCIpGX2rT9Me5xO5aR5Gz4FE3
bML7KyulS9tVyaEVBjFx1yDZxGTNooTQVTLIxBYbCVB1RrgiCGdLyC2okn79XHlk
olTZemq59Kn0B6t7BP7UQ0WCCCAs3RlbrQIZSrVh0HydoO0DHlJvttN2fc7bjeiM
Je/dSm14o+vjglkVpY1kbxzBHJOH7/XH5nqSQRoHoAZWdllWK4+iQtll63Un4A72
2KG0O5+nFrS3d166TEjDEInnEwR012Sod/8Gk72M0IZZhG/naqs5ekE=
=mn7E
-----END PGP SIGNATURE-----
Merge tag 'pm-7.3-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/rafael/linux-pm
Pull power management updates from Rafael Wysocki:
"As has been the case for quite some time, this set of changes is
dominated by cpufreq updates including intel-pstate and amd-pstate
driver updates, minor fixes and cleanups of other assorted cpufreq
drivers, schedutil governor updates, fixes of the Rust bindings, new
hardware support (IPQ5210 in qcom-nvmem), and some updates of self
tests related to cpufreq.
The second largest group of changes are cpuidle updates consisting of
intel_idle driver updates and ACPI processor idle driver updates, both
mostly related to ACPI _LPI support.
There are also updates related to system sleep, mostly in the
hibernation core code, two operating performance points (OPP) updates,
one runtime PM framework update, one power capping update, and some
tools updates including the addition of ACPI CPPC support to cpupower.
Specifics:
- Minor fixes and cleanups in assorted cpufreq drivers (Dan
Carpenter, Guru Das Srinagesh, Haoxiang Li, Karl Mehltretter, Sasha
Finkelstein, and Pan Chuang)
- Fix cpufreq table creation and bios_limits() callback in the Rust
bindings (Priya Bala Govindasamy)
- Add IPQ5210 support to qcom-nvmem driver (Varadarajan Narayanan)
- Adjust the .adjust_perf() cpufreq driver callback to allow the
maximum performance value to be passed to drivers and update the
intel_pstate driver to use it (Rafael Wysocki)
- Set policy->cur to the actual requested frequency in the
intel_pstate driver when the performance policy is used (Rafael
Wysocki)
- Simplify HWP handling on Broadwell processors in intel_pstate
(Rafael Wysocki)
- Fix setting minimum P-state at init time in intel_pstate (Rafael
Wysocki)
- Consolidate frequency values computation in intel_pstate and clean
up code in that driver (Rafael Wysocki)
- Add missing kernel-doc descriptions for structure and union members
in the amd-pstate driver (David Vernet)
- Handle missing policy in dynamic EPP callbacks in the amd-pstate
driver (EDAMAMEX)
- Introduce EXPORT_SYMBOL_FOR_PSTATE_UT() to export amd-pstate driver
symbols to the amd-pstate-ut subdriver (K Prateek Nayak)
- Add dynamic EPP as an "energy_performance_preference" mode in
amd-pstate, remove the "amd_dynamic_epp" kernel command line option
and the "dynamic_epp" sysfs attribute, and update the dynamic_epp
documentation accordingly (K Prateek Nayak)
- Add unit tests for CPPC Performance Priority and the "dynamic" EPP
mode in the amd-pstate driver (K Prateek Nayak)
- Set min_limit_freq based on bios_min_perf in amd-pstate and remove
the defensive check for bios_min_perf from it (K Prateek Nayak)
- Fix EPP return type and handle errors in amd-pstate during
initialization, toggle auto_sel in active mode on shared memory
systems, and cache the firmware programmed EPP value (Marco
Scardovi)
- Skip tests in amd-pstate-ut if the amd-pstate driver is not in
active use (Qianheng Peng)
- Replace sprintf() with sysfs_emit() in sysfs show in the cpufreq
schedutil governor and fix a self-contradictory comment in
sugov_iowait_apply() (Zhongqiu Han)
- Fix the usage example for the sampling_rate tunable of the ondemand
cpufreq governor in admin-guide (wangxiaodong)
- Avoid using deep idle states during initialization in the
intel_idle driver to work around device handling issues (Rafael
Wysocki)
- Fix and refactor the ACPI processor driver code related to ACPI
_LPI support and add ACPI _LPI support to intel_idle based on that
ACPI processor driver update (Rafael Wysocki)
- Backup and restore governor for cpufreq sptests (Yiwei Lin)
- Remove unnecessary sudo from quick_shuffle() and remove unused
local variables from switch_show_governor() in cpufreq selftests
(Jinseok Kim)
- Rename the PM core module parameter prefix to "pm" and allow the PM
transition (DPM) watchdog to be disabled by default (Tzung-Bi Shih)
- Fix off-by-one in wakelocks number limit check in the system sleep
sysfs interface (Haowen Tu)
- Remove kernel-doc markings from helper descriptions in the core
hibernation code (Adi Nata)
- Use %pe to print error pointer values in the hibernation core
(Ronan Marchal)
- Fix memory leak in snapshot_write_next() error path (Malaya Kumar
Rout)
- Delay allocating and linking the next swap_map_page in the
hibernation image saving code until another image page actually
needs to be recorded (Haesung Kim)
- Fix cleanup ordering around scope-based pointers in OPP (Gregor
Herburger).
- Use clk_get_optional() for optional clocks in OPP (Praveen Talari).
- Stop setting runtime_error on runtime resume callback failures to
allow drivers to recover from resume issues (Praveen Talari)
- Handle PMU registration failure during probe in the intel_rapl_tpmi
driver (Sumeet Pawnikar)
- Avoid optional imports in intel_pstate_tracer unless they are
really needed (Yousef Alhouseen)
- Add generic CPPC performance display to the cpupower utility, build
and call CPPC information on non-AMD processors, make cpupower
print kernel and hardware frequency information, and add libm to
cpupower for generic CPPC view (Jeremy Linton)
- Remove conditional return with no effect from cpupower (Sang-Heon
Jeon)"
* tag 'pm-7.3-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/rafael/linux-pm: (76 commits)
cpufreq: imx6q: fix out-of-bounds write when probed more than once
cpufreq: imx6q: fix devres accumulation across driver rebind
rust: cpufreq: Fix temporary write in Registration::bios_limit_callback
rust: cpufreq: Add CPUFREQ_TABLE_END as last table entry in TableBuilder::to_table
opp: Use clk_get_optional() to avoid leaving opp_table->clk as an error pointer
intel_idle: Avoid using deep idle states during initialization
cpupower: remove conditional return with no effect
cpufreq: intel_pstate: Adjust policy->cur in active mode to policy
cpufreq/amd-pstate: Document missing kernel-doc members
cpufreq/amd-pstate-ut: Add unit test for CPPC Performance Priority
cpufreq/amd-pstate-ut: Add unit test for "dynamic" EPP mode
cpufreq/amd-pstate: Reduce the scope of exported symbols
Documentation/amd-pstate: Update dynamic_epp documentation with new behavior
cpufreq/amd-pstate: Remove "amd_dynamic_epp" cmdline and "dynamic_epp" sysfs
cpufreq/amd-pstate: Add dynamic EPP as an "energy_performance_preference" mode
cpufreq/amd-pstate: Extract platform profile to EPP conversion into a helper
cpufreq/amd-pstate: Remove the defensive check for bios_min_perf
cpufreq/amd-pstate: Set min_limit_freq based on bios_min_perf
powercap: intel_rapl_tpmi: Handle PMU registration failure during probe
PM: sleep: Allow disabling DPM watchdog by default
...
When timers are migrated away from an offline CPU the debugobjects state
gets corrupted. The timer is accounted as inactive on deletion, but the
enqueue on the alive CPU lacks the activation call.
That used to work, but got broken when the trace point and the debug
objects call got separated. That change missed to fixup
migrate_timer_list().
Add the missing debug_timer_activate() invocation to fix it.
Fixes: dc1e7dc5ac ("timer: Move trace point to get proper index")
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Cc: stable@vger.kernel.org
Link: https://patch.msgid.link/87bjb0l7ha.ffs@fw13
Pull to receive the __arena argument conversion:
67f1f4a48c ("sched_ext: Pass kernel arena pointers to ops_cid callbacks")
a8dc810968 ("sched_ext: Convert sub-cap kfuncs to __arena cmask arguments")
a05c5b5cb5 ("sched_ext: Convert scx_bpf_cid_override() to __arena array arguments")
along with the bpf-next branch carrying the __arena argument support they
depend on.
Conflict in kernel/sched/ext/ext.c between:
c384ab8a0b ("sched_ext: Move the config-off sub-cap kfunc stubs into sub.c")
and:
a8dc810968 ("sched_ext: Convert sub-cap kfuncs to __arena cmask arguments")
which updated the stubs in their old ext.c location. Resolved by keeping
ext.c without the stubs and applying the prototype conversion to the
relocated stubs in sub.c.
Signed-off-by: Tejun Heo <tj@kernel.org>
Please consider pulling these changes from the signed vfs-7.3-rc1.netfs tag.
Thanks!
Christian
-----BEGIN PGP SIGNATURE-----
iHUEABYKAB0WIQRAhzRXHqcMeLMyaSiRxhvAZXjcogUCan7RJQAKCRCRxhvAZXjc
oq5VAPwLHEyDBJUM581Nsj9O7NnkSRhffjp4//Cp2jHRYkaR4wEAiceqF+rONVWK
VYCyLEDSU5TPP9EeON95qSzq0khbswo=
=kcGi
-----END PGP SIGNATURE-----
Merge tag 'vfs-7.3-rc1.netfs' of git://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs
Pull cachefiles ondemand removal from Christian Brauner:
"This sunsets cachefiles ondemand mode.
It was an effort to make fscache usable as a kernel cache for lazy
pulling. EROFS over fscache was its only in-tree user. fscache has
since become netfslib-oriented while EROFS never acts as a network
filesystem and EROFS over fscache has been removed.
So this cleans up the netfs, fscache and cachefiles side as well"
* tag 'vfs-7.3-rc1.netfs' of git://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs:
cachefiles,netfs: sunset ondemand mode
Please consider pulling these changes from the signed vfs-7.3-rc1.kthread tag.
Thanks!
Christian
-----BEGIN PGP SIGNATURE-----
iHUEABYKAB0WIQRAhzRXHqcMeLMyaSiRxhvAZXjcogUCan7RJQAKCRCRxhvAZXjc
ohHuAQCFeGhxqsb44KhuJaWXewiXbefr6eg31lXmxgSfg0DILAD/Tw/WmpiUbX5M
H+/jh6NmLKnQ/ADjlWFRy3R1K1Il3w4=
=jwPP
-----END PGP SIGNATURE-----
Merge tag 'vfs-7.3-rc1.kthread' of git://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs
Pull kthread vfs updates from Christian Brauner:
"This stops kernel threads from sharing filesystem state with
userspace. This work is about 3 cycles old and has been in -next
for about that time.
When the kernel boots init_task creates PID 1 and then kthreadd. From
that point every kthread and PID 1 share the same fs_struct. That is
why pivot_root() has to rewrite the fs_struct of all kthreads. The
rewriting exists so that kthreads can use init's filesystem state when
they want to. It also means userspace can move the ground out from
under the kernel.
PID 1 now gets a completely separate fs_struct. All kthreads are
anchored in a private SB_KERNMOUNT instance of nullfs that cannot be
mounted on and cannot be used to follow other mounts. Userspace init
can no longer affect kthread filesystem state and kthreads can no
longer affect userspace fs state without explicit opting in to that.
Path lookup from a kthread now fails by default. It makes it
deliberately hard to offload security sensitive operations into init's
filesystem state from a kthread.
Places that legitimately need to look something up there opt in
through the new scoped_with_init_fs() which temporarily overrides the
caller's fs_struct with init's. usermodehelpers remain the only kernel
tasks that genuinely share init's filesystem state, since they execute
random binaries in the root filesystem (excellent...).
The visible result is that /proc/2/root is a nullfs with an empty
mountinfo while /proc/1/root is the real root"
* tag 'vfs-7.3-rc1.kthread' of git://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs: (26 commits)
initramfs_test: use test init/exit hooks to override init fs
fs: stop rewriting paths for PF_EXITING | PF_DUMPCORE
fs: stop rewriting kthread fs structs
fs: start all kthreads in nullfs
nullfs: make nullfs multi-instance
devtmpfs: create private mount namespace
fs: add umh argument to struct kernel_clone_args
fs: stop sharing fs_struct between init_task and pid 1
af_unix: use scoped_with_init_fs() for coredump socket lookup
initramfs: use scoped_with_init_fs() for rootfs unpacking
pnfs/blocklayout: use scoped_with_init_fs() for SCSI device lookup
ksmbd: use scoped_with_init_fs() for VFS path operations
ksmbd: use scoped_with_init_fs() for filesystem info path lookup
ksmbd: use scoped_with_init_fs() for share path resolution
fs: use scoped_with_init_fs() for kernel_read_file_from_path_initns()
coredump: use scoped_with_init_fs() for coredump path resolution
btrfs: use scoped_with_init_fs() for update_dev_time()
scsi: target: use scoped_with_init_fs() for APTPL metadata
scsi: target: use scoped_with_init_fs() for ALUA metadata
crypto: ccp: use scoped_with_init_fs() for SEV file access
...
Every LOCALIO read and write is currently bounced through the dedicated
!WQ_MEM_RECLAIM nfslocaliod_workqueue. That bounce is only actually
required when the submitting context is a memory-reclaim context: LOCALIO
issues IO directly into a stacked local filesystem (e.g. XFS) which may in
turn flush its own !WQ_MEM_RECLAIM workqueue. Doing that from a
WQ_MEM_RECLAIM worker (most importantly writeback's wb_workfn on bdi_wq) or
an explicit PF_MEMALLOC reclaim task trips check_flush_dependency() and
risks a forward-progress deadlock, which is why commit b9f5dd57f4
("nfs/localio: use dedicated workqueues for filesystem read and write")
introduced the intermediate workqueue.
Outside of reclaim context -- ordinary application/task submission such as
O_DIRECT or fsync-driven writeback -- the workqueue hop buys nothing and
merely adds a context switch and scheduling latency per IO while discarding
the NFS client's inherent application-context parallelism.
Add current_is_workqueue_mem_reclaim(), which reports whether %current is a
WQ_MEM_RECLAIM worker using the same predicate check_flush_dependency()
warns on. Use it, together with the PF_MEMALLOC check, in the new
nfs_local_defer_io() helper to decide per-IO whether nfs_local_do_read()
and nfs_local_do_write() must defer to nfslocaliod_workqueue or may issue
the IO inline. Buffered writeback continues to bounce (wb_workfn is a
WQ_MEM_RECLAIM worker); O_DIRECT and app-context submission now run inline.
Running nfs_local_call_write() inline is safe: it already saves and
restores current->flags around the PF_LOCAL_THROTTLE|PF_MEMALLOC_NOIO it
sets and scopes the file opener's creds. The async O_DIRECT completion
path is likewise unaffected: when the underlying filesystem returns
-EIOCBQUEUED, the kiocb ki_complete callback (nfs_local_read_aio_complete /
nfs_local_write_aio_complete) can run in bottom-half context and so must
still defer the pgio completion (nfs_local_pgio_release -> rpc_call_done) to
nfsiod_workqueue via nfs_local_pgio_aio_complete(). That completion hop is
independent of how the IO was submitted, and this change leaves it as-is;
only the submission side stops unconditionally hopping through
nfslocaliod_workqueue.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Mike Snitzer <snitzer@kernel.org>
Signed-off-by: Trond Myklebust <trond.myklebust@hammerspace.com>
Please consider pulling these changes from the signed vfs-7.3-rc1.binfmt tag.
Thanks!
Christian
-----BEGIN PGP SIGNATURE-----
iHUEABYKAB0WIQRAhzRXHqcMeLMyaSiRxhvAZXjcogUCan7RJAAKCRCRxhvAZXjc
ovtnAQDiMF2CvH2vr/eiUEUNdDjeyzRKipkh3a9krO0XbFxpnAD8CZypBFw/oWOQ
7z3PDGhHp10ie6KYC45XbUc+a4b6vgQ=
=X9RZ
-----END PGP SIGNATURE-----
Merge tag 'vfs-7.3-rc1.binfmt' of git://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs
Pull binfmt updates from Christian Brauner:
"This contains a bunch of work for binfmt_misc. It fixes a bunch of
old bugs, reworks the locking, and then extends the format registry
so a binary type can be matched programmatically and its interpreter
computed per exec instead of being a fixed string recorded at
registration time.
This allows nixos and other to e.g., implement relocatable binaries
meaning the interpreter/dynamic loader can be determined
programatically, say found relative to the binary. The mechanism is
flexible and can support other policies:
- Handler lookup is now an rcu walk. An exec that matches no
binfmt_misc entry should now never write to a shared cacheline
- remove the VERBOSE_STATUS and USE_DEBUG compile time toggles
- convert the entry file to a seq_file which simplifies things quite
a bit and kills a lot of custom logic
- make flags proper enums
- rename struct Node to binfmt_misc_entry
- allow entries to be removed with unlink(2)
- Add the ability to attach bpf programs to binfmt_misc entries so
it's possible to dynamically choose the execution environment such
as the loader or interpreter on a per binary basis.
A handler is an instance of a binfmt_misc_ops struct_ops with a
->match() and a ->load() program. match() decides from the entry
lookup walk whether the handler applies under the same
registration-order. It can read file content as needed not only the
prefetched 256 bytes in bprm->buf.
load() then selects the interpreter and stages it through the new
bpf_binprm_set_interp(), bpf_binprm_set_interp_arg() and
bpf_binprm_set_flags() kfuncs.
Handlers are published in a registry keyed by the registering
task's user namespace and activated through the existing text
interface with a new 'B' type carrying the handler name:
echo ':origin:B::::nix:' > /proc/sys/fs/binfmt_misc/register
The permission and namespacing model is unchanged. Activating a
handler requires the same write access to an instance as any other
registration. A container mounting its own instance escapes the
host's entries exactly as before. The computed interpreter is
opened with open_exec() under the caller's credentials and goes
through full LSM vetting as the next binprm level. A program can
only ever redirect the caller to something the caller could exec
anyway.
- Two dispatch modes are added. So far the chosen interpreter owns
the whole process identity (argv[0], /proc/pid/cmdline,
/proc/self/exe all name interpreter information). So relocatable
find the dynamic linker instead. Also a binary passed to execveat()
as an inaccessible O_CLOEXEC fd cannot run at all and gdb trips
because AT_ENTRY and AT_PHDR do not match the exe file. So PIE
symbols are unrelocated.
This adds transparent dispatch which allows the interpreter to load
the binary through AT_EXECFD and leaves the argument vector exactly
as the caller built it and labels mm->exe_file and comm with the
binary. It also raises the AT_FLAGS_TRANSPARENT_INTERP aux vector
bit. The interpreter keeps control of mapping the binary.
The second mode is loader substitution. This allows a binary to be
executed natively and only the interpreter to be changed.
- Last, interpreters can be bound at registration time. Each
interpreter is opened by its own write with the credentials the
entry file was opened with. The program picks one per exec with
bpf_binprm_select_interp().
Ucounts are used to properly account for pre-opened interpreters
via /proc/sys/user/max_binfmt_misc_interpreters"
* tag 'vfs-7.3-rc1.binfmt' of git://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs: (63 commits)
binfmt_misc: document the pre-opened interpreter limit
selftests/exec: test the pre-opened interpreter limit
binfmt_misc: correctly account pre-opened interpreters
binfmt_misc: document interpreters bound by a 'B' entry
selftests/exec: test interpreters bound to a 'B' entry
binfmt_misc: let a 'B' entry bind its interpreters
binfmt_misc: carry pre-opened interpreters in struct binfmt_misc_interp
selftests/exec: share the bpf handler preconditions
binfmt_misc: document registering an entry disabled
selftests/exec: test registering an entry disabled
selftests/exec: let binfmt_flag_supported() return a bool
selftests/exec: check that a binfmt_misc instance cannot be pinned
binfmt_misc: let a register string create an entry disabled
binfmt_misc: document loader substitution
selftests/exec: test binfmt_misc loader substitution
binfmt_misc: let a bpf handler request loader substitution
binfmt_misc: add the 'L' loader substitution flag
binfmt_elf_fdpic: consume a stashed PT_INTERP substitute
binfmt_elf: consume a stashed PT_INTERP substitute
exec: carry a PT_INTERP substitute in struct linux_binprm
...
Please consider pulling these changes from the signed ipc-7.3-rc1.misc tag.
Thanks!
Christian
-----BEGIN PGP SIGNATURE-----
iHUEABYKAB0WIQRAhzRXHqcMeLMyaSiRxhvAZXjcogUCan7RJAAKCRCRxhvAZXjc
ooOoAP9pECC4lIvizRLhV+8id21aYyyHP6svcZDgMv7yI8E7qgD/Zaxt0hS/rsLY
IUA9iphoZ6pWyPyv/Lq6M0mk3wyoigM=
=KcXC
-----END PGP SIGNATURE-----
mergetag object d7337cad4d
type commit
tag kernel-7.3-rc1.misc
tagger Christian Brauner <brauner@kernel.org> 1786695972 +0200
kernel-7.3-rc1.misc
Please consider pulling these changes from the signed kernel-7.3-rc1.misc tag.
Thanks!
Christian
-----BEGIN PGP SIGNATURE-----
iHUEABYKAB0WIQRAhzRXHqcMeLMyaSiRxhvAZXjcogUCan7RJAAKCRCRxhvAZXjc
osV7AP0cdSyVJmxYRbryVuQykwbH83e+VyUwla8Qy4Lx7DaPqgEAg6MyH2kFZCKY
eahHCBwW7B5ef9x08zw1BpLwjhHNkQU=
=MhTX
-----END PGP SIGNATURE-----
Merge tags 'ipc-7.3-rc1.misc' and 'kernel-7.3-rc1.misc' of git://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs
Pull misc ipc and core updates from Christian Brauner:
- reject mq_notify() with a zero signal number
- fix coding style in the exit path
* tag 'ipc-7.3-rc1.misc' of git://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs:
mqueue: reject mq_notify with signo 0
* tag 'kernel-7.3-rc1.misc' of git://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs:
kernel: exit: fix coding style missing spaces
add_subprogs() rejects both BPF-to-BPF calls and BPF_PSEUDO_FUNC loads for
unprivileged programs. The latter loads a subprogram address for use as a
callback, but its Policy report currently describes it as a function call and
suggests avoiding calls that the program does not contain.
Select the operation and suggestion from the instruction kind. Preserve the
existing call wording for BPF_PSEUDO_CALL, and describe BPF_PSEUDO_FUNC as a
BPF function reference.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Link: https://lore.kernel.org/bpf/d02e6a6d3b2dc43a207b8ba836ce62497b250dede9252e7409c5212201c794b7@mail.kernel.org
Link: https://lore.kernel.org/bpf/20260816015746.2632990-14-memxor@gmail.com
GCC emits BTF line records with a file name and line number, but leaves the
source line string empty. bpf_diag_source() currently treats that empty string
as if the complete line record were unavailable, so diagnostics fall back to
an instruction number and discard the function, file, and line attribution.
Print the available source location before deciding whether source context can
be rendered. When source text is absent, omit only the source context and retain
the diagnostic annotation and instruction context.
Fixes: b9c5d822f6 ("bpf: Add source and instruction diagnostic context")
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://lore.kernel.org/bpf/20260816015746.2632990-12-memxor@gmail.com
The Call Type Safety diagnostics mishandle three kfunc argument classes.
BTF type ID 0 represents void, but btf_show_name() also uses zero to end
type traversal. A pointer that resolves to void therefore loses its pointee
name and is rendered as "()". End traversal directly for concrete terminal
types, but resolve referenced types before testing for ID zero, and name the
void terminal type explicitly. Format the complete parameter pointer type
for nullable kfunc arguments, so void pointers are reported as (void *).
Also add the missing structured report when an __szk memory-size argument is
not a verifier-known constant. Describe the generic bpf_refcount_acquire()
contract without deriving an object type from its void pointer prototype.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://lore.kernel.org/bpf/668871823f90f69896d3db27b56db2f53e481162.camel@gmail.com
Link: https://lore.kernel.org/bpf/20260816015746.2632990-7-memxor@gmail.com
The main diagnostic identifies the first outgoing stack slot as stack
argument 1 and the sixth function argument. The causal history instead
labels the same value as stack arg6, making it look like a different slot.
Render causal-history targets in the verifier's canonical stack-argument
location form. The first outgoing slot is now shown as *(R11-8), matching
reg_arg_name(), while the main diagnostic retains its fuller slot and
ordinal description.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://lore.kernel.org/bpf/eb1be5327d136b7e5bd6d68e76fef6de20c40790.camel@gmail.com
Link: https://lore.kernel.org/bpf/20260816015746.2632990-6-memxor@gmail.com
A cgroup getsockopt BPF program can shrink ctx->optlen after the
kernel getsockopt handler has run. The kernel-buffer variant, used by
TCP_ZEROCOPY_RECEIVE, only rejects values larger than the original
length.
If BPF writes a negative optlen, that value is accepted and propagated
back to the TCP getsockopt code. It can then be passed to
copy_to_sockptr() as a size_t and trigger the hardened usercopy
bytes > INT_MAX warning.
Reject negative ctx.optlen in __cgroup_bpf_run_filter_getsockopt_kern(),
matching the lower-bound validation already present in the sockptr-based
getsockopt hook.
Fixes: 9cacf81f81 ("bpf: Remove extra lock_sock for TCP_ZEROCOPY_RECEIVE")
Reported-by: Sechang Lim <rhkrqnwk98@gmail.com>
Signed-off-by: Junseo Lim <zirajs7@gmail.com>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Reviewed-by: Emil Tsalapatis <emil@etsalapatis.com>
Link: https://lore.kernel.org/bpf/187a4d756275aaaee5d65eecb63c1477b3b66554.1786448307.git.zirajs7@gmail.com
bpf_convert_ctx_accesses() turns a BPF_LDX into a BPF_PROBE_MEM one by
matching the type recorded for the insn against a list of exact pointer
types. The list cannot keep up with the flag combinations the verifier
produces, and a type which is missing from it ends up as a plain load
without an exception table entry, so a bad address panics the kernel
instead of being handled.
Two such types exist today and are reachable:
- PTR_TO_BTF_ID | PTR_UNTRUSTED | MEM_ALLOC | NON_OWN_REF
- PTR_TO_BTF_ID | PTR_UNTRUSTED | MEM_RCU
Rather than adding the two, just drop the list and state the property
itself in the default case of the switch. This is a superset of what
the list matched, the untrusted PTR_TO_MEM does not have to carry
MEM_RDONLY for it anymore, and it stays in sync with the verifier side
which uses the same match in save_aux_ptr_type() and reg_type_mismatch_ok().
Assert that a fault prone type which does not get the rewrite for whatever
reason is rejected at load time rather than left to fault at runtime to
catch any future cases.
Fixes: 1b12171533 ("bpf: Mark direct ld of stashed bpf_{rb,list}_node as non-owning ref")
Fixes: 6fcd486b3a ("bpf: Refactor RCU enforcement in the verifier.")
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://lore.kernel.org/bpf/20260814215301.709827-4-daniel@iogearbox.net
check_ptr_to_btf_access() allows the program to store before the default
BTF access path gets to reject a non read access. ac65c710cc ("bpf:
Reject writes through untrusted BTF pointers") closed that for a
PTR_UNTRUSTED pointer, but a bare PTR_TO_BTF_ID may fault on a dereference
just the same and is let through.
A BPF_LDX gets the BPF_PROBE_MEM rewrite in bpf_convert_ctx_accesses()
and a bad address is handled, but a BPF_STX does not and cannot, there
is no probed store to rewrite. The store is emitted as a plain one without
an exception table entry and a bad address panics the kernel.
A bpf_qdisc program can reach this, bpf_qdisc_btf_struct_access() permits a
write to Qdisc::limit and Qdisc::next_sched is a plain struct Qdisc pointer
which the walk turns into the compat type:
struct Qdisc *next = sch->next_sched;
next->limit = 1000;
BUG: kernel NULL pointer dereference, address: 0000000000000014
RIP: 0010:bpf_prog_c6e14e7f32c8e325_bpf_fifo_enqueue+0x3a/0x12b
Code: [...] bf e8 03 00 00 <89> 7e 14 41 8b 7f 14 [...]
Kernel panic - not syncing: Fatal exception in interrupt
Fix by widen the check to bpf_may_fault_on_deref() so that it covers both.
Fixes: 27ae7997a6 ("bpf: Introduce BPF_PROG_TYPE_STRUCT_OPS")
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://lore.kernel.org/bpf/20260814215301.709827-3-daniel@iogearbox.net
reg_type_mismatch_ok() enumerates the pointer types which must not
silently share a BPF_LDX with a different one, since the type recorded
for the insn drives a rewrite in bpf_convert_ctx_accesses().
f2362a57ae ("bpf: allow void* cast using bpf_rdonly_cast()") added
PTR_TO_MEM | MEM_RDONLY | PTR_UNTRUSTED as another type in need of one,
namely the BPF_PROBE_MEM rewrite, but did not add it there. Fix it by
adding the missing case to reg_type_mismatch_ok(), so that a PTR_TO_MEM
which may fault on deref is not mismatch ok anymore. The triage in
save_aux_ptr_type() then merges them.
Fixes: f2362a57ae ("bpf: allow void* cast using bpf_rdonly_cast()")
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://lore.kernel.org/bpf/20260814215301.709827-2-daniel@iogearbox.net
When the same BPF_LDX instruction is reached through paths that yield
different pointer types, save_aux_ptr_type() merges them into a single
type which is later used by bpf_convert_ctx_accesses() to decide whether
the load has to be rewritten into a BPF_PROBE_MEM one.
Before f2362a57ae ("bpf: allow void* cast using bpf_rdonly_cast()")
the merge only accepted two PTR_TO_BTF_ID pointers and unconditionally
fell back to PTR_TO_BTF_ID | PTR_UNTRUSTED, so the merged type was always
one that gets the BPF_PROBE_MEM rewrite. However, the mentioned commit
widened the merge to also cover a PTR_TO_MEM base and replaced the
fallback by a union of the PTR_UNTRUSTED and MEM_RDONLY flags.
A union of flags though cannot express the property the later rewrite
is built upon, some examples:
- PTR_TO_MEM merged with PTR_TO_BTF_ID | PTR_UNTRUSTED gets
PTR_TO_MEM | PTR_UNTRUSTED but only the MEM_RDONLY variant is valid
- PTR_TO_MEM merged with a plain PTR_TO_BTF_ID gets PTR_TO_MEM
dropping the rewrite the latter type would have gotten
- PTR_TO_MEM | MEM_RDONLY merged with a plain PTR_TO_BTF_ID gets
PTR_TO_MEM | MEM_RDONLY which is not rewritten either since only
its PTR_UNTRUSTED variant is
In all three cases a program can take the unsafe path at runtime with a
NULL or otherwise bad pointer and panic the kernel on the faulting load:
BUG: kernel NULL pointer dereference, address: 0000000000000038
RIP: 0010:bpf_prog_77531a87032eeaf1_mixed_mem_btf_id_type+0x4b/0x65
Call Trace:
<TASK>
bpf_test_run+0x20b/0x460
bpf_prog_test_run_skb+0x650/0xbe0
__sys_bpf+0xb96/0x3140
__x64_sys_bpf+0x2c/0x40
do_syscall_64+0xba/0x590
Kernel panic - not syncing: Fatal exception in interrupt
Note that the last two shapes have to be fixed right here, otherwise
the merged type retains nothing which marks the load as fault prone,
thus no rule in bpf_convert_ctx_accesses() can recover it. Fix it by
normalizing the merged type instead.
Reuse it in is_load_acq_unsafe() to avoid open coding, and trim the
overly verbose comment which is more of an implementation detail of
bpf_convert_ctx_accesses() anyway.
Fixes: f2362a57ae ("bpf: allow void* cast using bpf_rdonly_cast()")
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://lore.kernel.org/bpf/20260814215301.709827-1-daniel@iogearbox.net
Atomic RMW verification records an instruction pointer type only when the
current destination is PTR_TO_ARENA. A second path can therefore reach the
same instruction with an ordinary pointer without comparing it against the
saved arena type.
The post-verification fixup uses the saved type to rewrite the instruction
to BPF_PROBE_ATOMIC for every path. Record the actual destination type for
all atomic RMW paths so the existing mismatch check rejects incompatible
uses of one instruction.
Fixes: d503a04f8b ("bpf: Add support for certain atomics in bpf_arena to x86 JIT")
Signed-off-by: Yiyang Chen <chenyy23@mails.tsinghua.edu.cn>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://patch.msgid.link/20260816-bpf-next-038-mixed-atomic-v1-v2-1-4644c1886dbc@mails.tsinghua.edu.cn
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
time so that it receives the proper lag compensation
-----BEGIN PGP SIGNATURE-----
iQIzBAABCgAdFiEEzv7L6UO9uDPlPSfHEsHwGGHeVUoFAmqB/CQACgkQEsHwGGHe
VUq4Eg//ZeldqtFqUxohjcy5ZrgJ9dhdxwRfqZgYhZCSqTJHVLqAxWVAHnJZWfiz
Vh63rnt78rIPpVX6E/lZLnYm2VwhEO6P6piMzG4CVlh2pMySjhoRIQ3ufNAQRt0o
P79Y29rLhDhHkOaL+jjgSr+ePiDzerrkBfYHK0wJ+BAjphjWxML1wYyCGwhWk/Lu
KuXN/jzbEbAn2QWEwEy9KyxztzJlTYTE+l8jiGfRywAeZOBo8HaXg0HhuCwLnaXb
yPmarhof2/7XUdW/CBGYggLaXF+mW6VeMaiqdhxSKl48KMpIfPnBC99/YCJy6vmQ
pD+kOiysGSFy+3vMbTvjwOYV8T3g7LOpeVkY8KkVmAHUFVF9wBSyPULyooNuxGS9
2pBv6Uz2ojm3wMVk+gggt2VU5uVNLsn9IKpNObyuBRDkt3My4Jej3cQ89LeqyS8Z
q49JbAhEwCRfGpxq92WW6izMWjOnduhiTd9TXF/WoXVtcT9ZSbyxJ3sCxbTBrJxd
na7xln2xsR8w9+G91DmmNPRnBtBkBOJ6xRGacKDeV8dcfKcZGGH6/sRMN9mIhP8v
huCVazSwNCVWDaHn8o/ORBJ0dEJ/536a806ysYfB6MdNzZYtU8iepCj5Lr+uAG/Q
W49ftqYFst9o22wT2i4ZdTfrDrkbekVU67BLv9yu8VcmoQJgt90=
=bK3W
-----END PGP SIGNATURE-----
Merge tag 'sched_urgent_for_v7.2' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
Pull scheduler fix from Borislav Petkov:
- Make sure a delayed sched entity's runtime stats are updated at the
right time so that it receives the proper lag compensation
* tag 'sched_urgent_for_v7.2' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip:
sched: Update time before requeueing delayed entities
- Avoid use-after-free of an event's group leader by promoting detached
sibling events to standalone entities and correct related accounting and
state transitions
-----BEGIN PGP SIGNATURE-----
iQIzBAABCgAdFiEEzv7L6UO9uDPlPSfHEsHwGGHeVUoFAmqB6xQACgkQEsHwGGHe
VUpWFQ/+KyU6R2DC48ifpChzgTFmQ2gmIjy1IIsWSGmdSbyD5CEWar8hHDRq1X9r
peuQizTXTQ2Ze75atTMIBExR6eUFYb2sKA1HpobcGnB96cpK8e2vmxhvXoHgB3IZ
TYDXQ1RMPnbyTn2rCQwr5cgSq2Qe7w2tMXMQKzfmDxP1r7cE81zJVFAsRdVVZP75
kvlcCL43pIwzxzP7sDb9bU9lTkW2Sw83dLIp3jBd3iiUpIQpwqV+UG3/fkInV58n
L56cYFsovLbvWdxq4oj6cFwRBmvyrYKV0zkT+zW0SO2AzlVhfwCD/o74hxLwrN//
Gas0d51uQfWt+5M7s6T0KFQYBfClG4uoIi2yh7zXxWEXyhcNuvmAVrY3xiQxReIi
m88+ByWHfBc/mYTHKWJAqb8sHhJiktU52T55ktOJaPNGczA5+O/4alnOg3Kxvw7d
CXp1raxJLqDxvd7Ubu/LVjWY96ds+fAaC15ydC6Lh08b9LPhA8rRCavf7NeSOzOD
E1NA3QLS3TwbrTboaGqLMmzOLPkBBZ+28PUASg1ZpyfGnw91Ggv/gOVNUgr5PvMU
D/gzRCwEYfDdhOrIHzGrfJ1fwj4qFsZ6HSge3sHDj8/BCcQYj/zmlTKe0CHKbaiy
D7bkLNwdBc0z+eh5T7UfCYodUu0qiiek5Y0G3q8FVad0mFDYtUA=
=moR8
-----END PGP SIGNATURE-----
Merge tag 'perf_urgent_for_v7.2' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
Pull perf fixes from Borislav Petkov:
- Prevent the use of exited events as group leaders
- Avoid use-after-free of an event's group leader by promoting detached
sibling events to standalone entities and correct related accounting
and state transitions
* tag 'perf_urgent_for_v7.2' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip:
perf/core: Fix group leader use-after-free after sibling detach
perf: Reject exited events as group leaders
dequeue_task_scx() masks SCX_DEQ_CORE_SCHED_EXEC out of the
SCX_DEQ_SCHED_CHANGE decision, but the test can never fire: the incoming
flags are an int of generic DEQUEUE_* bits while the flag is bit 32, and the
core-sched execute path never goes through class dequeue anyway -
set_next_task_scx() calls ops_dequeue() with the flag directly. The test was
live when the SCX_DEQ_SCHED_CHANGE computation sat in ops_dequeue() and
became dead when 03f5304aad ("sched_ext: Pass full dequeue flags to
ops.quiescent()") moved the computation here. Drop it.
Signed-off-by: Tejun Heo <tj@kernel.org>
With sub-schedulers, tasks of different schedulers routinely share rqs and
SMT siblings, but scx_prio_less() consults ops.core_sched_before() only when
both tasks belong to the same scheduler. Every pair spanning two schedulers
falls back to the default ordering, so no scheduler can express ordering
across a scheduler boundary, including a root over its sub-schedulers'
tasks.
Order a pair spanning schedulers by the nearest common ancestor that
implements ops.core_sched_before(): both tasks are in its subtree, making
this the one op where a scheduler is called on tasks it delegated to its
sub-schedulers and may not be scheduling anymore. Same-scheduler pairs keep
using the owning scheduler's op so a parent never orders inside a subtree it
delegated. The op is skipped when the deciding scheduler is bypassing on
either task's CPU.
Update scx_qmap to fall back to the kernel's default ordering when handed a
delegated task it has no task_ctx for.
Signed-off-by: Tejun Heo <tj@kernel.org>
The default core-sched ordering runs the longest waiting task first by
comparing p->scx.core_sched_at stamps. The stamp is maintained under two
rules. touch_core_sched() stamps when a task starts waiting for a CPU and
when its slice runs out. If the scheduler implements
ops.core_sched_before(), touch_core_sched_dispatch() re-stamps on every
dispatch.
A comparison can see one stamp taken under each rule, which isn't a
meaningful ordering. The dispatch rule also buys little - it only aligns
bypass-mode comparisons with the local DSQ order. Multiple schedulers make
the mixed comparisons more common.
Wait time is what p->scx.runnable_at already tracks for the stall watchdog.
Delete core_sched_at with both touch functions and compare runnable_at in
the scx_prio_less() fallback.
runnable_at is refreshed only on enqueue and goes stale while a task keeps
occupying its CPU. Instead of re-stamping, order a running task after every
waiting task as it is the most recently serviced.
Signed-off-by: Tejun Heo <tj@kernel.org>
scx_prio_less() implements prio_less() semantics - %true means that @a is
the lower priority and should run after @b. ops.core_sched_before() is
documented to return %true when @a should run before @b. scx_prio_less()
returns the op's value as-is, inverting the documented semantics at runtime.
Call the op with the arguments swapped.
scx_qmap followed the wiring instead of the documentation and returned %true
for the younger task, so the two inversions canceled out and it behaved as
intended. Flip its comparison to match. scx_qmap is likely the only current
user in or out of the kernel tree. Any scheduler written the same way needs
the same flip, while schedulers following the documentation are fixed by
this change.
Fixes: 7b0888b7cc ("sched_ext: Implement core-sched support")
Cc: stable@vger.kernel.org # v6.12+
Signed-off-by: Tejun Heo <tj@kernel.org>
The EOPNOTSUPP stubs for the sub-cap kfuncs live in ext.c under #ifndef
CONFIG_EXT_SUB_SCHED while the real definitions live in sub.c. Move the
stubs into sub.c so all sub kfunc definitions live in one file. Pure code
move, no functional change.
Signed-off-by: Tejun Heo <tj@kernel.org>
futex_hash_allocate() allocates mm->futex.phash.ref without any locking.
Commit d9b05321e2 ("futex: Move futex_hash_free() back to __mmput()")
moved the allocation here and assumed that the process has just a single
thread at this point.
Commit ee9dce4436 ("futex: Drop CLONE_THREAD requirement for private
default hash alloc") widened need_futex_hash_allocate_default() to cover
any CLONE_VM clone, but left out vfork because the parent is suspended and
cannot race.
That no longer holds once vfork is nested. If a vfork child calls vfork
again and is then killed with SIGKILL, the parent is released from its
vfork wait and runs concurrently with the grandchild in the same mm.
Neither of them went through futex_hash_allocate_default().
When both call prctl(PR_FUTEX_HASH, PR_FUTEX_HASH_SET_SLOTS) at the same
time, each one sees mm->futex.phash.ref as NULL and stores its own percpu
counter. Only the last store survives. The counter stored first is no
longer reachable from the mm, so the references on it are not seen by
__futex_ref_atomic_end(). A private hash that still has references is then
considered dead and freed, and a task that still holds one of its buckets
writes into freed memory in futex_q_lock().
Store the counter once with cmpxchg() and let the loser free_percpu() its
own. The initial reference has to be taken before the store, otherwise
another task can install a private hash while the counter is still 0.
Fixes: d9b05321e2 ("futex: Move futex_hash_free() back to __mmput()")
Signed-off-by: Hyunwoo Kim <imv4bel@gmail.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Cc: stable@vger.kernel.org
Link: https://patch.msgid.link/ansrpP4ImE1MaBY9@v4bel
futex_exit_release() and futex_exec_release() are identical now. That means
also exit_mm_release() and exec_mm_release() are identical.
Consolidate the whole lot and remove the redundant copies.
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Reviewed-by: Kyle Zeng <kylebot@openai.com>
Acked-by: Peter Zijlstra <peterz@infradead.org>
The check for private futexes whether the waiter's mm, which is stored in
the futex_key and copied into the pi_state, is the same as the owner's mm
is not sufficient for exec(). exec() has a gap where the mm check fails to
give the correct answer:
exec()
...
exec_release_mm()
futex_exec_release()
tsk::futex::exit_state = EXITING;
cleanup_robust_list();
1) tsk::futex::exit_state = OK;
...
old_mm = tsk::mm;
2) tsk::mm = ->mm;
Between #1 and #2 the check for the mm is wrong as that mm is about to be
swapped out and eventually freed.
Plug this gap by:
1) Setting tsk::futex::exit_state to FUTEX_STATE_DEAD in
futex_exec_release()
2) Setting tsk::futex::exit_state to FUTEX_STATE_OK after
the mm has been switched.
From a futex point of view the task is dead after it finished the robust
list cleanup up to the point where it sets the state to OK again.
Fixes: 80367ad01d ("futex: Add basic infrastructure for local task local hash")
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Reviewed-by: Kyle Zeng <kylebot@openai.com>
Acked-by: Peter Zijlstra <peterz@infradead.org>
Cc: stable@vger.kernel.org
Add BPF kfuncs that allow BPF LSM programs to create and use sockets for
sending data. This provides a mechanism for BPF programs to emit
telemetry. For this first patch set, it's restricted to SOCK_DGRAM
socket types with IPPROTO_UDP protocol but could be easily extended to
SOCK_STREAM and IPPROTO_TCP in the future.
The API consists of five kfuncs:
bpf_ksock_create() - Create a socket (sleepable)
bpf_ksock_connect() - Connect socket to remote address (sleepable)
bpf_ksock_send() - Send data through the socket (sleepable)
bpf_ksock_acquire() - Acquire a reference to a socket context
bpf_ksock_release() - Release a reference (cleanup via
queue_rcu_work since sock_release sleeps)
The setup kfuncs bpf_ksock_create, bpf_ksock_connect, can be called from
SYSCALL programs only. While bpf_ksock_acquire, bpf_ksock_release and
bpf_ksock_send can be called from SYSCALL and LSM programs.
The implementation follows the established kfunc lifecycle pattern
(create/acquire/release with refcounting, kptr map storage, dtor
registration). The kernel socket is wrapped in a refcounted bpf_ksock
struct. Cleanup is deferred via queue_rcu_work() because sock_release()
may sleep.
The kfuncs are only compiled when CONFIG_INET is enabled, as they
specifically support AF_INET and AF_INET6 sockets.
The socket operations go through the expected LSM hooks instead of
by-passing them like many kernel sockets since those are created by BPF
programs and thus system users. Thus, the bpf_ksock_send() kfunc, which
is exposed to LSM progs has a verifier filter protection to avoid
recursion so that the whole bpf_kfunc_set kfunc set cannot be called in
a program attached to security_socket_sendmsg(). Also, because of the
LSM checks, we prevent the use of the kfuncs from asynchronous workqueue
as the current value would then be invalid.
In bpf_ksock_create(), we copy the arg values to avoid TOCTOU races
since the kfunc can sleep and the arg values could be stored in a map
that could be re-written by BPF progs or even userspace programs if the
map is mmaped.
Signed-off-by: Mahe Tardy <mahe.tardy@gmail.com>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Reviewed-by: Jiayuan Chen <jiayuan.chen@linux.dev>
Acked-by: Stanislav Fomichev <sdf@fomichev.me>
Acked-by: Song Liu <song@kernel.org>
Link: https://lore.kernel.org/bpf/20260813110540.103550-3-mahe.tardy@gmail.com
Augment selected helper and kfunc allowability failures with Policy reports.
These reports explain which requested operation is forbidden and why, without
adding path history for non-path-dependent policy checks.
Cover unprivileged bpf2bpf and kfunc use, helper program-type restrictions,
GPL-only helpers, helper-specific allow callbacks, kfunc allowability, and
destructive kfunc capability checks.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://patch.msgid.link/20260815064612.378577-15-memxor@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Augment selected whole-program and subprogram CFG validation failures with
Program Structure reports. These errors are structural rather than
path-dependent, so the reports focus on source and instruction context
instead of causal history.
Cover direct and indirect jumps outside the program or current subprogram,
unprivileged backedges, missing and out-of-range jump tables, targets in the
second half of an ldimm64, unreachable instructions, subprogram fallthrough,
and recursive bpf2bpf call graph edges.
Format long jump-range reasons directly in diagnostics.c, and keep the
fallthrough suggestion aligned with the verifier check by suggesting exit or
explicit jumps.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://patch.msgid.link/20260815064612.378577-14-memxor@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Augment selected sleepability and critical-section failures with Execution
Context Safety reports. Keep the existing verifier messages and add source
context, path history, and suggestions tied to the active context.
Use the context history recorded earlier to anchor causal paths to lock, IRQ,
RCU, and preempt regions instead of unrelated register updates.
Cover global calls while holding a lock, sleepable global function calls,
sleepable helpers, sleepable kfunc calls from disallowed contexts, operations
that exit while a context is still active, and unmatched context exits.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Link: https://patch.msgid.link/20260815064612.378577-13-memxor@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Augment selected Resource Lifetime Safety failures with structured diagnostics
while preserving the existing verifier messages.
Report unreleased references from check_reference_leak() using
reference-scoped diagnostic history, and add state reports for dynptr,
iterator, lock, and IRQ-flag lifetime misuse.
IRQ restore mismatch and out-of-order diagnostics use IRQ context-scoped
history when an IRQ-disabled region is active, so retained save/restore context
is still visible after per-state history removal.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Link: https://patch.msgid.link/20260815064612.378577-11-memxor@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Augment selected register-state verifier failures with Register Type Safety
reports. The existing verbose verifier messages remain in place; the new
reports add reason, source context, causal path, and suggestions.
Cover invalid pointer dereferences, unreadable registers, missing outgoing
stack arguments for bpf2bpf and kfunc calls, and rejected pointer arithmetic.
Use scoped diagnostic history so reports start from the latest relevant value
change and then show later branch outcomes.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://patch.msgid.link/20260815064612.378577-9-memxor@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Record verifier context transitions in the diagnostic history so later reports
can anchor causal paths to the critical section that made an operation invalid.
This covers lock, IRQ, RCU, and preempt regions without adding any new
verifier error reports. Category-specific commits decide where those recorded
events should be rendered.
Use context depth when selecting scoped history so nested regions anchor at the
outer active region, and fall back to the earliest retained event when the
matching entry was pruned.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://patch.msgid.link/20260815064612.378577-8-memxor@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Add reference acquire and release events to diagnostic history so Resource
Lifetime Safety reports can show the lifetime of a specific reference id along
the path.
Record acquisitions after the verifier assigns the reference id. Record
releases only after release_reference_nomark() succeeds, including the
kptr_xchg RCU conversion path and owning-to-non-owning conversion path that
consume an owning reference.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://patch.msgid.link/20260815064612.378577-7-memxor@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Record material register and outgoing stack argument changes so diagnostics can
explain how a value reached its current type, bounds, or unreadable state.
Store old and new register types, scalar ranges, tnum value and mask, map and
BTF type identity, and basic operand metadata in the environment-owned
diagnostic event stream.
Record invalidations when packet data moves, references are released, or
borrowed references leave their protected region. Register-scoped history
starts at the latest matching modification and then shows later branch
outcomes.
Also record fixed stack spills and overwrites, and tag register fills from
stack so register-scoped history can follow value flow through spilled stack
slots.
The type_is_map_ptr() helper previously lived as a static function in
kernel/bpf/log.c since commit 0c95c9fdb6 ("bpf: emit map name in register
state if applicable and available"). Move it verbatim to
include/linux/bpf_verifier.h as a static inline, next to the other type
classifiers, so diagnostics.c can reuse it without duplicating the case list.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Link: https://patch.msgid.link/20260815064612.378577-6-memxor@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Save the diagnostic event-log position with each verifier stack entry and
reset the environment-owned stream together with the normal verifier log
when a queued state is popped. Also reset the diagnostic stream after
successful subprogram verification even when level-2 logging preserves the
normal verifier log.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Link: https://patch.msgid.link/20260815064612.378577-5-memxor@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Add an environment-owned diagnostic history for verifier reports. Event
payloads keep the user-facing branch history shape, while storage lives
in bpf_verifier_env and follows the active verifier path.
Grow the event array geometrically up to a 64 MiB limit. Once storage
reaches the limit, or an allocation fails, overwrite the oldest event so
diagnostics retain the newest useful suffix without adding per-event
metadata.
Represent saved positions as absolute logical sequence numbers. A restore
truncates to a retained position. If its prefix has already been evicted,
clear the abandoned suffix and preserve the missing-history position. This
keeps marks stable across rotation without increasing their size.
Add the branch event renderer and branch recording.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Link: https://patch.msgid.link/20260815064612.378577-4-memxor@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Teach verifier diagnostics to annotate an instruction with BTF source
line information and nearby BPF instructions. The renderer keeps source
text in a fixed-width lane and prints instructions in a stable right-hand
gutter.
Wrap annotation text under the source line so long error labels remain
readable while the source and instruction lanes keep their fixed layout.
Keeping source and instruction context in one commit preserves the visual
layout contract that later diagnostic reports rely on.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Link: https://patch.msgid.link/20260815064612.378577-3-memxor@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Add the initial diagnostics renderer for verifier reports and wire it into
the BPF build. The helper emits the common failure header through the
verifier log.
Later patches add prose wrapping, reusable report sections, and source and
instruction context for category-specific diagnostics.
Gate the helpers on normal verifier log output from the start, so
BPF_LOG_STATS-only loads do not collect or render diagnostics.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://patch.msgid.link/20260815064612.378577-2-memxor@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
rate_limit_us is an unsigned int, while NSEC_PER_USEC is defined as
1000L. On 32-bit systems, the multiplication is therefore performed
using 32-bit unsigned arithmetic before the result is assigned to
freq_update_delay_ns.
For example, writing 4294968 to rate_limit_us wraps the delay from
4294968000 ns to 704 ns. This makes schedutil update far more often
than configured.
Add sugov_update_rate_limit_us() to widen rate_limit_us to s64 before
converting it to nanoseconds. Use the helper when updating the tunable
through sysfs and when starting the governor, so both paths perform the
conversion without overflow.
Fixes: 9bdcb44e39 ("cpufreq: schedutil: New governor based on scheduler utilization data")
Signed-off-by: Hui Su <sh_def@163.com>
Reviewed-by: Zhongqiu Han <zhongqiu.han@oss.qualcomm.com>
Cc: All applicable <stable@vger.kernel.org>
Link: https://patch.msgid.link/20260806142304.1761454-1-sh_def@163.com
Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
sched_class->balance() is gone from sched_ext and what balance_one() does is
run dispatch to produce something pickable. Update the balance-era names to
dispatch terms:
- balance_one() -> dispatch_one()
- SCX_RQ_IN_BALANCE -> SCX_RQ_IN_DISPATCH
No BPF scheduler reads the flag. The enum autogen headers gain the new name
with the old entry retained like other removed enumerators, zero-filling at
load time. No functional changes.
Signed-off-by: Tejun Heo <tj@kernel.org>
The fixup demoting a keep verdict when @prev is not on ext_sched_class
guarded against the rq-level SCX_RQ_BAL_KEEP flag going stale back when
balancing and picking were separate operations.
The verdict now travels in the return value, created and consumed in one
invocation against the @prev it evaluated, and every keep decision tests
SCX_TASK_QUEUED under the rq lock, which implies ext_sched_class as a class
switch dequeues first. Drop the fixup along with dispatch_core_pick()'s
copy.
Signed-off-by: Tejun Heo <tj@kernel.org>
kick_sync_wait_bal_cb() assumes it runs on the rq's CPU from the
__schedule() tail: the snapshots it compares against live in that CPU's
percpu area and the busy-wait runs with the rq lock dropped and IRQs
enabled.
However, dispatch can now drop the rq lock while the callback sits queued,
and rq lock takers in that window (the sched class change paths, the scx
task iterator) flush pending balance callbacks on release, running the
callback on a foreign CPU. Such a run compares against unrelated snapshots
and can deadlock when the executing CPU is itself a wait target.
Bail on a foreign CPU and leave the wait state alone. The wait only observes
progress that the resched kicks already guarantee and the rq's next wait
picks up the stale cpus_to_sync bits.
Fixes: 4c95380701 ("sched/ext: Fold balance_scx() into pick_task_scx()")
Cc: stable@vger.kernel.org # v6.19+
Signed-off-by: Tejun Heo <tj@kernel.org>
kernel/sched/ext/arena.c uses the gen_pool allocator, which is built only
when GENERIC_ALLOCATOR is set. SCHED_CLASS_EXT doesn't select it, so on
configs where nothing else does, the build fails to link:
build_policy.o: undefined reference to `gen_pool_create'
build_policy.o: undefined reference to `gen_pool_for_each_chunk'
build_policy.o: undefined reference to `gen_pool_destroy'
Fixes: 9eca087deb ("sched_ext: Sub-allocator over kernel-claimed BPF arena pages")
Reported-by: kernel test robot <lkp@intel.com>
Closes: https://lore.kernel.org/oe-kbuild-all/202608151315.tvN3X0Oq-lkp@intel.com/
Closes: https://lore.kernel.org/oe-kbuild-all/202608151632.3p91bTQj-lkp@intel.com/
Signed-off-by: Tejun Heo <tj@kernel.org>
scx_bpf_dsq_reenq() queues a deferred reenq (dru) that runs from
run_deferred(), not ops.dispatch(). If the DSQ is destroyed before the dru
runs, process_deferred_reenq_users() sees dsq->id == SCX_DSQ_INVALID and
hits the BUG_ON. destroy_dsq() doesn't flush pending drus, so just skip.
tj: Read dsq->id once with READ_ONCE(). Reading it separately in the INVALID
check and the BUG_ON would leave a window where destroy_dsq() can
invalidate the id between the two reads and still trigger the BUG_ON.
Fixes: 84b1a0ea0b ("sched_ext: Implement scx_bpf_dsq_reenq() for user DSQs")
Cc: stable@vger.kernel.org # v7.1+
Signed-off-by: Tao Cui <cuitao@kylinos.cn>
Signed-off-by: Tejun Heo <tj@kernel.org>
scx_bpf_events() always reads the root scheduler's event counters, so a
sub-scheduler program querying its own events silently gets the root's
instead and has no BPF-visible way to read its own (the per-scheduler sysfs
"events" file is the only interface). Resolve the scheduler from the calling
program with scx_prog_sched(). Unassociated programs follow the usual
scx_prog_sched() resolution: the root scheduler under a pre-sub-attach
compat root and zeroed counters otherwise.
Also fix up the malformed comment into proper kerneldoc.
Signed-off-by: Tejun Heo <tj@kernel.org>
scx_root_disable() invalidates each rq's clock before taking the rq lock.
scx_rq_clock_invalidate() is a plain read-modify-write of rq->scx.flags and
every other writer of the word runs under the rq lock, so the unlocked
update can race a concurrent flags update and lose one side's bits.
The invalidation doesn't matter in the first place. The cached clock is read
only by scx_bpf_now() from a loaded scheduler's BPF programs, nothing can
re-validate the clock while sched_ext is disabled as scx_rq_clock_update()
is gated on scx_enabled() too, and the usual rq lock cycles under the next
scheduler refresh or invalidate it before it's practically observable. Drop
the invalidation instead of fixing the locking.
v2: Description and comment updated - the invalidation is unnecessary rather
than subsumed by the rq lock cycle below.
Fixes: 3a9910b590 ("sched_ext: Implement scx_bpf_now()")
Signed-off-by: Tejun Heo <tj@kernel.org>
Cc: Changwoo Min <changwoo@igalia.com>
Changes:
Make expedited grace periods expedite normal RCU callbacks
Miscellaneous fixes:
* Improve diagnostic output with character task states.
* Mark accesses to inform KCSAN of concurrency design.
* Move from kmalloc() to kmalloc_obj().
* Documentation updates.
* Improve handling of RCU deferred quiescent states.
* Clean up unused function arguments and structure fields.
* Reduce show_rcu_gp_kthreads() stack space.
Tasks RCU updates:
* Clean up after SRCU re-implementation of Tasks Trace RCU.
* Mark accesses to inform KCSAN of concurrency design.
* Add ->lazy_timer status to diagnostic output.
* Remove an unnecessary memory barrier.
* Fix a data race, courtesy of KCSAN.
* Documentation updates.
* Convert cond_resched_tasks_rcu_qs() from macro to static inline
function.
SRCU updates:
* Add Rust helpers for SRCU.
* Avoid losing queued work at cleanup_srcu_struct() time.
Torture-test updates:
* Preparation work for immediate RCU priority deboosting.
* Test RCU readers from real interrupt handlers (as opposed to softirq).
* Simplify code through use of cpumask_next_wrap().
* Improve diagnostic output with character task states.
* Add rcutorture.nwriters parameter to allow lightweight stall testing,
and rcutorture.stall_only to make doing so easier.
* Test an RCU Tasks Trace grace period implying an RCU grace period.
* Make RCU Tasks Trace torturing track reader batches.
* Fix a data race, courtesy of KCSAN.
* Plug a shuffle_tmp_mask memory leak on kthread spawn failure.
This comment is missing a closing parenthesis:
* The in_atomic_preempt_off() check ensures that we come here holding
* the last preempt_count (which will get dropped once we return to
* __rcu_read_unlock().
This commit therefore adds it at the end of the sentence.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
An mmap-able BPF array map (BPF_F_MMAPABLE) has its backing memory
vmalloc'ed up front at map creation time. array_map_mmap() then wired up
the whole mapping eagerly via remap_vmalloc_range(), which calls
vm_insert_page() for every page of the map. For large maps this makes
every mmap() O(number of pages): an 8MiB map inserts 2048 PTEs per
mmap() and tears them all down again on munmap(), even when user space
only touches a few pages (or none at all).
Populate the mapping lazily instead, the same way the arena map already
does. array_map_mmap() now only performs the bounds check and returns,
leaving the PTEs unpopulated; pages are inserted on demand by a new
array_map_mmap_fault() handler. Because the memory is already resident,
the fault handler simply resolves the vmalloc page and hands it to the
fault path. This makes mmap() O(1), and munmap() proportional to the
number of pages that were actually faulted in rather than to the size of
the map.
The handler is reached through a new optional ->map_mmap_fault callback.
Maps that provide it get a vm_operations_struct with a .fault handler;
maps that populate their mapping eagerly keep the one they had. Both
share the same open/close callbacks, so the existing VMA accounting
(VM_MAYWRITE write-active tracking, freeze handling) stays centralized
rather than each map installing its own vm_operations_struct.
Callers that want the pages populated up front can still request that
explicitly with MAP_POPULATE. Kernel-side access to the map (via the
vmalloc address) is unaffected.
Time for one mmap()+munmap() of an 8MiB mmap-able array map:
before after
no MAP_POPULATE, no access 226us 1.1us
no MAP_POPULATE, access all pages 236us 1341us
MAP_POPULATE, no access 312us 493us
MAP_POPULATE, access all pages 318us 519us
Mapping without touching the data, which is what this change targets,
gets ~160x cheaper. Faulting in the whole mapping one page at a time is
more expensive than the eager remap_vmalloc_range() loop, so users that
do touch every page should ask for MAP_POPULATE. Note that MAP_POPULATE
is not free before this change either: it adds ~85us (226us => 312us)
for no benefit, as the mapping is already fully populated.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Song Liu <song@kernel.org>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Link: https://lore.kernel.org/bpf/20260814155623.111565-1-song@kernel.org
In overwrite mode ringbuf_avail_data_sz() picks the newer of the consumer
and overwrite positions before measuring how much data is available:
return prod_pos - max(cons_pos, over_pos);
max() is an ordering comparison, and consumer_pos, producer_pos and
overwrite_pos are unsigned long, i.e. 32-bit on 32-bit architectures,
where Documentation/bpf/ringbuf.rst allows them to wrap. Once one of the
two positions has wrapped and the other has not, max() returns the older
one: the result is then a modular difference close to 2^32, so the
function reports far more available data than the ring can hold. Pollers
using BPF_RB_AVAIL_DATA get a bogus figure, and epoll consumers can be
woken with nothing to read.
Compare distances rather than positions. prod_pos - X is the amount of
data produced since X for either position, wrap or no wrap, so the newer
position is simply the one with the smaller distance, which is also the
value the function wants to return.
64-bit hosts are unaffected in practice: their counters would need
16 EiB to wrap. Found by review of the same class of bug fixed in
"bpf: Fix pending_pos walk on 32-bit ring position wrap".
Signed-off-by: Israel Téllez García <i.tellez@btesa.com>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Link: https://lore.kernel.org/bpf/20260814124843.22041-3-i.tellez@btesa.com
The reservation path caches the position of the oldest not-yet-committed
record in rb->pending_pos and advances it past already committed records
on every reservation:
while (pend_pos < prod_pos) {
consumer_pos, producer_pos and pending_pos are unsigned long, i.e.
32-bit on 32-bit architectures, and Documentation/bpf/ringbuf.rst states
that these counters may wrap around there. Every other comparison in the
file is written as a difference, so modular arithmetic keeps them
correct across the wrap. This one is an ordering comparison, and it is
not wrap-safe.
Once producer_pos wraps past 2^32, prod_pos is small while pend_pos
still holds its pre-wrap value, so the loop condition is false and
pending_pos is never advanced again. Reservations keep succeeding for a
while, because bpf_ringbuf_has_space() uses differences, but
new_prod_pos - pend_pos grows as the producer advances, and once it
exceeds rb->mask every subsequent __bpf_ringbuf_reserve() call fails:
the kernel believes a pending record spans the whole buffer. The ring
never recovers, bpf_ringbuf_output() drops every event from then on, and
nothing is logged.
Observed on four armv7 devices (i.MX7 Dual, 6.6.52) running a
tracepoint-based collector with a 512 KiB ring and 160-byte records.
Every one of them stopped delivering after exactly 26846821 records and
4295491360 bytes had passed through the ring, at event rates between 441
and 862 records/s, that is after 8 h to 17 h of uptime: the trigger is
the byte count, not time or load. That figure is 2^32 plus 524064 bytes,
and the excess is one ring's worth of grace period, as expected while
new_prod_pos - pend_pos is still below rb->mask. The last reservation
that fits is the largest record boundary X with X + 160 <= 524287, and
since 2^32 mod 160 = 96 the boundaries after the wrap sit at
X = 64 (mod 160), giving X = 524064. Userspace kept consuming normally
until the producer stopped, then read zero records for good. With this
patch applied, one of the four devices took 10 GiB through the same ring
with no stall, while the three unpatched ones kept wedging at the same
byte count.
64-bit hosts are unaffected in practice: their counters would need
16 EiB to wrap.
Compare the two positions as a difference instead. pending_pos never
runs ahead of producer_pos, so the unsigned difference is the real
distance between them and stays correct across the wrap.
Fixes: cfa1a2329a ("bpf: Fix overrunning reservations in ringbuf")
Signed-off-by: Israel Téllez García <i.tellez@btesa.com>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Link: https://lore.kernel.org/bpf/20260814124843.22041-2-i.tellez@btesa.com
The rcu_read_delay() and srcu_read_delay() functions are suppose to
avoid (or at least to minimize) read-side delays during call_rcu()-based
forward-progress testing. Although rcu_read_delay() does account for
this by testing rcu_fwd_cb_nodelay, it only does so in some cases, and
srcu_read_delay() does not bother at all. And rcutorture testing does
occasionally turn up the MIN_FWD_CBS_LAUNDERED failure when running
forward-progress tests on Tree SRCU flavors.
This commit therefore makes both rcu_read_delay() and srcu_read_delay()
take an immediate exit if rcu_fwd_cb_nodelay is set.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Currently, rcu_torture_fwd_prog_cr() announces function entry,
but silently exits if rcu_fwd_emergency_stop is set or if there is no
->call() function available. This could cause confusion, leading people
to believe that rcu_torture_fwd_prog_cr() is on the job when it is not.
This commit therefore also announces the early exits from this function.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
If torture_shuffle_init() successfully allocates shuffle_tmp_mask but
then fails to create the torture_shuffle kthread, the cpumask is never
freed.
Free the cpumask directly on the kthread-creation error path.
Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Currently __this_cpu_inc() is used to increment elements of both
the rcu_torture_count[] and rcu_torture_batch[] arrays. However,
this can fail when the increments can happen in interrupt handlers,
as recently became possible. This commit therefore upgrades the uses
of __this_cpu_inc() to the interrupt-safe this_cpu_inc().
KCSAN located this issue.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
This commit adds the ->get_sp_seq and ->gp_diff fields to the
tasks_tracing_ops structure so that RCU Tasks Trace rcutorture runs will
track Reader Batch.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
An RCU Tasks Trace grace period is supposed to imply an RCU grace period,
and this implication is relied on by BPF. But this is not currently
tested. This commit therefore makes tasks_tracing_torture_read_lock()
sometimes use rcu_read_lock() instead of rcu_read_lock_trace(), thus
testing the required implication.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
This commit adds a stall_only module parameter that shuts off all
rcutorture kthreads other than the RCU CPU stall-warning test kthreads.
The purpose of this is to test production applictions' reactions to
CPU stalls, and with minimal additional overhead. Or you can omit the
stall-warning tests as well and get a heavy no-op, your choice!
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Believe it or not, there are people who would like to run rcutorture
without actually torturing RCU. For example, some people would like to
induce various types of stall warnings without placing any unnecessary
additional overhead on their systems running in production. And
rcutorture provides the stall_cpu, stall_cpu_holdoff, stall_no_softlockup,
stall_cpu_irqsoff, stall_cpu_block, and stall_cpu_repeat module parameters
in order to allow the user to force numerous types of stalls. In addition,
rcutorture provides a great number of other module parameters to allow the
user to reduce other overhead.
But unfortunately, there is no way to turn of the rcu_torture_writer()
portion of this torture test, which on my x86 laptop consumes somewhere
between 40% and 45% of a CPU. Although this is quite lightweight for a
torture test, it is not welcome on systems running production workloads.
This commit therefore adds an nwriters module parameter that defaults
to 1 but can be set to 0 in order to disable the rcu_torture_writer()
portion of the torture test, but that cannot be set to any other value
(that is what the fakewriters module parameter is for!). This reduces
the overhead to well under 1% of a CPU, which is much more likely to be
compatible with production workloads.
Reported-by: Breno Leitao <leitao@debian.org>
Reported-by: Puranjay Mohan <puranjay@kernel.org>
Reported-by: Usama Arif <usama.arif@linux.dev>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Use the kernel's standard symbolic task-state representation instead of
printing raw hexadecimal task-state values.
Suggested-by: Zqiang <qiang.zhang@linux.dev>
Co-developed-by: Wang Lian <lianux.mm@gmail.com>
Signed-off-by: Wang Lian <lianux.mm@gmail.com>
Signed-off-by: Kunwu Chan <kunwu.chan@gmail.com>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
The rcu_torture_preempt() function uses cpumask_next(), and if that
returns an out-of-bounds result, re-invokes cpumask_next() on -1.
Which is exactly what cpumask_next_wrap() does. This commit therefore
saves a couple of lines by instead using cpumask_next_wrap().
This was reported by metacode when asked to look for opportunities
to use cpumask_next_wrap() in kernel/rcu.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Although rcutorture has long had the irqreader module parameter, this
parameter results only in RCU readers in softirq handlers, specifically,
timers. This commit therefore uses smp_call_function_single() to test
RCU readers in real hardware interrupt handlers, thus providing the full
effect from the irqreader module parameter.
However, consistency/debug checks must account for the possibility that
the smp_call_function_single() handler function is directly invoked
from the idle loop, in which case, for example, in_hardirq() will
return false. This commit uses a per-CPU variable to record being in
the rcu_torture_irq() smp_call_function_single() handler function.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
This commit adds a check for failure to have fully deboosted a
multi-segmented RCU reader at the end of the full read-side critical
section. This check only happens for fully task-level readers, because
a a handler might have interrupted an already-boosted task-level RCU
reader, and a reader in that handler could then cause false positives.
The first failed check (due to an RCU reader that was not immediately
deboosted) causes a splat, but only when the disabled-by-default
deboost_timeliness_check module parameter is enabled. Regardless of the
value of this parameter, it produces a list of the segments making up that
RCU reader following a "Slow-deboost rcutorture reader segments" heading.
Subsequent failures fail silently, all in the name of keeping console
output down to a dull roar.
Although most uses of RCU priority boosting serve as debugging aids,
this might change, and in fact might already have changed. And allowing
(for example) RCU priority boosting to persist until the next scheduler
tick could cause an aggressively real-time system to miss sub-millisecond
deadlines. So we do need to find this sort of problem during testing,
and preferably not in the field.
The name and type of the newly added rcu_torture_ops function pointer
(named "->is_task_rcu_boosted()") may need to change should other
end-of-reader checks be needed. But let's start simple.
Oh, and Claude figured out that rcu_is_task_rcu_boosted() could be
lockless. Perhaps there is hope for AI yet! ;-)
[ paulmck: Apply Akira Yokosawa feedback. ]
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
scx_fork() initializes tasks when scx_init_task_enabled is set, but
scx_cancel_fork() only exits them when scx_enabled() is true. A fork
that fails in the enable window (between releasing scx_fork_rwsem and
setting __scx_enabled) runs ops.init_task() but never ops.exit_task().
Gate scx_cancel_fork() on scx_init_task_enabled.
Fixes: 4269c603cc ("sched_ext: Enable scx_ops_init_task() separately")
Cc: stable@vger.kernel.org # v6.12+
Signed-off-by: fangqiurong <fangqiurong@kylinos.cn>
Signed-off-by: Tejun Heo <tj@kernel.org>
- inlines.h: scx_bpf_dispatch() doesn't exist; the comment means
scx_bpf_sub_dispatch()
- internal.h: name %SCX_DEQ_SCHED_CHANGE instead of the never-defined
%SCX_DEQ_SAVE
- internal.h: @name shows up in the ops file in the scheduler's sysfs
directory, not a "kernel.sched_ext_ops" sysctl
Signed-off-by: Tao Cui <cuitao@kylinos.cn>
Signed-off-by: Tejun Heo <tj@kernel.org>
scx_process_sync_ecaps() sets up the dispatch context for
ops.sub_ecaps_updated() in the target cpu's pcpu context recovered from the
llist node. However, the context is per executing cpu: the dispatch kfuncs
resolve it with this_cpu_ptr() and the dispatch buffer lives in it. What the
dispatches target is determined by the rq recorded in the context, not by
which cpu's context it is. Under core scheduling the pick runs balance_one()
for sibling rqs, so a sync processed for a sibling invokes the op with the
executing cpu's context not set up and its dispatch kfuncs misoperate on a
NULL or stale rq.
Set up the executing cpu's dsp_ctx instead, matching scx_dispatch_sched().
The recorded rq keeps the dispatches targeting the synced cpu.
Fixes: b81a6c018c ("sched_ext: Add sub_ecaps_updated() effective-cap change notifier")
Reported-by: David Carlier <devnexen@gmail.com>
Link: https://lore.kernel.org/all/20260813045931.8691-1-devnexen@gmail.com/
Signed-off-by: Tejun Heo <tj@kernel.org>
ring_buffer_per_cpu::mapped tracks if a ring-buffer is either mapped by
user-space or if it is a persistent buffer. We already have user_mapped
for the former and ring_meta for the latter. Get rid of mapped and
instead create rb_is_static(). A static ring-buffer cannot be resized,
swapped or have its pages extracted.
Link: https://patch.msgid.link/20260813131152.3589632-10-vdonnefort@google.com
Signed-off-by: Vincent Donnefort <vdonnefort@google.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
The 'cpus' field in struct trace_buffer became useless in commit
8e7b58c27b ("ring-buffer: Just update the subbuffers when changing their
allocation order"). Remove it
Link: https://patch.msgid.link/20260813131152.3589632-9-vdonnefort@google.com
Fixes: 8e7b58c27b ("ring-buffer: Just update the subbuffers when changing their allocation order")
Signed-off-by: Vincent Donnefort <vdonnefort@google.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
The ring buffer order can be dynamically modified and temporarily
disables writing to do so. It is therefore safe to use the updated value
to calculate the maximum event size which can be written onto the ring
buffer.
However, notice it is hardly making any difference for trace_marker
because of the TRACE_MARKER_MAX_SIZE limit. For an 8KiB subbuf size,
trace_marker can take 4096 characters while it can 'only' take 4054
bytes for smaller subbufs.
Link: https://patch.msgid.link/20260813131152.3589632-8-vdonnefort@google.com
Signed-off-by: Vincent Donnefort <vdonnefort@google.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
ring_buffer_alloc_read_page() is racy with ring_buffer_subbuf_order_set,
it can allocate a reader page with an outdated order. This isn't a big
issue, the user can still re-allocate a new reader page and try again.
However, what is more problematic is if the value of subbuf_order
changes in the middle of ring_buffer_alloc_read_page(). In that case,
bpage->order might not match the actual allocated memory.
Use bpage->order for the allocation to prevent this race.
Cc: stable@vger.kernel.org
Link: https://patch.msgid.link/20260813131152.3589632-6-vdonnefort@google.com
Fixes: bce761d757 ("ring-buffer: Read and write to ring buffers with custom sub buffer size")
Reported-by: Sashiko <sashiko-bot@kernel.org>
Signed-off-by: Vincent Donnefort <vdonnefort@google.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
trace_buffer subbuf_size is read lockless in ring_buffer_read_page() and
ring_buffer_read_start(), while it can simultaneously be resized with
ring_buffer_subbuf_order_set().
Instead of trace_buffer::subbuf_size, use bpage::order in
ring_buffer_read_start() and ring_buffer_read_page().
In ring_buffer_read_start(), even with resize_disabled, there is still a
possibility of a race with a buffer modification. Hold the trace_buffer
mutex to synchronise with any pending ring buffer order modification.
trace_buffer::subbuf_size is now actually useless, remove it. Also,
create accessors rb_subbuf_capacity() and rb_page_capacity() which
return the actual size available for storing events, while
rb_subbuf_size() returns the actual subbuf page-size.
Cc: stable@vger.kernel.org
Link: https://patch.msgid.link/20260813131152.3589632-5-vdonnefort@google.com
Fixes: f9b94daa54 ("ring-buffer: Set new size of the ring buffer sub page")
Reported-by: Sashiko <sashiko-bot@kernel.org>
Closes: https://sashiko.dev/#/patchset/20260805153225.2096152-1-vdonnefort%40google.com # patch 1
Acked-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
Signed-off-by: Vincent Donnefort <vdonnefort@google.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
Discarding a cached reader page after a concurrent ring buffer resize
uses the new global subbuf_order for the free_pages() call. This
mismatched order may crashes the kernel or leaks memory because the cached
page was allocated under the old size.
Save the actual free_page order alongside the page address to ensure we
always refer to the correct value and do not rely on the potentially
stalled cpu_buffer->subbuf_order value. The simplest is to make
free_page a buffer_data_read_page which already covers exactly what we
need: a page address and a page order.
Cc: stable@vger.kernel.org
Link: https://patch.msgid.link/20260813131152.3589632-4-vdonnefort@google.com
Fixes: 8e7b58c27b ("ring-buffer: Just update the subbuffers when changing their allocation order")
Signed-off-by: Vincent Donnefort <vdonnefort@google.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
arm64 passes zero as the stack pointer while walking BPF frames, so
bpf_throw() leaves stale KASAN stack poison after jumping to the
exception callback.
Use the frame pointer as the fallback stack watermark.
Fixes: e74cb1b422 ("arm64: stacktrace: Implement arch_bpf_stack_walk() for the BPF JIT")
Signed-off-by: Mykyta Yatsenko <yatsenko@meta.com>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Tested-by: Ihor Solodrai <ihor.solodrai@linux.dev>
Link: https://lore.kernel.org/bpf/20260812-hello_world-v1-1-c3c2ddcb362d@meta.com
Please consider pulling these changes from the signed vfs-7.2-rc8.fixes tag.
Thanks!
Christian
-----BEGIN PGP SIGNATURE-----
iHUEABYKAB0WIQRAhzRXHqcMeLMyaSiRxhvAZXjcogUCan7RJAAKCRCRxhvAZXjc
olPOAP9C1jX75LIMjyoJb9lpVAvCdlrvf0wcD88NXVBylueILAEA6GC8/lVpvUX8
nSFAZojyykVsPitfUdsbLagltXwLqgU=
=/Rxi
-----END PGP SIGNATURE-----
Merge tag 'vfs-7.2-rc8.fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs
Pull vfs fixes from Christian Brauner:
- Don't warn when a mount is completed from another user namespace.
fsopen() records the caller's user namespace in fc->user_ns and
hands back an ordinary file descriptor. The task that calls
fsconfig(FSCONFIG_CMD_CREATE) doesn't have to be the one that
created the context, and mount_capable() lets it through as long
as the caller has CAP_SYS_ADMIN over fc->user_ns, which anyone in
an ancestor namespace does. So fc->user_ns != current_user_ns()
is something an unprivileged user can arrange.
Both overlayfs and binfmt_misc WARN_ON() that. Overlayfs already
has the same check as a plain error return in ovl_parse_param().
Drop the WARN_ON() and just refuse. Add selftests for both cases.
- Reject pid allocations through dead ancestor pid namespaces.
Require PIDNS_ADDING in every namespace that will receive the pid
before publishing any of them. That preserves the invariant that
free_pid() never decrements pid_allocated in a namespace whose
child_reaper is no longer live. The existing ENOMEM behavior is
unchanged.
* tag 'vfs-7.2-rc8.fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs:
pid: reject allocations through dead ancestor pid namespaces
selftests/filesystems: test completing a context from another user namespace
binfmt_misc: don't warn when the mount is completed from another user namespace
ovl: don't warn when the mount is completed from another user namespace
When a fair task is enqueued, we must update curr and more precisely
its vruntime before placing the enqueued task so avg vruntime will take
into account the last exec phase.
Example:
TA is an always running task in cgroup G0.
TB is a short running task (cyclictest) in cgroup G1.
The lag of TB always increases up the clamp limit because TB is placed
before TA(curr) is updated (since the last tick). When curr(TA) is
finally updated, its last exec phase provide positive lag to TB
Because TA and TB don't belong to the same group, enqueue_hierarchy() will not
update TA's entity when updating curr but only G0's entity at root level.
The same applies when dequeuing.
This is because update_curr() uses ->h_curr, rather than ->curr, and therefore,
while it is invoked on the root cfs_rq, which contains all the eevdf bits, it
does not do the right thing.
Fixes: 85570f10a4 ("sched/eevdf: Move to a single runqueue")
Signed-off-by: Vincent Guittot <vincent.guittot@linaro.org>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/20260812125039.1717249-1-vincent.guittot@linaro.org
In order to compute the right lag, it is required to update time to 'now'.
Without this, the delayed entity might appear younger than it really is and
receive less compensation for having waited.
Signed-off-by: Vincent Guittot <vincent.guittot@linaro.org>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
The function trace_event_update_all() does a scan of events looking to
replace enums with their values in the strings that get exported to the
event format files. It's run at boot up on all events and again when a
module loads.
The issue is that when a module loads, it still runs on *all* events.
There's no reason to process every event when a module loads as the
previous events have already been processed. Only execute on the events
that are loaded with the module.
Link: https://patch.msgid.link/20260813204226.29563591@gandalf.local.home
Fixes: 3673b8e4ce ("tracing: Allow for modules to convert their enums to values")
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
The check that prevents userspace from sending siginfo with si_code values
reserved to the kernel is duplicated across do_rt_sigqueueinfo(),
do_rt_tgsigqueueinfo() and do_pidfd_send_signal(). Move the check into a
helper so the rule lives in one place.
Link: https://lore.kernel.org/20260806133013.4341-1-include@grrlz.net
Signed-off-by: Bradley Morgan <include@grrlz.net>
Reviewed-by: Andrew Morton <akpm@linux-foundation.org>
Acked-by: Oleg Nesterov <oleg@redhat.com>
Cc: Christian Brauner <brauner@kernel.org>
Cc: Thomas Gleixner <tglx@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
cmd_attr_register_cpumask() and cmd_attr_deregister_cpumask() differed
only in which attribute they parsed and which action they passed on, so
take both as arguments. __free(free_cpumask_var) then removes the goto.
No functional change.
Link: https://lore.kernel.org/20260728202104.17839-3-include@grrlz.net
Signed-off-by: Bradley Morgan <include@grrlz.net>
Cc: Balbir Singh <bsingharora@gmail.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Patch series "taskstats: tidy up the cpumask command path".
Two small cleanups from reading kernel/taskstats.c. No functional change in
either one.
This patch (of 2):
taskstats_user_cmd() only calls the cpumask handlers after checking the
same info->attrs[] entry, so parse() never sees a NULL attribute. Drop
the check and its odd "return 1", which no caller tested for anyway.
No functional change.
Link: https://lore.kernel.org/20260728202104.17839-1-include@grrlz.net
Link: https://lore.kernel.org/20260728202104.17839-2-include@grrlz.net
Signed-off-by: Bradley Morgan <include@grrlz.net>
Cc: Balbir Singh <bsingharora@gmail.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
The verifier keeps per-subprogram metadata in three parallel arrays:
subprog_info, func_info, and func_info_aux. Dead code elimination can
remove whole subprograms, and adjust_subprog_starts_after_remove()
shifts subprog_info and func_info to close the gap, but leaves
func_info_aux in place. From that point on, func_info_aux[i] no longer
describes subprogram i.
Shift func_info_aux together with func_info so the three arrays stay
aligned after subprogram removal.
Reported-by: Sashiko <sashiko-bot@kernel.org>
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Link: https://lore.kernel.org/bpf/20260808064523.DE3E71F000E9@smtp.kernel.org
Link: https://lore.kernel.org/bpf/20260812231506.3558128-1-memxor@gmail.com
The following sequence may leads race between event_define_fields()
and update_event_fields():
CPU0 (loads module A) CPU1 (loads module B)
=============================== ===============================
load_module(A) load_module(B)
notifier_call_chain notifier_call_chain
trace_module_notify trace_module_notify
mutex_lock(&event_mutex) trace_event_update_all()
trace_module_add_events(A) down_write(&trace_event_sem)
__register_event(call_A)
__add_event_to_tracers(call_A)
event_define_fields(call_A)
for each f: list_for_each_entry(field,
list_add(&f->link, &class->fields, link)
&class->fields) field = class->fields->next;
Where access to the class->fields is not protected by the event_mutex in
trace_event_update_all().
This produces the following panic:
Unable to handle kernel access ... at virtual address 0000000000000018
pc : update_event_fields+0xf8/0x368
Call trace:
update_event_fields+0xf8/0x368
trace_event_update_all+0x7c/0x2b4
trace_module_notify+0x4c/0x1dc
notifier_call_chain+0x84/0x168
blocking_notifier_call_chain_robust+0x64/0xd4
load_module+0x10c8/0x123c
__arm64_sys_finit_module+0x230/0x31c
Fix by taking event_mutex in trace_event_update_all() before
trace_event_sem.
Cc: stable@vger.kernel.org
Fixes: b3bc8547d3 ("tracing: Have TRACE_DEFINE_ENUM affect trace event types as well")
Link: https://patch.msgid.link/2e5730d2-c631-da41-3a3a-ae35bb4895f3@allwinnertech.com
Signed-off-by: Michael Wu <michael@allwinnertech.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
A module-only event filter such as ":mod:foo" is cached with a NULL
event_mod->match when foo has not been loaded. If a later write tries to
remove a specific match from the same module, remove_cache_mod() passes
the NULL cached match to strcmp(), causing a NULL pointer dereference.
The issue can be reproduced from userspace:
echo ':mod:trace_events_kunit_missing' > /sys/kernel/tracing/set_event
echo '!foo_bar:mod:trace_events_kunit_missing' >> /sys/kernel/tracing/set_event
The second write must be a concatenation (">>") to not include O_TRUNC as
that would cause ftrace_clear_events() to clear the cached modules lines.
The crash was reproduced on x86_64 QEMU while KUnit workers contended on
the event tracing path:
BUG: kernel NULL pointer dereference, address: 0000000000000000
#PF: supervisor read access in kernel mode
RIP: 0010:strcmp+0x10/0x30
Call Trace:
__ftrace_set_clr_event_nolock+0x373/0x4a0
ftrace_set_clr_event+0xf0/0x180
ftrace_event_write+0xdf/0x110
vfs_write+0xf6/0x440
ksys_write+0x68/0xe0
do_syscall_64+0xf9/0x540
entry_SYSCALL_64_after_hwframe+0x77/0x7f
Check event_mod->match before comparing it, consistent with the existing
NULL checks for the cached system and event fields. The mismatched removal
continues to return -EINVAL; a broad cached module filter is removed with
"!:mod:<module>".
Cc: stable@vger.kernel.org
Link: https://patch.msgid.link/20260811173902.1927376-2-sh_def@163.com
Fixes: b355247df1 ("tracing: Cache \":mod:\" events for modules not loaded yet")
Reported-by: syzbot+4d3143c8e28f6266c636@syzkaller.appspotmail.com
Closes: https://lore.kernel.org/lkml/6a7a6b7f.9c11d2ce.289b96.00f8.GAE@google.com/
Signed-off-by: Hui Su <sh_def@163.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
Introduce global percpu data, inspired by the commit
6316f78306 ("Merge branch 'support-global-data'"). It enables the
definition of global percpu variables in BPF, similar to the
include/linux/percpu-defs.h::DEFINE_PER_CPU() macro.
For example, in BPF, it is able to define a global percpu variable like:
int data SEC(".percpu");
With this patch, tools like retsnoop [1] and bpfsnoop [2] can simplify
their BPF code for handling LBRs. The code can be updated from
static struct perf_branch_entry lbrs[1][MAX_LBR_ENTRIES] SEC(".data.lbrs");
to
static struct perf_branch_entry lbrs[MAX_LBR_ENTRIES] SEC(".percpu.lbrs");
This eliminates the need to retrieve the CPU ID using the
bpf_get_smp_processor_id() helper.
Additionally, by reusing global percpu data map, sharing information
between tail callers and callees or freplace callers and callees becomes
simpler compared to reusing percpu_array maps.
Links:
[1] https://github.com/anakryiko/retsnoop
[2] https://github.com/bpfsnoop/bpfsnoop
Signed-off-by: Leon Hwang <leon.hwang@linux.dev>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://lore.kernel.org/bpf/20260813152324.97937-4-leon.hwang@linux.dev
In the next commit, percpu_array map will add map_direct_value_addr
support.
IOW, it will add a map_type check in the iff condition of the
bpf_map_direct_read() code block, which will reduce the code block
readability.
Hence, factor out check_map_mem_read helper to improve the readability,
and the maintainability for the percpu_array map case.
Signed-off-by: Leon Hwang <leon.hwang@linux.dev>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://lore.kernel.org/bpf/20260813152324.97937-3-leon.hwang@linux.dev
There are many adjacent blank lines in kernel/bpf/ that have accumulated
over time.
Drop them for cleanup.
No functional changes intended.
Signed-off-by: Leon Hwang <leon.hwang@linux.dev>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Reviewed-by: Emil Tsalapatis <emil@etsalapatis.com>
Link: https://lore.kernel.org/bpf/20260813152324.97937-2-leon.hwang@linux.dev
kick_pool_pick() reads and writes p->wake_cpu while the scheduler can
update it concurrently. KCSAN reports:
BUG: KCSAN: data-race in kick_pool_pick+0xf8/0x2d8
race at unknown origin, with read to 0xffff000663229da4 of 4 bytes by
task 1817002 on cpu 40:
kick_pool_pick+0xf8/0x2d8
process_scheduled_works+0x2bc/0x888
worker_thread+0x394/0x548
kthread+0x1b8/0x1f0
ret_from_fork+0x10/0x20
value changed: 0x0000002b -> 0x0000002f
The race is harmless. wake_cpu is a best-effort placement hint: every
writer stores a valid CPU id and the wakeup path validates it through
select_task_rq(), so a stale value only affects which CPU the worker
wakes up on.
Mark both accesses with READ_ONCE() and WRITE_ONCE() to document that
they are intentionally racy and to stop the compiler from reloading or
tearing them.
Signed-off-by: Breno Leitao <leitao@debian.org>
Reviewed-by: Bradley Morgan <include@grrlz.net>
Signed-off-by: Tejun Heo <tj@kernel.org>
When testing a linux-next kernel with commit 59bd1d914b ("memblock:
warn when freeing reserved memory before memory map is initialized"),
the following warning was hit when there was a "nohz_full" kernel boot
parameter.
Cannot free reserved memory because of deferred initialization of the memory map
WARNING: mm/memblock.c:904 at __free_reserved_area+0xde/0xf0, CPU#0: swapper/0/0
:
Call Trace:
<TASK>
memblock_phys_free+0xcb/0x100
housekeeping_init+0x14c/0x170
start_kernel+0x207/0x450
x86_64_start_reservations+0x24/0x30
x86_64_start_kernel+0xda/0xe0
common_startup_64+0x13e/0x141
</TASK>
IOW, we shouldn't free memblock allocated memory so early
in the boot process when memory map isn't fully initialized in
deferred_init_memmap().
Fix it by saving the housekeeping cpumask memblock memory to be
freed into a llist free list in housekeeping_init() and add a new
housekeeping_late_init() helper to defer the actual freeing of memblock
memory to when initcall's are being processed. The cpumask memblock
memory is treated as a llist_node with the size of a "long" type which
is also smallest cpumask size that can be allocated.
The non-atomic version of the llist APIs are used as there is no
contention.
This commit depends on the presence of commit 7c2eee9c13 ("memblock:
don't touch memblock arrays when memblock_free() is called late")
to prevent a KASAN UAF bug report [1].
[1] https://lore.kernel.org/lkml/20260505051821.1107133-1-longman@redhat.com/
Fixes: 27c3a5967f ("sched/isolation: Convert housekeeping cpumasks to rcu pointers")
Signed-off-by: Waiman Long <longman@redhat.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Frederic Weisbecker <frederic@kernel.org>
Reviewed-by: Phil Auld <pauld@redhat.com>
Link: https://patch.msgid.link/20260701195810.477326-1-longman@redhat.com
Under heavy concurrent DMA traffic on CoCo VMs, inc_used_and_hiwater()
performs an atomic_long_add_return() plus a CAS loop on the global
used_hiwater, and dec_used() performs an atomic_long_sub() on total_used.
All CPUs contend on the same cacheline, causing measurable throughput
degradation at scale.
Historically these counters were only compiled in under CONFIG_DEBUG_FS,
which means production kernels with debugfs paid the atomic overhead
unconditionally. Make the tracking boot-time opt-in instead so that it
is disabled by default with near-zero overhead via static_call, and can
be enabled via "swiotlb=track_hiwater" parameter on demand for debugging.
Note that when CONFIG_DEBUG_FS is enabled but hiwater tracking is disabled,
the "io_tlb_used" metric reports an approximate value rather than an
instantaneously exact one.
Suggested-by: Fan Du <fan.du@intel.com>
Signed-off-by: Jun Miao <jun.miao@intel.com>
Co-developed-by: Fan Du <fan.du@intel.com>
Signed-off-by: Fan Du <fan.du@intel.com>
Tested-by: chenhuguanshen <chenhgs@chinatelecom.cn>
Signed-off-by: chenhuguanshen <chenhgs@chinatelecom.cn>
Reviewed-by: Michael Kelley <mhklinux@outlook.com>
Tested-by: Michael Kelley <mhklinux@outlook.com>
Link: https://lore.kernel.org/r/20260812070459.637077-1-frankchen158@126.com
Signed-off-by: Marek Szyprowski <m.szyprowski@samsung.com>
The commit 7619a0ee93 ("bpf: Mark existing lock-safe kfuncs with KF_SPINLOCK_SAFE")
dropped some helpers in verifier, which also eliminated the use of the
following kfuncs from the special_kfunc_list:
* bpf_arena_reserve_pages
* bpf_stream_vprintk
* bpf_stream_print_stack
So, drop them from the special_kfunc_list.
Signed-off-by: Leon Hwang <leon.hwang@linux.dev>
Link: https://lore.kernel.org/bpf/20260812164843.55601-1-leon.hwang@linux.dev
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
The dup/restore of insn_aux_data was introduced to resolve the
inconsistency between insnsi and insn_aux_data arrays, which occurs
on the failure path where insnsi was rolled back to the original
state before constants blinding, while insn_aux_data was not.
After JIT failure, there is only one user, bpf_clear_insn_aux_data(),
that requires insnsi and insn_aux_data to be synchronized. It accesses
both insnsi and insn_aux_data using the same array size and index.
However, the access to insnsi in bpf_clear_insn_aux_data() is not
necessary. It is checked to skip the second slot of an ldimm64 instruction,
whose jt is never set and can be absorbed into the jt check itself.
So remove the access to insnsi from bpf_clear_insn_aux_data(), and add a
specific length field for insn_aux_data to allow it to have a different
length from the insnsi array. Then remove dup/restore of insn_aux_data.
Signed-off-by: Xu Kuohai <xukuohai@huawei.com>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://lore.kernel.org/bpf/5a4528f019c8d2638c019a2f37475cccc16a9503.1785240296.git.xukuohai@huawei.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
bpf_trampoline_multi_detach() always returns 0 and the sole caller
ignores the return value. Change it to return void and drop the
WARN_ON_ONCE at the call site.
Signed-off-by: Hui Zhu <zhuhui@kylinos.cn>
Acked-by: Leon Hwang <leon.hwang@linux.dev>
Acked-by: Jiri Olsa <jolsa@kernel.org>
Link: https://lore.kernel.org/bpf/12beba657f5c9e86a016a097750209287a2f262a.1786412280.git.zhuhui@kylinos.cn
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
When bpf_trampoline_update() fails before modify_fentry_multi()/
unregister_fentry_multi() is called, cur_image is unchanged
(cur_image == old_image) and ftrace still calls into it. Freeing
old_image in that case causes a UAF.
Only free old_image when it differs from cur_image.
Fixes: aef4dfa790 ("bpf: Add bpf_trampoline_multi_attach/detach functions")
Signed-off-by: Hui Zhu <zhuhui@kylinos.cn>
Acked-by: Leon Hwang <leon.hwang@linux.dev>
Acked-by: Jiri Olsa <jolsa@kernel.org>
Link: https://lore.kernel.org/bpf/aaa3829e11e2e26bcd3bda9ee6df7a0101a718ac.1786412280.git.zhuhui@kylinos.cn
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Stack depth statistics list captured depths in subprogram-number order,
while per-verification instruction counts are reported separately. Since
libbpf determines subprogram numbers, it is hard to associate either
statistic with its subprogram name or see where verifier work is spent.
Now that self and inclusive instruction counts are available for every
subprogram, keep the combined maximum stack depth on its own line and print
one uniform record for each subprogram. Represent the main program as
subprog 0, then classify each record as main, global, or static before
reporting insns_self, insns_total, and stack depth.
The aggregate processed count is the sum of all self counts, while each
total shows verifier work rooted at that subprogram.
When no subprogram name is available, print <unknown>. Keep the existing
aggregate "processed ... insns" record unchanged for compatibility.
Suggested-by: Andrii Nakryiko <andrii@kernel.org>
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://patch.msgid.link/20260812221925.3358041-4-memxor@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Asynchronous callbacks are explored as fresh frame-zero verifier states,
so normal callee-to-caller accounting cannot propagate their instruction
budget to the main or global subprogram whose verification scheduled them.
The callback exploration still happens within the same do_check_common()
invocation as that independent verification root. Record
env->insn_processed at do_check_common() entry and override the root's
inclusive count with the delta before returning. This includes all directly
and transitively scheduled asynchronous callbacks in the root's total
without maintaining a separate accounting call stack.
Static subprogram and callback totals remain local to their synchronous call
paths. Their self counts continue to account for each processed instruction
exactly once.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://patch.msgid.link/20260812221925.3358041-3-memxor@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
The verifier currently records one instruction count for the main program
and each global subprogram checked independently. Static subprograms are
explored within callers, so their verification cost cannot be reported
separately.
Track both self and inclusive instruction counts for every subprogram.
Charge each processed instruction as self work to the current subprogram and
to a path-local subtotal in its function frame. When a function returns, add
the callee subtotal to its inclusive count and to its parent subtotal. Fold
any remaining frames when a path terminates or is pruned.
Instruction subtotals are accounting state, not semantic verifier state.
Clear them when a verifier state is copied so work before a path fork is
charged once, rather than again when a saved branch is explored. If copying
a saved state fails before all frames are allocated, skip missing frames
while folding the current path.
This generic frame accounting also records self and inclusive totals when an
asynchronous callback starts as a fresh frame-zero state. It does not yet
charge that independently explored callback path back to the main or global
exploration root which scheduled it. That will be done in subsequent
changes.
This does not change the verification statistics output format. It only
prepares the counters for per-subprogram reporting.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://patch.msgid.link/20260812221925.3358041-2-memxor@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
audit_del_rule() is used for both netlink deletion templates and internal
fsnotify autoremove. The former passes a parsed template which owns a
temporary tree reference; the latter passes the installed entry itself.
The unconditional audit_put_tree() at the end of audit_del_rule() assumes
the template case. For mixed AUDIT_DIR plus AUDIT_EXE rules, an fsnotify
autoremove event therefore drops the installed rule's live tree reference.
Repeating this across rules sharing the same tree can free the tree while
another rule still references it, and a later autoremove dereferences the
freed pathname while comparing rules.
Move the temporary-tree put to audit_rule_change(), the caller that owns
deletion templates. Keep it in the AUDIT_DEL_RULE cleanup so both
successful deletion and -ENOENT still release the parser-owned tree.
Cc: stable@kernel.org
Fixes: 34d99af52a ("audit: implement audit by executable")
Assisted-by: Codex:gpt-5
Signed-off-by: Jérémy Jean <Jeremy.Jean@oss.cyber.gouv.fr>
Reviewed-by: Ricardo Robaina <rrobaina@redhat.com>
Tested-by: Ricardo Robaina <rrobaina@redhat.com>
[PM: dropped unnecessary comment for line length reasons]
Signed-off-by: Paul Moore <paul@paul-moore.com>
scx_bpf_cid_override() predates the cid-form arena transition and takes its
arrays as verifier-checked mem+size buffers, forcing scx_qmap to keep the
cpu_to_cid and shard_start arrays in writable bss while the rest of its
state lives in the arena. Unify on arena arguments before cid-form
schedulers start seeing real use.
BPF now translates between BPF and kernel arena addresses for __arena
arguments. Take the arrays as __arena arguments, with the counts passed in
entries. The counts now size the snapshot copies and are bounds-checked
before them.
scx_qmap moves the arrays into struct qmap_arena. As the arena is mmapped at
load, the loader populates them between load and attach instead of before
load.
The arena argument address translation is currently implemented only on
x86-64. Schedulers calling this kfunc load only there for now.
Signed-off-by: Tejun Heo <tj@kernel.org>
The sub-cap kfuncs take their cmask arguments as __ign pointers. The values
cross the kfunc boundary as unchecked scalars and scx_cmask_ref_init()
rebases them into the arena by hand.
BPF now translates between BPF and kernel arena addresses for __arena
arguments. Tag the cmask arguments __arena so the kfuncs receive kernel
addresses and scx_cmask_ref_init() loses the hand-rolled conversion. The
optional denied_out keeps its NULL not-provided signal via
__arena__nullable. The mandatory masks use plain __arena.
scx_qmap's call sites drop the (void *)(long) casts since the BPF-side
declarations type the cmask arguments __arena and take arena pointers
directly.
The arena argument address translation is currently implemented only on
x86-64. Schedulers calling these kfuncs load only there for now.
Signed-off-by: Tejun Heo <tj@kernel.org>
The cid-form set_cmask() and sub_caps_updated() callbacks receive cmasks
that the kernel builds in the arena, and the kernel converts the kernel
addresses to the BPF arena pointer form by hand before each call.
BPF now translates between BPF and kernel arena addresses for __arena
arguments. Tag the arguments __arena in the cfi stubs and the ops_cid member
declarations and pass the kernel arena addresses directly, dropping the
manual scx_kaddr_to_arena() conversions and the now-unused helper. The
delivered value is unchanged and existing BPF-side code works as before.
The arena argument address translation is currently implemented only on
x86-64. cid-form schedulers implementing these callbacks load only there for
now.
Signed-off-by: Tejun Heo <tj@kernel.org>
scx_bpf_cid_to_cpu(), scx_bpf_cpu_to_cid() and scx_bpf_cid_topo() live in
the scx_kfunc_ids_cid set, but scx_kfunc_context_filter() doesn't check
that set. The filter's first test treats any kfunc outside its known sets
as non-SCX and allows it, so these three kfuncs can be called from any
struct_ops program - e.g. a TCP congestion control program.
Add scx_kfunc_ids_cid to the filter's known sets, matching how in_any and
in_idle are handled.
Fixes: e9b55af47e ("sched_ext: Add topological CPU IDs (cids)")
Assisted-by: Z.ai:glm-5.2
Signed-off-by: fangqiurong <fangqiurong@kylinos.cn>
Signed-off-by: Tejun Heo <tj@kernel.org>
check_atomic_rmw() open codes the mapping from a BPF_ATOMIC to the register
it reads the old value into, the BPF_STX case of insn_def_regno() open codes
the very same mapping a second time, the const folding and the liveness
transfer functions a third and a fourth time, and BPF JITs need it as well
to know which register a faulting BPF_PROBE_ATOMIC has to clear.
Add a small helper so that all of them can share it. No functional change.
The BPF_LOAD_ACQ case is there for the JITs, which do walk all instruction
classes. const_reg_xfer() loses its explicit BPF_ATOMIC mode test since the
helper checks class and mode itself; the BPF_PROBE_ATOMIC it additionally
accepts cannot be seen there as it is only set from bpf_do_misc_fixups(),
that is, after const folding has run. arg_track_xfer() keeps its mode test
since that also guards the stack clearing next to it.
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://patch.msgid.link/20260811131600.506721-1-daniel@iogearbox.net
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
wq_node_nr_active() warns and returns NULL when @wq is not unbound, but
every caller dereferences the result right away, so the WARN_ON_ONCE()
only moves the oops one frame up, as raised by Tejun.
Fix it by BUGing_ON() instead of this silly WARN_ON_ONCE();
Fixes: b72fdc6510 ("workqueue: account nr_active by the backing pool")
Suggested-by: Tejun Heo <tj@kernel.org>
Signed-off-by: Breno Leitao <leitao@debian.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
wq->cpu_pwq holds RCU-protected pwq pointers, but the percpu allocation
path fills it in with plain loads and stores, which sparse flags:
kernel/workqueue.c:5682:57: sparse: incorrect type in initializer (different address spaces) @@ expected struct pool_workqueue **pwq_p @@ got struct pool_workqueue [noderef] __rcu ** @@
Allocate the array as __rcu pointers and publish each pwq with
rcu_assign_pointer() once it is initialized and linked, the order
install_unbound_pwq() uses.
The warnings are not new: commit 79f23600bc ("workqueue: factor out
get_percpu_pool()") only turned the flagged assignment into an
initializer.
Reported-by: kernel test robot <lkp@intel.com>
Closes: https://lore.kernel.org/oe-kbuild-all/202608120931.tvTzq1gD-lkp@intel.com/
Signed-off-by: Breno Leitao <leitao@debian.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
workqueue_congested() fetches the pwq out of wq->cpu_pwq with a plain
load, so sparse complains about the dropped __rcu:
kernel/workqueue.c:6304:13: sparse: incorrect type in assignment (different address spaces) @@ expected struct pool_workqueue *pwq @@ got struct pool_workqueue [noderef] __rcu * @@
A pwq is released with kfree_rcu() and the read is protected by the
surrounding preempt_disable(), which is what
commit fd5081f4ef ("workqueue: Remove redundant rcu_read_lock/unlock() in
workqueue_congested()") relied on when it dropped the rcu_read_lock()
here.
Use the rcu_dereference_sched() helper to make that explicit.
Reported-by: kernel test robot <lkp@intel.com>
Closes: https://lore.kernel.org/oe-kbuild-all/202608120931.tvTzq1gD-lkp@intel.com/
Signed-off-by: Breno Leitao <leitao@debian.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
Pull to receive:
c10b216a07 ("sched/core: Handle pick_task() releasing the rq lock")
f3629c63a4 ("sched/core: Make core-sched flips wait for in-flight selections")
ffaab58d21 ("sched_ext: Replace SCX_RQ_BAL_KEEP with a dispatch verdict return")
3dd52416e4 ("sched_ext: Fix this_rq() assumptions in dispatch kfuncs")
f2da958711 ("sched_ext: Count rq lock releases in rq->scx.lock_drop_seq")
d954004205 ("sched_ext: Fix rq->core_pick corruption under core scheduling")
for the pending core scheduling follow-ups and to resolve the conflicts
with the code reorganization and cap gate work on for-7.3.
ffaab58d21 converts scx_dispatch_sched() to a dispatch verdict return
which for-7.3 moved from ext.c into inlines.h. Resolved by applying the
conversion to the relocated copy and combining balance_one()'s verdict
returns with the scx_task_can_stay_on_cpu() gate from the cap work.
ffaab58d21 and 3dd52416e4 update scx_bpf_sub_dispatch() which
for-7.3 moved into sub.c. Resolved by applying the scx_locked_rq()
switch and the verdict test to the sub.c copy.
f2da958711 instruments the open-coded lock releases in
consume_remote_task() which for-7.3 folded into switch_rq_lock().
Resolved by keeping the accounting in switch_rq_lock() which covers all
its callers.
d954004205 widens the put_prev_task_scx() WARN suppression to all
core-sched rqs on the same condition that for-7.3 gated with
scx_task_can_stay_on_cpu(). Resolved by combining both.
Signed-off-by: Tejun Heo <tj@kernel.org>
alloc_pid() checks PIDNS_ADDING only on the leaf pid namespace before
making a new struct pid visible in every ancestor namespace. That is
insufficient when an unborn descendant pid namespace outlives an
ancestor whose init task has already exited. The descendant can still be
initialized later through setns(), and the new pid is then published
into the dead ancestor as well.
Keep the existing ENOMEM behavior, but require PIDNS_ADDING to be set in
every namespace that will receive the new pid before publishing any of
them. This preserves the invariant that free_pid() never decrements
pid_allocated in a namespace whose child_reaper is no longer live.
Fixes: a3bdc23ba8 ("pid_namespace: allow opening pid_for_children before init was created")
Signed-off-by: Jérémy Jean <Jeremy.Jean@oss.cyber.gouv.fr>
Reviewed-by: Pavel Tikhomirov <ptikhomirov@virtuozzo.com>
Signed-off-by: Christian Brauner (Amutable) <brauner@kernel.org>
Commit 16a1d96835 ("mm/slab: remove mm/slab.c and slab_def.h")
removed the SLAB allocator, the only allocator that implemented cpuset
slab spreading. Commit 61a182ab61 ("cgroup/cpuset: Remove
cpuset_do_slab_mem_spread()") then removed the last task_spread_slab()
caller. Commit 3ab67a9ce8 ("cgroup/cpuset: Mark memory_spread_slab as
obsolete") marked the legacy control obsolete.
cpuset still updates PFA_SPREAD_SLAB when tasks attach to a legacy
cpuset and walks all tasks in a cpuset when memory_spread_slab changes.
Remove the unused task flag and its helpers, and make spread task
updates depend only on memory_spread_page.
Keep the memory_spread_slab control and CS_SPREAD_SLAB state so legacy
users retain the existing write, readback and inheritance behavior.
Update the comments and documentation to describe only page-cache
spreading as functional.
Assisted-by: LLM
Signed-off-by: Guopeng Zhang <zhangguopeng@kylinos.cn>
Reviewed-by: Waiman Long <longman@redhat.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
Core scheduling's pick_next_task() picks what to run on every SMT sibling of
the core in a single pass under the shared core-wide rq lock. The selection
state is consistent only while the lock is held continuously, so
->pick_task() originally could not release it. However, since 4c95380701
("sched/ext: Fold balance_scx() into pick_task_scx()"), sched_ext runs
dispatch from inside the pick and dispatching can drop the rq lock. To
support this, pick_next_task() has been updated to restart the whole
selection when a pick returns RETRY_TASK after releasing the lock.
When selections on the same core interleave through the dropped lock, they
corrupt each other's state: one clears the other's rq->core_pick leading to
a NULL deref, or invalidates its keep-the-previous-task decision leaving a
dequeued task running, which deadlocks the next wakeup and matches the
reported hard hangs. A cookied ping-pong load on an SMT machine makes the
interleavings frequent and kills the kernel within seconds.
Fix it by making the pick return RETRY_TASK whenever dispatch released the
rq lock, so that a selection only ever commits picks made under a
continuously held lock. The previous patch's rq->scx.lock_drop_seq counts
the releases. A dispatch that touched nothing never releases the lock and
its verdict, including "nothing to run", stands: retries are bounded, each
following a dispatch that actually did something, and an idle CPU does not
loop.
If another dispatch is already in flight on the rq, skip dispatching and
pick from what is already queued locally - the in-flight dispatch has
released the lock, so its own selection will retry and re-pick this rq,
while returning RETRY_TASK here would only spin on the lock that dispatch
needs to finish.
Balance callbacks must run in the context that queued them, so they can only
be queued on the CPU's own rq. When dispatching for another rq, run the
deferred work directly instead - that rq may consume all its picks through
the core-sched fast path and never queue the callback itself.
The put_prev_task_scx() warning about a runnable task being left behind
assumed that dispatch ran as part of the very pick that is switching away.
That now only holds on the non-core path, so gate it and drop the
cookie-match test, which is always true without core scheduling, from its
condition.
Fixes: 4c95380701 ("sched/ext: Fold balance_scx() into pick_task_scx()")
Cc: stable@vger.kernel.org # v6.19+
Reported-by: ElXreno <elxreno@gmail.com>
Link: https://github.com/sched-ext/scx/issues/3715
Signed-off-by: Tejun Heo <tj@kernel.org>
Under core scheduling, pick_next_task() selects for all SMT siblings under
one continuous hold of the shared core-wide rq lock, and sched_ext's
dispatch can release that lock from inside the pick. In preparation for
making the core-sched pick detect the releases and retry, add
rq->scx.lock_drop_seq and bump it at every site that can release an rq lock
while a dispatch may be in flight. The counter is only maintained while core
scheduling is enabled. No functional changes.
Fixes: 4c95380701 ("sched/ext: Fold balance_scx() into pick_task_scx()")
Cc: stable@vger.kernel.org # v6.19+
Signed-off-by: Tejun Heo <tj@kernel.org>
Under core scheduling, dispatch runs from within the core-wide pick and can
target a sibling rq, so ops.dispatch() may execute on a CPU different from
the dispatched rq's. Several kfunc paths assumed the two always coincide:
- scx_dsq_move() decided whether an rq lock is held by testing this_rq()'s
rq flags and lock-danced accordingly. A dispatch for a sibling took the
unlocked-context branch and acquired the source rq lock on top of the
already held dispatched rq lock which could deadlock.
- scx_bpf_sub_dispatch() dispatched this_rq() with its stashed
sub_dispatch_prev, which is NULL when dispatching for a sibling.
- finish_dispatch(), scx_bpf_dsq_reenq() and scx_bpf_dsq_nr_queued()
resolved SCX_DSQ_LOCAL to this CPU's local DSQ rather than the dispatched
rq's. The latter two are callable from other rq-locked operations too,
where SCX_DSQ_LOCAL now likewise resolves to the op's rq. This changes
behavior also without core scheduling, e.g. for ops.enqueue() running a
remote wakeup on the waking CPU, and is intended: which CPU happens to
execute an operation is incidental, the op's rq is what it is operating
on, and the resolution now matches the insert side where SCX_DSQ_LOCAL
dispatches land on the task's rq.
Use the rq tracked by scx_locked_rq(), which is set to the dispatched rq
around ops invocations and NULL in unlocked contexts.
Fixes: 4c95380701 ("sched/ext: Fold balance_scx() into pick_task_scx()")
Cc: stable@vger.kernel.org # v6.19+
Signed-off-by: Tejun Heo <tj@kernel.org>
SCX_RQ_BAL_KEEP tells the pick to keep running the previous task, a leftover
from when balancing and picking were separate operations. An rq-level flag
only works while dispatches and picks pair up one to one, which core
scheduling breaks: selections interleave through dispatch's lock drops and a
pick can consume a stale flag, keeping a task that has since been dequeued.
Fixing core scheduling support requires the decision to travel with the
dispatch that made it. Make scx_dispatch_sched() and balance_one() return an
explicit verdict instead and drop the flag's plumbing from the tools autogen
enum headers.
Also factor the pick-side invocation, its follow-up queueing and the
post-dispatch checks out of do_pick_task_scx() into dispatch_pick(). No
functional changes intended.
v2: Drop the SCX_RQ_BAL_KEEP plumbing from the tools autogen enum headers
as well (Andrea).
Fixes: 4c95380701 ("sched/ext: Fold balance_scx() into pick_task_scx()")
Cc: stable@vger.kernel.org # v6.19+
Signed-off-by: Tejun Heo <tj@kernel.org>
Core scheduling's pick_next_task() operates on all sibling rqs under one
acquisition of the shared core-wide lock. A ->pick_task() that releases the
rq lock leaves every sibling __lock momentarily free, letting
__sched_core_flip(false) complete mid-selection and rebind rq_lockp() under
it. The selection resumes on the split locks, touching sibling state it no
longer protects, and __schedule() finally releases a lock that was never
taken while leaking the one that was.
Count in-flight core-wide selections in the leader's rq->core_pick_in_flight
and make __sched_core_flip() wait for the count to drain. The count only
changes under the shared lock, which the flip holds while sampling, so no
other ordering is needed. The wait can repeat while selections overlap, but
the flip backs off between samples and flips are rare cookie-lifetime
events.
sched_core_cpu_deactivate() moves the count to the new leader - a stale copy
left behind would bias it forever if that CPU later returns as its own
leader.
Fixes: 539f65125d ("sched: Add core wide task selection and scheduling")
Cc: stable@vger.kernel.org # v5.14+
Signed-off-by: Tejun Heo <tj@kernel.org>
Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Core scheduling's pick_next_task() breaks when a ->pick_task()
implementation can release the rq lock. The selection state derived on entry
is only valid while the lock is held continuously. Once a pick can drop the
lock, an interleaving selection can invalidate all of it: the single-CPU
fast path can commit an uncookied pick although the core went cookied during
the release, and forceidle committed by the interleaving selection skews the
restarted pass's accounting.
Fix it by restarting the whole selection when a pick returns RETRY_TASK
after releasing the lock: a single restart point above the state derivation
replaces the per-loop restart labels, so a retry picks up state committed by
interleaving selections and accounts and resets forceidle like a fresh
selection would.
need_sync and fi_before latch across retries. Clock validity can't be
re-derived - there is no program-ordered way to tell whether the own and
core rq clocks are still updated after the lock was released, as other
lockers' pin cycles may or may not have invalidated them. When restarting,
clear core_clock_updated so that the sibling loop re-updates the core rq,
and update the own rq clock if invalidated.
Fixes: 4c95380701 ("sched/ext: Fold balance_scx() into pick_task_scx()")
Cc: stable@vger.kernel.org # v6.19+
Signed-off-by: Tejun Heo <tj@kernel.org>
Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org>
In the cleanup_srcu_struct() function, when iterating over per-cpu's
srcu_data, timer_delete_sync(&sdp->delay_work) is called to cancel the
delayed work before doing flush_work(&sdp->work).
However, suppose that timer_delete_sync() returns 1, which means that it
successfully deleted an pending timer before it had a chance to fire.
But this also means that the sdp->work will not be queued, so that the
subsequent flush_work(&sdp->work) will returns immediately without waiting
for anything. Taken together, all of this means that any recently queued
SRCU callbacks to not be invoked, which can result in memory leaks,
hangs, or worse.
Fix this by checking the return value of timer_delete_sync(), if it
returns 1, explicitly queue sdp->work so that the callbacks will be
invoked and the following flush_work() will correctly wait for all of
those callbacks to finish executing.
[ Zqiang: Apply feedback from Breno Leitao and kernel test robot. ]
Signed-off-by: Zqiang <qiang.zhang@linux.dev>
Tested-by: kernel test robot <oliver.sang@intel.com>
Reviewed-by: Frederic Weisbecker <frederic@kernel.org>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
An iterator stack slot can be MEM_RCU or PTR_UNTRUSTED. These states
must not be equal, or the verifier can prune an unsafe path.
Compare the pointer type for STACK_ITER slots.
Fixes: dfab99df14 ("bpf: teach the verifier to enforce css_iter and task_iter in RCU CS")
Signed-off-by: Ning Ding <dingning04@gmail.com>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://patch.msgid.link/20260811035955.132989-2-dingning04@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
When a function-like macro expands to an expression, that expression
doesn't need a semicolon after it. All uses have been verified to
have their own semicolons.
This was found using the following Coccinelle semantic patch:
@r@
identifier i : script:ocaml() { String.lowercase_ascii i = i };
expression e;
@@
*#define i(...) e;
Link: https://patch.msgid.link/20260801191002.1383835-6-Julia.Lawall@inria.fr
Signed-off-by: Julia Lawall <Julia.Lawall@inria.fr>
Acked-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
hrtimers use ktime_t in their implementation and API.
The timer list performs a lot of unnecessary conversion to nanoseconds
which make the code harder to read and are also wrong in case the values
ever become negative.
Remove the conversions.
Signed-off-by: Thomas Weißschuh (Schneider Electric) <thomas.weissschuh@linutronix.de>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Link: https://patch.msgid.link/20260803-auxclock-nanosleep-prep-v2-5-910cbd485390@linutronix.de
'%Ld' and '%Lu' are GNU extensions. While they do work for kernel code,
checkpatch complains about them all the time. Replace them with the
standard placeholders for 'long long' types, namely '%lld' and '%llu'.
Signed-off-by: Thomas Weißschuh (Schneider Electric) <thomas.weissschuh@linutronix.de>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Link: https://patch.msgid.link/20260803-auxclock-nanosleep-prep-v2-4-910cbd485390@linutronix.de
Use an explicitly sized type to make the code a bit more consistent
with other fields of the datastructure and other sequence counters.
Signed-off-by: Thomas Weißschuh (Schneider Electric) <thomas.weissschuh@linutronix.de>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Link: https://patch.msgid.link/20260803-auxclock-nanosleep-prep-v2-2-910cbd485390@linutronix.de
Re-arranging hrtimer_interrupt() switched the retry path to a local
counter and dropped the update of cpu_base->nr_retries, leaving the
field exported via /proc/timer_list stuck at zero.
Increment nr_retries only when another pass through the expiry loop
is started; the third attempt that falls through to hang handling is
still accounted by nr_hangs alone.
Fixes: 2889243848 ("hrtimer: Re-arrange hrtimer_interrupt()")
Signed-off-by: Liang Hao <haohlliang@gmail.com>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Link: https://patch.msgid.link/20260731150408.19554-1-haohlliang@gmail.com
The enable_data variable gets freed on most error paths in
event_enable_trigger_parse(). Use free() to free it and just before
returning normally, call retain_and_null_ptr(enable_data) just before a
successful exit to keep it from being freed. On success, the enable_data
is assigned to the trigger_data->private_data field.
Also add a comment to why event_trigger_free(trigger_data) is being called
before a successful exit.
Link: https://patch.msgid.link/20260807113558.0ff14e96@gandalf.local.home
Reviewed-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
WARN_ONCE() splats once per call site, so only the first offending event
registered is ever reported. The tree currently has six:
ice_{rx,tx}_dim_template, two hfi1 txq events, mtu3_ep and edma_log_io.
Whichever registers first hides the rest, and each has to be found again
on the next boot.
Add a pr_warn() next to the WARN_ONCE() so every offender is listed, the
same way test_event_printk() already pairs WARN_ON_ONCE() with pr_warn()
for unsafe %p* dereferences. The WARN_ONCE() stays so the condition still
fails tests and panics under panic_on_warn.
Link: https://patch.msgid.link/20260806215256.1680267-1-devnexen@gmail.com
Suggested-by: Steven Rostedt <rostedt@goodmis.org>
Signed-off-by: David Carlier <devnexen@gmail.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
On native sparc64, struct __kernel_old_timeval contains a four-byte hole
after tv_usec because tv_sec is 64-bit while __kernel_suseconds_t is 32-bit.
put_itimerval() fills only the named fields in a stack-allocated
__kernel_old_itimerval and copies the entire object to userspace, so
getitimer() can expose the two padding holes.
Zero-initialize the aggregate before assigning the fields so implicit
padding is deterministic before it crosses the user/kernel boundary.
Signed-off-by: Jérémy Jean <Jeremy.Jean@oss.cyber.gouv.fr>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Assisted-by: Codex:gpt-5
Cc: stable@vger.kernel.org
Link: https://patch.msgid.link/20260809190428.1523014-1-Jeremy.Jean@oss.cyber.gouv.fr
'select' does not work on config options in a 'choice', so currently the
'select VIRT_CPU_ACCOUNTING_GEN' for NO_HZ_FULL is dead, with the choice
option VIRT_CPU_ACCOUNTING_GEN only being enabled when NO_HZ_FULL=y because
the other choice members depend on NO_HZ_FULL=n.
Remove the dead select, and encode this relationship as a default of the
choice, instead.
This dead select was found by kconfirm, a static analysis tool for Kconfig.
Signed-off-by: Julian Braha <julianbraha@gmail.com>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Reviewed-by: Bradley Morgan <include@grrlz.net>
Reviewed-by: Nicolas Schier <nsc@kernel.org>
Link: https://patch.msgid.link/20260801160140.2391000-1-julianbraha@gmail.com
Writing 0 to kernel.ftrace_enabled has not reliably disabled ftrace
for years (FTRACE_OPS_FL_PERMANENT users already block it, and more
callers rely on ftrace always being on). Refuse the write instead of
leaving it in an inconsistent "disables some, not all" state: return
-EOPNOTSUPP and log a message. Reads and enabling (writing 1) are
unaffected.
Update the docs to note the deprecation up front.
Link: https://patch.msgid.link/20260806153000.4184871-2-andrey.grodzovsky@crowdstrike.com
Suggested-by: Steven Rostedt <rostedt@goodmis.org>
Signed-off-by: Andrey Grodzovsky <andrey.grodzovsky@crowdstrike.com>
Acked-by: Song Liu <song@kernel.org>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
Pull bpf-next d114bb9893 ("Merge branch
'add-arena-argument-support-to-kfuncs-and-struct_ops'") to make the __arena
and __arena__nullable kfunc and struct_ops argument suffixes available. The
suffixed arguments will be used to convert sched_ext kfuncs and struct_ops
callbacks that currently pass arena pointers as scalars and rebase them by
hand.
apply_wqattrs_commit() updates node_nr_active->max unconditionally.
wq->node_nr_active[] is only allocated for unbound workqueues, so guard
the call before per-cpu workqueues start using this path.
No functional change: only unbound workqueues reach apply_wqattrs_*()
today.
Signed-off-by: Breno Leitao <leitao@debian.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
This allocates a pwq and binds it to the pool @attrs asks for.
Which pool that is becomes a property of the attrs (once per-cpu becomes
an affinity scope).
Remove the 'unbound" from the function name, given it will be bigger
than unbound.
No functional change.
Signed-off-by: Breno Leitao <leitao@debian.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
The attrs are where the affinity scope lives, and a per-cpu workqueue
will need one once per-cpu becomes a scope rather than a separate
backend. Allocate them unconditionally.
wq_dump.py used a non-NULL wq->attrs as its test for an unbound
workqueue, which no longer holds; test WQ_UNBOUND there instead.
Signed-off-by: Breno Leitao <leitao@debian.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
The unbound prefix says which workqueues currently have the field rather
than what it holds, and the next patch allocates it for every workqueue.
Rename it first so that change stays a single line.
tools/workqueue/wq_dump.py reads the field by name, so rename it there
too.
wq_sysfs_unbound_attrs[] keeps its name: it is the set of sysfs files
that only unbound workqueues expose.
No functional change.
Signed-off-by: Breno Leitao <leitao@debian.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
workqueue_online_cpu() and workqueue_offline_cpu() decide whether a
workqueue needs a pod affinity update by testing wq->unbound_attrs for
NULL, which is only meaningful because the attrs are allocated for
unbound workqueues alone.
Test the flag instead, so the attrs can later be allocated for every
workqueue.
No functional change.
Signed-off-by: Breno Leitao <leitao@debian.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
pwq_tryinc_nr_active() and pwq_dec_nr_active() choose between the shared
per-node nr_active and the plain per-pwq one by testing
wq_node_nr_active() for NULL.
Test the backing pool with is_percpu_pool() instead, so the accounting
follows the pool that runs the work rather than the workqueue type.
No functional change.
Signed-off-by: Breno Leitao <leitao@debian.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
Add is_percpu_pool() and test the pool directly for per cpu. Convert the
other open-coded pool->cpu checks -- in put_unbound_pool(),
pool_allowed_cpus() and the workqueue watchdog -- to the same helper.
No functional change.
Signed-off-by: Breno Leitao <leitao@debian.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
Move the per-cpu pwq allocation loop out of alloc_and_link_pwqs() into a
helper. The inner allocation-failure path now returns -ENOMEM and the
caller jumps to the existing enomem cleanup, equivalent to the previous
goto.
No functional change.
Signed-off-by: Breno Leitao <leitao@debian.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
Move the static per-cpu worker_pool lookup in alloc_and_link_pwqs() into
a helper, get_percpu_pool(), so the lookup can be shared by other
pool-selection paths.
No functional change.
Signed-off-by: Breno Leitao <leitao@debian.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
If the auxiliary clock is disabled during tk_get_aux_ts64() but is enabled
before tks->clock_valid is checked, then uninitialized stackdata will be
used in the calculations and indirectly leaked to userspace.
The same race window also exists after this change and also for the core
timekeeper. But in these cases the only effect would be incorrect
adjustments and this is userspace's responsibility to avoid this.
Fixes: 4eca49d0b6 ("timekeeping: Prepare do_adtimex() for auxiliary clocks")
Signed-off-by: Thomas Weißschuh (Schneider Electric) <thomas.weissschuh@linutronix.de>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Cc: stable@vger.kernel.org
Link: https://patch.msgid.link/20260731-timekeeping-aux-adjtimex-return-v1-1-b7fea4692886@linutronix.de
cpuset_partition_show() reads cs->prs_err without cpuset_mutex using
READ_ONCE(). The field is documented as not lock protected, but several
updates to live cpusets still use plain stores.
Convert the remaining prs_err stores on live cpusets to WRITE_ONCE().
Fixes: 0c7f293efc ("cgroup/cpuset: Add cpuset.cpus.exclusive.effective for v2")
Assisted-by: LLM
Signed-off-by: Guopeng Zhang <zhangguopeng@kylinos.cn>
Reviewed-by: Waiman Long <longman@redhat.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
The timekeeper update path uses a bulk memcpy() to synchronize the
timekeeper structure, which is not guaranteed to be atomic. This allows for
torn reads in ktime_get_real_seconds() on 64-bit systems, where the
sequence counter protection is bypassed for performance.
To prevent reading a torn 64-bit xtime_sec value, enforce atomic-like
access by using WRITE_ONCE() for the critical field before the bulk
memcpy() in timekeeping_update_from_shadow(). Correspondingly, use
READ_ONCE() in ktime_get_real_seconds() to ensure a fresh, consistent load
from memory.
[ tglx: Format changelog and add comment ]
Reported-by: syzbot+72789cd1697965e714ca@syzkaller.appspotmail.com
Signed-off-by: Dennis Moshegov <dennis@xzync.uk>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Link: https://patch.msgid.link/20260724154405.70-1-dennis@xzync.uk
Closes: https://syzkaller.appspot.com/bug?extid=72789cd1697965e714ca
With the changes that enable preempt count to track IRQ disabling
nesting, we don't have enough bits in 32-bit preempt count
implementation, as a result we move NMI nesting bits out of the 32-bit
preempt count. However on the architectures that can support 64-bit
preempt count implementation, we can keep the NMI nesting bits in the
32-bit preempt count and avoid maintaining NMI nesting bits outside of
the same cache line.
Therefore HAS_SEPARATE_PREEMPT_RESCHED_BITS is introduced to allow
architectures to select this. Note that under this Kconfig, preempt
count is maintained in a 64-bit word however preempt_count() still
remains as an int because all the effective bits still fit in
(previously we mask out NEED_RESCHED bit in preempt_count()). This
should make no functional changes for existing preempt_count() users.
Enable this for x86_64 along with the introduction of the Kconfig.
[boqun: Undo the __preempt_count_{add,sub}() optimization in 32-bit
preempt count since it may introduce {over,under}flow]
Originally-by: Peter Zijlstra <peterz@infradead.org>
Signed-off-by: Boqun Feng <boqun@kernel.org>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/20260804161447.84806-11-boqun@kernel.org
Currently preempt_count() is always a non-negative int on all archs
(PREEMPT_NEED_RESCHED archs will mask out the MSB when returning
preempt_count()), hence the checking in __cant_migrate() is in fact just
checking whether preempt_count() is 0 or not. In a future change, we are
going to use all the 32 bits of preempt_count(), which would make
negative int values possible from preempt_count(). Therefore convert the
"> 0" comparison into a zero check to prepare for the future change.
No functional changes are intended.
Signed-off-by: Boqun Feng <boqun@kernel.org>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/20260804161447.84806-10-boqun@kernel.org
The preempt_offset is always 0 in all the callsites of __cant_sleep(),
hence remove it. It also allows us to clear up the code a bit by
no longer using a "preempt_count() > .." comparison.
Signed-off-by: Boqun Feng <boqun@kernel.org>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/20260804161447.84806-9-boqun@kernel.org
The semantics of various IRQ disabling guards match what
*_irq_{disable,enable}() provide, i.e. the interrupt disabling is
properly nested, therefore it's OK to switch to use
*_irq_{disable,enable}() primitives.
[boqun: Adjust the user-side changes in do_sched_cfs_*_timer() provided
by Peter and Lyude]
Signed-off-by: Boqun Feng <boqun@kernel.org>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/20260804161447.84806-8-boqun@kernel.org
While making changes to the refcounted interrupt patch series, at some
point on my local branch I broke something and ended up writing some kunit
tests for testing refcounted interrupts as a result. So, let's include
these tests now that we have refcounted interrupts.
Signed-off-by: Lyude Paul <lyude@redhat.com>
Signed-off-by: Boqun Feng <boqun@kernel.org>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/20260804161447.84806-7-boqun@kernel.org
Currently the nested interrupt disabling and enabling is represented by
_irqsave() and _irqrestore() APIs, which are relatively unsafe, for
example:
<interrupts are enabled as beginning>
spin_lock_irqsave(l1, flag1);
spin_lock_irqsave(l2, flag2);
spin_unlock_irqrestore(l1, flags1);
<l2 is still held but interrupts are enabled>
// accesses to interrupt-disable protected data will cause races
This is even easier to trigger with guard facilities:
unsigned long flag2;
scoped_guard(spin_lock_irqsave, l1) {
spin_lock_irqsave(l2, flag2);
}
// l2 locked but interrupts are enabled.
spin_unlock_irqrestore(l2, flag2);
(Hand-to-hand locking critical sections are not uncommon for a
fine-grained lock design)
And because of this unsafety, Rust cannot easily wrap the
interrupt-disabling locks in a safe API, which complicates the design.
To resolve this, introduce a new set of interrupt disabling APIs:
* local_interrupt_disable();
* local_interrupt_enable();
They work like local_irq_save() and local_irq_restore() except that 1)
the outermost local_interrupt_disable() call saves the interrupt state
into a per-CPU variable, so that the outermost local_interrupt_enable()
can restore the state, and 2) a per-CPU counter is added to record the
nest level of these calls, so that interrupts are not accidentally
enabled inside the outermost critical section.
Also add the corresponding spin_lock primitives: spin_lock_irq_disable()
and spin_unlock_irq_enable(), as a result, code as follows:
spin_lock_irq_disable(l1);
spin_lock_irq_disable(l2);
spin_unlock_irq_enable(l1);
// Interrupts are still disabled.
spin_unlock_irq_enable(l2);
doesn't have the issue that interrupts are accidentally enabled.
This also makes the wrapper of interrupt-disabling locks on Rust easier
to design.
[boqun: Apply Peter's feedback and fix spell errors reported by Ingo]
[boqun: Address the duplicate spin_acquire() spotted by sashiko]
Co-developed-by: Lyude Paul <lyude@redhat.com>
Signed-off-by: Lyude Paul <lyude@redhat.com>
Signed-off-by: Boqun Feng <boqun@kernel.org>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/20260804182657.87716-1-boqun@kernel.org
fprobe_remove_ips() manually duplicates the unregister and filter-removal
logic for both graph and ftrace ops. Simplify it by delegating to the
existing fprobe_graph_remove_ips() and fprobe_ftrace_remove_ips() helpers.
Link: https://lore.kernel.org/all/178528139798.102586.5349128066643420018.stgit@devnote2/
Assisted-by: Antigravity:gemini-3.6-flash
Signed-off-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
Reviewed-by: Steven Rostedt <rostedt@goodmis.org>
The futex state is used to prevent a waiter from attaching to the lock
owner while the owner runs the futex cleanup in exit() or exec().
Only the state transition from FUTEX_STATE_OK to FUTEX_STATE_EXITING must
be done with the task's pi_lock held, the transition away from
FUTEX_STATE_EXITING has no serialization requirements on the writer side,
but it's completely non obvious why. It's magically protected by
exit_pi_state(), which operates under tsk::pi_lock, as that's the state
which has to be correct when the waiter observes the new state.
OTOH, taking the pi_lock in futex_cleanup_end() is not a performance issue
because at that point the lock should be uncontended in the vast majority
of cases.
Aside of that the handling of FUTEX_STATE_EXITING in attach_to_pi_owner()
and handle_exit_race() is confusing at best.
Protect the store in futex_cleanup_end() with tsk::pi_lock, handle
FUTEX_STATE_EXITING in attach_to_pi_owner() explicitly and document how
this is supposed to work.
Reported-by: Peter Zijlstra <peterz@infradead.org>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Reviewed-by: Kyle Zeng <kylebot@openai.com>
Acked-by: Peter Zijlstra <peterz@infradead.org>
Cc: stable@vger.kernel.org
A private futex key borrows the waiter's mm without taking an mm_users
reference. Nevertheless, attach_to_pi_owner() currently accepts an owner
from a different address space and copies the private key into the owner's
PI state.
When that owner exits, exit_pi_state_list() uses the saved key to find the
hash bucket and acquires a reference to the waiter's private hash. If the
last user of the waiter's mm exits concurrently, futex_hash_free() frees
the hash while the owner still uses its bucket and reference.
Prevent this by validating in attach_to_pi_owner() that, for private
futexes, the owner mm and waiter mm are the same. Perform the check with
the owner's pi_lock held and after validating owner::futex::state to
serialize against a concurrent PI-state exit cleanup.
[ tglx: Amended comment ]
Fixes: 80367ad01d ("futex: Add basic infrastructure for local task local hash")
Signed-off-by: Kyle Zeng <kylebot@openai.com>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Acked-by: Peter Zijlstra <peterz@infradead.org>
Assisted-by: Codex:gpt-5.6-sol
Cc: stable@vger.kernel.org
- Fix use-after-free in eventfs_remove_rec()
The freeing of the eventfs_inode children used list_for_each_entry() where
the child is freed via srcu, but there's still a chance that it gets freed.
It should be using list_for_each_entry_safe().
- Fix eventfs_inode SRCU use of list in freeing
The iterator uses an SRCU protected list walk on the eventfs inodes. The
eventfs inode uses its "list" field in a union with the RCU list head.
When the inode gets added to the SRCU list it immediately corrupts the
list pointer and can cause an issue with the iterator. Move the RCU list
head to be shared with the children list head which allows the iterator to
check the parent inode if is freed before referencing the child. Have the
iterator check the parent "is_freed" field and break out if it is set.
Also add memory barriers to make sure the ordering is correct.
- Fix various RCU synchronization issues with direct_functions
Updates to direct_functions have some missing RCU protection and
synchronization. Restructure the code a bit to make sure updates to the
direct_functions are protected.
- Remove an unneeded comma from a scope_guard()
There's a spurious comma in a scope_guard(). Remove it.
- Fix race in per CPU buffer swap in the ring buffer
When a per CPU buffer swap happens, it must make sure that it doesn't
occur while a writer is active. Instead it returns an -EBUSY. But there's
a small race window when a writer moves from one sub-buffer to the next
that it resets the "committing" counter. If a swap happens at that moment,
the buffer used for the commit of an event will not match the buffer the
event is actually on. Instead of using the "committing" counter, use the
recursive detection counter that does not get reset when the writer
crosses sub-buffers.
- Fix off-by-one in ftrace_free_mem()
The function ftrace_free_mem() gets an "end_ptr" as a parameter that is
exclusive to the rang to be freed. But its value is used to search for the
records that expects an inclusive value. Subtract one from the parameter
to convert it to an inclusive range.
- Disable resizing of the ring buffer for persistent buffers
Resizing the persistent buffer has undefined behavior. Prevent it from
being resized.
- Disable changing ring buffer subbuf order when resizing is disabled
The ring buffer subbuffer order can not be changed during resizing. Use
that instead of just checking if the buffer is mapped as mapped buffers
also have resizing disabled.
- Initialize subbuf_order of reader pages when they are created
In rb_allocate_cpu_buffer() the bpage->order is not updated to the current
subbuf_order leaving it as zero. This value is used when the page is freed.
- Fix test_ringbuffer() to test for ERR_PTR before calling kthread_stop()
The rb_threads[] array is assigned the output of kthread_run_on_cpu()
which could return an ERR_PTR. At the end of the test, all threads in the
array are cleaned up by kthread_stop() passing in the value in the array
if it isn't zero. But if the array contains an ERR_PTR, kthread_stop()
will not be able to handle it properly.
-----BEGIN PGP SIGNATURE-----
iIoEABYKADIWIQRRSw7ePDh/lE+zeZMp5XQQmuv6qgUCanicmBQccm9zdGVkdEBn
b29kbWlzLm9yZwAKCRAp5XQQmuv6quonAP9HgM214Bt43edhuJb3oFy9fdS+sqYZ
RIJ9q90iMDUH0AEAk3650lu7u80YniD4INKNrz5QMM2EbIMaNqtqwkS8uwQ=
=tZwi
-----END PGP SIGNATURE-----
Merge tag 'trace-v7.2-rc6' of git://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace
Pull tracing fixes from Steven Rostedt:
- Fix use-after-free in eventfs_remove_rec()
The freeing of the eventfs_inode children used list_for_each_entry()
where the child is freed via srcu, but there's still a chance that it
gets freed. It should be using list_for_each_entry_safe().
- Fix eventfs_inode SRCU use of list in freeing
The iterator uses an SRCU protected list walk on the eventfs inodes.
The eventfs inode uses its "list" field in a union with the RCU list
head. When the inode gets added to the SRCU list it immediately
corrupts the list pointer and can cause an issue with the iterator.
Move the RCU list head to be shared with the children list head which
allows the iterator to check the parent inode if is freed before
referencing the child. Have the iterator check the parent "is_freed"
field and break out if it is set. Also add memory barriers to make
sure the ordering is correct.
- Fix various RCU synchronization issues with direct_functions
Updates to direct_functions have some missing RCU protection and
synchronization. Restructure the code a bit to make sure updates to
the direct_functions are protected.
- Remove an unneeded comma from a scope_guard()
There's a spurious comma in a scope_guard(). Remove it.
- Fix race in per CPU buffer swap in the ring buffer
When a per CPU buffer swap happens, it must make sure that it doesn't
occur while a writer is active. Instead it returns an -EBUSY. But
there's a small race window when a writer moves from one sub-buffer
to the next that it resets the "committing" counter. If a swap
happens at that moment, the buffer used for the commit of an event
will not match the buffer the event is actually on. Instead of using
the "committing" counter, use the recursive detection counter that
does not get reset when the writer crosses sub-buffers.
- Fix off-by-one in ftrace_free_mem()
The function ftrace_free_mem() gets an "end_ptr" as a parameter that
is exclusive to the rang to be freed. But its value is used to search
for the records that expects an inclusive value. Subtract one from
the parameter to convert it to an inclusive range.
- Disable resizing of the ring buffer for persistent buffers
Resizing the persistent buffer has undefined behavior. Prevent it
from being resized.
- Disable changing ring buffer subbuf order when resizing is disabled
The ring buffer subbuffer order can not be changed during resizing.
Use that instead of just checking if the buffer is mapped as mapped
buffers also have resizing disabled.
- Initialize subbuf_order of reader pages when they are created
In rb_allocate_cpu_buffer() the bpage->order is not updated to the
current subbuf_order leaving it as zero. This value is used when the
page is freed.
- Fix test_ringbuffer() to test for ERR_PTR before calling
kthread_stop()
The rb_threads[] array is assigned the output of kthread_run_on_cpu()
which could return an ERR_PTR. At the end of the test, all threads in
the array are cleaned up by kthread_stop() passing in the value in
the array if it isn't zero. But if the array contains an ERR_PTR,
kthread_stop() will not be able to handle it properly.
* tag 'trace-v7.2-rc6' of git://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace:
ring-buffer: Fix crash passing ERR_PTR to kthread_stop()
ring-buffer: Initialise reader page order in rb_allocate_cpu_buffer()
ring-buffer: Prevent subbuf order change when resizing is disabled
ring-buffer: Prevent resizing of persistent ring buffer
ftrace: Fix off-by-one fentry site disable in ftrace_free_mem()
ring-buffer: Use current_context for safe per-CPU buffer swap
ftrace: Drop extra comma in trace_buffered_event_enable
ftrace: Protect direct_functions in update_ftrace_direct_mod
ftrace: Protect direct_functions in update_ftrace_direct_del
ftrace: Protect direct_functions in ftrace_find_rec_direct
eventfs: Use children field for rcu head and add memory barriers
eventfs: Fix use-after-free in eventfs_remove_rec()
In test_ringbuffer()'s out_free cleanup loop, the check
`!rb_threads[cpu]` only catches NULL entries and misses entries that
hold an ERR_PTR.
rb_threads[] is static, so unassigned slots are NULL. But when
kthread_run_on_cpu() fails for a cpu, it stores ERR_PTR(-ENOMEM) (or
-EINTR) in rb_threads[cpu] before the creation loop jumps to out_free.
That entry is non-NULL, so the old `!ptr` check does not break, and the
cleanup proceeds to call kthread_stop() on the ERR_PTR. kthread_stop()
then dereferences the bogus pointer, crashing the kernel during the
late_initcall self-test.
crash logs:
BUG: kernel NULL pointer dereference, address: 000000000000001c
Oops: 0002 [#1] SMP NOPTI
CPU: 1 PID: 1 Comm: swapper/0 Not tainted 7.2.0-rc6-dirty #7 PREEMPT(lazy)
RIP: 0010:kthread_stop+0x2e/0x220
RBX: fffffffffffffff4
CR2: 000000000000001c
Call Trace:
<TASK>
test_ringbuffer+0x1ec/0x650
do_one_initcall+0x6c/0x2c0
kernel_init_freeable+0x21d/0x420
kernel_init+0x15/0x1c0
ret_from_fork+0x21b/0x320
</TASK>
Kernel panic - not syncing: Fatal exception
Cc: stable@vger.kernel.org
Fixes: 64ed3a049e ("ring-buffer: make use of the helper function kthread_run_on_cpu()")
Link: https://patch.msgid.link/20260807154145.2846521-2-sh_def@163.com
Signed-off-by: Hui Su <sh_def@163.com>
Reviewed-by: Vincent Donnefort <vdonnefort@google.com>
Acked-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
In rb_allocate_cpu_buffer(), bpage->order was omitted, leaving it as 0.
This is an issue for a ring-buffer with subbufs bigger than PAGE_SIZE if
when freed: free_buffer_page() relies on this value. Align the value
with the actual allocation size (buffer::subbuf_order).
Cc: stable@vger.kernel.org
Fixes: f9b94daa54 ("ring-buffer: Set new size of the ring buffer sub page")
Link: https://patch.msgid.link/20260806211306.3704194-4-vdonnefort@google.com
Signed-off-by: Vincent Donnefort <vdonnefort@google.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
When a module's init text is freed, do_init_module() calls
ftrace_free_mem() with a half-open [start, end) range. However the
ftrace_cmp_recs() comparator treats the upper bound as inclusive, as all
its other users do, passing 'ip + size - 1'. So ftrace_free_mem() can
delete a record sitting exactly at 'end', which is outside the freed
range.
For a kernel without CFI or IBT, the first record of a function is at
the function start, which for the first function in a module is also the
base of its text allocation. As the module allocator packs its regions,
that address is often the 'end' passed by a neighboring module's
do_init_module(), causing the first function's ftrace location to get
disabled, preventing an attempt to livepatch it:
livepatch: failed to find location for function 'pcspkr_probe'
Convert the exclusive end to the inclusive 'end - 1' the comparator
expects, and return early for an empty range to avoid the subtraction
from underflowing when the init text size is zero.
Cc: stable@vger.kernel.org
Fixes: 42c269c88d ("ftrace: Allow for function tracing to record init functions on boot up")
Link: https://patch.msgid.link/1b5ccfa8095bdb1277f84af1c2c2e2205aca03ae.1785992188.git.jpoimboe@kernel.org
Signed-off-by: Josh Poimboeuf <jpoimboe@kernel.org>
Acked-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
The ring_buffer_swap_cpu() function currently checks the per-CPU
committing counter to determine if a buffer is actively being written to
before performing the swap. However, there exists a race window where
this check can be bypassed:
ring_buffer_lock_reserve
cpu_buffer = buffer->buffers[cpu]; // cpu_buffer_a
rb_reserve_next_event
rb_start_commit // inc committing
if (unlikely(READ_ONCE(cpu_buffer->buffer) != buffer)) {...}
__rb_reserve_next
rb_move_tail
rb_end_commit(cpu_buffer); // dec committing => 0
/* interrupt hits here, successfully swaps! */
local_inc(&cpu_buffer->committing);
ring_buffer_unlock_commit
cpu_buffer = buffer->buffers[cpu]; // cpu_buffer_b
rb_commit
rb_end_commit
RB_WARN_ON(cpu_buffer, !local_read(&cpu_buffer->committing))
// triggers warning
The committing counter can temporarily drop to 0 during a single write
operation (within rb_move_tail), creating a window where swap can
succeed even though the write is still in progress. This leads to
inconsistent buffer state and triggers the RB_WARN_ON in rb_commit().
Replace the committing counter check with current_context checks, which
are set at the entry of ring_buffer_lock_reserve() and remain valid
throughout the entire write operation, providing a reliable indicator of
buffer busy state during swap.
Cc: stable@vger.kernel.org
Fixes: 4239c38fe0 ("ring-buffer: Process commits whenever moving to a new page.")
Link: https://patch.msgid.link/20260803005640.2445666-2-wutengda@huaweicloud.com
Signed-off-by: Tengda Wu <wutengda@huaweicloud.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
that can cause stuck tasks (Yao Kai)
Signed-off-by: Ingo Molnar <mingo@kernel.org>
-----BEGIN PGP SIGNATURE-----
iQJFBAABCgAvFiEEBpT5eoXrXCwVQwEKEnMQ0APhK1gFAmp3gQ8RHG1pbmdvQGtl
cm5lbC5vcmcACgkQEnMQ0APhK1h0CQ//Wq3AnGGWisG9OAyn22xxlkh1lK4RqUeE
uAYuLQmIAw481YjiVg7U0QTcyHrMrDHK5j902oc1Zd7Cyc+IqBaaZVPs63Vq0onh
KAP19tbFz2w9D/mxyTkuxEekJ42w8a7bk5cHFL1rw0t/rSA51LixzbdW7DloC2vg
So+quAtUcaXI8831ljnikN4OdNQOCLJCA9MemTpncIMuyL4BmhOXDRMwXneevg74
f6BOqrgbvKEgrBsVhWzGeDdq5ZHekmGVrHQeOxlDaQy/rZS+VE3VJwtMBwZvhn6X
wU3CQdvkvDOeQvqXyfWxTbqhk7AJrIL5FKQrp5ZlhECw2WnPOUHwjAHQ8dKHaJEw
wbv58RALcJ33s+PWy+0tYmOP4SPyICpQEmdp+SCaR0N4N/LtAScz05XAGnJ4S97+
t8LBmmJFmkxz4rDbTdawBV+sulDX/y+8xYu0/CZJhAyp1hEW9ajyMbRj5gVGOomT
xKyyQAUTUUznStscc4hgTNVd1UAhuqUYlMNCJsEJOHNHvnq5qHT+ezPoZt5X5qd6
cDHCs5b+agQ/PIpW1vIiulCypelckAqvs+XdE0Pv3uPlNVjMLfZCTeyaOoXqIqI7
LpXkgR0UxkWmFf8vGncpPHjDCh3YqGgN66iE1qKohO1L9/uSxwwxEUOJIcPaOFtX
C07TnEwMdK0=
=3w7m
-----END PGP SIGNATURE-----
Merge tag 'locking-urgent-2026-08-08' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
Pull futex fix from Ingo Molnar:
- Fix race in futex_pivot_pending() during private hash resize
that can cause stuck tasks (Yao Kai)
* tag 'locking-urgent-2026-08-08' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip:
futex: Fix race in futex_pivot_pending() during private hash resize
Move NMI nesting tracking from the preempt_count bits to a separate
per-CPU counter (nmi_nesting). This is to free up the NMI bits in the
preempt_count, allowing those bits to be repurposed for other uses.
Reduce NMI_BITS from 4 to 1, using it only to detect if we're in an NMI.
The per-CPU counter currently caps nesting at 15.
[boqun: Address Steven Rostedt's comment on the BUG_ON() condition]
[boqun: Use preempt_count_set() in __nmi_exit() to avoid underflow]
Suggested-by: Boqun Feng <boqun@kernel.org>
Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com>
Signed-off-by: Lyude Paul <lyude@redhat.com>
Signed-off-by: Boqun Feng <boqun@kernel.org>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/20260121223933.1568682-3-lyude@redhat.com
Link: https://patch.msgid.link/20260804161447.84806-2-boqun@kernel.org
Move the event_define_fields() call in event_create_dir() before the
eventfs directory creation. Previously, a failure after directory
creation wouldn't clean up eventfs_inode because the error path didn't
call eventfs_remove_dir(). This eliminates the need to clean up the
eventfs directories if event_define_fields() fails.
Link: https://patch.msgid.link/20260715135231.338535-3-ashelat@redhat.com
Signed-off-by: Anubhav Shelat <ashelat@redhat.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
Drop the extra comma in "scoped_guard()" to cleanup the code.
Link: https://patch.msgid.link/20260730150411.88667-5-leon.hwang@linux.dev
Acked-by: Jiri Olsa <jolsa@kernel.org>
Signed-off-by: Leon Hwang <leon.hwang@linux.dev>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
Fix accessing the __rcu pointer direct_functions with RCU protection.
Cc: stable@vger.kernel.org
Link: https://patch.msgid.link/20260730150411.88667-2-leon.hwang@linux.dev
Fixes: d05cb47066 ("ftrace: Fix modification of direct_function hash while in use")
Acked-by: Jiri Olsa <jolsa@kernel.org>
Suggested-by: Steven Rostedt <rostedt@goodmis.org>
Signed-off-by: Leon Hwang <leon.hwang@linux.dev>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
Boot-time tracing currently supports kprobe-events and synthetic-events
under per-event configuration options.
Extend boot-time tracing to support newly added dynamic probe types:
- event probes (eprobe) under the "eprobes" event group
- function probes (fprobe) under the "fprobes" event group
- tracepoint probes (tprobe) under the "tracepoints" or "tprobes"
event group
To support this cleanly, update dyn_event_create() in trace_dynevent.c
so that passing NULL as the type parameter delegates to
create_dyn_event(), allowing generic creation of any registered
dynamic event type from a raw command string.
Update Documentation/trace/boottime-trace.rst accordingly to describe
the new per-event bootconfig options.
Link: https://lore.kernel.org/all/178613905149.259829.18185480460810689421.stgit@devnote2/
Assisted-by: Antigravity:gemini-3.6-flash
Signed-off-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
Changes in v3:
- Check return values of strscpy() and snprintf() in trace_boot_add_probe_event()
to prevent silent buffer truncation when constructing probe event strings.
Changes in v2:
- Fix raw command detection logic for eprobes, fprobes, and tprobes
by requiring ':' or isspace() after type prefix.
- Consolidate duplicate loop logic into trace_boot_add_probe_event()
helper function.
Reject tracing and freplace attachments to a target program with arena
context arguments. The struct_ops indirect trampoline converts those
arguments before entering the target, so a generic tracing trampoline
would otherwise expose arena offsets using the target BTF pointer type.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://patch.msgid.link/20260808003938.3486067-14-memxor@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
A struct_ops callback cannot receive an arena pointer directly, so
passing one takes two steps. The pointer arrives as a bare u64 that the
callback casts, and because the two sides address the arena through
different bases it also has to be rebased by hand on the way in.
Add the __arena and __arena__nullable stub argument suffixes to make this
convenient. The callback declares the parameter as an arena pointer,
receives it as a PTR_TO_ARENA register, and dereferences it directly,
while the kernel caller just passes the natural kernel arena address
(kaddr). The trampoline converts the value while saving the arguments
into the BPF ctx, ctx[slot] = (u32)(kaddr - kern_vm_start), so the
program never sees a kernel address and nothing rewrites the ctx after
the fact. The converted value keeps the upper 32 bits clear as the JITs
require of arena pointer registers and behaves like any cast_kern'ed
arena pointer, so cast_user recovers the full user-visible address.
__arena converts unconditionally and the kernel caller must not pass
NULL. __arena__nullable preserves NULL, tested on the full 64-bit kernel
pointer, and surfaces to the verifier as PTR_TO_ARENA (but not as a
PTR_TO_ARENA | PTR_MAYBE_NULL). The reason is that PTR_TO_ARENA in the
program's type state already encompasses NULL-ness, so it is not
meaningful to force a NULL check for the program.
The composite suffix intentionally ends in __nullable. Classify
__arena__nullable before the generic suffix so scalar arena pointees do
not take the generic nullable BTF pointer path.
This patch adds the generic side. prepare_arg_info() records arena and
nullable argument flags in the struct_ops function model, and
bpf_tramp_arena_base() returns the arena base for a single-program
struct_ops indirect trampoline. Only that trampoline converts: its
program's arena is fixed at generation time. Generic trampolines can mix
programs with different arenas and reject arena context arguments
defensively, which is unreachable today as only struct_ops programs
carry them. Architectures that do not implement the conversion are
gated out at verification time with bpf_jit_supports_arena_args().
Co-developed-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://patch.msgid.link/20260808003938.3486067-6-memxor@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Passing an arena pointer to a kfunc takes two steps today. There is no
arena pointer argument type, so the pointer crosses the boundary as a
bare scalar, and the kfunc then offsets it by the arena base and casts
it before it can touch the memory. Every such kfunc open-codes the same
translation.
Add the __arena and __arena__nullable argument suffixes to make this more
convenient. The kfunc declares the parameter by its real pointer type
and dereferences it directly, with the JIT rebasing the value at the
call site, rN = kern_vm_start + (u32)rN. No bounds check is needed: the
u32 offset stays within the guard-padded arena kernel mapping, and a
fault on an unpopulated page recovers through the per-arena scratch
page. A suffixed argument accepts a PTR_TO_ARENA or scalar register,
matching global subprog arena arguments.
__arena rebases unconditionally, so the kfunc never sees NULL and a
value with zero in the low 32 bits arrives as the arena base.
__arena__nullable preserves NULL for optional arguments by skipping the
rebase when the truncated value, arena offset 0, is zero. Keeping the
plain form NULL-free saves the NULL test on every call.
The double separator makes the annotations composable:
__arena__nullable also ends in __nullable and naturally follows the
common nullable argument path. Plain __arena follows that path too for
verifier type checking because both forms accept a constant zero; the
function-model flag still determines whether the JIT preserves NULL or
rebases it to the arena base.
This patch adds the verifier side: the suffixes are recognized in
check_kfunc_args() and distilled into argument flags in the function
model stored in the kfunc descriptor. JITs retrieve the model while
emitting the call, avoiding per-call state in insn_aux_data.
JITs declare support with bpf_jit_supports_arena_args() and verification
fails with -ENOTSUPP elsewhere.
Co-developed-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://patch.msgid.link/20260808003938.3486067-5-memxor@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
The kfunc descriptors include argument prototypes generated while calls are
collected. Some argument classifications need program auxiliary state derived
from referenced maps, such as the arena associated with the program.
This avoids a footgun in get_kfunc_arg_type() checks where we do
validation on whether program has prog->aux->arena and it hasn't been
resolved yet.
check_and_resolve_insns() records used maps and populates that state. It must
remain after bpf_check_btf_info(), which applies kernel-side CO-RE relocations,
so that instruction validation and the program tag observe the relocated
instruction stream.
Move only add_kfuncs() after instruction and resource resolution. Subprogram
discovery and validation remain before the full BTF phase because that phase
needs the complete subprogram layout. Add a short comment describing the
resource resolution phase at the call site.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Reviewed-by: Amery Hung <ameryhung@gmail.com>
Link: https://patch.msgid.link/20260808003938.3486067-4-memxor@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
add_subprog_and_kfunc() combines two operations with different ordering
requirements. Subprogram discovery must precede validation of func_info and
line_info, while kfunc descriptors are only needed by the verifier after its
initial program setup is complete.
Split the helper into add_subprogs() and add_kfuncs() so each operation can be
placed according to its actual dependencies. Keep both calls adjacent and in
their existing phase for now, and add short comments describing their roles.
No functional change is intended for valid programs.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Reviewed-by: Amery Hung <ameryhung@gmail.com>
Link: https://patch.msgid.link/20260808003938.3486067-3-memxor@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
BTF processing is split around subprogram discovery. The first phase gets
program BTF and imports func_info because a BTF-tagged exception callback
may not be referenced by any instruction. Subprogram discovery needs this
metadata to find it.
The later phase validates func_info and line_info against the complete
subprogram table and applies CO-RE relocations. This split breaks a real
dependency cycle rather than merely running the same checks early.
Rename bpf_check_btf_info_early() and check_btf_func_early() to preparation
names that reflect this role. Add short call-site comments to make the two
phases and their responsibilities clear.
No functional change is intended.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Reviewed-by: Amery Hung <ameryhung@gmail.com>
Link: https://patch.msgid.link/20260808003938.3486067-2-memxor@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
After the previous commit bpf_is_reg64() is only used in a context
where destination register's property is queried, and only for
instructions for which insn_def_regno() >= 0. Hence, simplify the
function by:
- removing unused parameters;
- removing code paths considering BPF_JMP{,32} instructions;
- streamlining the condition expressions.
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Acked-by: Daniel Borkmann <daniel@iogearbox.net>
Link: https://lore.kernel.org/bpf/20260807-static-zext-v4-6-b6c270013c77@gmail.com
As reported in the thread [1], the verifier's 32-bit operations zero
extension logic is broken. This logic is responsible for correct
semantics of 32-bit operations on s390 architecture.
According to BPF semantics, operation `w1 += 1` is supposed to zero
extend the upper half of the register `r1`. On s390 the JIT relies on
the verifier emitting explicit zero extension before such operations.
The verifier attempts to minimize the amount of zero extensions
inserted by tracking whether upper halves of the 64-bit registers are
ever used. Previously such tracking worked as follows:
- bpf_reg_state->subreg_def field was set by do_check_insn()
for each operation defining lower but not the upper halves
of the register.
- Whenever an operation reading the whole register was verified,
the verifier checked register's subreg_def and set
bpf_insn_aux_data->zext_dst flag as true via a call to
mark_insn_zext() function.
- After the verification was complete, a special pass
bpf_opt_subreg_zext_lo32_rnd_hi32() extended 32-bit operations
with bpf_insn_aux_data->zext_dst set as true by adding
explicit zero extension.
Note that the logic above relies on bpf_reg_state->subreg_def,
which is a property of a current verifier state.
Before the commit [2] two additional steps happened:
- The verifier tracked upper and lower register halves' liveness as
flags REG_LIVE_READ{32,64} in bpf_reg_state->live.
- The function propagate_liveness() called mark_insn_zext()
in order to transfer the knowledge about which registers have
their upper halves alive (and thus might require zero extension).
The commit [2] removed the two steps described above,
hence making possible a situation like below:
- The register's upper half is set and is used on some verification
path P1 and the register happens not to be marked as precise.
- The checkpoint C is created while processing some instruction
between register initialization and usage.
- On some other verification path P2 the register's upper half is not
initialized and that path ends hitting the checkpoint C.
- In such a case the register's initialization on path P2 would lack
zext_dst mark, making it possible for the program to inject
an arbitrary value in the register's upper half.
This commit replaces subreg_def based logic with computing zext_dst
statically, as a part of the bpf_compute_live_registers() analysis:
- The analysis now tracks usage of upper and lower halves of the
registers separately.
- If some instruction defines a 32-bit subregister, but not the whole
register, *and* the upper half of the register is alive after that
instruction, the instruction is marked as zext_dst.
There is one notable drop in precision: whenever a BPF subprogram is
called, all 64 bits of parameter registers are presumed to be used.
The assumption is that such a drop in precision would not inflict
a noticeable performance penalty.
[1] https://lore.kernel.org/bpf/CAGKGUv=sOuqQtA1Ub-5JXfA4FPosJFYKAQE4B79cK+P1erxqtg@mail.gmail.com/
[2] commit 107e169799 ("bpf: disable and remove registers chain based liveness")
Fixes: 107e169799 ("bpf: disable and remove registers chain based liveness")
Reported-by: Min-gyu Kim <gimm78064@gmail.com>
Reported-by: STAR Labs SG <info@starlabs.sg>
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Acked-by: Daniel Borkmann <daniel@iogearbox.net>
Link: https://lore.kernel.org/bpf/CAGKGUv=sOuqQtA1Ub-5JXfA4FPosJFYKAQE4B79cK+P1erxqtg@mail.gmail.com/
Link: https://lore.kernel.org/bpf/20260807-static-zext-v4-5-b6c270013c77@gmail.com
Extend compute_live_registers() to track upper and lower register
halves' liveness separately. This is mostly straightforward:
- use/def masks are extended to track 2 bits per register;
- compute_insn_live_regs() is updated to properly track these
2 bits according to the instruction semantics.
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Acked-by: Daniel Borkmann <daniel@iogearbox.net>
Link: https://lore.kernel.org/bpf/20260807-static-zext-v4-4-b6c270013c77@gmail.com
The following patches are going to remove bpf_is_reg64() users from
everywhere except fixups.c, and also make it dependent on functions
local to fixups.c. Move the function before hand to simplify the
review. Non functional change.
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Acked-by: Daniel Borkmann <daniel@iogearbox.net>
Link: https://lore.kernel.org/bpf/20260807-static-zext-v4-3-b6c270013c77@gmail.com
bpf_do_misc_fixups() converts the following address space cast
instructions to 32-bit moves:
- cast from address space 1 (user) to address space 0 (kernel)
- cast from address space 0 (kernel) to address space 1 (user)
iff associated arena map has a BPF_F_NO_USER_CONV flag.
Extract a predicate detecting such instructions for use in the
following patches.
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Acked-by: Daniel Borkmann <daniel@iogearbox.net>
Link: https://lore.kernel.org/bpf/20260807-static-zext-v4-2-b6c270013c77@gmail.com
At the moment there are more callsites that want bpf_verbose_insn() to
not print a newline after the instruction, than callsites that want a
newline. Drop '\n' from disasm.c. Non-functional change.
The changes in bpftool are verified by writing a bpf program using a
variety of instructions and comparing `prog dump xlated` output in the
following modes: plain, opcodes, visual, visual opcodes. The output
before and after the changes is identical.
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Reviewed-by: Quentin Monnet <qmo@kernel.org>
Acked-by: Daniel Borkmann <daniel@iogearbox.net>
Link: https://lore.kernel.org/bpf/20260807-static-zext-v4-1-b6c270013c77@gmail.com
stack_map_get_build_id_offset() introduced a per-CPU irq_work to defer
mmap_read_unlock() from NMI context, and bpf_find_vma() later reused the
same mmap_unlock_work. Both callers only check whether the work is busy
before taking mmap_lock, so a nested caller can reuse the slot before the
first caller queues it. Two read locks may then be acquired while only one
deferred unlock runs, leaking a read lock and blocking exit_mmap().
Reserve the per-CPU slot before mmap_read_trylock(). Use the same wrapper
in stackmap and bpf_find_vma() so both callers release the reservation on
trylock failure. Keep rejecting the slot while the irq_work remains busy.
Release it after the irq_work callback unlocks the mm.
Fixes: eac9153f2b ("bpf/stackmap: Fix deadlock with rq_lock in bpf_get_stack()")
Reported-by: syzbot+cdd6c0925e12b0af60cc@syzkaller.appspotmail.com
Reported-by: sashiko-bot@kernel.org
Signed-off-by: Sanghyun Park <sanghyun.park.cnu@gmail.com>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Closes: https://syzkaller.appspot.com/bug?extid=cdd6c0925e12b0af60cc
Closes: https://lore.kernel.org/r/20260630033745.B80201F000E9@smtp.kernel.org
Link: https://lore.kernel.org/bpf/20260805031425.2157475-2-sanghyun.park.cnu@gmail.com
In mmio_print_mark(), a raw C cast (struct print_entry *)entry is used to
obtain the print_entry pointer.
Use the standard trace_assign_type() macro instead, matching the usage in
mmio_print_rw() and mmio_print_map().
Link: https://patch.msgid.link/178524301013.56416.9116249028160618790.stgit@devnote2
Assisted-by: Antigravity:gemini-3.6-flash
Signed-off-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
Syzkaller reported a storage null-ptr-deref issue after replacing prog.
This occurs in the following scenario:
1. prog A, an empty prog, is attached to a cgrp.
2. prog B uses BPF_MAP_TYPE_PERCPU_CGROUP_STORAGE and calls the
bpf_get_local_storage helper.
3. link_update is called to replace prog A with prog B.
The reason is that __cgroup_bpf_replace fails to alloc and assign the
required cgrp storage for the incoming replacement prog. Consequently,
the new prog inherits an uninit storage, leading to null-ptr-deref panic
when kick the new prog.
Fix this by rejecting a link update if new_prog's cgroup storage is
incompatible with link->prog.
Fixes: 0c991ebc8c ("bpf: Implement bpf_prog replacement for an active bpf_cgroup_link")
Signed-off-by: Pu Lehui <pulehui@huawei.com>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Reviewed-by: Amery Hung <ameryhung@gmail.com>
Acked-by: Leon Hwang <leon.hwang@linux.dev>
Link: https://lore.kernel.org/bpf/20260728132336.2857800-1-pulehui@huaweicloud.com [0]
Link: https://lore.kernel.org/bpf/f87b53c0-8f00-45a6-82db-8242fa9b143f@huaweicloud.com [1]
Link: https://lore.kernel.org/bpf/20260807104403.1013064-1-pulehui@huaweicloud.com
Merge updates related to system sleep handling and runtime PM, a power
capping update, and power management tools updates for 7.3-rc1:
- Rename the PM core module parameter prefix to "pm" and allow the PM
transition (DPM) watchdog to be disabled by default (Tzung-Bi Shih)
- Fix off-by-one in wakelocks number limit check in the system sleep
sysfs interface (Haowen Tu)
- Remove kernel-doc markings from helper descriptions in the core
hibernation code (Adi Nata)
- Use %pe to print error pointer values in the hibernation core (Ronan
Marchal)
- Fix memory leak in snapshot_write_next() error path (Malaya Kumar
Rout)
- Delay allocating and linking the next swap_map_page in the
hibernation image saving code until another image page actually
needs to be recorded (Haesung Kim)
- Stop setting runtime_error on runtime resume callback failures to
allow drivers to recover from resume issues (Praveen Talari)
- Handle PMU registration failure during probe in the intel_rapl_tpmi
driver (Sumeet Pawnikar)
- Avoid optional imports in intel_pstate_tracer unless they are really
needed (Yousef Alhouseen)
- Add generic CPPC performance display to the cpupower utility, build
and call CPPC information on non-AMD processors, make cpupower print
kernel and hardware frequency information, and add libm to cpupower
for generic CPPC view (Jeremy Linton)
- Remove conditional return with no effect from cpupower (Sang-Heon
Jeon)
* pm-sleep:
PM: sleep: Allow disabling DPM watchdog by default
PM: sleep: Rename module parameters prefix to "pm"
PM: hibernate: swap: defer linking the next map page
PM: hibernate: Fix memory leak in snapshot_write_next() error path
PM: hibernate: Use %pe to print error pointer values
PM: hibernate: Remove kernel-doc markings from helper descriptions
PM: sleep: Fix off-by-one in wakelocks number limit check
* pm-runtime:
PM: runtime: Only set runtime_error on suspend callback failures
* pm-powercap:
powercap: intel_rapl_tpmi: Handle PMU registration failure during probe
* pm-tools:
cpupower: remove conditional return with no effect
tools/power: intel_pstate_tracer: avoid optional imports for help
cpupower: Add libm to cpupower for generic CPPC view
cpupower: Print kernel and hardware frequency information
cpupower: Build and call CPPC information on non-AMD processors
cpupower: Add generic CPPC performance display
Merge cpufreq updates for 7.3-rc1:
- Minor fixes and cleanups in assorted cpufreq drivers (Dan Carpenter,
Guru Das Srinagesh, Haoxiang Li, Karl Mehltretter, Sasha Finkelstein,
and Pan Chuang)
- Fix cpufreq table creation and bios_limits() callback in the Rust
bindings (Priya Bala Govindasamy)
- Add IPQ5210 support to qcom-nvmem driver (Varadarajan Narayanan)
- Adjust the .adjust_perf() cpufreq driver callback to allow the
maximum performance value to be passed to drivers and update the
intel_pstate driver to use it (Rafael Wysocki)
- Set policy->cur to the actual requested frequency in the intel_pstate
driver when the performance policy is used (Rafael Wysocki)
- Simplify HWP handling on Broadwell processors in intel_pstate (Rafael
Wysocki)
- Fix setting minimum P-state at init time in intel_pstate (Rafael
Wysocki)
- Consolidate frequency values computation in intel_pstate and clean up
code in that driver (Rafael Wysocki)
- Add missing kernel-doc desciptions for structure and union members in
the amd-pstate driver (David Vernet)
- Handle missing policy in dynamic EPP callbacks in the amd-pstate
driver (EDAMAMEX)
- Introduce EXPORT_SYMBOL_FOR_PSTATE_UT() to export amd-pstate driver
symbols to the amd-pstate-ut subdriver (K Prateek Nayak)
- Add dynamic EPP as an "energy_performance_preference" mode in
amd-pstate, remove the "amd_dynamic_epp" kernel command line option
and the "dynamic_epp" sysfs attribute, and update the dynamic_epp
documentation accordingly (K Prateek Nayak)
- Add unit tests for CPPC Performance Priority and the "dynamic" EPP
mode in the amd-pstate driver (K Prateek Nayak)
- Set min_limit_freq based on bios_min_perf in amd-pstate and remove
the defensive check for bios_min_perf from it (K Prateek Nayak)
- Fix EPP return type and handle errors in amd-pstate during
initialization, toggle auto_sel in active mode on shared memory
systems, and cache the firmware programmed EPP value (Marco Scardovi)
- Skip tests in amd-pstate-ut if the amd-pstate driver is not in active
use (Qianheng Peng)
- Replace sprintf() with sysfs_emit() in sysfs show in the cpufreq
schedutil governor and fix a self-contradictory comment in
sugov_iowait_apply() (Zhongqiu Han)
- Fix the usage example for the sampling_rate tunable of the ondemand
cpufreq governor in admin-guide (wangxiaodong)
* pm-cpufreq: (40 commits)
cpufreq: imx6q: fix out-of-bounds write when probed more than once
cpufreq: imx6q: fix devres accumulation across driver rebind
rust: cpufreq: Fix temporary write in Registration::bios_limit_callback
rust: cpufreq: Add CPUFREQ_TABLE_END as last table entry in TableBuilder::to_table
cpufreq: intel_pstate: Adjust policy->cur in active mode to policy
cpufreq/amd-pstate: Document missing kernel-doc members
cpufreq/amd-pstate-ut: Add unit test for CPPC Performance Priority
cpufreq/amd-pstate-ut: Add unit test for "dynamic" EPP mode
cpufreq/amd-pstate: Reduce the scope of exported symbols
Documentation/amd-pstate: Update dynamic_epp documentation with new behavior
cpufreq/amd-pstate: Remove "amd_dynamic_epp" cmdline and "dynamic_epp" sysfs
cpufreq/amd-pstate: Add dynamic EPP as an "energy_performance_preference" mode
cpufreq/amd-pstate: Extract platform profile to EPP conversion into a helper
cpufreq/amd-pstate: Remove the defensive check for bios_min_perf
cpufreq/amd-pstate: Set min_limit_freq based on bios_min_perf
cpufreq: apple-soc: Calculate frequency as a 64-bit value
kselftest: cpufreq: Backup and restore governor for sptests
selftests/cpufreq: Remove unnecessary sudo from quick_shuffle()
selftests/cpufreq: Remove unused local variables from switch_show_governor()
cpufreq/amd-pstate: handle missing policy in dynamic EPP callbacks
...
Commit 9c63e84db2 ("sched/core: Disable SD_PREFER_SIBLING on asymmetric
CPU capacity domains") removed the SD_PREFER_SIBLING from the domains with
asymmetric capacity. This was done to avoid spreading tasks to sibling
scheduling groups with less capacity, but this does not happen: checks for
capacity in update_sd_pick_busiest(), sched_balance_find_src_group(), and
sched_balance_find_src_rq() prevent migrations from high- to low-capacity
CPUs if the busiest group is not overloaded.
The cluster topology is a notable example: some systems have scheduling
domains spanning CPUs of asymmetric capacity, grouped into two or more
equal-capacity clusters sharing an L2 cache. When CONFIG_SCHED_CLUSTER is
enabled, SD_PREFER_SIBLING is needed in the domain to spread load across
these clusters.
CPUs with spare capacity, big or small, have always helped overloaded
groups. Once the overloading condition disappears, misfit load will still
be used to move high-utilization tasks to bigger CPUs if they have spare
capacity.
Adding the SD_PREFER_SIBLING flag shifts load balancing in shared-LLC
domains from equalizing the number of idle CPUs to equalizing the number
of running tasks. This enables migrations among clusters from newly-idle
load balance, where the outgoing task is already dequeued but the CPU
has not yet transitioned to idle.
Signed-off-by: Ricardo Neri <ricardo.neri-calderon@linux.intel.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Vincent Guittot <vincent.guittot@linaro.org>
Tested-by: Christian Loehle <christian.loehle@arm.com>
Tested-by: Andrea Righi <arighi@nvidia.com>
Link: https://patch.msgid.link/20260720-rneri-fix-cas-clusters-v6-6-bb500bf4afd4@linux.intel.com
sched_balance_find_src_rq() avoids selecting a runqueue with a single
running task as busiest if doing so results in migrating the task to a
CPU with less than ~5% of extra capacity. It also unintentionally
prevents migrations between CPUs of identical capacity.
When CONFIG_SCHED_CLUSTER is enabled, load should be balanced across
clusters of CPUs with the same capacity. Allowing migration between CPUs
of identical capacity is necessary to meet this goal.
Use get_actual_cpu_capacity() to reflect architectural capacity as well
as diminished capacity due to hardware or cpufreq pressure. Guard this
check with the sched_cluster_active static key so that systems without
cluster topology are unaffected.
Signed-off-by: Ricardo Neri <ricardo.neri-calderon@linux.intel.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Christian Loehle <christian.loehle@arm.com>
Reviewed-by: Vincent Guittot <vincent.guittot@linaro.org>
Tested-by: Christian Loehle <christian.loehle@arm.com>
Tested-by: Andrea Righi <arighi@nvidia.com>
Link: https://patch.msgid.link/20260720-rneri-fix-cas-clusters-v6-5-bb500bf4afd4@linux.intel.com
In domains with asymmetric capacity, identifying misfit load in a
scheduling group is not useful when the destination CPU cannot help (i.e.,
its capacity exceeds the group's maximum CPU capacity by less than ~5%). In
such cases, it also prevents load balance among clusters of equal capacity
when CONFIG_SCHED_CLUSTER is enabled. This happens because
update_sd_pick_busiest() skips candidate groups of type misfit_task if the
destination CPU has similar capacity.
Skipping misfit load accounting in this situation allows the group to be
classified as has_spare or fully_busy and lets load balancing proceed. Keep
marking scheduling groups as overloaded when misfit tasks are present. The
sg_overloaded flag propagates to the root domain and allows bigger CPUs in
it to help via newly idle balance.
Signed-off-by: Ricardo Neri <ricardo.neri-calderon@linux.intel.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Christian Loehle <christian.loehle@arm.com>
Reviewed-by: Chen Yu <yu.c.chen@intel.com>
Reviewed-by: Vincent Guittot <vincent.guittot@linaro.org>
Tested-by: Christian Loehle <christian.loehle@arm.com>
Tested-by: Andrea Righi <arighi@nvidia.com>
Link: https://patch.msgid.link/20260720-rneri-fix-cas-clusters-v6-4-bb500bf4afd4@linux.intel.com
update_sd_pick_busiest() may incorrectly select a fully_busy group as the
busiest group when its per-CPU capacity exceeds that of the destination
CPU. This happens because the type of busiest group is initialized to
group_has_spare and allows the fully_busy group to win the type comparison.
update_sd_pick_busiest() should not choose a candidate scheduling group
with at most one runnable task if its per-CPU capacity is greater than that
of the destination CPU. Such a check already exists, but it is done too
late: after the type comparison, preventing a subsequent fully_busy group
of equal per-CPU capacity from being correctly selected.
Move this check to occur before comparing group types.
Fixes: 0b0695f2b3 ("sched/fair: Rework load_balance()")
Signed-off-by: Ricardo Neri <ricardo.neri-calderon@linux.intel.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Christian Loehle <christian.loehle@arm.com>
Reviewed-by: Chen Yu <yu.c.chen@intel.com>
Reviewed-by: Tim Chen <tim.c.chen@linux.intel.com>
Reviewed-by: Vincent Guittot <vincent.guittot@linaro.org>
Tested-by: Christian Loehle <christian.loehle@arm.com>
Tested-by: Andrea Righi <arighi@nvidia.com>
Link: https://patch.msgid.link/20260720-rneri-fix-cas-clusters-v6-3-bb500bf4afd4@linux.intel.com
The argument sg_overloaded of update_sg_lb_stats() is only consumed when
balancing at the root domain. It only makes sense to update it in such a
case. Commit 3229adbe78 ("sched/fair: Do not compute overloaded status
unnecessarily during lb") updated the logic accordingly but missed the case
in which the root domain has the SD_ASYM_CPUCAPACITY flag. Fix this.
Fixes: 3229adbe78 ("sched/fair: Do not compute overloaded status unnecessarily during lb")
Reported-by: Chen Yu <yu.c.chen@intel.com>
Signed-off-by: Ricardo Neri <ricardo.neri-calderon@linux.intel.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Vincent Guittot <vincent.guittot@linaro.org>
Tested-by: Christian Loehle <christian.loehle@arm.com>
Tested-by: Andrea Righi <arighi@nvidia.com>
Link: https://patch.msgid.link/20260720-rneri-fix-cas-clusters-v6-2-bb500bf4afd4@linux.intel.com
When picking a busiest CPU with only one running task, the function
sched_balance_find_src_rq() skips candidate CPUs if the destination CPU has
less than ~5% extra capacity. This condition only holds if all the SMT
siblings of a CPU are idle.
SMT siblings share the computing resources of a physical core and this
results in reduced capacity if more than one sibling is busy.
Skipping a CPU as described would prevent the load balancer from pulling
tasks from a scheduling group previously and correctly identified as
group_smt_balance (i.e., one with more than one task running).
Do not skip a candidate CPU of similar capacity if it has busy SMT
siblings.
Signed-off-by: Ricardo Neri <ricardo.neri-calderon@linux.intel.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: K Prateek Nayak <kprateek.nayak@amd.com>
Reviewed-by: Vincent Guittot <vincent.guittot@linaro.org>
Tested-by: Andrea Righi <arighi@nvidia.com>
Link: https://patch.msgid.link/20260720-rneri-fix-cas-clusters-v6-1-bb500bf4afd4@linux.intel.com
find_new_ilb() selects the first idle housekeeping CPU without
considering whether another thread is running on the same physical core.
On an SMT system, the idle load balancer can therefore activate both
siblings even when another housekeeping CPU has an entirely idle core.
On most SMT systems, this is not problematic because the idle load
balancer is a short-lived activity and the transient wakeup of a sibling
has negligible performance impact.
However, this can be particularly costly on NVIDIA Olympus cores used in
Vera. Briefly activating an otherwise idle sibling can reduce the
performance available to the other sibling and this effect does not
necessarily end once the activated sibling becomes idle: after the ILB
finishes and its CPU enters WFI, full single-thread performance is
restored only after the sibling has remained idle for a qualification
interval (10 Ki cycles on the tested Vera system). Repeated short
sibling wakeups can therefore sustain the interference even with little
actual overlap.
Prevent this by preferring an idle housekeeping CPU whose entire SMT
core is idle. Retain the first idle CPU as a fallback when no fully idle
core is available, so NOHZ balancing continues to make forward progress.
Once a partially busy core has been examined, skip its remaining SMT
siblings to avoid repeating the core-idle check on wide SMT systems.
Tests performed using an ad hoc GEMM benchmark running one CPU-intensive
task per SMT core within its CPU affinity mask improved from
approximately 6.2 TFLOP/s to 9.4 TFLOP/s.
Note that this preference may wake a fully idle physical core instead of
using an idle sibling of an active core, potentially increasing ILB
wakeup latency or energy consumption on some architectures. It may also
scan additional CPUs before selecting the one to run the ILB. The
selection falls back to the first idle CPU when no fully idle SMT core
is available. Non-SMT systems continue to select the first idle
housekeeping CPU.
Signed-off-by: Andrea Righi <arighi@nvidia.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Mete Durlu <meted@linux.ibm.com>
Reviewed-by: Vincent Guittot <vincent.guittot@linaro.org>
Link: https://patch.msgid.link/20260804151324.918020-1-arighi@nvidia.com
perf_group_detach() handles leader and sibling detach differently. When the
group leader is detached, all siblings are promoted to singleton events and
their group_leader pointer is reset to themselves. When a sibling is
detached, it is removed from the leader's sibling_list, but its
group_leader pointer is left pointing at the old leader.
That is harmless when the sibling is being closed and freed immediately, as
in the DETACH_DEAD path. It is not safe when the sibling is detached but
kept alive, such as during CPU hotplug with DETACH_GROUP. In that case the
sibling is removed from the context, while its file descriptor can still
keep it alive.
A typical failing sequence is:
- A group contains leader L and sibling S.
- CPU hot-unplug detaches S with DETACH_GROUP, removing it from
L->sibling_list but leaving S->group_leader == L.
- L is later closed and freed.
- A PERF_IOC_FLAG_GROUP ioctl on S follows S->group_leader and
dereferences the freed leader.
This was reproduced by running the perf event fuzzer, CPU hotplug, and a
stress workload concurrently:
Unable to handle kernel paging request at virtual address 006b6b6b6b6b6cdb
CPU: 2 PID: 12489 Comm: perf_fuzzer 6.18.7 PREEMPT
pc : perf_ioctl+0x34c/0xc68
x20: ffffff89a3fa2c70 x8 : 6b6b6b6b6b6b6b6b
Code: 943c4a0e 340047a0 f9404a94 f9411e88 (f940b908)
Call trace:
perf_ioctl+0x34c/0xc68 (P)
__arm64_sys_ioctl+0xa0/0xf4
invoke_syscall+0x58/0xe4
el0_svc_common+0xa8/0xdc
do_el0_svc+0x1c/0x28
el0_svc+0x40/0xc0
el0t_64_sync_handler+0x68/0xdc
el0t_64_sync+0x1c4/0x1c8
The fault happened in perf_ioctl(), where perf_event_for_each() follows
the stale group_leader pointer and perf_event_for_each_child() then
dereferences the freed leader's context.
Fix the use-after-free by promoting the detached sibling to a singleton.
Also fix __event_disable() cgroup accounting and event state change.
Fixes: 8a49542c05 ("perf_events: Fix races in group composition")
Assisted-by: PatchWise:gpt-5.5
Signed-off-by: Aditya Chillara <aditya.chillara@oss.qualcomm.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
Cc: stable@vger.kernel.org
Link: https://patch.msgid.link/20260807-fix-group-leader-uaf-v3-1-b0c2310c9a0d@oss.qualcomm.com
perf_event_remove_on_exec() sets remove-on-exec events to the EXIT state
and detaches their group relationships. The event's file descriptor can
remain open, however, and perf_event_open() currently accepts that event
as a group leader because its early validation rejects only REVOKED and
DEAD events.
A new sibling can consequently be linked to the detached leader. When
the leader is closed, perf_group_detach() observes that its
PERF_ATTACH_GROUP bit is already clear and skips the new sibling. The
sibling then retains a group_leader pointer to the freed event.
Reject group leaders in the EXIT state. Perform the check while holding
the shared context mutex so that an exec in the target task cannot detach
the leader between validation and group attachment.
[peterz: make the earlier test fully consistent]
Fixes: 037a3c43ed ("perf/core: Detach event groups during remove_on_exec")
Assisted-by: Codex:gpt-5.6-sol
Signed-off-by: Kyle Zeng <kylebot@openai.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/20260806205655.75722-1-kylebot@openai.com
queued_spin_unlock() gates its contended_release trace call behind a
static branch, so a NOP sits on the unlock path even while the
tracepoint is disabled. Removing that requires replacing the unlock
implementation only while contended_release is enabled, which needs a
callback when the tracepoint is toggled.
Convert contended_release to TRACE_EVENT_FN() and add weak no-op
arch_contended_release_trace_reg()/arch_contended_release_trace_unreg()
hooks.
The default hooks are empty, so this is a no-op until an architecture
overrides them.
No functional change intended.
Signed-off-by: Dmitry Ilvokhin <d@ilvokhin.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Acked-by: Juergen Gross <jgross@suse.com>
Link: https://patch.msgid.link/1c2fcccfb584c075c02890c484f22c76a1948bf1.1785778551.git.d@ilvokhin.com
Unlike mutex and rw_semaphore, qspinlock has no owner field, so "perf
lock contention --lock-owner" cannot attribute a contended spinlock to
its holder. The waiter-side contention_begin event records that a
spinlock is contended, but not by whom. Firing contended_release in the
holder's context at unlock is the only way to capture the holder of a
contended spinlock.
Combine the contention check, trace call and release in an out-of-line
queued_spin_release_traced() so the compiler need not preserve the lock
pointer in a callee-saved register across the call.
The check in queued_spin_unlock() is paid on every unlock, even while
the tracepoint is disabled: a static-branch NOP on x86_64, and a few
more instructions to manage a stack frame elsewhere. Gate it behind
CONFIG_QUEUED_SPINLOCKS_TRACE_CONTENDED_RELEASE (default n) so nobody
pays for a tracepoint they do not use. Sleeping locks fire
contended_release regardless.
On x86 this generic path is used only with PARAVIRT_SPINLOCKS=n (e.g.
defconfig). PARAVIRT_SPINLOCKS=y kernels keep the paravirt static_call
unlock and are wired up separately.
All below are with the QUEUED_SPINLOCKS_TRACE_CONTENDED_RELEASE option
enabled.
_raw_spin_unlock(), x86_64 defconfig, GCC 11, tracepoint compiled in but
disabled. The unlock is the single 'movb'. The only instruction added to
the executed path is the 2-byte static-branch NOP. The CALL to the
traced helper and the JMP back are emitted out of line and are reached
only once the static branch is patched on:
endbr64 ; 4 bytes
xchg %ax,%ax ; 2 static-branch NOP
; (added)
movb $0x0,(%rdi) ; 3 unlock (single store)
A: decl %gs:__preempt_count ; 7
je B ; 2
jmp __x86_return_thunk ; 5
call queued_spin_release_traced ; 5 out of line, reached
; only when the
; tracepoint is on
jmp A ; 2 (added)
B: call __SCT__preempt_schedule ; 5
jmp __x86_return_thunk ; 5
Baseline is the same stream without the NOP and the out-of-line
CALL/JMP: 31 bytes vs 40 (+9 bytes).
Binary size impact on x86_64, defconfig: +680 bytes (+0.00%), since all
standard configs out-of-line unlock. Architectures with inlined unlock
(s390 (always), csky and loongarch (both when !PREEMPTION)) will see a
bigger increase in binary size.
On the same path (x86_64, PARAVIRT_SPINLOCKS=n) with the tracepoint
disabled, a _raw_spin_unlock()-heavy nginx workload [1] shows no
measurable difference between baseline and patched kernels in
throughput, latency, cycles, instructions, IPC, or L1 instruction-cache
misses (kernel and total): all deltas stay within run-to-run noise.
Unlike x86, on arm64 the frame setup code (STP, MOV and LDP) lands on
the executed path in addition to static-branch NOP. Binary size impact
on arm64, defconfig: +932 bytes (+0.00%).
The _raw_spin_unlock()-heavy nginx workload reflects the larger hot
path: L1 instruction-cache misses rise ~1.4% (kernel and total) and
instruction count ~0.4%, consistent with the per-unlock frame.
cpu_cycles, throughput and latency show no measurable change and are
within run-to-run noise.
Architectures with fully custom qspinlock implementations (e.g.
PowerPC) are not covered by this change.
[1]: https://lore.kernel.org/all/aiphFXe_TPNPxZ_n@shell.ilvokhin.com/
Signed-off-by: Dmitry Ilvokhin <d@ilvokhin.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Acked-by: Juergen Gross <jgross@suse.com>
Link: https://patch.msgid.link/0d998e22a0c595f670cfc6725bb683323aced5cb.1785778551.git.d@ilvokhin.com
A task performing a custom private hash resize can remain blocked in
uninterruptible sleep indefinitely. The hung-task detector reports:
INFO: task futex-resizer:314 blocked for more than 10 seconds.
task:futex-resizer state:D stack:14824 pid:314 tgid:312 ppid:311
Call Trace:
__schedule+0x521/0xf30
schedule+0x22/0xa0
futex_hash_allocate+0x3db/0x490
__do_sys_prctl+0x6f5/0xbd0
do_syscall_64+0xf9/0x530
entry_SYSCALL_64_after_hwframe+0x77/0x7f
Kernel panic - not syncing: hung_task: blocked tasks
futex_pivot_pending() allows the resize request to continue when
either no replacement hash is pending (hash_new == NULL) or the current
hash reference count has reached zero.
After the final-reference wake, another futex task can complete the
pivot between the two observations:
T1 T2
futex_hash_allocate()
wait_var_event(mm, ...)
futex_pivot_pending(mm)
hash_new != NULL
futex_hash()
futex_ref_get(old) -> false
futex_pivot_hash(mm)
hash_new = NULL
__futex_pivot_hash(mm, new)
rcu_assign_pointer(hash, new)
fph = rcu_dereference(hash) /* new */
futex_ref_is_dead(fph) -> false
schedule()
The pivot changes the state from hash_new != NULL with a dead current
hash to hash_new == NULL with a live current hash. Because
futex_pivot_pending() reads hash_new and hash without serialization,
the resize task can observe hash_new in the pre-pivot state and hash in
the post-pivot state, causing futex_pivot_pending() to return false even
though the pivot has completed. The task then goes to sleep after the
wakeup has already been consumed.
Serialize state reads in futex_pivot_pending() using futex_mm_phash::lock.
This guarantees that futex_pivot_pending() observes hash_new and hash
atomically, eliminating the race condition.
Fixes: bd54df5ea7 ("futex: Allow to resize the private local hash")
Suggested-by: Peter Zijlstra <peterz@infradead.org>
Signed-off-by: Yao Kai <yaokai34@huawei.com>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Cc: stable@vger.kernel.org
Link: https://patch.msgid.link/20260804125530.3933754-1-yaokai34@huawei.com
- Fix BPF verifier to preserve full pointer state for commuted
scalar += pointer arithmetic (Yiyang Chen, Eduard Zingerman)
- Fix a use-after-free of request sockets in the BPF TCP
iterator batching (Jose Fernandez)
- Fix a use-after-free of sk_redir in the BPF sockmap send
verdict path (Chengfeng Ye)
- Fix a netns reference imbalance in the BPF conntrack kfuncs
(Chengfeng Ye)
- Fix bpf_get_fsverity_digest() dynptr assumptions and silent
digest truncation (Eric Biggers)
- Fix bpf_tcp_{gen,check}_syncookie to check sk_state before
sk_protocol to make sure it is a full socket (Luxiao Xu)
- Fix rqspinlock to reset the tail when preserving the queue
on deadlock (Kumar Kartikeya Dwivedi)
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
-----BEGIN PGP SIGNATURE-----
iIsEABYKADMWIQTFp0I1jqZrAX+hPRXbK58LschIgwUCanXeCRUcZGFuaWVsQGlv
Z2VhcmJveC5uZXQACgkQ2yufC7HISIMfLQD9EWzi5MVBTcvg0XsHY1GZZBZUpfwo
VCrfPm9vHAVuqQ0A/0D9vWVRf1UEk9ccn+ebVKPuTuydGnDRR0Qovuca4gQF
=jZY8
-----END PGP SIGNATURE-----
Merge tag 'bpf-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf
Pull BPF fixes from Daniel Borkmann:
- Fix BPF verifier to preserve full pointer state for commuted
scalar += pointer arithmetic (Yiyang Chen, Eduard Zingerman)
- Fix a use-after-free of request sockets in the BPF TCP iterator
batching (Jose Fernandez)
- Fix a use-after-free of sk_redir in the BPF sockmap send verdict
path (Chengfeng Ye)
- Fix a netns reference imbalance in the BPF conntrack kfuncs
(Chengfeng Ye)
- Fix bpf_get_fsverity_digest() dynptr assumptions and silent
digest truncation (Eric Biggers)
- Fix bpf_tcp_{gen,check}_syncookie to check sk_state before
sk_protocol to make sure it is a full socket (Luxiao Xu)
- Fix rqspinlock to reset the tail when preserving the queue
on deadlock (Kumar Kartikeya Dwivedi)
* tag 'bpf-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf:
rqspinlock: Reset tail when preserving queue on deadlock
bpf: Check sk_state before sk_protocol in bpf_tcp_*_syncookie
fsverity: Fix silent truncation in bpf_get_fsverity_digest()
fsverity: Fix bpf_get_fsverity_digest() dynptr assumptions
bpf: tcp: Fix use-after-free in bpf_iter_tcp_established_batch()
bpf: Fix netns reference imbalance in conntrack kfuncs
bpf, sockmap: Fix sk_redir use-after-free in send verdict
selftests/bpf: Cover commuted pointer state propagation
bpf: Propagate untrusted pointer state in commuted arithmetic
bpf: Preserve pointer state for commuted arithmetic
bpf: Simplify sanitize_err() signature
A BPF_LOAD_ACQ is not rewritten to a BPF_PROBE_MEM load by the verifier,
unlike a regular BPF_LDX, so the JIT emits a plain load with no exception
table entry and a fault panics the kernel instead of being handled.
Reject the source pointer types that a BPF_LDX would have had that fault
protection applied to, i.e. the ones bpf_convert_ctx_accesses() turns
into BPF_PROBE_MEM: a bare PTR_TO_BTF_ID, PTR_TO_BTF_ID | PTR_UNTRUSTED,
PTR_TO_BTF_ID | MEM_ALLOC | PTR_UNTRUSTED and PTR_TO_MEM | MEM_RDONLY |
PTR_UNTRUSTED.
This is reachable e.g. by loading ->mm out of a trusted task_struct
yields an untrusted pointer to mm_struct, and it is NULL for a kernel
thread:
[...]
SEC("tp_btf/sched_switch")
int BPF_PROG(demo, bool preempt, struct task_struct *prev,
struct task_struct *next)
{
struct mm_struct *mm = next->mm; /* untrusted */
out_ldx = (__u64)mm->pgd; /* BPF_LDX */
out_acq = load_acquire(&mm->pgd); /* BPF_LOAD_ACQ */
return 0;
}
[...]
Both dereference the same pointer, but only the BPF_LDX is protected
(x86-64 JIT, jump targets shown prog-relative):
[...]
; out_ldx = (__u64)mm->pgd;
17: movq $-10485760, %r10
1e: movq %rsi, %r11
21: addq $184, %r11
28: subq %r10, %r11
2b: movabsq $140737498841088, %r10
35: cmpq %r10, %r11
38: ja 0x3e <-- kernel addr?
3a: xorl %edi, %edi <-- no: dst = 0, skip the load
3c: jmp 0x45
3e: movq 184(%rsi), %rdi <-- yes: load + extable entry
[...]
; load_acquire(&mm->pgd)
53: movq %rsi, %rdi
56: movq 184(%rdi), %rax <-- no check, no extable entry
[...]
Note that BPF_PROBE_MEM is not visible in a bpftool xlated dump, as
bpf_insn_prepare_dump() rewrites it back to BPF_MEM.
A PTR_TRUSTED pointer is deliberately not on the list. Such a load is
not converted either, but it does not need to be, since the pointer is
guaranteed live, so load-acquire from it stays allowed.
The check is gated on BPF_LOAD_ACQ so that atomic RMW and store-release
error messages are unchanged; writes (RMW / store-release) to such
pointers are already rejected elsewhere, so only load-acquire needs this.
Fixes: 880442305a ("bpf: Introduce load-acquire and store-release instructions")
Reported-by: STAR Labs SG <info@starlabs.sg>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Link: https://lore.kernel.org/bpf/20260806201047.333389-1-daniel@iogearbox.net
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
These comments describing the page allocator are out of date:
- __alloc_pages() is no longer a public API and has no business being
described outside of mm/.
- The `wait` variable is gone.
It may be out of date for other reasons too but this patch is just
fixing the issues that stood out.
To fix it:
- Instead of referring to a specific function, instead to "the page
allocator"
- Completely drop out-of-date details of that function's internal
behaviour, since they were irrelevant anyway.
Link: https://lore.kernel.org/20260715-spin-trylock-followup-v3-2-fc4d246f705d@google.com
Signed-off-by: Brendan Jackman <jackmanb@google.com>
Suggested-by: Zi Yan <ziy@nvidia.com>
Link: https://lore.kernel.org/all/DJP11T5V7BDW.2FZZZ8R6LOY4I@nvidia.com/
Reviewed-by: Zi Yan <ziy@nvidia.com>
Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Acked-by: Tejun Heo <tj@kernel.org>
Cc: David Hildenbrand <david@kernel.org>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Michal Koutný <mkoutny@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Waiman Long <longman@redhat.com>
Cc: Brendan Jackman <brendan.jackman@linux.dev>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Currently, the destruction of the waiter queue is suppressed for
rqspinlock in cases where a deadlock is detected. Deadlock checks happen
relatively frequently (on entry for AA, within 1ms for ABBA), and waiter
threads may not be involved in locking scenarios involving deadlocks.
Thus, it is useful to not flush the queue and let other waiters take a
stab at acquiring the lock after we detect a deadlock and exit.
However, we need to follow the same logic as what we did previously for
the waitq_timeout label: reset the tail, and if we cannot, signal the
next waiter appropriately. In case of deadlocks, this signal would just
mark the MCS node as unlocked, and in case of timeouts, it would signal
RES_TIMEOUT_VAL. The difference thus is in the value propagated, which
decides whether the queue remains active or gets flushed.
Not doing the tail reset, and waiting for the next waiter can lead to
cases where we are the final waiter, and thus no next waiter arrives,
leading to intermittent stalls in this path. Once the next waiter does
join, we will be unblocked. In the theoretical case when the next waiter
never joins, we risk stalling indefinitely.
This can only happen for ABBA deadlocks, since entry into the wait queue
is guarded with AA checks. A precise sequence of executions leading up
to this scenario can be:
CPU 0 holds lock A.
CPU 1 holds lock B.
CPU 2 attempts lock B, becomes the pending waiter for B.
CPU 0 attempts lock B. B has locked+pending bits set, thus CPU 0 queues.
CPU 1 attempts lock A.
CPU 0 detects an ABBA deadlock.
Once deadlock detection happens for CPU 0, it will sit waiting for the
next waiter in the queue to populate node->next, which will experience
delays until such a waiter arrives.
Fix this by adjusting the logic for the check for deadlocks preceding
the waitq_timeout label. It would make sense to consolidate code for
both cases and use 'ret' to distinguish the value being propagated, but
that is left as an exercise for a future refactoring task to avoid diff
noise in this patch.
Fixes: 7bd6e5ce5b ("rqspinlock: Disable queue destruction for deadlocks")
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Link: https://patch.msgid.link/20260802021759.1139457-1-memxor@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Add an optional reclaim callback to struct dmem_cgroup_region. When
dmem.max is set below the current usage of a cgroup pool, the new limit
is applied immediately (so that concurrent allocations are throttled
while reclaim is in progress) and then the driver is asked to evict
memory to bring usage back below the limit.
Reclaim is attempted up to a bounded number of times. No error is
returned to userspace if usage remains above the limit after reclaim,
and a pending signal will abort the reclaim loop early. This matches
the behavior of memory.max in the memory cgroup controller.
Also honor O_NONBLOCK so that if that flag is set during the
max value write, no reclaim is initiated. The idea is to avoid
charging the reclaim cost to the writer of the max value.
v2:
- Write max before reclaim is attempted (Maarten)
- Let signals abort the reclaim without error (Maarten)
- If a new max value is written with the O_NONBLOCK flag,
reclaim is not attempted (Maarten)
- Extract region from the pool parameter rather than
passing it explicitly to set_resource_xxx().
v3:
- Use an rw_semaphore (unregister_sem) to protect reclaim callbacks
against concurrent region unregistration: readers (reclaim) hold the
read side; dmem_cgroup_unregister_region() takes the write side to
drain in-flight callbacks before returning. (Sashiko-bot)
v5:
- Rebased on the introduction of struct dmem_cgroup_init.
- Use nonblock=true in reset_all_resource_limits() to avoid sleeping
inside rcu_read_lock() in dmemcs_offline(). (Sashiko-bot)
- Compare usage against the truncated limit value stored in cnt.max,
not the original u64. (Sashiko-bot)
- Use a DMEM_MAX_RECLAIM_RETRIES (16) retry budget instead of 5, matching
the memcg controller's MAX_RECLAIM_RETRIES. Only -ENOSPC (no progress)
counts against the retry budget; other errors terminate the loop
immediately.
v6:
- Fix dmem_cgroup_ops->reclaim docstring: -ENOSPC does not stop reclaim
immediately but is retried up to DMEM_MAX_RECLAIM_RETRIES times; only
other negative errors terminate the loop. (Sashiko-bot)
v7:
- Replace the per-region rw_semaphore with a static SRCU domain
(dmemcg_srcu). SRCU is a better fit than rwsem for this use: it
avoids the per-region lock overhead on every reclaim call, and
synchronize_srcu() at unregister time is a rare operation. (Maarten)
- Trim in-function comments to focus on what rather than how.
Assisted-by: GitHub_Copilot:claude-sonnet-4.6
Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Reviewed-by: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
Tested-by: Thadeu Lima de Souza Cascardo <cascardo@igalia.com>
Link: https://patch.msgid.link/20260725100036.2372-4-thomas.hellstrom@linux.intel.com
Signed-off-by: Maarten Lankhorst <dev@lankhorst.se>
Replace the bare u64 size argument to dmem_cgroup_register_region() and
drmm_cgroup_register_region() with a const struct dmem_cgroup_init *
pointer. The struct currently carries only the size field, but using a
struct makes the API extensible: future callers can supply additional
initialization parameters without adding more positional arguments.
Update all in-tree callers (amdgpu, xe) to use a compound-literal
initializer.
v5:
- Commit introduced.
Assisted-by: GitHub_Copilot:claude-sonnet-4.6
Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Reviewed-by: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
Tested-by: Thadeu Lima de Souza Cascardo <cascardo@igalia.com>
Link: https://patch.msgid.link/20260725100036.2372-3-thomas.hellstrom@linux.intel.com
Acked-by: Dave Airlie <airlied@redhat.com>
Acked-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Maarten Lankhorst <dev@lankhorst.se>
param_set_charp() stores charp parameters in allocated memory after slab is
available, and releases the previous value when the parameter is updated.
The previous value is released before the replacement allocation succeeds.
If kmalloc_parameter() fails, the setter returns -ENOMEM with the parameter
left as NULL.
Failing zswap's compressor update before zswap is initialized can later
trigger:
BUG: kernel NULL pointer dereference, address: 0000000000000000
RIP: 0010:strcmp+0x10/0x30
Call Trace:
zswap_setup+0x3b1/0x490
zswap_enabled_param_set+0x5b/0xa0
param_attr_store+0x93/0xe0
module_attr_store+0x1c/0x30
kernfs_fop_write_iter+0x116/0x1f0
Allocate and copy the replacement first, then replace the parameter value
only after allocation succeeds.
Fixes: e180a6b775 ("param: fix charp parameters set via sysfs")
Cc: stable@vger.kernel.org
Signed-off-by: Jiacheng Yu <yujiacheng3@huawei.com>
Reviewed-by: Petr Pavlu <petr.pavlu@suse.com>
Signed-off-by: Petr Pavlu <petr.pavlu@suse.com>
In elf_validity_cache_sechdrs, section sizes and offsets are validated,
unless the section type is SHT_NULL or SHT_NOBITS.
Later, elf_validity_cache_secstrings and elf_validity_cache_index_str
access the section name table (.shstrtab) and symbol string table
(.strtab) headers without first ensuring that their types are
SHT_STRTAB. If a section type is SHT_NULL or SHT_NOBITS, sh_offset has
not been validated and may reference out-of-bounds memory when
dereferenced in elf_validity_cache_secstrings or
elf_validity_cache_strtab.
Validate that both string section headers are of type SHT_STRTAB before
caching them.
Cc: stable@vger.kernel.org
Signed-off-by: Thiébaud Weksteen <tweek@google.com>
Reviewed-by: Aaron Tomlin <atomlin@atomlin.com>
Reviewed-by: Petr Pavlu <petr.pavlu@suse.com>
Signed-off-by: Petr Pavlu <petr.pavlu@suse.com>
Disabling preemption or local IRQs keeps the current CPU in an RCU
read-side critical section, but in_rcu_cs() does not account for either
state. The verifier therefore rejects safe kptr accesses and invalidates
pointers when another RCU source ends.
Include preemption-disabled and IRQ-disabled state in in_rcu_cs().
Invalidate RCU-protected pointers on RCU unlock, preempt enable, or IRQ
restore only after the final protection ends.
Signed-off-by: Ning Ding <dingning04@gmail.com>
Link: https://lore.kernel.org/bpf/20260805233940.3966981-2-dingning04@gmail.com
[ kkd: Simplify was_in_rcu_cs on spin unlock and adjust the selftest. ]
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Use strcmp() instead of strlen()+memcmp() to compare module names in
kmod_dup_request_lookup(), since all strings are NUL-terminated.
Reviewed-by: Aaron Tomlin <atomlin@atomlin.com>
Signed-off-by: Petr Pavlu <petr.pavlu@suse.com>
Use scope-based cleanup helpers for kmod_dup_mutex and kmod_req to shorten
the code and to clarify where the lock is taken in
kmod_dup_request_exists_wait().
Signed-off-by: Petr Pavlu <petr.pavlu@suse.com>
The kmod dups code preallocates kmod_dup_req before taking kmod_dup_mutex
to avoid allocating memory while holding the lock. This provides little
benefit, since the allocation is fast and can safely be done under the
lock. On the other hand, it leads to unnecessary allocations when the
request turns out to be a duplicate and slightly complicates the code.
Allocate kmod_dup_req only when needed and introduce a helper function
alloc_kmod_req() to initialize the structure.
Reviewed-by: Aaron Tomlin <atomlin@atomlin.com>
Signed-off-by: Petr Pavlu <petr.pavlu@suse.com>
The kmod dups code uses RCU to ensure that a kmod_dup_req instance is freed
only after it is no longer referenced. When releasing an instance, the
kmod_dup_request_delete() function removes the kmod_dup_req from the
dup_kmod_reqs list, waits via synchronize_rcu() and finally frees it.
However, this doesn't work correctly because parallel users referencing the
instance in kmod_dup_request_exists_wait() don't enter an RCU read-side
critical section. This can result in a use-after-free.
The kmod_dup_request_exists_wait() function may need to hold a valid
reference to a kmod_dup_req instance across a blocking wait until the
corresponding modprobe command completes. This makes it unsuitable for RCU.
Fix the issue by changing the lifecycle management of kmod_dup_req to use
reference counting.
Fixes: 8660484ed1 ("module: add debugging auto-load duplicate module support")
Reviewed-by: Aaron Tomlin <atomlin@atomlin.com>
Signed-off-by: Petr Pavlu <petr.pavlu@suse.com>
When kmod_dup_request_announce() announces the completion of
a request_module() call to duplicate waiters, it queues a work item to
invoke kmod_dup_request_complete(), and only that function calls
complete_all().
This adds an arbitrary delay that is unnecessary and provides little
benefit. Call complete_all() directly from kmod_dup_request_announce()
instead.
Signed-off-by: Petr Pavlu <petr.pavlu@suse.com>
Historically, various parameter-handling code kept pointers into
module::args, most notably the charp support. However, in 2009,
commit e180a6b775 ("param: fix charp parameters set via sysfs") changed
charp parameters to kstrdup() the input string as well. As a result,
module::args now mostly wastes memory.
The last users that still pointed into module::args have now been cleaned
up, so remove this data.
Reviewed-by: Aaron Tomlin <atomlin@atomlin.com>
Signed-off-by: Petr Pavlu <petr.pavlu@suse.com>
module_total_size() returns unsigned int but uses a signed int
accumulator. While the result is numerically correct, the type
mismatch is misleading.
Change the accumulator to unsigned int to match the return type.
Signed-off-by: Naveen Kumar Chaudhary <naveen.osdev@gmail.com>
Reviewed-by: Sami Tolvanen <samitolvanen@google.com>
[ppavlu: correct the commit title]
Signed-off-by: Petr Pavlu <petr.pavlu@suse.com>
Both try_add_failed_module() and kmod_dup_request_exists_wait() use
memcpy() with strlen() to copy module names into fixed-size
char[MODULE_NAME_LEN] buffers. Neither performs a bounds check on the
copy. Current callers always pass names originating from
mod->name (itself char[MODULE_NAME_LEN]), so this is not exploitable
today. However both functions accept a plain const char * with no
documented length contract, making them latent buffer overflows if a
future caller passes a longer string.
Replace memcpy() with strscpy() in both sites, which bounds the copy
to MODULE_NAME_LEN and always NUL-terminates.
Signed-off-by: Naveen Kumar Chaudhary <naveen.osdev@gmail.com>
Reviewed-by: Petr Pavlu <petr.pavlu@suse.com>
Signed-off-by: Petr Pavlu <petr.pavlu@suse.com>
The comment wrongly references to /sys/modules/XYZ/parameters/ directory
instead of /sys/module/XYZ/parameters/. Fix it.
Signed-off-by: Zenghui Yu <zenghui.yu@linux.dev>
Reviewed-by: Aaron Tomlin <atomlin@atomlin.com>
Acked-by: Randy Dunlap <rdunlap@infradead.org>
Signed-off-by: Petr Pavlu <petr.pavlu@suse.com>
The symbol lookup code assumes the queried address resides in either
MOD_TEXT or MOD_INIT_TEXT. This breaks for addresses in other module
memory regions (e.g. rodata or data), resulting in incorrect upper
bounds and wrong symbol size.
Select the module memory region the address belongs to instead of
hardcoding text sections. Also initialize the lower bound to the start
of that region, as searching from address 0 is unnecessary.
Cc: stable@vger.kernel.org
Signed-off-by: Stanislaw Gruszka <stf_xl@wp.pl>
Reviewed-by: Petr Pavlu <petr.pavlu@suse.com>
Signed-off-by: Petr Pavlu <petr.pavlu@suse.com>
The verifier currently keeps a hard-coded list of kfuncs that may be
called while holding a bpf_spin_lock. With KF_SPINLOCK_SAFE available,
retaining this list creates two sources of truth and requires verifier
changes whenever another lock-safe kfunc is added.
Mark every kfunc currently accepted by kfunc_spin_allowed() with
KF_SPINLOCK_SAFE. This covers the graph, numeric iterator, resource
spin lock, arena, and stream kfuncs.
Remove the obsolete category checks and make kfunc_spin_allowed() rely
solely on the kfunc registration metadata. This preserves the behavior
of existing kfuncs while using the same mechanism for built-in and
module kfuncs.
Signed-off-by: Kaitao Cheng <chengkaitao@kylinos.cn>
Acked-by: Leon Hwang <leon.hwang@linux.dev>
Link: https://lore.kernel.org/bpf/20260805153340.34776-3-kaitao.cheng@linux.dev
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Introduce the KF_SPINLOCK_SAFE kfunc metadata flag in BTF so kfuncs may
be explicitly marked as safe to call while holding bpf_spin_lock.
Allow kfuncs defined in kernel modules to be marked with KF_SPINLOCK_SAFE.
Example: BTF_ID_FLAGS(func, $kfunc_name, KF_SPINLOCK_SAFE)
Signed-off-by: Kaitao Cheng <chengkaitao@kylinos.cn>
Acked-by: Leon Hwang <leon.hwang@linux.dev>
Link: https://lore.kernel.org/bpf/20260805153340.34776-2-kaitao.cheng@linux.dev
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
DECLARE_EVENT_CLASS emitted __bpf_trace_btf_ids_<call> through
BTF_ID_LIST_GLOBAL, i.e. a global symbol named after the event class.
The class name is not unique across the kernel, so the symbol
multiply-defines whenever two translation units instantiate the same
class.
Switch to the file-local BTF_ID_LIST: the list is reached
only through the event_class_<call>.btf_ids pointer, initialised in the
same unit, so tracefs readers never reference the symbol by name and
resolve_btfids still fills the now-local .BTF_ids entries.
The handcrafted syscall classes are the one cross-unit consumer: give
them their own local BTF_ID_LIST rather than importing the generated
sys_{enter,exit} lists.
Link: https://patch.msgid.link/20260730-b4-fix_btf_tracefs-v2-1-6b66da8dc103@meta.com
Fixes: eadc0725ab ("tracing: Expose tracepoint BTF ids via tracefs")
Reported-by: Mark Brown <broonie@kernel.org>
Closes: https://lore.kernel.org/all/ff58b01c-3f5e-4d55-be82-609d2faaf12e@sirena.org.uk/
Signed-off-by: Mykyta Yatsenko <yatsenko@meta.com>
Acked-by: Andrii Nakryiko <andrii@kernel.org>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
bloom_map_alloc() has two 32-bit-specific problems when the computed
bitmap reaches the U32_MAX fallback case.
First, BITS_TO_BYTES(U32_MAX) is evaluated with 32-bit arithmetic. The
addition performed by DIV_ROUND_UP wraps, so the map allocates only the
fixed-size bloom filter object while keeping bitset_mask == U32_MAX.
Subsequent updates can then write past the allocated object.
Second, fixing only the allocation size is not sufficient. The bloom hash
is a u32, but set_bit() takes a signed long bit number and x86 test_bit()
eventually feeds the index to variable_test_bit(long, ...). On 32-bit
kernels, hashes in [0x80000000, U32_MAX] therefore become negative bit
offsets. x86 bt/bts with a memory operand interpret those offsets relative
to the supplied base, so a map with bitset_mask == U32_MAX can read or
write before bloom->bitset even after allocating the full 512 MiB bitmap.
Keep the U32_MAX fallback, but split each hash into a word pointer and an
in-word bit number before calling test_bit() or set_bit(). The bitops
argument is then always in [0, BITS_PER_LONG - 1], while BIT_WORD(h) still
selects the intended word in the full bitmap.
Compute the bitset size from (u64)bitset_mask + 1 before passing the final
size to bpf_map_area_alloc(). This fixes the original under-allocation and
keeps the allocated storage consistent with the addressable bitset.
Exploitation note: local privilege escalation is possible on a 32-bit x86
kernel using the under-allocation bug from a binary with CAP_BPF.
Fixes: 9330986c03 ("bpf: Add bloom filter map implementation")
Signed-off-by: Jérémy Jean <Jeremy.Jean@oss.cyber.gouv.fr>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Cc: stable@vger.kernel.org
Link: https://lore.kernel.org/bpf/20260805060228.2703051-1-Jeremy.Jean@oss.cyber.gouv.fr
Assisted-by: Codex:gpt-5
When CONFIG_FUNCTION_ERROR_INJECTION is disabled, a sleepable tracing prog
is allowed to attach to '__x64_'-alike prefix symbols.
It is because the verifier does not verify whether the symbol is a kernel
function or a bpf prog. That said, a sleepable tracing prog is allowed to
attach to a bpf prog target whose name has '__x64_'-alike prefix.
For example, a sleepable fentry prog attaches to a '__x64_sys_nop' XDP
prog, and copies buffer from a user pointer with bpf_copy_from_user()
helper. After attaching the XDP prog to lo interface, the kernel BUG
could be triggered by 'ping -c 1 -W 1 127.0.0.1':
[ 3.460756] BUG: sleeping function called from invalid context at kernel/bpf/trampoline.c:1324
Fix it by disallowing sleepable prog always when its target
btf is not a kernel's btf.
Fixes: 16d9c56606 ("bpf: Always allow sleepable programs on syscalls")
Signed-off-by: Leon Hwang <leon.hwang@linux.dev>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Acked-by: Viktor Malik <vmalik@redhat.com>
Link: https://lore.kernel.org/bpf/20260805150810.34907-2-leon.hwang@linux.dev
There's no need to modify the trace object bpf_get_stackid_pe, we just
need to pass the needed callchain length in separate argument.
This way we can have callchain pointers const and remove the trace->nr
modification and restoration.
Assisted-by: Codex:GPT-5.5
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Link: https://lore.kernel.org/bpf/20260803210149.296496-13-jolsa@kernel.org
There's no need to modify the trace object bpf_get_stack_pe, we just
need to pass the needed callchain length in separate argument.
This way we can have callchain pointers const and remove the trace->nr
modification and restoration.
Assisted-by: Codex:GPT-5.5
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Link: https://lore.kernel.org/bpf/20260803210149.296496-12-jolsa@kernel.org
get_perf_callchain() returns a per-CPU perf_callchain_entry buffer and
releases its recursion slot via put_callchain_entry() before returning,
so nothing keeps the entry reserved while __bpf_get_stack() consumes
it below.
A preemptible BPF program (e.g. a non-sleepable raw tracepoint program
on a PREEMPT kernel, which runs under migrate_disable() but not
preempt_disable()) can be scheduled out between obtaining the entry
and the copy. Another task scheduled on the same CPU then reuses the
same per-CPU buffer and overwrites trace->nr with a larger value.
copy_len is then computed from the inflated trace->nr and can exceed
the caller's buffer, causing an out-of-bounds write in the memcpy()
and in the build_id path.
The rcu_read_lock() taken here alone does not prevent this. It is
only taken on the may_fault path, and under CONFIG_PREEMPT_RCU it does
not disable preemption; it merely keeps perf's callchain buffer array
alive (freed via call_rcu()) and does nothing to stop another task
from reusing the entry.
Disable preemption around obtaining the callchain entry and copying
it into the caller's buffer, so the entry cannot be reused underneath
us and trace->nr stays bounded by max_depth. Build ID resolution may
fault and is therefore deferred until after preemption is re-enabled;
by then the instruction pointers have already been copied into buf,
so it operates only on that private copy. Note, preempt_disable() also
subsumes the buffer-lifetime guarantee the rcu_read_lock() provided,
since a preempt-disabled section is an RCU read-side critical section
for the callchain buffers' call_rcu() reclaim.
Fixes: c195651e56 ("bpf: add bpf_get_stack helper")
Reported-by: Tao Chen <chen.dylane@linux.dev>
Reported-by: STAR Labs SG <info@starlabs.sg>
Signed-off-by: Daniel Borkmann <borkmann@iogearbox.net>
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Cc: stable@vger.kernel.org
Link: https://lore.kernel.org/bpf/20260803210149.296496-11-jolsa@kernel.org
Closes: https://lore.kernel.org/bpf/20260206090653.1336687-1-chen.dylane@linux.dev/
[ changed Fixes: commit ]
Both bpf_get_task_stack and bpf_get_task_stack_sleepable helpers that
use __bpf_get_task_stack have buf defined as ARG_PTR_TO_UNINIT_MEM
argument and we should initialize the buf on every return path.
Adding missing buf memset for __bpf_get_task_stack fail paths. This
provides deterministic buffer contents, which is useful when the buffer
is used directly as a map key.
Fixes: 06ab134ce8 ("bpf: Refcount task stack in bpf_get_task_stack")
Fixes: b992f01e66 ("bpf: Guard against accessing NULL pt_regs in bpf_get_task_stack()")
Reported-by: Sashiko <sashiko-bot@kernel.org>
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Link: https://lore.kernel.org/bpf/20260803210149.296496-10-jolsa@kernel.org
Now with the new callchain_* helper functions we can process trace_in
case directly in bpf_get_stack_pe function and remove it from
__bpf_get_stack which makes things easier for preemption fix in
following change.
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Link: https://lore.kernel.org/bpf/20260803210149.296496-9-jolsa@kernel.org
The new callchain_finalize function calls the build-id retrieval
(if needed) and zeroes the buffer. This makes things easier for
preemption fix in following change.
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Link: https://lore.kernel.org/bpf/20260803210149.296496-8-jolsa@kernel.org
The new callchain_store function stores trace entries buffer into
user supplied buffer. It covers both just-ip and buildid data.
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Link: https://lore.kernel.org/bpf/20260803210149.296496-7-jolsa@kernel.org
Replacing __bpf_get_stackid calls with sequence of following functions:
stackid_fastpath
stackid_new_bucket
stackid_install
This makes code more structured and allows us to easily disable
preemption only in bpf_get_stackid in following changes.
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Link: https://lore.kernel.org/bpf/20260803210149.296496-5-jolsa@kernel.org
The new stackid_fastpath does the fast stack hash and trace check, that
does not need new bucket allocation. It covers both just-ip and buildid
code paths.
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Link: https://lore.kernel.org/bpf/20260803210149.296496-3-jolsa@kernel.org
The new stackid_init function stores all the necessary bits for stackid
trace and it will be used by other functions in following changes.
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Link: https://lore.kernel.org/bpf/20260803210149.296496-2-jolsa@kernel.org
syzbot is reporting KCOV state corruption on PREEMPT_RT kernels, for the
temporary storage used for saving/restoring remote KCOV state is currently
allocated as the per-CPU area.
On PREEMPT_RT kernels, softirq handlers run as preemptible task threads
(e.g., ksoftirqd). If a softirq context preempts a task running a remote
KCOV session, it safely saves the task's state into the per-CPU area.
However, if that softirq thread is subsequently preempted by a higher-
priority softirq thread on the same CPU, the second softirq will overwrite
the same per-CPU area, permanently destroying the original task's KCOV
state.
Fix this data corruption by moving the temporary storage from the per-CPU
area to the per-thread area. Since each softirq thread now owns its own
task context, nested softirq preemption no longer causes data overwrites.
Note that while the temporary storage is now on a per-thread basis, the
per-CPU kcov_percpu_data.lock must be retained, for we need to ensure that
kcov_remote_start() and kcov_remote_stop() operate atomically without
racing against asynchronous interrupts that manipulate the current task's
KCOV state.
It is likely that GFP_KERNEL allocation by vmalloc_node() in kcov_init()
has already called panic() before returning NULL, for there will be no
OOM-killable userspace processes when __init function of built-in module
runs. But this patch also fixes crashing the kernel when vmalloc_node()
in kcov_init() returned NULL, for kcov_init() left per-CPU irq_area == NULL
but kcov_remote_start() depends on per-CPU irq_area != NULL, resulting in
(1) doing vmalloc() in kcov_remote_start() despite !in_task() context
(2) out-of-array-bounds access if (1) succeeded but
kcov->remote_size < CONFIG_KCOV_IRQ_AREA_SIZE
(3) always leak memory allocated by (1), eventually killing all
OOM-killable userspace processes
problems.
Link: https://lore.kernel.org/43552d09-2ce2-4b19-b0d3-a2d1ab952145@I-love.SAKURA.ne.jp
Reported-by: syzbot+3f51ad7ac3ae57a6fdcc@syzkaller.appspotmail.com
Closes: https://syzkaller.appspot.com/bug?extid=3f51ad7ac3ae57a6fdcc
Reported-by: syzbot+47cf95ca1f9dcca872c8@syzkaller.appspotmail.com
Closes: https://syzkaller.appspot.com/bug?extid=47cf95ca1f9dcca872c8
Reported-by: syzbot+8a173e13208949931dc7@syzkaller.appspotmail.com
Closes: https://syzkaller.appspot.com/bug?extid=8a173e13208949931dc7
Reported-by: syzbot+90984d3713722683112e@syzkaller.appspotmail.com
Closes: https://syzkaller.appspot.com/bug?extid=90984d3713722683112e
Analyzed-by: AI Mode in Google Search (no mail address)
Fixes: 5ff3b30ab5 ("kcov: collect coverage from interrupts")
Signed-off-by: Tetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp>
Reviewed-by: Alexander Potapenko <glider@google.com>
Cc: Alan Stern <stern@rowland.harvard.edu>
Cc: Andrey Konovalov <andreyknvl@gmail.com>
Cc: Christoph Hellwig <hch@infradead.org>
Cc: Clark Williams <williams@redhat.com>
Cc: Dmitry Vyukov <dvyukov@google.com>
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Cc: Marco Elver <elver@google.com>
Cc: Mark Brown <broonie@kernel.org>
Cc: Roman Gushchin <roman.gushchin@linux.dev>
Cc: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Cc: <stable@vger.kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
parse() hands nla_strscpy() len as dstsize, and nla_strscpy() copies at
most dstsize - 1 bytes. When the attr payload comes in without a trailing
NUL, srclen == len >= dstsize and the last character of the cpumask string
gets cut off. Register "0-15" and you are silently listening on "0-1",
exit data for the rest never shows up.
The bug only bites when the sender doesn't NUL terminate the payload;
senders that include the NUL were always fine (srclen gets decremented for
the trailing NUL, so srclen < dstsize). Thats probably why this survived
20 years. And the policy is NLA_STRING, not NLA_NUL_STRING, so a payload
without the trailing NUL is legit input here.
Skip the kmalloc/nla_strscpy dance entirely and use nla_strdup(), which
already allocates srclen + 1 and terminates. The nla_len() bounds checks
stay as they were.
Link: https://lore.kernel.org/EC49FE41-7F5F-41E0-A07A-ABEB8ECA514D@grrlz.net
Fixes: f9fd8914c1 ("[PATCH] per-task delay accounting taskstats interface: control exit data through cpumasks")
Signed-off-by: Bradley Morgan <include@grrlz.net>
Reported-by: Oleg Deomi <oleg.deomi@gmail.com>
Closes: https://lore.kernel.org/CAByWkfZ6b1=3H9pwkz-dDQOs9cZaF-HYQ6b9Yb0=Hq2r1Vv_Pw@mail.gmail.com
Reviewed-by: Andrew Morton <akpm@linux-foundation.org>
Cc: Balbir Singh <bsingharora@gmail.com>
Cc: <stable@vger.kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Once destroy() returns the stack slot is no longer tracked as iterator
state, so zeroing it is dead work. Make the kfunc a no-op and inline the
call to a single BPF_JA 0 (the fixup can't drop the instruction outright,
so emit a nop; the JITs elide it).
Suggested-by: Andrii Nakryiko <andrii@kernel.org>
Signed-off-by: Puranjay Mohan <puranjay@kernel.org>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Link: https://lore.kernel.org/bpf/20260804134601.2305303-5-puranjay@kernel.org
bpf_iter_num_next() runs on every bpf_for() iteration, so inlining it
drops a call from the loop body. R1 points to the iterator; the returned
pointer to s->cur is R1 itself, since s->cur is first.
s->cur and s->end are int, so the kfunc's s->cur + 1 >= s->end is a
signed 32-bit compare and the inlined code needs no sign extension.
Signed-off-by: Puranjay Mohan <puranjay@kernel.org>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Link: https://lore.kernel.org/bpf/20260804134601.2305303-4-puranjay@kernel.org
bpf_for() expands to the bpf_iter_num_{new,next,destroy}() kfuncs, which
the verifier emits as regular calls. They are tiny and only touch the
8-byte on-stack iterator state, so open-code them in bpf_fixup_kfunc_call()
like the other special kfuncs there.
Start with bpf_iter_num_new(): R1 points to the iterator, R2/R3 hold
start/end. The inlined sequence mirrors the kfunc and returns the same
-EINVAL / -E2BIG / 0.
start > end is rejected first, so end - start fits in a u32; range-check
it as u32 on both sides ((u32)(end - start) in the kfunc). A movsx-based
check would emit a cpuv4 instruction that some JITs (x86-32, mips32,
sparc64) decode as a plain move and get wrong.
The emitted instructions are plain BPF, so the interpreter path stays
correct and no jit_required marking is needed.
Signed-off-by: Puranjay Mohan <puranjay@kernel.org>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Link: https://lore.kernel.org/bpf/20260804134601.2305303-3-puranjay@kernel.org
The comment on the s->cur + 1 >= s->end check claims the (s64) cast is
needed to avoid overflow when s->cur == s->end == INT_MAX. It isn't:
s->cur + 1 is computed in int and wraps before the cast, so the cast
changes nothing (INT_MAX + 1 compares the same either way).
The wraparound is the point. bpf_iter_num_new() sets s->cur = start - 1,
which wraps to INT_MAX for start == INT_MIN, and the wrapping s->cur + 1
brings it back to start. (s64)s->cur + 1 would instead break iterators
starting at INT_MIN.
Drop the cast and reword the comment. No functional change; the wrap is
well-defined under -fno-strict-overflow.
Signed-off-by: Puranjay Mohan <puranjay@kernel.org>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Link: https://lore.kernel.org/bpf/20260804134601.2305303-2-puranjay@kernel.org
CONFIG_SYSCTL used to make sense as a separate hidden bool before commit
61a47c1ad3 ("sysctl: Remove the sysctl system call"); it was selected
by both CONFIG_SYSCTL_SYSCALL and CONFIG_PROC_SYSCTL.
Today CONFIG_PROC_SYSCTL is the only selector, so the two are always
equal. Kill the hidden bool, rename the PROC_SYSCTL prompt to SYSCTL,
and s/CONFIG_PROC_SYSCTL/CONFIG_SYSCTL/ tree-wide.
Signed-off-by: Oleg Nesterov <oleg@redhat.com>
Signed-off-by: Joel Granados <joel.granados@kernel.org>
cad_pid is global, and kill_cad_pid() is only used in the root namespace.
However, due to pid_table_root_permissions(), a non-root user can unshare
pid/user namespaces and modify it from the child namespace. This makes no
sense and is simply wrong.
Move it to kern_reboot_table[] where it logically belongs; this ensures
that only GLOBAL_ROOT_UID can read/modify this sysctl.
Note that this patch doesn't preserve "#ifdef CONFIG_PROC_SYSCTL" around
the "cad_pid"; CONFIG_PROC_SYSCTL selects CONFIG_SYSCTL, so it is always
set when kern_reboot_table[] is compiled.
Cc: stable@vger.kernel.org
Fixes: e054bcbe7e ("sysctl: move cad_pid into kernel/pid.c")
Signed-off-by: Oleg Nesterov <oleg@redhat.com>
Acked-by: Alexey Gladkov <legion@kernel.org>
Reviewed-by: Bradley Morgan <include@grrlz.net>
Reviewed-by: Pavel Tikhomirov <ptikhomirov@virtuozzo.com>
Signed-off-by: Joel Granados <joel.granados@kernel.org>
Fix kernel-doc warnings in kernel/sysctl.c by adding Returns.
Signed-off-by: Randy Dunlap <rdunlap@infradead.org>
Signed-off-by: Joel Granados <joel.granados@kernel.org>
Add colon ":" after argument name where it is missing
Add doc for proc_int_conv and proc_dointvec_conv
Signed-off-by: Joel Granados <joel.granados@kernel.org>
Remove "_minmax" from proc_doulongvec_minmax_conv as it does not enforce
min/max limits but serves as a generic converter for unsigned long
vectors. Update function declaration in sysctl.h, definition in
sysctl.c, and caller in jiffies.c accordingly.
Signed-off-by: Joel Granados <joel.granados@kernel.org>
Make do_proc_vec static and parametrize by proc_vec_type enum which
defines the type being processed and selects which converter is "live".
Signed-ness and size are calculated based on proc_vec_type and
table->data is now walked as raw bytes and advanced by the element size;
the converter still performs the actual typed load/store. Pass converter
as a union to avoid a cast from void*. The public
proc_do{int,uint,ulong}vec_conv() prototypes and all converter
signatures in kernel/, fs/ and the header are therefore unchanged.
Remove do_proc_doulongvec_minmax. proc_doulongvec_minmax_conv uses a
converter callback passed by the caller instead of conversions based on
conv{mul,div}. Create uni and bi-direction converters for milliseconds
to jiffies in proc_doulongvec_ms_jiffies_minmax; which is the only user
of proc_doulongvec_minmax_conv.
Replace do_proc_douintvec{,_w,_r} functions with a call to do_proc_vec.
Disallow vectors for uint by returning -EINVAL when more than one
element is detected.
Signed-off-by: Joel Granados <joel.granados@kernel.org>
Updates all douintvec converter function signatures to include a bool
*negp parameter. This is a preparation commit required to eventually run
all converters under the same function. The negp argument will be
ignored as it is not relevant for the uint type. Note that
do_proc_uint_conv_pipe_maxsz in pipe.c is also modified.
Signed-off-by: Joel Granados <joel.granados@kernel.org>
Move the converter assignment out of do_proc_dointvec into the caller.
Both the test for NULL and the assignment are meant to stay within the
sysctl.c context. This is in preparation of using a typed macro to for
the integer proc vector function.
Signed-off-by: Joel Granados <joel.granados@kernel.org>
check_atomic_load() calls check_load_mem() before atomic_ptr_type_ok().
For a load-acquire that fetches into its own source register (dst_reg ==
src_reg), check_load_mem() overwrites src_reg's type with the type of the
loaded value, so the subsequent atomic_ptr_type_ok() no longer sees the
source pointer and fails to reject the disallowed types (ctx, pkt,
flow_keys, sock).
Since bpf_convert_ctx_accesses() does not rewrite atomic loads, the raw
access to the underlying kernel object is left in place. The destination
type is taken from the ctx access itself, so a load-acquire of the sk
field of struct __sk_buff for example leaves the register typed as
PTR_TO_SOCK_COMMON_OR_NULL, which type_is_sk_pointer() does not match
either, while it actually holds unconverted struct sk_buff bytes. Once
the NULL check has passed this is a type confusion, not just a leak of
kernel data.
Validate src_reg with check_reg_arg() and check the source pointer type
with atomic_ptr_type_ok() before the load again, mirroring
check_atomic_rmw(). Out-of-range register numbers are already rejected
earlier by check_and_resolve_insns() (commit 503d21ef8e ("bpf: Do
register range validation early")), and the only exemption there,
is_stack_arg_ldx(), requires BPF_LDX | BPF_MEM | BPF_DW and thus never
matches a BPF_ATOMIC insn. atomic_ptr_type_ok() can therefore not
dereference register state out of bounds, that is, the out-of-bounds
read addressed by the Fixes commit below does not reappear (as proven
also via selftest).
Fixes: c03bb2fa32 ("bpf: Fix out-of-bounds read in check_atomic_load/store()")
Reported-by: STAR Labs SG <info@starlabs.sg>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://lore.kernel.org/bpf/20260804201917.253491-1-daniel@iogearbox.net
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
This helps to find a common subtree of two resources, which is important
when determining whether it's helpful to evict one resource in favor of
another.
To facilitate this, add a common helper to find the ancestor of two
cgroups using each cgroup's ancestor array.
Tested-by: Thadeu Lima de Souza Cascardo <cascardo@igalia.com>
Reviewed-by: Maarten Lankhorst <dev@lankhorst.se>
Reviewed-by: Timur Kristóf <timur.kristof@gmail.com>
Signed-off-by: Natalie Vock <natalie.vock@gmx.de>
Link: https://patch.msgid.link/20260804-dmemcg-aggressive-protect-v8-2-07af96681bf8@gmx.de
Callers can use this feedback to be more aggressive in making space for
allocations of a cgroup if they know it is protected.
These are counterparts to memcg's mem_cgroup_below_{min,low}.
Reviewed-by: Maarten Lankhorst <dev@lankhorst.se>
Reviewed-by: Timur Kristóf <timur.kristof@gmail.com>
Signed-off-by: Natalie Vock <natalie.vock@gmx.de>
Link: https://patch.msgid.link/20260804-dmemcg-aggressive-protect-v8-1-07af96681bf8@gmx.de
Be consistent in using vma_start_pgoff() and vma_end_pgoff(), which clearly
indicates which part of the VMA the page offset refers to and aids
greppability.
This is part of a broader series laying the ground to provide a virtual
page offset for MAP_PRIVATE-file backed anon folios.
No functional change intended.
Link: https://lore.kernel.org/20260710-b4-pre-scalable-cow-v2-19-2a5aa403d977@kernel.org
Signed-off-by: Lorenzo Stoakes <ljs@kernel.org>
Acked-by: Marek Szyprowski <m.szyprowski@samsung.com> # for kernel/dma
Reviewed-by: Gregory Price <gourry@gourry.net>
Acked-by: Pedro Falcato <pfalcato@suse.de>
Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Cc: Ackerley Tng <ackerleytng@google.com>
Cc: David Hildenbrand (Arm) <david@kernel.org>
Cc: Kai Huang <kai.huang@intel.com>
Cc: SJ Park <sj@kernel.org>
Cc: Thomas Zimmermann <tzimmermann@suse.de>
Cc: Liam R. Howlett (Oracle) <liam@infradead.org>
Cc: Zi Yan <ziy@nvidia.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
The family of vma_interval_tree_() functions manipulate the
address_space (which, of course, is generally referred to as 'mapping')
reverse mapping, but are named the 'VMA' interval tree.
VMAs may be mapped by an anon_vma, an address_space, or both. Therefore
calling the mapping interval tree a 'VMA' interval tree is rather
confusing.
This is also inconsistent with the anon_vma_interval_tree_*() functions
which explicitly reference the rmap object to which they pertain.
Rename the vma_interval_tree_*() functions to mapping_rmap_tree_*() to
correct this.
We will rename the anon rmap functions similarly in a subsequent patch.
No functional change intended.
Link: https://lore.kernel.org/20260710-b4-pre-scalable-cow-v2-8-2a5aa403d977@kernel.org
Signed-off-by: Lorenzo Stoakes <ljs@kernel.org>
Reviewed-by: Gregory Price <gourry@gourry.net>
Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Reviewed-by: Zi Yan <ziy@nvidia.com>
Cc: Ackerley Tng <ackerleytng@google.com>
Cc: David Hildenbrand (Arm) <david@kernel.org>
Cc: Kai Huang <kai.huang@intel.com>
Cc: Marek Szyprowski <m.szyprowski@samsung.com>
Cc: Pedro Falcato <pfalcato@suse.de>
Cc: SJ Park <sj@kernel.org>
Cc: Thomas Zimmermann <tzimmermann@suse.de>
Cc: Liam R. Howlett (Oracle) <liam@infradead.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
The file-backed mapping interval tree functions vma_interval_tree_*()
accept a raw rb_root_cached pointer to determine the tree in which they are
operating.
However, in each case, this is always associated with an address_space data
type.
So simply pass a pointer to that instead to simplify the code, and more
clearly differentiate between these operations and those concerning
anonymous mappings.
While we're here, make the generated interval tree functions static as they
do not need to be used externally (any previously existing external users
have now been removed).
We also rename VMA parameters from 'node' to 'vma' as calling this a node
is simply confusing, update the input index types to pgoff_t since they
reference page offsets and rename the parameters to pgoff_start and
pgoff_last.
No functional change intended.
Link: https://lore.kernel.org/20260710-b4-pre-scalable-cow-v2-6-2a5aa403d977@kernel.org
Signed-off-by: Lorenzo Stoakes <ljs@kernel.org>
Reviewed-by: Pedro Falcato <pfalcato@suse.de>
Reviewed-by: Gregory Price <gourry@gourry.net>
Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Reviewed-by: Zi Yan <ziy@nvidia.com>
Cc: Ackerley Tng <ackerleytng@google.com>
Cc: David Hildenbrand (Arm) <david@kernel.org>
Cc: Kai Huang <kai.huang@intel.com>
Cc: Marek Szyprowski <m.szyprowski@samsung.com>
Cc: SJ Park <sj@kernel.org>
Cc: Thomas Zimmermann <tzimmermann@suse.de>
Cc: Liam R. Howlett (Oracle) <liam@infradead.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
mm/internal.h becomes more and more bloated.
Move declarations related to vmalloc to a new mm/vmalloc.h header.
No functional changes.
Link: https://lore.kernel.org/20260709-internal-h-v2-3-695631425968@kernel.org
Signed-off-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Acked-by: Muchun Song <muchun.song@linux.dev>
Acked-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Acked-by: Lorenzo Stoakes <ljs@kernel.org>
Acked-by: Pratyush Yadav <pratyush@kernel.org>
Acked-by: SJ Park <sj@kernel.org>
Cc: Alexander Graf <graf@amazon.com>
Cc: Alexander Potapenko <glider@google.com>
Cc: Brendan Jackman <jackmanb@google.com>
Cc: Brendan Jackman <brendan.jackman@linux.dev>
Cc: Dennis Zhou <dennis@kernel.org>
Cc: Dmitry Vyukov <dvyukov@google.com>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Marco Elver <elver@google.com>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Oscar Salvador <osalvador@suse.de>
Cc: Pasha Tatashin <pasha.tatashin@soleen.com>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Tejun Heo <tj@kernel.org>
Cc: "Uladzislau Rezki (Sony)" <urezki@gmail.com>
Cc: Zi Yan <ziy@nvidia.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Patch series "mm: split a couple of headers from internal.h", v2.
mm/internal.h becomes more and more bloated.
Split declarations related to mm_init, memblock, vmalloc and sparse into
new headers.
This patch (of 3):
mm/internal.h becomes more and more bloated.
Move declarations for related to mm/mm_init.c and mm/memblock.c to a new
mm/mm_init.h header.
No functional changes.
[rppt@kernel.org: split stubfs from internal.h to mm_init.h]
Link: https://lore.kernel.org/alJd1BLypyK9Mpaw@kernel.org
Link: https://lore.kernel.org/20260709-internal-h-v2-0-695631425968@kernel.org
Link: https://lore.kernel.org/20260709-internal-h-v2-1-695631425968@kernel.org
Signed-off-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Acked-by: Muchun Song <muchun.song@linux.dev>
Acked-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Acked-by: Lorenzo Stoakes <ljs@kernel.org>
Acked-by: Pratyush Yadav <pratyush@kernel.org>
Acked-by: SJ Park <sj@kernel.org>
Cc: Alexander Graf <graf@amazon.com>
Cc: Alexander Potapenko <glider@google.com>
Cc: Brendan Jackman <jackmanb@google.com>
Cc: Brendan Jackman <brendan.jackman@linux.dev>
Cc: Dennis Zhou <dennis@kernel.org>
Cc: Dmitry Vyukov <dvyukov@google.com>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Marco Elver <elver@google.com>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Oscar Salvador <osalvador@suse.de>
Cc: Pasha Tatashin <pasha.tatashin@soleen.com>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Tejun Heo <tj@kernel.org>
Cc: "Uladzislau Rezki (Sony)" <urezki@gmail.com>
Cc: Zi Yan <ziy@nvidia.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
A potential invalid storage access issue can occur after replacing a
cgroup bpf prog.
This occurs in the following scenario:
1. prog1 with storage is attached to a cgroup in multi-attach mode.
2. prog1 is replaced with prog2 using BPF_F_REPLACE in multi-attach
mode, but fails midway (e.g. in bpf_trampoline_link_cgroup_shim or
update_effective_progs).
3. A new prog3 is attached to the cgroup in multi-attach mode.
The reason is that __cgroup_bpf_attach overwrites pl->storage with the
new storage prior to attachment completion. When attachment fails
midway, the cleanup path calls bpf_cgroup_storages_free(new_storage) to
free the newly allocated storage, but fails to restore pl->storage back
to old_storage.
Consequently, the still-active prog1 holds invalid or dangling storage
pointers, leading to an invalid memory access when prog1 executes and
calls bpf_get_local_storage. Additionally, original pl->flags and
cgrp->bpf.flags[atype] are left unrestored.
Fix this by saving old_pl_flags, old_storage, and old_flags prior to the
update, and properly restoring all of them in the cleanup path on error.
Fixes: 7d9c342789 ("bpf: Make cgroup storages shared between programs on the same cgroup")
Reported-by: Sashiko <sashiko-bot@kernel.org>
Signed-off-by: Pu Lehui <pulehui@huawei.com>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Reviewed-by: Emil Tsalapatis <emil@etsalapatis.com>
Link: https://lore.kernel.org/bpf/20260803013934.4036646-1-pulehui@huaweicloud.com
Commit 8da2a88383 ("rv: Add KUnit tests for some DA/HA monitors")
introduced a division of a 64-bit value by 1000 in the nomiss KUnit
test. This does not compile on 32-bit systems, as standard division of
64-bit values leads to an undefined reference to __udivdi3.
Fix the build on 32-bit systems by using div_u64().
Fixes: 8da2a88383 ("rv: Add KUnit tests for some DA/HA monitors")
Reported-by: kernel test robot <lkp@intel.com>
Closes: https://lore.kernel.org/oe-kbuild-all/202608020311.hYjqOG5k-lkp@intel.com
Reviewed-by: Nam Cao <namcao@linutronix.de>
Link: https://lore.kernel.org/r/20260803150622.322806-1-gmonaco@redhat.com
Signed-off-by: Gabriele Monaco <gmonaco@redhat.com>
blkdev.h gets included in various places outside the block layer just
for struct blk_plug and related plugging functions.
Split blk_plug into a separate helper to reduce the amount of code
that needs to get rebuilt when blkdev.h changes and to slightly
reduce compile times.
In io_uring this requires pulling in a few other headers explicitly that
previously were implicitly included through blkdev.h.
Signed-off-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Christian Brauner (Amutable) <brauner@kernel.org>
Reviewed-by: Johannes Thumshirn <johannes.thumshirn@wdc.com>
Reviewed-by: Damien Le Moal <dlemoal@kernel.org>
Link: https://patch.msgid.link/20260804125524.740996-1-hch@lst.de
Signed-off-by: Jens Axboe <axboe@kernel.dk>
futex_private_hash_put() drops the reference to fph before evaluating
fph->mm for wake_up_var(). futex_ref_put() enables preemption again before
returning. If that put drops the final reference and the task is preempted,
another task can pivot to the replacement hash and free the old hash after
an RCU grace period. The first task then reads fph->mm from the freed
allocation when it resumes.
KASAN reports a slab-use-after-free in futex_private_hash_put(), with the
read at offset 24 in a freed kmalloc-512 allocation. The allocation and
free stacks point to futex_hash_allocate() and the RCU free path,
respectively.
Load the mm pointer while the fph reference is still held and pass the
saved value to wake_up_var(). wake_up_var() uses the pointer as a waitqueue
key and does not dereference the mm through it.
Fixes: bd54df5ea7 ("futex: Allow to resize the private local hash")
Signed-off-by: Felix Hoffmann <f3lix.dev@gmx.de>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Cc: stable@vger.kernel.org
Link: https://patch.msgid.link/20260731155024.1150011-1-f3lix.dev@gmx.de
In a sleepable BPF program, a spin lock can provide the only RCU protection
for a kptr. The final bpf_spin_unlock() ends that protection, but the
verifier leaves the pointer valid. Another CPU can then free the object
before the pointer is used. A capability-limited runtime PoC triggered a
task_struct use-after-free in __bpf_get_task_stack().
Record whether the program is in an RCU-protected context before releasing
the lock. Invalidate RCU-protected pointers only when the unlock leaves the
final such context. This preserves valid pointers in non-sleepable programs
and inside an explicit RCU read-side section.
Fixes: 5861d1e8db ("bpf: Allow bpf_spin_{lock,unlock} in sleepable progs")
Assisted-by: Codex:gpt-5.6-sol
Assisted-by: ChatGPT:GPT-5.6-Pro
Signed-off-by: Ning Ding <dingning04@gmail.com>
Link: https://lore.kernel.org/bpf/20260803112615.3362122-2-dingning04@gmail.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
HugeTLB pages can be preserved memory. So they are never allocated from
scratch. Instead, they are allocated from the memory blocks with no
preserved memory. These areas are detected at runtime on each boot.
But since they are allocated via memblock, they show up as RSRV_KERN,
and blow up the scratch size when scratch scale is in use.
All hugetlb pages are marked RSRV_HUGETLB. Subtract their size from
RSRV_KERN when calculating scratch sizes.
Signed-off-by: Pratyush Yadav (Google) <pratyush@kernel.org>
Link: https://patch.msgid.link/20260801084833.1897543-23-pratyush@kernel.org
Signed-off-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Motivation
==========
The scratch space is allocated by the first kernel in the KHO chain, and
is reused by all subsequent kernels. The size of the space is either set
via the commandline by the system administrator or by calculating the
amount of memory used by the kernel and adding a multiplier. In either
case, the scratch space is a heuristic and is liable to fill up and fail
allocation if a kernel uses more memory than expected.
In addition, gigantic huge pages (usually 1 GiB) are allocated via
memblock, and in a KHO boot that memory comes from the scratch space. In
hypervisors it is common to dedicate a major part of the system's memory
to gigantic hugepages for VM memory.
If this memory needs to come from scratch space, then scratch needs to
be greater than the memory needed for huge pages, which is impractical.
In addition, hugepages can be preserved memory. Allocating them from
scratch violates the assumption that scratch contains no preserved
memory.
Methodology
===========
Discover areas that don't contain any preserved memory at boot by
walking the preserved memory radix tree. Mark them as scratch to allow
allocations from them. This makes KHO more resilient to memory pressure
and allows supporting huge page preservation.
Since the preserved memory radix tree mixes both physical address and
order into a single key, and does not track table pages, it is difficult
to identify free areas from it directly. Walk the tree and digest it
down into another radix tree. The latter tracks blocks of
KHO_SCRATCH_EXT_BLKSIZE (1 GiB as of now) granularity. Then walk the
digested tree and mark the areas between the present keys as scratch.
Performance
===========
The discovery algorithm traverses the preserved memory radix tree
exactly once. While it does use memory for the digested radix tree,
since the blocks are split by 1 GiB, a single bitmap with 4k pages can
track up to 32 TiB of memory. So there are likely to be very few radix
tree pages used in this tracking. For systems with all physical memory
below 32 TiB, this should result in a total of 6 pages being
used (KHO_TREE_MAX_DEPTH == 6).
An alternate way of achieving this would be to call kho_mem_retrieve()
earlier in boot and mark all the KHO preservations as reserved. But that
can blow up memblock.reserved with a bunch of 4K pages scattered
everywhere, which will reduce performance of subsequent allocations.
Since the free blocks are tracked in chunks of 1 GiB, this won't blow up
memblock.memory as much.
There is no inherent reason for using 1 GiB as the discovered block
size. This can be changed later if needed. Currently, KHO is mainly
targeted for server grade systems with hundreds of gigabytes to
terabytes of memory. So 1 GiB is a reasonable granularity for those
systems. For smaller systems this doesn't work as well, but we can
arrive at a better heuristic when we have concrete use cases.
Practical evaluation
====================
The testing is done on a x86_64 qemu VM running under KVM with 64G
memory and 12 CPUs. The machine pre-allocates 50 1G pages.
Since the performance scales with how busy the radix tree is, tests are
done with 2 preservation patterns: first with two 1M memfds, second with
two 1G memfds, both using 4k pages.
Test case 1 - 1M memfd
~~~~~~~~~~~~~~~~~~~~~~
This test case has two memfds with 1M memory each in 4k pages, plus
other preservations from LUO core and other KHO users.
This is how the radix tree stats look like:
radix_nodes: 0x13
nr_preservations: 0x214
mem_preserved: 0x227000
per order preservations:
order 0: 0x20f
order 1: 0x4
order 4: 0x1
and this is how long it takes to extend the scratch after KHO boot:
KHO: KHO extend time: 47 us
KHO: KHO extend total mem: 0xe6c17b000 (~57G)
Test case 2 - 1G memfd
~~~~~~~~~~~~~~~~~~~~~~
This test case has two memfds with 1G memory each in 4k pages, plus
other preservations from LUO core and other KHO users.
This is how the radix tree stats look like:
radix_nodes: 0x28
nr_preservations: 0x80816
mem_preserved: 0x80829000
per order preservations:
order 0: 0x80811
order 1: 0x4
order 4: 0x1
and this is how long it takes to extend the scratch after KHO boot:
KHO: KHO extend time: 22514 us
KHO: KHO extend total mem: 0xd3f200000 (~52G)
Signed-off-by: Pratyush Yadav (Google) <pratyush@kernel.org>
Link: https://patch.msgid.link/20260801084833.1897543-19-pratyush@kernel.org
Signed-off-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Currently the preserved memory radix tree is initialized from
kho_memory_init(), which happens relatively late in MM init. In a coming
patch, the tree will be used from kho_memory_init_early(). Move the tree
initialization there.
Simplify some of the code in kho_mem_retrieve() by getting rid of the
err variable and jumping to err directly.
Signed-off-by: Pratyush Yadav (Google) <pratyush@kernel.org>
Link: https://patch.msgid.link/20260801084833.1897543-16-pratyush@kernel.org
Signed-off-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
In a future patch, mm init will use kho_scratch_overlap() for deciding
the migrate type of pageblocks it initializes. The earliest user
currently is free_area_init(). kho_scratch_overlap()
relies on kho_scratch pointer being initialized. Introduce
kho_memory_init_early() to do this.
kho_populate() would normally be a good place to do this, but
unfortunately, phys_to_virt() does not work at that point on ARM64. So
we need yet another initialization function.
Signed-off-by: Pratyush Yadav (Google) <pratyush@kernel.org>
Link: https://patch.msgid.link/20260801084833.1897543-15-pratyush@kernel.org
Signed-off-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
