mirror of
https://github.com/torvalds/linux.git
synced 2026-10-23 23:51:02 +02:00
Linux kernel source tree
Since kfree_rcu() can be called under pi_lock (a raw spinlock in the
scheduler), kfree_rcu() itself should never allocate memory with
__GFP_KSWAPD_RECLAIM as waking up kswapd ends up acquiring pi_lock,
which leads to a deadlock.
Reproducing the issue even intentionally was not straightforward.
The set_cpus_allowed_force() path that is called under pi_lock is
exercised very rarely, and competing tasks that allocate memory will
most likely wake kswapd up.
Therefore the existence of the deadlock was verified with a modified
kernel that has a lockdep map for waking up kswapd:
======================================================
WARNING: possible circular locking dependency detected
7.2.0-rc1-slab-for-next+ #10 Not tainted
------------------------------------------------------
git/6660 is trying to acquire lock:
ffff8e524533c550 (&p->pi_lock){-.-.}-{2:2}, at: _raw_spin_lock_irqsave+0x12/0x20
but task is already holding lock:
ffff8e57bffff1a0 (&pgdat->kswapd_wait){....}-{3:3}, at: _raw_spin_lock_irqsave+0x12/0x20
which lock already depends on the new lock.
the existing dependency chain (in reverse order) is:
-> #2 (&pgdat->kswapd_wait){....}-{3:3}:
__lock_acquire+0x5a4/0xc50
lock_acquire.part.0+0xb7/0x240
lock_acquire+0x70/0x170
__raw_spin_lock_irqsave+0x44/0x80
_raw_spin_lock_irqsave+0x12/0x20
__wake_up_common_lock+0x31/0xa0
__wake_up+0x20/0x40
kswapd_wakeup_wake+0x7e/0x110
wakeup_kswapd+0x295/0x320
wake_all_kswapds+0xac/0x1a0
__alloc_pages_slowpath.constprop.0+0x282/0xf80
__alloc_frozen_pages_noprof+0x32a/0x360
alloc_slab_page+0x2e/0x160
allocate_slab+0x82/0x420
new_slab+0x52/0xb0
refill_objects+0x13d/0x190
refill_sheaf+0x5d/0xd0
__pcs_replace_empty_main+0x230/0xb10
[...]
-> #1 (kswapd_wakeup){-.-.}-{0:0}:
__lock_acquire+0x5a4/0xc50
lock_sync.part.0+0x75/0x100
lock_sync+0x36/0x70
might_wakeup_kswapd+0x61/0xa0
__kfree_rcu_sheaf+0x33/0xd90
kvfree_call_rcu+0x1d4/0x3b0
set_cpus_allowed_force+0x163/0x220
cpuset_cpus_allowed_fallback+0x18b/0x240
select_fallback_rq+0x1e6/0x250
[...]
-> #0 (&p->pi_lock){-.-.}-{2:2}:
check_prev_add+0xe6/0xe00
validate_chain+0x51e/0x6e0
__lock_acquire+0x5a4/0xc50
lock_acquire.part.0+0xb7/0x240
lock_acquire+0x70/0x170
__raw_spin_lock_irqsave+0x44/0x80
_raw_spin_lock_irqsave+0x12/0x20
try_to_wake_up+0x77/0xa90
default_wake_function+0x27/0x60
autoremove_wake_function+0x23/0xb0
__wake_up_common+0xb8/0x170
__wake_up_common_lock+0x51/0xa0
__wake_up+0x20/0x40
kswapd_wakeup_wake+0x7e/0x110
wakeup_kswapd+0x295/0x320
wake_all_kswapds+0xac/0x1a0
__alloc_pages_slowpath.constprop.0+0x282/0xf80
__alloc_frozen_pages_noprof+0x32a/0x360
alloc_slab_page+0x2e/0x160
allocate_slab+0x82/0x420
new_slab+0x52/0xb0
refill_objects+0x13d/0x190
refill_sheaf+0x5d/0xd0
__pcs_replace_empty_main+0x230/0xb10
[...]
other info that might help us debug this:
Chain exists of:
&p->pi_lock --> kswapd_wakeup --> &pgdat->kswapd_wait
Possible unsafe locking scenario:
CPU0 CPU1
---- ----
lock(&pgdat->kswapd_wait);
lock(kswapd_wakeup);
lock(&pgdat->kswapd_wait);
lock(&p->pi_lock);
*** DEADLOCK ***
3 locks held by git/6660:
#0: ffff8e519f0e9e20 (&type->i_mutex_dir_key#6){++++}-{4:4}, at: lookup_slow+0x2d/0x60
#1: ffffffff8a31c460 (kswapd_wakeup){-.-.}-{0:0}, at: kswapd_wakeup_wake+0x4d/0x110
#2: ffff8e57bffff1a0 (&pgdat->kswapd_wait){....}-{3:3}, at: _raw_spin_lock_irqsave+0x12/0x20
[...]
Fix this by always avoiding waking up kswapd in __kfree_rcu_sheaf().
Note that there are two paths that might wake up kswapd:
1) __kfree_rcu_sheaf()
// __GFP_KSWAPD_RECLAIM might wake up kswapd
-> alloc_empty_sheaf(GFP_NOWAIT)
2) __kfree_rcu_sheaf()
// Let's say __kfree_rcu_sheaf() doesn't pass GFP_NOWAIT
-> alloc_empty_sheaf(__GFP_NOWARN)
-> kmalloc_flags()
-> slab_alloc_node()
-> alloc_from_pcs()
-> __pcs_replace_empty_main()
// Free a sheaf in an allocation path when the sheaf becomes empty
// and refilling the sheaf fails
-> free_empty_sheaf()
-> slab_free()
-> free_to_pcs()
-> __pcs_replace_full_main()
// However free path always assumes it's safe to wake up kswapd
-> alloc_empty_sheaf(GFP_NOWAIT)
Drop __GFP_KSWAPD_RECLAIM in both cases. Note that the kfree_rcu() is
not the only user of free_to_pcs() path, but it should be fixed as it
can be invoked under pi_lock.
Reported-by: Sashiko <sashiko-bot@kernel.org>
Closes: https://sashiko.dev/#/patchset/20260831-b4-kfree_rcu_hotfix-v1-1-4f0fb882638b%40kernel.org
Fixes:
|
||
|---|---|---|
| arch | ||
| block | ||
| certs | ||
| crypto | ||
| Documentation | ||
| drivers | ||
| fs | ||
| include | ||
| init | ||
| io_uring | ||
| ipc | ||
| kernel | ||
| lib | ||
| LICENSES | ||
| mm | ||
| net | ||
| rust | ||
| samples | ||
| scripts | ||
| security | ||
| sound | ||
| tools | ||
| usr | ||
| virt | ||
| .clang-format | ||
| .clippy.toml | ||
| .cocciconfig | ||
| .editorconfig | ||
| .get_maintainer.ignore | ||
| .gitattributes | ||
| .gitignore | ||
| .mailmap | ||
| .pylintrc | ||
| .rustfmt.toml | ||
| COPYING | ||
| CREDITS | ||
| Kbuild | ||
| Kconfig | ||
| MAINTAINERS | ||
| Makefile | ||
| README | ||
Linux kernel ============ The Linux kernel is the core of any Linux operating system. It manages hardware, system resources, and provides the fundamental services for all other software. Quick Start ----------- * Report a bug: See Documentation/admin-guide/reporting-issues.rst * Get the latest kernel: https://kernel.org * Build the kernel: See Documentation/admin-guide/quickly-build-trimmed-linux.rst * Join the community: https://lore.kernel.org/ Essential Documentation ----------------------- All users should be familiar with: * Building requirements: Documentation/process/changes.rst * Code of Conduct: Documentation/process/code-of-conduct.rst * License: See COPYING Documentation can be built with make htmldocs or viewed online at: https://www.kernel.org/doc/html/latest/ Who Are You? ============ Find your role below: * New Kernel Developer: Getting started with kernel development * Academic Researcher: Studying kernel internals and architecture * Security Expert: Hardening and vulnerability analysis * Backport/Maintenance Engineer: Maintaining stable kernels * System Administrator: Configuring and troubleshooting * Maintainer: Leading subsystems and reviewing patches * Hardware Vendor: Writing drivers for new hardware * Distribution Maintainer: Packaging kernels for distros * AI Coding Assistant: LLMs and AI-powered development tools For Specific Users ================== New Kernel Developer -------------------- Welcome! Start your kernel development journey here: * Getting Started: Documentation/process/development-process.rst * Your First Patch: Documentation/process/submitting-patches.rst * Coding Style: Documentation/process/coding-style.rst * Build System: Documentation/kbuild/index.rst * Development Tools: Documentation/dev-tools/index.rst * Kernel Hacking Guide: Documentation/kernel-hacking/hacking.rst * Core APIs: Documentation/core-api/index.rst Academic Researcher ------------------- Explore the kernel's architecture and internals: * Researcher Guidelines: Documentation/process/researcher-guidelines.rst * Memory Management: Documentation/mm/index.rst * Scheduler: Documentation/scheduler/index.rst * Networking Stack: Documentation/networking/index.rst * Filesystems: Documentation/filesystems/index.rst * RCU (Read-Copy Update): Documentation/RCU/index.rst * Locking Primitives: Documentation/locking/index.rst * Power Management: Documentation/power/index.rst Security Expert --------------- Security documentation and hardening guides: * Security Documentation: Documentation/security/index.rst * LSM Development: Documentation/security/lsm-development.rst * Self Protection: Documentation/security/self-protection.rst * Reporting Vulnerabilities: Documentation/process/security-bugs.rst * CVE Procedures: Documentation/process/cve.rst * Embargoed Hardware Issues: Documentation/process/embargoed-hardware-issues.rst * Security Features: Documentation/userspace-api/seccomp_filter.rst Backport/Maintenance Engineer ----------------------------- Maintain and stabilize kernel versions: * Stable Kernel Rules: Documentation/process/stable-kernel-rules.rst * Backporting Guide: Documentation/process/backporting.rst * Applying Patches: Documentation/process/applying-patches.rst * Subsystem Profile: Documentation/maintainer/maintainer-entry-profile.rst * Git for Maintainers: Documentation/maintainer/configure-git.rst System Administrator -------------------- Configure, tune, and troubleshoot Linux systems: * Admin Guide: Documentation/admin-guide/index.rst * Kernel Parameters: Documentation/admin-guide/kernel-parameters.rst * Sysctl Tuning: Documentation/admin-guide/sysctl/index.rst * Tracing/Debugging: Documentation/trace/index.rst * Performance Security: Documentation/admin-guide/perf-security.rst * Hardware Monitoring: Documentation/hwmon/index.rst Maintainer ---------- Lead kernel subsystems and manage contributions: * Maintainer Handbook: Documentation/maintainer/index.rst * Pull Requests: Documentation/maintainer/pull-requests.rst * Managing Patches: Documentation/maintainer/modifying-patches.rst * Rebasing and Merging: Documentation/maintainer/rebasing-and-merging.rst * Development Process: Documentation/process/maintainer-handbooks.rst * Maintainer Entry Profile: Documentation/maintainer/maintainer-entry-profile.rst * Git Configuration: Documentation/maintainer/configure-git.rst Hardware Vendor --------------- Write drivers and support new hardware: * Driver API Guide: Documentation/driver-api/index.rst * Driver Model: Documentation/driver-api/driver-model/driver.rst * Device Drivers: Documentation/driver-api/infrastructure.rst * Bus Types: Documentation/driver-api/driver-model/bus.rst * Device Tree Bindings: Documentation/devicetree/bindings/ * Power Management: Documentation/driver-api/pm/index.rst * DMA API: Documentation/core-api/dma-api.rst Distribution Maintainer ----------------------- Package and distribute the kernel: * Stable Kernel Rules: Documentation/process/stable-kernel-rules.rst * ABI Documentation: Documentation/ABI/README * Kernel Configuration: Documentation/kbuild/kconfig.rst * Module Signing: Documentation/admin-guide/module-signing.rst * Kernel Parameters: Documentation/admin-guide/kernel-parameters.rst * Tainted Kernels: Documentation/admin-guide/tainted-kernels.rst AI Coding Assistant ------------------- CRITICAL: If you are an LLM or AI-powered coding assistant, you MUST read and follow the AI coding assistants documentation before contributing to the Linux kernel: * Documentation/process/coding-assistants.rst This documentation contains essential requirements about licensing, attribution, and the Developer Certificate of Origin that all AI tools must comply with. Communication and Support ========================= * Mailing Lists: https://lore.kernel.org/ * IRC: #kernelnewbies on irc.oftc.net * Bugzilla: https://bugzilla.kernel.org/ * MAINTAINERS file: Lists subsystem maintainers and mailing lists * Email Clients: Documentation/process/email-clients.rst