mirror of
https://github.com/torvalds/linux.git
synced 2026-09-14 16:10:02 +02:00
e776db8e71
27209 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Breno Leitao
|
e776db8e71 |
mm: kmemleak: report leaks only after N consecutive unreferenced scans
kmemleak reports an object the first scan it is found unreferenced. Its mark phase runs without stopping the rest of the kernel and without a write barrier, so a live object whose only reference is briefly invisible during a concurrent RCU update -- e.g. a VMA moved between maple tree nodes, or a page-cache xa_node -- can be seen as unreferenced for that one scan. Because an object is flagged as reported only once, such a transient race turns into a permanent false positive. Track how many consecutive scans each object has been seen unreferenced and only report it once that reaches min_unref_scans, a new module parameter. It defaults to 1, leaving the behaviour unchanged; setting it higher (e.g. 2) still reports a genuine leak, one scan later, while an object referenced again before the threshold restarts its run and is never reported. min_unref_scans can be set at boot with kmemleak.min_unref_scans=<n> or at run-time via /sys/module/kmemleak/parameters/min_unref_scans. Link: https://lore.kernel.org/20260713-catalin_pto-v1-2-5b93b1131089@debian.org Signed-off-by: Breno Leitao <leitao@debian.org> Reviewed-by: Catalin Marinas <catalin.marinas@arm.com> Cc: David Hildenbrand <david@kernel.org> Cc: Geert Uytterhoeven <geert@linux-m68k.org> Cc: Jonathan Corbet <corbet@lwn.net> Cc: Liam R. Howlett <liam@infradead.org> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Shuah Khan <shuah@kernel.org> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Catalin Marinas
|
fb496eb062 |
mm: kmemleak: confirm suspected leaks with a second scan
Patch series "mm: kmemleak: reduce transient false positives by confirming leaks". This series combines two kmemleak enhancements that were originally submitted separately but both required rebasing after commit |
||
|
David Hildenbrand (Arm)
|
534b19bbb6 |
mm/gup_test: keep longterm pin state per file
The pin longterm test currently stores its data globally, shared among
multiple concurrent users of the interface (multiple open file descriptors
-> multiple "struct file"'s). That makes the gup_test interface
problematic to use concurrently: two users, such as concurrent selftest
runs, can interfere with the same longterm pin state.
While this has not been observed as a problem so far in practice, let's
just handle it cleanly. There could be a way to trigger selftest failures
by e.g., running the cow.c and gup_longerm.c selftests concurrently, but
we usually run them sequentially. Let's add a "Fixes" tag to be safe, but
not need to CC stable.
Link: https://lore.kernel.org/20260810-gup_test_data-v1-1-fb1d41be5bb4@kernel.org
Fixes:
|
||
|
Hui Su
|
8790303cba |
kasan: fix cache shrink race with CPU hotplug
kasan_quarantine_remove_cache() first invokes per_cpu_remove_cache() on
all online CPUs. Each callback moves objects belonging to the cache from
cpu_quarantine to the CPU's shrink_qlist, where they can later be freed
from task context.
kmem_cache_destroy() invokes the quarantine removal path while holding
cpus_read_lock(), but kmem_cache_shrink() does not. The latter can
therefore race with CPU offlining as follows:
kmem_cache_shrink() CPU hotplug
------------------- -----------
on_each_cpu()
CPU1 moves objects to
CPU1's shrink_qlist
on_each_cpu() returns
CPU1 goes offline
kasan_cpu_offline()
drains cpu_quarantine
leaves shrink_qlist untouched
for_each_online_cpu()
skips CPU1
The objects left on CPU1's shrink_qlist are not returned to the slab
allocator. This may prevent kmem_cache_shrink() from releasing slabs that
would otherwise become empty. If CPU1 remains offline, a later
kmem_cache_destroy() also skips the list and can report that the cache
still contains objects.
An intermittent occurrence was observed with a virtio-9p filesystem. The
mount and umount commands both returned 0, but the kernel logged the
following during the userspace-triggered teardown:
[ 2994.380134][ T111] BUG 9p-fcall-cache-1 (Tainted: G B ): Objects remaining on __kmem_cache_shutdown()
[ 2994.381140][ T111] Object 0xff11000004361118 @offset=4376
[ 2994.381607][ T111] Allocated in p9_fcall_init+0x201/0x400 age=19564 cpu=1 pid=104
[ 2994.382591][ T111] p9_fcall_init+0x201/0x400
[ 2994.382810][ T111] p9_tag_alloc+0x12f/0x700
[ 2994.382982][ T111] p9_client_prepare_req+0x102/0x3e0
[ 2994.383165][ T111] p9_client_rpc+0x1ab/0xa50
[ 2994.383334][ T111] p9_client_getattr_dotl+0xb0/0x1a0
[ 2994.383515][ T111] v9fs_vfs_getattr_dotl+0x115/0x360
[ 2994.383719][ T111] vfs_getattr_nosec+0x22c/0x3a0
[ 2994.383910][ T111] vfs_statx+0xd7/0x170
[ 2994.384062][ T111] vfs_fstatat+0x45/0x80
[ 2994.384215][ T111] __do_sys_newfstatat+0x84/0xe0
[ 2994.384386][ T111] do_syscall_64+0x115/0x6a0
[ 2994.384566][ T111] entry_SYSCALL_64_after_hwframe+0x77/0x7f
[ 2994.399720][ T111] WARNING: mm/slub.c:1244 at __kmem_cache_shutdown+0x363/0x500, CPU#0: busybox/111
[ 2994.405655][ T111] Call Trace:
[ 2994.406325][ T111] kmem_cache_destroy+0x73/0x1b0
[ 2994.406630][ T111] p9_client_destroy+0x271/0x3c0
[ 2994.407210][ T111] v9fs_session_close+0x3c/0x260
[ 2994.407409][ T111] v9fs_kill_super+0x48/0x90
[ 2994.407584][ T111] deactivate_locked_super+0xa3/0x160
[ 2994.407778][ T111] cleanup_mnt+0x1dd/0x3e0
Thus, a successful umount left objects in the 9p fcall cache and prevented
the cache from being destroyed cleanly.
Per-CPU shrink_qlist storage exists for every possible CPU, and each list
is protected by its own raw spinlock. Iterate over possible CPUs so that
a list populated before its CPU went offline is drained as well.
for_each_possible_cpu() can do more work than for_each_online_cpu(), but
this change only affects CONFIG_KASAN_GENERIC kernels. The extra work is
limited to cache shrink and cache destruction paths and does not affect
the normal allocation/free fast path. It adds one raw-spinlock-protected
scan of each possible CPU's shrink list. These lists are normally empty;
a non-empty list is traversed to remove objects belonging to the cache
being shrunk or destroyed.
Link: https://lore.kernel.org/20260808031459.3032812-1-sh_def@163.com
Fixes:
|
||
|
Hongfu Li
|
097492865f |
mm/cma: remove stray newline from auto-generated CMA area name
When no name is supplied, cma_new_area() generates names with format
"cma%d\n", introducing an unintended newline character ('\n') in the CMA
name.
Most CMA regions are created with explicit names, so this path is seldom
hit. The newline only creates cosmetic noise in debug logs, traces and
debugfs with no functional impact.
Link: https://lore.kernel.org/20260810093215.91419-1-hongfu.li@linux.dev
Signed-off-by: Hongfu Li <lihongfu@kylinos.cn>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Reviewed-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Reviewed-by: SJ Park <sj@kernel.org>
Reviewed-by: Anshuman Khandual <anshuman.khandual@arm.com>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
||
|
Yunzhao Li
|
1f2b4b28aa |
mm/zswap: use ratelimited stats flush in zswap_shrinker_count()
zswap_shrinker_count() calls mem_cgroup_flush_stats(), which takes the
global cgroup rstat lock synchronously. On machines with many CPUs and
NUMA nodes, this creates severe lock contention in the kswapd reclaim
path:
- Multiple kswapd threads (one per NUMA node) run concurrently.
- do_shrink_slab() invokes zswap_shrinker_count() for each
memcg-aware shrinker pass.
- Each call flushes the full cgroup rstat hierarchy under the global
lock.
On AMD EPYC 9684X machines (96 cores, 192 threads, 12 NUMA nodes) running
production workloads with zswap enabled, perf shows 2.88% of kernel cycles
in osq_lock contention from this path:
2.88% [k] osq_lock
--__mutex_lock.constprop.0
--__cgroup_rstat_lock
--cgroup_rstat_flush_locked
--cgroup_rstat_flush
--zswap_shrinker_count
do_shrink_slab
shrink_slab
shrink_node
balance_pgdat
kswapd
84% of kswapd kernel cycles are spent in
shrink_slab -> zswap_shrinker_count -> cgroup_rstat_flush, not in actual
page reclaim (shrink_lruvec).
Controlled A/B on identical hardware and workload:
shrinker=Y: 2.88% osq_lock, memory PSI 1.58%
shrinker=N: 0.00% osq_lock, memory PSI 0.57%
eBPF-based rstat lock wait measurement across 8 production metals
confirms the contention splits cleanly along shrinker enablement:
shrinker=Y: 50-250x more contended lock acquisitions (248/s vs 1.1/s)
shrinker=N: baseline lock wait (0.0017 s/s vs 1.04 s/s)
zswap_shrinker_count() only produces a heuristic estimate, scaled by
compression ratio via mult_frac(). The actual writeback happens in
zswap_shrinker_scan(). Slightly stale stats are acceptable here.
Switch to mem_cgroup_flush_stats_ratelimited(), which only flushes if
the periodic 2-second flusher is one full cycle late. This matches the
approach already used in prepare_scan_control() (mm/vmscan.c) for the
same reclaim path.
After applying this patch, rstat flush latency and lock wait time on
shrinker=Y machines dropped to the same level as shrinker=N controls,
while the zswap shrinker continues to function (pool size remains
bounded under the max_pool_percent cap).
Previously discussed:
- Chengming Zhou (Dec 2023): rstat contention from
zswap_shrinker_count [1]
- Shakeel Butt (Aug 2024): zswap_shrinker_count still uses sync
flush [2]
- Yosry Ahmed (Aug 2024): suggested eliminating in-kernel
flushers [3]
- Jesper Dangaard Brouer (Sep 2024): cgroup/rstat V11 patch [4]
Link: https://lore.kernel.org/20260702180908.150136-1-yunzhao@cloudflare.com
Link: https://lore.kernel.org/linux-mm/20231206103935.3440502-1-zhouchengming@bytedance.com/ [1]
Link: https://lore.kernel.org/linux-mm/CALvZod7LFxLCxVpOFH8b2Ppm8T40HPGMKQwX_=NPCWB_mFW+oQ@mail.gmail.com/ [2]
Link: https://lore.kernel.org/linux-mm/CAJD7tkYvFyOSX+rP_FKGBhxvZiCDxtpsNp-c5CGOA-4Bq9oXSg@mail.gmail.com/ [3]
Link: https://lore.kernel.org/linux-mm/172616070094.2055617.17676042522679701515.stgit@firesoul/ [4]
Suggested-by: Jesper Dangaard Brouer <hawk@kernel.org>
Signed-off-by: Jesper Dangaard Brouer <hawk@kernel.org>
Signed-off-by: Yunzhao Li <yunzhao@cloudflare.com>
Tested-by: Yunzhao Li <yunzhao@cloudflare.com>
Acked-by: Johannes Weiner <hannes@cmpxchg.org>
Acked-by: Jesper Dangaard Brouer <hawk@kernel.org>
Acked-by: Nhat Pham <nphamcs@gmail.com>
Cc: Chengming Zhou <chengming.zhou@linux.dev>
Cc: Shakeel Butt <shakeel.butt@linux.dev>
Cc: Yosry Ahmed <yosry@kernel.org>
Cc: Yunzhao Li <yunzhao@cloudflare.com>
Cc: Sourav Panda <souravpanda@google.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
||
|
Hongfu Li
|
b090524f77 |
mm/swap: fix swap_cluster_lock() !CONFIG_SWAP stub signature mismatch
The !CONFIG_SWAP stub for swap_cluster_lock() has mismatched prototype: it has an extra unused irq argument and uses pgoff_t instead of unsigned long for offset. All callers are under CONFIG_SWAP so the extra parameter is dead. Delete the unused stub function entirely. Link: https://lore.kernel.org/20260717071104.73467-1-hongfu.li@linux.dev Signed-off-by: Hongfu Li <lihongfu@kylinos.cn> Reviewed-by: Baoquan He <baoquan.he@linux.dev> Acked-by: Kairui Song <kasong@tencent.com> Cc: Barry Song <baohua@kernel.org> Cc: Chris Li <chrisl@kernel.org> Cc: Hongfu Li <lihongfu@kylinos.cn> Cc: Kemeng Shi <shikemeng@huaweicloud.com> Cc: Nhat Pham <nphamcs@gmail.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Christoph Hellwig
|
c01e6df60e |
mm/vmstat: add NRSWP{IN,OUT} counters
Count how many swap I/Os we cause. Due to batching this can be different than the current counter number of pages written/read, and tracking this information is useful to see how efficient the batching is. The counters are added at the end of enum vm_event_item and the vmstat_text array under the assumption that the order of fields in /proc/vmstat is an ABI. If that is not the case, they could be grouped with the other swap counters. Link: https://lore.kernel.org/20260713093350.2154226-8-hch@lst.de Signed-off-by: Christoph Hellwig <hch@lst.de> Reviewed-by: Nhat Pham <nphamcs@gmail.com> Cc: Baolin Wang <baolin.wang@linux.alibaba.com> Cc: Baoquan He <baoquan.he@linux.dev> Cc: Barry Song <baohua@kernel.org> Cc: Chris Li <chrisl@kernel.org> Cc: Kairui Song <kasong@tencent.com> Cc: Kemeng Shi <shikemeng@huaweicloud.com> Cc: Youngjun Park <youngjun.park@lge.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Christoph Hellwig
|
0df74c1158 |
mm/swap: remove SWP_FS_OPS
Provide a swap_fs_activate helper that directly sets up swap_fs_ops, and a flag in struct swap_ops to indicate of NOFS swapping is allowed. Link: https://lore.kernel.org/20260713093350.2154226-7-hch@lst.de Signed-off-by: Christoph Hellwig <hch@lst.de> Cc: Baolin Wang <baolin.wang@linux.alibaba.com> Cc: Baoquan He <baoquan.he@linux.dev> Cc: Barry Song <baohua@kernel.org> Cc: Chris Li <chrisl@kernel.org> Cc: Kairui Song <kasong@tencent.com> Cc: Kemeng Shi <shikemeng@huaweicloud.com> Cc: Nhat Pham <nphamcs@gmail.com> Cc: Youngjun Park <youngjun.park@lge.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Christoph Hellwig
|
563597895e |
mm/swap: use swap_ops to register swap device's methods
This simplifies codes and makes logic clearer. And also makes later any new swap device type being added easier to handle. Currently there are two types of swap devices: fs and bdev. [hch@lst.de: updated for the new submit and can_merge abstraction] Link: https://lore.kernel.org/20260713093350.2154226-6-hch@lst.de Signed-off-by: Baoquan He <baoquan.he@linux.dev> Signed-off-by: Christoph Hellwig <hch@lst.de> Suggested-by: Chris Li <chrisl@kernel.org> Reviewed-by: Nhat Pham <nphamcs@gmail.com> Cc: Baolin Wang <baolin.wang@linux.alibaba.com> Cc: Barry Song <baohua@kernel.org> Cc: Kairui Song <kasong@tencent.com> Cc: Kemeng Shi <shikemeng@huaweicloud.com> Cc: Youngjun Park <youngjun.park@lge.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Christoph Hellwig
|
4e915b16de |
mm/swap: remove count_swpout_vm_event
There is only one caller left, so merge it into that. Link: https://lore.kernel.org/20260713093350.2154226-5-hch@lst.de Signed-off-by: Christoph Hellwig <hch@lst.de> Reviewed-by: Baoquan He <baoquan.he@linux.dev> Reviewed-by: Nhat Pham <nphamcs@gmail.com> Cc: Baolin Wang <baolin.wang@linux.alibaba.com> Cc: Barry Song <baohua@kernel.org> Cc: Chris Li <chrisl@kernel.org> Cc: Kairui Song <kasong@tencent.com> Cc: Kemeng Shi <shikemeng@huaweicloud.com> Cc: Youngjun Park <youngjun.park@lge.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Christoph Hellwig
|
dda8fb68b5 |
mm/swap: also use struct swap_iocb for block I/O
Block I/O benefits from batching just as much as remote file systems. Extend struct swap_iocb to support building a bio on the fly as well, and rewrite the block based swap code for it. This especially benefits submit_bio based drivers that do not have the block plugging available, but also saves allocating extra bios for blk-mq drivers. Add a pre-allocated bio to struct swap_iocb in a union with kiocb used for file system based swap so that struct swap_iocb can be used for all swap I/O, and initialize the pool for it unconditionally. Various low-level bdev and fs functions are now replaced with a unified can_merge/add/submit scheme. Note that the block based swap code now uses the same memcg-based check previously added for file system based swap as well. Link: https://lore.kernel.org/20260713093350.2154226-4-hch@lst.de Signed-off-by: Christoph Hellwig <hch@lst.de> Cc: Baolin Wang <baolin.wang@linux.alibaba.com> Cc: Baoquan He <baoquan.he@linux.dev> Cc: Barry Song <baohua@kernel.org> Cc: Chris Li <chrisl@kernel.org> Cc: Kairui Song <kasong@tencent.com> Cc: Kemeng Shi <shikemeng@huaweicloud.com> Cc: Nhat Pham <nphamcs@gmail.com> Cc: Youngjun Park <youngjun.park@lge.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Christoph Hellwig
|
8f29aa226f |
mm/swap: introduce struct swap_io_ctx
Generalize the context currently provided by double pointers to struct swap_iocb to an on-stack context. This cleans up the code and prepares for adding more fields and supporting batching multiple folios into a single bio for block-based swap as well. This new swap_io_ctx is required for all functions using it, the old way of allowing a NULL iocb for some callers is removed to keep the interface consistent. To reduce code duplication caused by this, a new swap_cache_read_folio_sync helper is added to consolidate the code to call swap_cache_read_folio with a local swap_io_ctx. The unpug helpers are renamed to use the submit wording as they are generalized. Link: https://lore.kernel.org/20260713093350.2154226-3-hch@lst.de Signed-off-by: Christoph Hellwig <hch@lst.de> Reviewed-by: Nhat Pham <nphamcs@gmail.com> Cc: Baolin Wang <baolin.wang@linux.alibaba.com> Cc: Baoquan He <baoquan.he@linux.dev> Cc: Barry Song <baohua@kernel.org> Cc: Chris Li <chrisl@kernel.org> Cc: Kairui Song <kasong@tencent.com> Cc: Kemeng Shi <shikemeng@huaweicloud.com> Cc: Youngjun Park <youngjun.park@lge.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Christoph Hellwig
|
a8efc69a65 |
shmem: provide a shmem_write_folio wrapper
Patch series "better block swap batching and a different take on swap_ops v5". This series makes use of the swap_iocb for block as well so that it doesn't do inefficient single-bio I/O, and then rebases the swap_ops from Baoquan on top of the now very different method structure. When running doing kernels builds, which is a workload that doesn't really do much THP anonymous memory it still gets 2x clustering for writeout and 1.2x for reading back swap in. The overall times do not actually change, though. This patch (of 7): Provide a wrapper for the shmem abuses in drm to prepare for swap I/O refactoring by keeping swap_iocb handling entirely contained in mm/. Link: https://lore.kernel.org/20260713093350.2154226-1-hch@lst.de Link: https://lore.kernel.org/20260713093350.2154226-2-hch@lst.de Signed-off-by: Christoph Hellwig <hch@lst.de> Reviewed-by: Baoquan He <baoquan.he@linux.dev> Reviewed-by: Nhat Pham <nphamcs@gmail.com> Reviewed-by: Baolin Wang <baolin.wang@linux.alibaba.com> Acked-by: Chris Li <chrisl@kernel.org> Reviewed-by: Kairui Song <kasong@tencent.com> Cc: Kemeng Shi <shikemeng@huaweicloud.com> Cc: Barry Song <baohua@kernel.org> Cc: Youngjun Park <youngjun.park@lge.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Zhiling Zou
|
a44730dd05 |
mm: shmem: reject page-aligned fallocate end overflow
shmem_fallocate() validates offset + len with inode_newsize_ok(), but then
rounds that end offset up to a page boundary before entering the
preallocation loop.
For a valid request ending at MAX_LFS_FILESIZE, such as offset = 0 and len
= LLONG_MAX, adding PAGE_SIZE - 1 to the validated end can overflow the
signed loff_t used for the rounded end calculation. If that wrapped value
is then converted into a page index, shmem_fallocate() can enter the folio
allocation loop with an invalid range.
Use check_add_overflow() when calculating the page-aligned end, and fail
before entering the allocation loop if the rounded end cannot be
represented.
Link: https://lore.kernel.org/1929a466735dcbb9438936ff50b7a4fc2332a8a4.1785377919.git.zhilinz@nebusec.ai
Fixes:
|
||
|
Abhishek Bapat
|
33588e0b81 |
alloc_tag: add accuracy based filtering to ioctl
Extend the allocinfo filtering mechanism to allow users to filter tags based on their accuracy. [abhishekbapat@google.com: move `inaccurate` filtering criteria from `struct allocinfo_tag` to `struct allocinfo_filter`] Link: https://lore.kernel.org/e4e49ec4a5960292aeeb9e196526c18dc95228a2.1785867739.git.abhishekbapat@google.com Link: https://lore.kernel.org/396a5e4bc3b2990223ab355f2cd3ceb6aa15499e.1783532853.git.abhishekbapat@google.com Signed-off-by: Abhishek Bapat <abhishekbapat@google.com> Acked-by: Hao Ge <hao.ge@linux.dev> Acked-by: Suren Baghdasaryan <surenb@google.com> Cc: Jonathan Corbet <corbet@lwn.net> Cc: Kent Overstreet <kent.overstreet@linux.dev> Cc: Sourav Panda <souravpanda@google.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Abhishek Bapat
|
6f6769ea88 |
alloc_tag: add size-based filtering to ioctl
Extend the allocinfo filtering mechanism to allow users to filter tags based on the total number of bytes allocated [min_size, max_size]. The size range is inclusive. Filtering by size involves retrieving allocinfo per-CPU counters, which is an expensive operation. Hence, the performance of size-based filtering will be worse than other filters. Link: https://lore.kernel.org/0a7653b70ae0d64e967fbea0e933bc35f8ac656e.1783532853.git.abhishekbapat@google.com Signed-off-by: Abhishek Bapat <abhishekbapat@google.com> Acked-by: Hao Ge <hao.ge@linux.dev> Acked-by: Suren Baghdasaryan <surenb@google.com> Cc: Jonathan Corbet <corbet@lwn.net> Cc: Kent Overstreet <kent.overstreet@linux.dev> Cc: Sourav Panda <souravpanda@google.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Abhishek Bapat
|
5732a4e4c1 |
alloc_tag: add ioctl filters to /proc/allocinfo
Extend the capability of the IOCTL mechanism to filter allocations based on tag's module name, function name, file name and line number. Link: https://lore.kernel.org/6a6100c0c58cb2911f39126b9fe177a8c17db16f.1783532853.git.abhishekbapat@google.com Signed-off-by: Abhishek Bapat <abhishekbapat@google.com> Acked-by: Hao Ge <hao.ge@linux.dev> Acked-by: Suren Baghdasaryan <surenb@google.com> Cc: Jonathan Corbet <corbet@lwn.net> Cc: Kent Overstreet <kent.overstreet@linux.dev> Cc: Sourav Panda <souravpanda@google.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Suren Baghdasaryan
|
1d581ab234 |
alloc_tag: add ioctl to /proc/allocinfo
Patch series "alloc_tag: introduce IOCTL-based filtering for MAP", v8. Currently, memory allocation profiling data is primarily exposed through /proc/allocinfo. While useful for manual inspection, this text-based interface poses challenges for production monitoring and large-scale analysis: 1. Userspace must parse large amounts of text to extract specific fields. 2. To find specific tags, userspace must read the entire dataset, requiring many context switches and high data copying. 3. The kernel currently aggregates per-CPU counters for every allocation size, even those the user intends to filter out immediately. This series introduces a new IOCTL-based binary interface for allocinfo that supports kernel-side filtering. By allowing the user to specify a filter mask, we significantly reduce the work performed in-kernel and the amount of data transferred to userspace. The IOCTL mechanism was chosen for allocinfo to address the per-CPU counter aggregation bottleneck. A traditional read() operation must report the total allocation count and sizes for every code tag in the system. Doing so requires iterating across all CPUs to sum their per-CPU counters for thousands of tags, which introduces substantial runtime overhead. The IOCTL interface allows userspace to push selective filtering criteria directly into the kernel before the per-CPU counter aggregation. The kernel aggregates per-CPU counters only for a small subset of tags that match the filter. This results in significant performance improvement. Beyond fast filtered retrieval, the IOCTL foundation allows introducing a context capture mechanism in the future to capture the context for specific allocations. Performance measurements were conducted on an Intel Xeon Platinum 8481C (224 CPUs) with caches dropped before each run. The IOCTL mechanism shows a ~20x performance improvement for filtered queries. The kernel avoids the expensive per-CPU counter aggregation (alloc_tag_read) for any tags that fail the initial string or location filters. Scenario 1: Specific File Filtering (arch/x86/events/rapl.c) 1. Traditional (cat /proc/allocinfo | grep): 22ms (sys) 2. IOCTL Interface: 1ms (sys) Scenario 2: Compound Filtering (Filename + Size) 1. Traditional: (cat ... | grep | awk): 21ms (sys) 2. IOCTL Interface: 1ms (sys) Scenario 3: Size-Based Filtering (min_size = 1MB) 1. Traditional: (cat ... | awk): 21ms (sys) 2. IOCTL Interface: 14ms (sys) This patch (of 6): Add the following ioctl commands for /proc/allocinfo file: ALLOCINFO_IOC_CONTENT_ID - gets content identifier which can be used to check whether the file content has changed specifically due to module load/unload. Every time a module is loaded / unloaded, the returned value will be different. By comparing the identifier value at the beginning and at the end of the content retrieval operation, users can validate retrieved information for consistency. ALLOCINFO_IOC_GET_AT - gets the record at the specified position. This is the position of a record in /proc/allocinfo. ALLOCINFO_IOC_GET_NEXT - gets the record next to the last retrieved one. If no records were previously retrieved, returns the first record. Note, function file and module names often have the same prefixes, therefore when filtering for them, we compare the last 64 characters to minimize the chances of name collisions. [akpm@linux-foundation.org: include compat.h, per Suren] Closes: https://lore.kernel.org/oe-kbuild-all/202607091820.qbjlGhKK-lkp@intel.com/ Link: https://lore.kernel.org/cover.1783532853.git.abhishekbapat@google.com Link: https://lore.kernel.org/15596de2607ef13e7c77c6d74763f4ae992ec475.1783532853.git.abhishekbapat@google.com Signed-off-by: Suren Baghdasaryan <surenb@google.com> Signed-off-by: Abhishek Bapat <abhishekbapat@google.com> Acked-by: Hao Ge <hao.ge@linux.dev> Cc: Jonathan Corbet <corbet@lwn.net> Cc: Kent Overstreet <kent.overstreet@linux.dev> Cc: Sourav Panda <souravpanda@google.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Sourav Panda
|
34e0849142 |
mm/hugetlb_cma: support percentage-based hugetlb_cma reservation
Currently, hugetlb_cma reservation only supports absolute sizes (e.g., hugetlb_cma=2G or hugetlb_cma=0:1G,1:1G). This can be restrictive in heterogeneous environments or when deploying common kernel command lines across machines with different memory capacities. Add support for percentage-based hugetlb_cma reservation (e.g., hugetlb_cma=20% or hugetlb_cma=0:20%,1:10%). The percentage is calculated against the total memory (for global settings) or against the node-specific memory (for node-specific settings) using memblock APIs during early boot. Link: https://lore.kernel.org/20260807040003.2156630-1-souravpanda@google.com Signed-off-by: Sourav Panda <souravpanda@google.com> Acked-by: Usama Arif <usama.arif@linux.dev> Cc: David Hildenbrand <david@kernel.org> Cc: David Rientjes <rientjes@google.com> Cc: Frank van der Linden <fvdl@google.com> Cc: Greg Thelen <gthelen@google.com> Cc: Muchun Song <muchun.song@linux.dev> Cc: Oscar Salvador <osalvador@suse.de> Cc: Suren Baghdasaryan <surenb@google.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Mike Rapoport (Microsoft)
|
3774c56cc3 |
drivers/base, mm: move arch_numa.c to mm/
arch_numa.c implements boot time discovery and initialization of NUMA topology on architectures that select GENERIC_ARCH_NUMA (currently arm64 and riscv). Since this is step in the initialization of the memory management subsystem, it's logical to have arch_numa.c in mm/ alongside numa.c, numa_memblks.c and numa_emulation.c. Move arch_numa.c to mm/ and add its F: entry to "MEMBLOCK AND MEMORY MANAGEMENT INITIALIZATION" in MAINTAINERS. Link: https://lore.kernel.org/20260806-arch-numa-v1-1-968ec128121e@kernel.org Signed-off-by: Mike Rapoport (Microsoft) <rppt@kernel.org> Acked-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org> Acked-by: Lorenzo Stoakes (ARM) <ljs@kernel.org> Acked-by: Vlastimil Babka (SUSE) <vbabka@kernel.org> Acked-by: David Hildenbrand (Arm) <david@kernel.org> Acked-by: Danilo Krummrich <dakr@kernel.org> Cc: Albert Ou <aou@eecs.berkeley.edu> Cc: Alexandre Ghiti <alex@ghiti.fr> Cc: Catalin Marinas <catalin.marinas@arm.com> Cc: Liam R. Howlett <liam@infradead.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Palmer Dabbelt <palmer@dabbelt.com> Cc: "Rafael J. Wysocki" <rafael@kernel.org> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Will Deacon <will@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Hao Jia
|
6f34b4126b |
mm/zswap: support batch writeback in shrink_memcg()
Currently, shrink_memcg() writes back at most one entry per-node during
its traversal. This makes shrink_worker() inefficient, as it must
repeatedly re-enter shrink_memcg() to make any substantial progress.
Under high memory pressure, this can cause the writeback speed to be too
slow to keep up with refaults, leading to zswap store failures and forcing
pages to skip zswap and go directly to disk, which results in an LRU
inversion.
To address this, extend the per-node scan budget in shrink_memcg() from a
single entry to up to SWAP_CLUSTER_MAX pages, enabling batch writeback for
both the shrink_worker() and zswap_store() paths.
Test Setup:
- Total memory: 32 GB, 1 NUMA node.
- zswap settings: accept_threshold_percent=50, shrinker_enabled=N.
Test Case 1:
Set max_pool_percent=1, allocate 512MB of anonymous pages, and fill them
with random data (to avoid compression). Then, use cgroup memory.reclaim
to force a large amount of anonymous pages into zswap. At an interval of
2ms, allocate a 4K anonymous page where the first 4 bytes are random numbers
and the rest are zeros, and then trigger reclamation of this 4K page through
cgroup memory.reclaim. When the pool threshold is reached, shrink_memcg()
will be triggered.
The test data after running for 120s is as follows:
Baseline Patched
shrink_worker wakeups 5,363 169
shrink_memcg calls 11,373,201 350,703
written_back pages 40,212 40,241
zswap_store calls 161,190 163,753
store succeeded (ret=1) 102,743 117,183
store rejected (ret=0) 58,447 46,570
store reject rate ~36% ~28%
pool_limit_hit delta 55,826 33,760
pswpout 98,659 86,811
pswpin 2 0
Test Case 2:
We evaluated the following two sub-configurations using stress-ng inside
a cgroup capped at memory.max=1G for 120 seconds:
Test Case 2a (max_pool_percent=1): Continuously triggers the global
zswap pool limit, thereby waking up shrink_worker() to perform asynchronous
shrinking.
Test Case 2b (zswap.max=320M, max_pool_percent=50): Continuously triggers
the cgroup's zswap.max limit, thereby invoking synchronous shrinking.
Command executed for both setups:
bash -c 'echo $$ > /sys/fs/cgroup/zswaptest/cgroup.procs ; \
exec stress-ng --vm 4 --vm-bytes 4G --vm-keep --vm-method rand-set -t \
120s -q'
Test Case 2a (max_pool_percent=1):
Baseline Patched
shrink_worker wakeups 5,640 1,308
shrink_memcg calls 8,481,500 3,140,972
written_back pages 260 468,216
zswap_store calls 2,742,756 2,011,269
store succeeded (ret=1) 934,640 947,988
store rejected (ret=0) 1,808,116 1,063,281
store reject rate ~66% ~52%
pool_limit_hit delta 1,181,310 196,882
pswpout 1,808,376 1,531,497
pswpin 4,288,497 3,635,365
Test Case 2b (zswap.max=320M, max_pool_percent=50):
Baseline Patched
shrink_worker wakeups 0 0
shrink_memcg calls 687,608 54,002
written_back pages 639,176 846,663
zswap_store calls 1,224,222 1,228,548
store succeeded (ret=1) 992,816 1,208,123
store rejected (ret=0) 231,431 20,425
store reject rate ~19% ~2%
pool_limit_hit delta 0 0
pswpout 870,745 867,360
pswpin 1,707,823 1,216,814
Under identical workloads and runtimes, batched zswap shrinking exhibits a
significant reduction in both shrink_worker() wakeups and shrink_memcg()
calls. Furthermore, the sharp drop in both pswpin and zswap_store()
rejections demonstrates that batching zswap shrink operations effectively
mitigates zswap_store() failures caused by hitting the pool limit. This
significantly prevents pages from bypassing zswap and falling back
directly to disk, thereby reducing LRU inversion.
Link: https://lore.kernel.org/20260806070943.95542-3-jiahao.kernel@gmail.com
Signed-off-by: Hao Jia <jiahao1@lixiang.com>
Suggested-by: Yosry Ahmed <yosry@kernel.org>
Suggested-by: Johannes Weiner <hannes@cmpxchg.org>
Acked-by: Yosry Ahmed <yosry@kernel.org>
Acked-by: Nhat Pham <nphamcs@gmail.com>
Cc: Chengming Zhou <chengming.zhou@linux.dev>
Cc: Michal Hocko <mhocko@kernel.org>
Cc: Michal Koutný <mkoutny@suse.com>
Cc: Muchun Song <muchun.song@linux.dev>
Cc: Roman Gushchin <roman.gushchin@linux.dev>
Cc: Shakeel Butt <shakeel.butt@linux.dev>
Cc: Tejun Heo <tj@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
||
|
Hao Jia
|
dc8458f43f |
mm/zswap: fix global shrinker when memory cgroup is disabled
Patch series "mm/zswap: Fixes and improves the zswap shrink", v4.
This series fixes and improves the zswap global shrinker
(shrink_worker()): Patch 1: Fix missing global shrinker when memory cgroup
is disabled. Patch 2: Extend shrink_memcg() to support batch writeback
and thereby improving the writeback efficiency in the shrink_worker() and
zswap_store() paths.
This patch (of 2):
Zswap writeback when the global pool limit is hit fails when memory cgroup
is disabled. The pool remains full until it is organically drained by
swapins or memory freeing, leading to zswap store failures and pages
bypassing getting written directly to the backing swap device, causing LRU
inversion (hotter pages with higher fault latency).
This happens because mem_cgroup_iter() always returns NULL when memory
cgroups are disabled. As a result, the global shrinker shrink_worker()
repeatedly takes empty walks. After MAX_RECLAIM_RETRIES failed attempts,
the worker gives up without writing back any pages.
Therefore, when memory cgroup is disabled, fall through with the !memcg
branch and shrink the root memcg directly.
With memcg disabled, shrink_memcg() only returns -ENOENT when the root LRU
is empty, which means the total pages are already below thr. In the
absence of heavy concurrent zswap stores, the loop then safely bails out
via the zswap_total_pages() <= thr check; otherwise, it will resume
shrinking the memcg after processing the reschedule check. For any other
return value from shrink_memcg(), the loop is guaranteed to terminate,
either after MAX_RECLAIM_RETRIES failures or once the threshold is met.
This is a potential performance regression for people using zswap
without memcg that was introduced by the commit in "Fixes".
Link: https://lore.kernel.org/20260806070943.95542-1-jiahao.kernel@gmail.com
Link: https://lore.kernel.org/20260806070943.95542-2-jiahao.kernel@gmail.com
Fixes:
|
||
|
Ye Liu
|
54cc9b3847 |
mm: debug_page_alloc: fix NULL buf in debug_guardpage_minorder_setup
If the kernel command line includes "debug_guardpage_minorder" without an
equals sign (i.e., no value is provided), the early parameter parser
passes a NULL buf pointer to the setup function.
kstrtouint() does not perform a NULL check on its input and calls directly
into kstrtoull() which dereferences s[0] unconditionally, leading to a
NULL pointer dereference and early boot crash.
Additionally, the error path's pr_err("%s", buf) would also crash with a
NULL format argument.
Link: https://lore.kernel.org/20260806004556.2633049-1-ye.liu@linux.dev
Fixes:
|
||
|
Ye Liu
|
8be7c167be |
mm/show_mem: fix format string inconsistencies and type mismatches
Fix five format string issues in show_free_areas() and __show_mem(): 1-2. reserved_highatomic and free_highatomic: %luKB -> %lukB The uppercase "KB" is inconsistent with all other fields in the same output block and with /proc/meminfo convention. 3. local_pcp: %ukB -> %lukB with explicit (unsigned long) cast per_cpu_pages.count is int, so K(count) yields int. Using %u was a signed/unsigned mismatch. Cast to unsigned long and use %lu for consistency with all other K() usages in the file. 4. total pagecache pages: %ld -> %lu global_node_page_state() returns unsigned long. Using %ld is a signedness mismatch caught by gcc -Wformat-signedness. 5. hwpoisoned pages: %lu -> %ld atomic_long_read() returns long (signed). Using %lu is a signedness mismatch caught by gcc -Wformat-signedness. Verified with: make KCFLAGS="-Wformat -Wformat-signedness" mm/show_mem.o Link: https://lore.kernel.org/20260805021556.1908807-1-ye.liu@linux.dev Signed-off-by: Ye Liu <liuye@kylinos.cn> Acked-by: Vlastimil Babka (SUSE) <vbabka@kernel.org> Acked-by: Johannes Weiner <hannes@cmpxchg.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Muchun Song
|
e73aeb8a41 |
mm/sparse: keep mem_section_usage_size() internal
mem_section_usage_size() is only needed by sparsemem implementation code
after commit
|
||
|
Abhishek Bapat
|
afff109c2f |
alloc_tag: expose boot-time compression configuration
Currently, userspace has limited visibility into the exact active runtime
state of memory allocation profiling and its page extension compression
('sysctl.vm.mem_profiling={0|1|never}[,compressed]').
While reading the sysctl provides basic on/off status, it is currently
impossible for userspace to natively determine whether page-tag
compression was successfully enabled without scraping dmesg boot logs.
Add a new read-only sysctl representing how compression was configured
at boot time.
Link: https://lore.kernel.org/c795f8089f82841e8a6e00d7ca286da2b23aeb7b.1785950530.git.abhishekbapat@google.com
Signed-off-by: Abhishek Bapat <abhishekbapat@google.com>
Acked-by: Suren Baghdasaryan <surenb@google.com>
Cc: Hao Ge <hao.ge@linux.dev>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
||
|
Ye Liu
|
bb3e3c5c2d |
mm: debug_page_alloc: fix type mismatch for debug_guardpage_minorder
The debug_guardpage_minorder local variable is declared as unsigned int, but debug_guardpage_minorder_setup() uses unsigned long and kstrtoul() to parse the value. Use kstrtouint() with unsigned int local variable to match the actual type of _debug_guardpage_minorder. Also fix the format specifier from %lu to %u accordingly. Link: https://lore.kernel.org/20260805093108.2352900-1-ye.liu@linux.dev Signed-off-by: Ye Liu <liuye@kylinos.cn> Reviewed-by: Andrew Morton <akpm@linux-foundation.org> Cc: Johannes Weiner <hannes@cmpxchg.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Sang-Heon Jeon
|
0ddb8bb85b |
mm/page_ext: remove pgdat_page_ext_init()
pgdat_page_ext_init() sets pgdat->node_page_ext to NULL only on FLATMEM. FLATMEM depends on !NUMA, so the pgdat is always the zero-initialized contig_page_data and the store has no effect. So remove the call site, the unused function and its declaration. No functional change. Link: https://lore.kernel.org/20260804151145.3419768-3-ekffu200098@gmail.com Signed-off-by: Sang-Heon Jeon <ekffu200098@gmail.com> Acked-by: Zi Yan <ziy@nvidia.com> Acked-by: David Hildenbrand (Arm) <david@kernel.org> Cc: Johannes Weiner <hannes@cmpxchg.org> Cc: Liam R. Howlett <liam@infradead.org> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Sang-Heon Jeon
|
34568000f3 |
mm/Kconfig: make FLATMEM depend on !NUMA
Patch series "mm/page_ext: remove pgdat_page_ext_init()", v2. pgdat_page_ext_init() has no effect on FLATMEM. The pgdat is always the zero-initialized contig_page_data, because no architecture supports FLATMEM + NUMA. That constraint is only implicit in the arch Kconfig files. So patch 1 makes it explicit in mm/Kconfig, and patch 2 removes pgdat_page_ext_init(). No functional change. This patch (of 2): FLATMEM + NUMA is not supported by any architecture and fails to build. The constraint is only implicit in the arch Kconfig files. So make it explicit in mm/Kconfig. No functional change. Link: https://lore.kernel.org/20260804151145.3419768-1-ekffu200098@gmail.com Link: https://lore.kernel.org/20260804151145.3419768-2-ekffu200098@gmail.com Signed-off-by: Sang-Heon Jeon <ekffu200098@gmail.com> Suggested-by: Zi Yan <ziy@nvidia.com> Acked-by: Zi Yan <ziy@nvidia.com> Acked-by: David Hildenbrand (Arm) <david@kernel.org> Acked-by: Mike Rapoport (Microsoft) <rppt@kernel.org> Cc: Johannes Weiner <hannes@cmpxchg.org> Cc: Liam R. Howlett <liam@infradead.org> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Lorenzo Stoakes (ARM)
|
6fd3e592c0 |
mm: add some missing includes to mm-local headers
There are a number of internal headers local to mm/ which reference functions and data types without including the relevant headers. mm/vma.h is a special case that intentionally does not include additional headers, but the others are not. This breaks tooling like clangd (which is where I noticed this), though the build is OK due to the C files including the headers happening to include required dependencies. It's better to be explicit about dependencies anyway, so add the missing includes and fix clangd as a bonus. Link: https://lore.kernel.org/20260804-fix-some-local-headers-v1-1-a7beb173c116@kernel.org Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org> Acked-by: Vlastimil Babka (SUSE) <vbabka@kernel.org> Acked-by: David Hildenbrand (Arm) <david@kernel.org> Acked-by: Zi Yan <ziy@nvidia.com> Reviewed-by: Barry Song <baohua@kernel.org> Reviewed-by: Baoquan He <baoquan.he@linux.dev> Cc: Chris Li <chrisl@kernel.org> Cc: Johannes Weiner <hannes@cmpxchg.org> Cc: Kairui Song <kasong@tencent.com> Cc: Kemeng Shi <shikemeng@huaweicloud.com> Cc: Liam R. Howlett <liam@infradead.org> Cc: Michal Hocko <mhocko@kernel.org> Cc: Mike Rapoport <rppt@kernel.org> Cc: Muchun Song <muchun.song@linux.dev> Cc: Nhat Pham <nphamcs@gmail.com> Cc: Oscar Salvador <osalvador@suse.de> Cc: Roman Gushchin <roman.gushchin@linux.dev> Cc: Shakeel Butt <shakeel.butt@linux.dev> Cc: Suren Baghdasaryan <surenb@google.com> Cc: "Uladzislau Rezki (Sony)" <urezki@gmail.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Uladzislau Rezki (Sony)
|
b9183788a2 |
mm/vmalloc: do not warn on -ENOMEM from va_alloc()
Since vmalloc() accepts non-blocking GFP flags, allocation requests may fail when callers pass restrictive GFP masks. va_clip() may return -ENOMEM when its GFP_NOWAIT fallback allocation fails during NE_FIT_TYPE splitting. This is an expected failure, so va_alloc() should return the error without triggering a kernel splat. Link: https://lore.kernel.org/20260802104627.63892-1-urezki@gmail.com Signed-off-by: Uladzislau Rezki (Sony) <urezki@gmail.com> Reported-by: syzbot+61c997e6be1d9bb300ba@syzkaller.appspotmail.com Closes: https://lore.kernel.org/6a6d3cbd.6ce73036.24301b.000e.GAE@google.com Reviewed-by: Anshuman Khandual <anshuman.khandual@arm.com> Reviewed-by: Baoquan He <baoquan.he@linux.dev> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Hongfu Li
|
a44ab4bd1e |
ksm: update comments and docs to reference folio->mapping
The KSM code already stores and checks the stable node key via folio->mapping, but the comment in ksm_get_folio() and the reverse mapping documentation in ksm.rst still refer to page->mapping. This is a pure wording update to match the folio-based implementation. No functional change is intended. Link: https://lore.kernel.org/20260805105927.41987-1-hongfu.li@linux.dev Signed-off-by: Hongfu Li <lihongfu@kylinos.cn> Acked-by: David Hildenbrand (Arm) <david@kernel.org> Reviewed-by: Xu Xin <xu.xin16@zte.com.cn> Reviewed-by: Dongliang Mu <dzm91@hust.edu.cn> Cc: Alex Shi <alexs@kernel.org> Cc: Chengming Zhou <chengming.zhou@linux.dev> Cc: David Hildenbrand <david@kernel.org> Cc: Jonathan Corbet <corbet@lwn.net> Cc: Liam R. Howlett <liam@infradead.org> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Cc: Yanteng Si <si.yanteng@linux.dev> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Longlong Xia
|
3c37cac718 |
mm/ksm: avoid missing ksmd wakeups in ksm_enter
__ksm_enter() decides whether ksmd needs a wakeup by checking if the mm
slot list is empty before inserting the new slot.
The empty check is currently outside ksm_mmlist_lock. Another CPU can
remove the last slot and let ksmd go back to sleep after the unlocked
check, while this CPU inserts a new slot and skips the wakeup based on the
stale result.
Take ksm_mmlist_lock before checking the list so the empty-to-nonempty
transition and the insertion are observed as one critical section.
Link: https://lore.kernel.org/20260805132736.1063408-1-xialonglong2025@163.com
Fixes:
|
||
|
Jianlin Shi
|
8282cb36d0 |
mm/page_alloc: only update lowmem_reserve_ratio on sysctl write
lowmem_reserve_ratio_sysctl_handler() ignores the return value of proc_dointvec_minmax() and always calls setup_per_zone_lowmem_reserve(), even for read operations. Fix three issues: 1. Propagate errors from proc_dointvec_minmax() instead of always returning success. For example, writing non-integer garbage to the sysctl now returns an error instead of silently succeeding with unchanged values. 2. Only call setup_per_zone_lowmem_reserve() when the sysctl is actually written, matching the write-only refresh pattern of min_free_kbytes and watermark_scale_factor handlers. 3. On write, parse into a temporary ratio[] array and only copy into sysctl_lowmem_reserve_ratio[] and refresh derived state after the full vector is validated. This avoids leaving the ratio array partially updated while skipping setup when proc_dointvec_minmax() returns an error on a later element (suggested by Andrew Morton). Drop the manual "< 1 -> 0" sanitization loop and set .extra1 = SYSCTL_ZERO on the ctl_table entry so proc_dointvec_minmax() enforces the minimum on write; negative values now return -EINVAL instead of being silently coerced to 0 (suggested by Vlastimil Babka). [akpm@linux-foundation.org: add comment, per hannes] Link: https://lore.kernel.org/anSRGASe5FIrqwlg@cmpxchg.org Link: https://lore.kernel.org/linux-mm/tencent_FFD4F4D728AAE8A8AE0AF277A59854A29A06@qq.com/ Link: https://lore.kernel.org/tencent_A860C873956A52E26AD8D309A308A241BA08@qq.com Signed-off-by: Jianlin Shi <shijianlin11@foxmail.com> Acked-by: Vlastimil Babka (SUSE) <vbabka@kernel.org> Cc: Brendan Jackman <jackmanb@google.com> Cc: Johannes Weiner <hannes@cmpxchg.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Zi Yan <ziy@nvidia.com> Cc: Joel Granados <joel.granados@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Eric Biggers
|
c310a8932a |
mm/swap: reject swapon() on filesystem-level encrypted files
ext4 and f2fs don't prevent filesystem-level encrypted files from being set up directly as swap files. In this case, encryption is bypassed. No one should be doing this, vs. the methods of encrypted swap that actually do work (such as swapping to a dm-crypt device, or swapping to a loopback device on top of a filesystem-level encrypted file). Nevertheless, to prevent user error, make swapon() explicitly reject this case. Document this behavior in fscrypt.rst as well. Link: https://lore.kernel.org/20260803180426.3123-1-ebiggers@kernel.org Fixes: |
||
|
Artem Lytkin
|
272b0d84b1 |
mm/vmalloc: make vm_struct.nr_pages an unsigned long
vm_struct::nr_pages is an unsigned int, and the file keeps deriving byte counts from it as nr_pages << PAGE_SHIFT. A shift is evaluated in the type of its promoted left operand, so those are 32-bit arithmetic and wrap at 4 GiB of bytes, which is 2^20 pages. Every site depends on a cast being remembered; vmap() has one, two recent commits did not. vread_iter() then computes a size of zero for a 4 GiB VM_ALLOC area and /proc/kcore returns it as zeros while reporting a successful read, which drgn, crash or gdb cannot tell from real memory, and the vrealloc() grow-in-place check declines a request that would have fit. Widen the field so the class of bug goes away instead of one site at a time. Everything feeding or consuming it widens too: vm_area_alloc_pages() and its accumulators, nr_small_pages, new_nr_pages and old_nr_pages, the index range of vm_area_free_pages(), and three page indexes that were plain int. Five casts go. Two prints needed fixing as well, %u in vmalloc_dump_obj() and %d for the unsigned field in vmalloc_info_show(). No bug report behind this, I found it reading the code. The 4 GiB wrap needs only a machine with over 4 GiB of memory. Neither larger threshold is a practical concern: 2^32 pages, where the field itself truncates, is 16 TiB and beyond what hardware can populate, and 2^31, where the plain int indexes break, is 8 TiB and larger than anything in the tree asks for. The int *nr cursor in the mapping path is unchanged and is separate work. Users outside mm/vmalloc.c need no change either. Those handing the count to a narrower parameter cannot drive it near 2^31, and kho_preserve_vmalloc() stores it into a 32-bit ABI field that still receives the same low bits; above 2^32 pages the truncation just moves out of vm_struct into that store. sizeof(struct vm_struct) on x86-64 stays 72 bytes with CONFIG_HAVE_ARCH_HUGE_VMALLOC=n and goes from 72 to 80 with it enabled, both inside the kmalloc-96 bucket it already comes from. Link: https://lore.kernel.org/20260801114915.115224-1-iprintercanon@gmail.com Fixes: |
||
|
David Hildenbrand (Arm)
|
fc6415a384 |
mm/gup: factor out LRU cache draining for folio into lru_cache_drain_for_folio()
KVM with guest_memfd wants to remove any folio references due to LRU caches, as it really must only allow to convert folios from shared to private when there are no unexpected folio references (e.g., from GUP references). So, to drive the refcount down, it needs a way to flush the LRU caches. Let's factor out what we have in lru_cache_drain_for_folio(). Document it, and also mention that concurrent folio (un)mapping might, in theory, miss detecting LRU cache references. Keep obtaining the expected refcount twice to minimize the possibility. For the current and future user that should work, and we don't really have a better alternative: we could detect if the mapcount changed, but it would still be racy and add more complexity with questionable benefit. Maybe there is a chance to avoid the draining entirely in the future, by avoiding extra references from the LRU cache: Hugh thinks there might be a way. But for the time being, this handling is unfortunately required. Make folio_may_be_lru_cached() accept a const pointer so lru_cache_drain_for_folio() can accept a const pointer as well. Link: https://lore.kernel.org/20260806-lru_cache_drain_for_folio-v1-1-c6287d295e99@kernel.org Signed-off-by: David Hildenbrand (Arm) <david@kernel.org> Reviewed-by: Fuad Tabba <fuad.tabba@linux.dev> Cc: Ackerley Tng <ackerleytng@google.com> Cc: Baoquan He <baoquan.he@linux.dev> Cc: Barry Song <baohua@kernel.org> Cc: Chris Li <chrisl@kernel.org> Cc: Jason Gunthorpe <jgg@ziepe.ca> Cc: John Hubbard <jhubbard@nvidia.com> Cc: Kairui Song <kasong@tencent.com> Cc: Kemeng Shi <shikemeng@huaweicloud.com> Cc: Liam R. Howlett <liam@infradead.org> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Nhat Pham <nphamcs@gmail.com> Cc: Peter Xu <peterx@redhat.com> Cc: Sean Christopherson <seanjc@google.com> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
David Hildenbrand (Arm)
|
078e1a0fc4 |
mm/gup: fix always draining LRU caches in collect_longterm_unpinnable_folios()
folio_may_be_lru_cached() is currently only true for small folios, and for small folios FOLL_PIN adds GUP_PIN_COUNTING_BIAS references instead of 1 in try_grab_folio()/try_grab_folio_fast(). Consequently, our folio_ref_count(folio) != folio_expected_ref_count(folio) + 1 check in collect_longterm_unpinnable_folios() will currently always identify "reference mismatch" and first drain the local LRU cache to then drain the LRU cache on all CPUs, as collect_longterm_unpinnable_folios() is really called after pinning the folios with FOLL_PIN. Add a comment because the current code is not quite intuitive: we used to drain only to make sure the folio_isolate_lru() would succeed. But then we also started draining to make later migration more reliable. We'll refactor that code soon a bit, to also make it usable in other context where we really want to remove any references from LRU caches. Let's add CC stable, because having an easy way for excessive LRU cache draining on all CPUs does not sound right. In common scenarios we don't expect to ever have to drain. Link: https://lore.kernel.org/20260731-check_and_migrate_movable_folios-v1-1-e0002d7b791e@kernel.org Fixes: |
||
|
Zi Yan
|
c299a2285d |
mm/huge_memory: use folio's memcg inside __folio_split()
Patch series "Honor XA_FLAGS_ACCOUNT in xas_split_alloc() and charge to folio's memcg", v3. __GFP_ACCOUNT is needed for xarray node allocation accounting when XA_FLAGS_ACCOUNT is set. Commit |
||
|
Arvind Yadav
|
34a00895d0 |
mm/migrate_device: clear stale mapping after freeing swapcache
__migrate_device_pages() reads the folio mapping before calling
folio_free_swap(). When folio_free_swap() succeeds, the folio is removed
from the swap cache, but the saved mapping still points to swap_space.
Passing the stale mapping to folio_migrate_mapping() makes it use the
mapped-folio path for a folio that is no longer in swapcache. It can then
operate on swap_space.i_pages with invalid reference accounting,
eventually triggering a folio reference count BUG.
After a successful split, nr still contains the number of pages in the
original large folio, although each resulting page is now a separate
order-0 folio. Reset nr to 1 so each split folio is processed separately,
including its own swapcache removal and mapping lookup.
Refresh the saved mapping after folio_free_swap() so the current folio
state is used during migration.
Link: https://lore.kernel.org/20260728062832.1107127-1-arvind.yadav@intel.com
Fixes:
|
||
|
Wilson Felipe Pereira
|
7822fa5f4f |
mm: zswap: drop list_lru param from zswap_lru_add() and _del()
Since zswap_lru_add() and zswap_lru_del() are only called with the global zswap_list_lru, remove the redundant list_lru argument and use zswap_list_lru directly. Link: https://lore.kernel.org/20260731053721.1412304-1-wfelipe@google.com Signed-off-by: Wilson Felipe Pereira <wfelipe@google.com> Acked-by: Johannes Weiner <hannes@cmpxchg.org> Acked-by: Yosry Ahmed <yosry@kernel.org> Reviewed-by: SJ Park <sj@kernel.org> Acked-by: Nhat Pham <nphamcs@gmail.com> Cc: Chengming Zhou <chengming.zhou@linux.dev> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Sang-Heon Jeon
|
8380671909 |
mm/sparse: correct init section annotations
The !SPARSEMEM_EXTREME stub of sparse_index_init() has no annotation but the SPARSEMEM_EXTREME variant is __meminit. So mark the stub __meminit too. mminit_validate_memmodel_limits() is only called by memory_present(), which is __init. So mark it __init. sparse_usagebuf and sparse_usagebuf_end are only used by sparse_init_early_section(), sparse_usage_init() and sparse_usage_fini(), which are all __init. So mark them __initdata. Link: https://lore.kernel.org/20260731164758.1210668-1-ekffu200098@gmail.com Signed-off-by: Sang-Heon Jeon <ekffu200098@gmail.com> Reviewed-by: Andrew Morton <akpm@linux-foundation.org> Acked-by: David Hildenbrand (Arm) <david@kernel.org> Acked-by: Mike Rapoport (Microsoft) <rppt@kernel.org> Reviewed-by: Anshuman Khandual <anshuman.khandual@arm.com> Cc: Liam R. Howlett <liam@infradead.org> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Pratyush Mallick
|
7a39f03bc9 |
mm/page_reporting: add page_reporting_delay_ms module parameter
Free page reporting currently hardcodes a 2-second interval between reports. This rigid delay cannot accommodate diverse guest workloads. This patch introduces a module parameter, page_reporting_delay_ms (default: 2000), allowing users to tune the reporting rate: - Lower values enable aggressive memory reclamation by returning unused pages to the host immediately. - Higher values help batch pages during spiky allocation/free churn, reducing hypercalls and nested page fault overheads. Setting the delay to 0 is safe and execution is strictly gated by: - reporting is only triggered by high-order page frees. - expensive hypercalls are bounded by a slot capacity watermark check before proceeding. Link: https://lore.kernel.org/20260731193705.2902728-1-pratmal@google.com Signed-off-by: Pratyush Mallick <pratmal@google.com> Reviewed-by: SJ Park <sj@kernel.org> Acked-by: Lorenzo Stoakes (ARM) <ljs@kernel.org> Acked-by: David Hildenbrand (Arm) <david@kernel.org> Cc: Anshuman Khandual <anshuman.khandual@arm.com> Cc: Brendan Jackman <jackmanb@google.com> Cc: Greg Thelen <gthelen@google.com> Cc: Johannes Weiner <hannes@cmpxchg.org> Cc: Jonathan Corbet <corbet@lwn.net> Cc: Liam R. Howlett <liam@infradead.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: SeongJae Park <sj@kernel.org> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Guillaume Morin
|
5120b1e048 |
hugetlb: only adjust reservation during unmapping if mapcount is 0
Since |
||
|
Alexander Gordeev
|
62e39381b7 |
mm: use proper PTE accessor in move_ptes()
Follow the pattern established by commit |
||
|
Shakeel Butt
|
6b0d108336 |
memcg: bypass the reclaim and oom killer for dying tasks once oom_reaper is done
At Meta, we are seeing instances where an OOM killed job is stuck in the
exit path for several hours. In one particular case, the job was stuck
for more than 8 hours and I had to manually remove the memory.max limits
to allow the process to exit.
The job was a single process job and had ~55 GiB memory.max and zswap
enabled. It had almost 0 anon in memory and ~111 GiB in zswap compressed
to ~51 GiB zswap pool (i.e. almost all of memory.current was zswap).
Nothing was left on the LRUs to reclaim.
On further inspection, I observed ~20k threads of that process stuck with
the following stack:
[<0>] mem_cgroup_out_of_memory+0x4e/0xa0
[<0>] charge_memcg+0x8bf/0x990
[<0>] mem_cgroup_swapin_charge_folio+0x4e/0x80
[<0>] __read_swap_cache_async+0x10c/0x260
[<0>] swapin_readahead+0x116/0x3f0
[<0>] do_swap_page+0x13c/0x1ce0
[<0>] handle_mm_fault+0x61d/0x11f0
[<0>] do_user_addr_fault+0x3e7/0x6d0
[<0>] exc_page_fault+0x8f/0x110
[<0>] asm_exc_page_fault+0x22/0x30
[<0>] __get_user_8+0x14/0x20
[<0>] futex_cleanup+0x27/0x1c0
[<0>] futex_exit_release+0x47/0x60
[<0>] do_exit+0x107/0x940
[<0>] do_group_exit+0x81/0xa0
[<0>] get_signal+0x2b1/0x6e0
[<0>] arch_do_signal_or_restart+0x1a/0x1c0
[<0>] exit_to_user_mode_loop+0xa8/0x1c0
[<0>] do_syscall_64+0x152/0x250
[<0>] entry_SYSCALL_64_after_hwframe+0x4b/0x53
In addition the dmesg was filled with "Out of memory and no killable
processes..." messages.
I have no idea why oom reaper was not able to reap/unmap the process. My
guess is that since oom reaper tries to acquire mmap_lock in read mode
limited number of times and then gives up, there might be a thread of that
process which had mmap_lock in write mode at that time.
My initial suspicion was the futex_cleanup and kernel page fault causing
infinite fault and charge retries but that was put to rest in previous
discussions happened on similar problem [1].
My current theory is that it is just a simple slow serialization behind
the oom_lock. Unlike page allocator, memcg charge code takes the oom_lock
without the "try". Though memcg oom code uses mutex_lock_killable(), note
that in the call stack get_signal() consumes SIGKILL (or
sigdelset(SIGKILL)) before calling do_group_exit(). So this
mutex_lock_killable() is just a mutex_lock() here. Therefore 10s of
thousands of threads are waiting on oom_lock and one by one they get
-EFAULT from get_user() in the futex cleanup code and bails out.
Discussion from [1] led to commit
|
||
|
Guopeng Zhang
|
9477820c63 |
mm: memcg: stop reclaim when a limit update is superseded
kernfs serializes file operations only per open file, so separate open files can update the same memory.high or memory.max file concurrently. Both handlers store the new limit before synchronous reclaim, but continue to use the writer's local target in the reclaim loop. If another writer raises or removes the limit, the first writer can continue reclaiming toward a stale target. For memory.max, this can leave the writer looping indefinitely once reclaim retries are exhausted. The OOM path sees sufficient margin under the current limit and returns true without killing, while the writer still compares usage against its stale target and records another OOM event. Check the current limit at the start of each reclaim iteration and stop if it no longer matches the writer's target. Reproducer: Populate a cgroup with anonymous memory and disable swapping. Lower memory.max from one open file, then restore it to "max" through another open file after the new limit becomes visible. Without the patch, the first writer remains blocked and repeatedly increments the OOM event counter. With the patch, it returns normally. This was not motivated by a reported production workload. We found it through automated randomized testing for our cgroup observability work and reduced it to the reproducer above. Link: https://lore.kernel.org/20260724021805.1234583-1-guopeng.zhang@linux.dev Fixes: |
||
|
Usama Arif
|
7b9f4e5f81 |
mm/vmscan: reduce lru_lock contention via vmstat-derived scan-balance cost
The anon/file scan balance in get_scan_count() is driven by two scalars in
struct lruvec, anon_cost and file_cost, accumulated by every reclaim
producer under lruvec->lru_lock. The acquisition sites for cost work
specifically are:
- shrink_inactive_list() re-takes lru_lock at function exit purely
to call lru_note_cost_unlock_irq() with (nr_pageout, nr_scanned -
nr_reclaimed). One acquisition per inactive shrink.
- shrink_active_list() does the same with (0, nr_rotated). One
acquisition per active shrink.
- workingset_refault() takes the lock via folio_lruvec_lock_irq()
purely to record the refault cost. One acquisition per refault.
- prepare_scan_control() takes lru_lock just to snapshot the two
scalars into sc->{anon,file}_cost.
- lru_note_cost_unlock_irq() itself walks parent_lruvec and
re-acquires lru_lock on each ancestor to propagate the update,
adding O(memcg-depth) acquisitions per producer call.
This hurts because lru_lock is already a heavy contention point on
memory-heavy workloads: every isolate_lru_folios(), move_folios_to_lru()
and folio_add_lru() takes it. The cost work itself is trivial (two scalar
bumps and one comparison), but it contends with and causes contention for
actual LRU manipulation. The parent_lruvec() walk also multiplies
cost-update overhead by memcg hierarchy depth.
The balance formula for anon and file, respectively, is this:
cost = nr_io * SWAP_CLUSTER_MAX + nr_rotated
Instead of recording cost and running averaging logic directly when these
events occur, snapshot running vmstat counters once per reclaim cycle and
derive the balance from event deltas since the last run.
Use PGROTATE_* from the preceding patch for the rotation input.
WORKINGSET_RESTORE_* and NR_VMSCAN_WRITE provide the remaining event
counters. Charge NR_VMSCAN_WRITE through lruvec stats so all inputs can
be sampled per lruvec and aggregated through the memcg hierarchy. This is
overall cheaper and has fewer lock acquisition sites.
Moving accumulation and decay to the reclaim side also improves the cost
model across reclaim gaps. With producer-side decay, events that happen
while reclaim is idle still age each other before reclaim ever samples the
costs. If a workload refaults a large anon set and then a smaller file
set before reclaim runs again, the later file activity can age the earlier
anon activity out of the cost model. The new scheme observes the whole
between-reclaim delta and decays anon and file proportionally, so the
scan-balance history better represents what happened since the last
reclaim pass.
A dedicated per-lruvec spinlock, cost_lock, serialises the delta
extraction, the cost->count update and the halving loop against concurrent
reclaimers in the same memcg+node.
NR_VMSCAN_WRITE is accounted at writeout(), so reclaim_stat.nr_pageout is
no longer needed and is removed.
memcg-v1's memory.stat anon_cost/file_cost is now sourced from
cost[].count instead of the removed lruvec anon_cost/file_cost fields.
The reported values only refresh when prepare_scan_control() runs and are
bounded at ~lrusize/4 by the halving loop; the scan-balance signal they
express is unchanged.
Under pure MGLRU the scan-balance signal itself is not consumed (both
prepare_scan_control() and get_scan_count() are short-circuited on the
MGLRU paths, and MGLRU's own type/tier selection comes from
read_ctrl_pos() on lrugen->{avg_refaulted,avg_total,refaulted,evicted},
not from anon_cost/file_cost). NR_VMSCAN_WRITE naturally covers writeout
from either reclaim implementation. The preceding patch also bumps
PGROTATE_{ANON,FILE} from evict_folios(), so rotation-driven reclaim work
is accounted consistently across both implementations.
Link: https://lore.kernel.org/20260727162550.2032-4-usama.arif@linux.dev
Signed-off-by: Usama Arif <usama.arif@linux.dev>
Acked-by: Shakeel Butt <shakeel.butt@linux.dev>
Acked-by: Johannes Weiner <hannes@cmpxchg.org>
Acked-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Cc: Axel Rasmussen <axelrasmussen@google.com>
Cc: Baoquan He <baoquan.he@linux.dev>
Cc: Chris Li <chrisl@kernel.org>
Cc: David Hildenbrand <david@kernel.org>
Cc: David Rientjes <rientjes@google.com>
Cc: Kairui Song <kasong@tencent.com>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Muchun Song <muchun.song@linux.dev>
Cc: Nhat Pham <nphamcs@gmail.com>
Cc: Roman Gushchin <roman.gushchin@linux.dev>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Wei Xu <weixugc@google.com>
Cc: Yuanchu Xie <yuanchu@google.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
||
|
Usama Arif
|
1b089def0f |
mm/vmscan: add pgrotate_anon and pgrotate_file vmstat counters
Reclaim can spend substantial work on an LRU type without immediately
reclaiming or demoting a corresponding amount of memory. Record this work
in PGROTATE_ANON and PGROTATE_FILE.
For classic LRU reclaim:
- Inactive-list reclaim adds nr_scanned - nr_reclaimed to the
corresponding anon/file counter when isolation succeeds.
- Active-list reclaim adds referenced executable file folios that
are retained on the active list to PGROTATE_FILE. Active anon
reclaim does not contribute this component.
For MGLRU, add the number of initially isolated pages that remain
unreclaimed after both the initial and retry passes to the counter for the
selected anon/file type.
These counters are distinct from the existing pgrotated vm event.
pgrotated records an actual move to the inactive-list tail, primarily
after reclaim-marked writeback completes or failed invalidation leaves a
folio for accelerated reclaim. PGROTATE_ANON and PGROTATE_FILE measure
reclaim cost and do not imply that a folio moved to an LRU tail.
A subsequent patch will consume these counters for anon/file scan
balancing.
Link: https://lore.kernel.org/20260727162550.2032-3-usama.arif@linux.dev
Signed-off-by: Usama Arif <usama.arif@linux.dev>
Acked-by: Shakeel Butt <shakeel.butt@linux.dev>
Acked-by: Johannes Weiner <hannes@cmpxchg.org>
Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Cc: Axel Rasmussen <axelrasmussen@google.com>
Cc: Baoquan He <baoquan.he@linux.dev>
Cc: Chris Li <chrisl@kernel.org>
Cc: David Hildenbrand <david@kernel.org>
Cc: David Rientjes <rientjes@google.com>
Cc: Kairui Song <kasong@tencent.com>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Muchun Song <muchun.song@linux.dev>
Cc: Nhat Pham <nphamcs@gmail.com>
Cc: Roman Gushchin <roman.gushchin@linux.dev>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Wei Xu <weixugc@google.com>
Cc: Yuanchu Xie <yuanchu@google.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|