mirror of
https://github.com/torvalds/linux.git
synced 2026-10-17 06:20:02 +02:00
master
27352 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Linus Torvalds
|
18fbf5151d |
mm.git review status for linus..mm-stable
Everything: Total patches: 171 Reviews/patch: 1.83 Reviewed rate: 82% Excluding selftests: Total patches: 149 Reviews/patch: 1.77 Reviewed rate: 80% Excluding selftests and maple_tree: Total patches: 129 Reviews/patch: 1.99 Reviewed rate: 89% Summary of patch series in this merge: - "mm/rmap: index MAP_PRIVATE file-backed folios by anonymous pgoff" (Lorenzo Stoakes): Index MAP_PRIVATE file-backed folios by their anonymous page offset to resolve confusion around reverse mapping for zeroed and CoW'd file-backed memory. Use this new VMA anonymous page offset tracking to eliminate index conflicts and lay the foundation for scalable CoW performance improvements. - "promote mapped executable folios after first usage for MGLRU" (Baolin Wang): Make MGLRU's protection of mapped executable file folios more reliable. Follow the classical LRU's logic, promoting mapped executable file folios after their first usage to give executable code a better chance to stay in memory and improve workload performance. - "mm: vmscan: fix node reclaim ignoring swappiness parameter" (Ridong Chen): Fix per-node proactive reclaim interface's ignoring the swappiness parameter when CONFIG_MEMCG is disabled by consolidating sc_swappiness() into a single function that checks proactive_swappiness regardless of kernel configuration. - "mm/vmscan: reduce lru_lock contention via vmstat-derived scan-balance cost" (Usama Arif): Reduce lru_lock contention in the reclaim path by deriving scan-balance costs from vmstat counters rather than lock-acquired producer updates. Read and decay these cost signals on the reclaim side under a dedicated per-lruvec lock, reducing total LRU lock wait time by over 60% without impacting scan throughput. - "zram: fix zram issues reported by sashiko" (Sergey Senozhatsky): Fix two low-risk zram bugs which Sashiko spotted in drive-by review. - "Honor XA_FLAGS_ACCOUNT in xas_split_alloc() and charge to folio's memcg" (Zi Yan): Fix xas_split_alloc() by enabling target folio memcg charging during splits and adding the missing __GFP_ACCOUNT flag for proper XArray node memory accounting. - "selftests/mm: use pattern matching in .gitignore" (Pratyush Mallick): Replace hardcoded binary names in selftests/mm/.gitignore with a generic pattern-matching rule to automatically ignore generated test files and avoid manual updates when adding new tests. - "mm/page_ext: remove pgdat_page_ext_init()" (Sang-Heon Jeon): Make the incompatibility between FLATMEM and NUMA explicit in mm/Kconfig and remove the unused pgdat_page_ext_init() function. - "zram: fix zstd error paths and add parameter validation" (Haoqin Huang): Clean up zram compression backends by removing redundant error cleanup, adding parameter and dictionary validation, auto-prefixing algorithm error logs, and resetting parameters prior to reinitialization. - "zram: fix stale scan bounds after reinitialization" (Longlong Xia): Prevent out-of-bounds slot accesses during concurrent zram resets by moving table scan bound calculations under dev_lock in writeback_store() and read_block_state(). - "add anon mTHP collapse test cases" (Baolin Wang): Extend selftests helper functions to support arbitrary page orders and add new test cases and options for mTHP collapse in khugepaged. - "selftests/mm: Handle unsupported and transient test conditions" (Muhammad Usama Anjum): Update MM selftests to report a SKIP status instead of a failure when required kernel or filesystem features are unsupported, while adding retry logic for transient page migration errors. - "mm/zswap: Fixes and improves the zswap shrink" (Hao Jia): Fix the missing zswap global shrinker when CONFIG_MEMCG is disabled and extend shrink_memcg() to support batch writeback for improved writeback efficiency. - "alloc_tag: introduce IOCTL-based filtering for MAP" (Suren Baghdasaryan): Introduce an IOCTL-based binary interface for memory allocation profiling that enables kernel-side filtering before per-CPU counter aggregation. This eliminates the text-parsing overhead of /proc/allocinfo and provides up to a 20x speedup by transferring only filtered allocation data to userspace. - "better block swap batching and a different take on swap_ops v5" (Christoph Hellwig): Refactor block swap I/O to use swap_iocb for batching instead of single-bio requests and rebase the swap_ops interface, achieving faster swap throughput during kernel builds. - "mm: kmemleak: reduce transient false positives by confirming leaks" (Catalin Marinas): Reduce false-positive kmemleak reports by combining two kmemleak enhancements that add a second confirmation scan and a configurable minimum unreferenced scan count module parameter. - "mm: kmemleak: default min_unref_scans to 2 for verbose kernels" (Breno Leitao): Auto-scanning kernels can generate false-positive memory leak reports on single scans, so this patch defaults min_unref_scans to 2 when CONFIG_DEBUG_KMEMLEAK_VERBOSE is enabled to require a second confirming scan. - "swap_ops updates" (Christoph Hellwig): Batching I/O for synchronous swap devices causes performance regressions and filesystem-based swap suffers from double-indirection overhead. This series resolves both issues by reintroducing per-folio writes for synchronous swap and allowing filesystems to directly export their own swap_ops. - "mm/khugepaged: several cleanups" (Nico Pache): khugepaged accumulated redundant state-checking patterns and outdated comments following mTHP integration. Introduce dedicated helpers for PTE validation and event counting while refreshing the internal documentation. - "maple_tree: lock checking and clean ups" (Liam Howlett): Syzbot reports incorrectly blame memory management exit paths for locking bugs, maple tree erase operations risk allocation failures without gfp flags and internal documentation lacks clarity. Improve lock error detection, update docs, fix race and allocation edge cases and optimize erase allocations using a fallback to GFP_KERNEL | GFP_NOFAIL. -----BEGIN PGP SIGNATURE----- iHUEABYKAB0WIQTTMBEPP41GrTpTJgfdBJ7gKXxAjgUCao9nJQAKCRDdBJ7gKXxA jk/9AQDlfevYJuSJmzAI8bt8ISG+/TfXMtIZC/MdbHqtQVYWPQD8Cvm3DUZsdGB/ Gloq/HBFuMPgE8p2pwUIthdgnTPNvAc= =c+Nb -----END PGP SIGNATURE----- Merge tag 'mm-stable-2026-08-26-15-22' of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm Pull more MM updates from Andrew Morton: - "mm/rmap: index MAP_PRIVATE file-backed folios by anonymous pgoff" (Lorenzo Stoakes) Index MAP_PRIVATE file-backed folios by their anonymous page offset to resolve confusion around reverse mapping for zeroed and CoW'd file-backed memory. Use this new VMA anonymous page offset tracking to eliminate index conflicts and lay the foundation for scalable CoW performance improvements. - "promote mapped executable folios after first usage for MGLRU" (Baolin Wang) Make MGLRU's protection of mapped executable file folios more reliable. Follow the classical LRU's logic, promoting mapped executable file folios after their first usage to give executable code a better chance to stay in memory and improve workload performance. - "mm: vmscan: fix node reclaim ignoring swappiness parameter" (Ridong Chen) Fix per-node proactive reclaim interface's ignoring the swappiness parameter when CONFIG_MEMCG is disabled by consolidating sc_swappiness() into a single function that checks proactive_swappiness regardless of kernel configuration. - "mm/vmscan: reduce lru_lock contention via vmstat-derived scan-balance cost" (Usama Arif) Reduce lru_lock contention in the reclaim path by deriving scan-balance costs from vmstat counters rather than lock-acquired producer updates. Read and decay these cost signals on the reclaim side under a dedicated per-lruvec lock, reducing total LRU lock wait time by over 60% without impacting scan throughput. - "zram: fix zram issues reported by sashiko" (Sergey Senozhatsky) Fix two low-risk zram bugs which Sashiko spotted in drive-by review. - "Honor XA_FLAGS_ACCOUNT in xas_split_alloc() and charge to folio's memcg" (Zi Yan) Fix xas_split_alloc() by enabling target folio memcg charging during splits and adding the missing __GFP_ACCOUNT flag for proper XArray node memory accounting. - "selftests/mm: use pattern matching in .gitignore" (Pratyush Mallick) Replace hardcoded binary names in selftests/mm/.gitignore with a generic pattern-matching rule to automatically ignore generated test files and avoid manual updates when adding new tests. - "mm/page_ext: remove pgdat_page_ext_init()" (Sang-Heon Jeon) Make the incompatibility between FLATMEM and NUMA explicit in mm/Kconfig and remove the unused pgdat_page_ext_init() function. - "zram: fix zstd error paths and add parameter validation" (Haoqin Huang) Clean up zram compression backends by removing redundant error cleanup, adding parameter and dictionary validation, auto-prefixing algorithm error logs, and resetting parameters prior to reinitialization. - "zram: fix stale scan bounds after reinitialization" (Longlong Xia) Prevent out-of-bounds slot accesses during concurrent zram resets by moving table scan bound calculations under dev_lock in writeback_store() and read_block_state(). - "add anon mTHP collapse test cases" (Baolin Wang) Extend selftests helper functions to support arbitrary page orders and add new test cases and options for mTHP collapse in khugepaged. - "selftests/mm: Handle unsupported and transient test conditions" (Muhammad Usama Anjum) Update MM selftests to report a SKIP status instead of a failure when required kernel or filesystem features are unsupported, while adding retry logic for transient page migration errors. - "mm/zswap: Fixes and improves the zswap shrink" (Hao Jia) Fix the missing zswap global shrinker when CONFIG_MEMCG is disabled and extend shrink_memcg() to support batch writeback for improved writeback efficiency. - "alloc_tag: introduce IOCTL-based filtering for MAP" (Suren Baghdasaryan) Introduce an IOCTL-based binary interface for memory allocation profiling that enables kernel-side filtering before per-CPU counter aggregation. This eliminates the text-parsing overhead of /proc/allocinfo and provides up to a 20x speedup by transferring only filtered allocation data to userspace. - "better block swap batching and a different take on swap_ops v5" (Christoph Hellwig) Refactor block swap I/O to use swap_iocb for batching instead of single-bio requests and rebase the swap_ops interface, achieving faster swap throughput during kernel builds. - "mm: kmemleak: reduce transient false positives by confirming leaks" (Catalin Marinas) Reduce false-positive kmemleak reports by combining two kmemleak enhancements that add a second confirmation scan and a configurable minimum unreferenced scan count module parameter. - "mm: kmemleak: default min_unref_scans to 2 for verbose kernels" (Breno Leitao) Auto-scanning kernels can generate false-positive memory leak reports on single scans, so this patch defaults min_unref_scans to 2 when CONFIG_DEBUG_KMEMLEAK_VERBOSE is enabled to require a second confirming scan. - "swap_ops updates" (Christoph Hellwig) Batching I/O for synchronous swap devices causes performance regressions and filesystem-based swap suffers from double-indirection overhead. This series resolves both issues by reintroducing per-folio writes for synchronous swap and allowing filesystems to directly export their own swap_ops. - "mm/khugepaged: several cleanups" (Nico Pache) khugepaged accumulated redundant state-checking patterns and outdated comments following mTHP integration. Introduce dedicated helpers for PTE validation and event counting while refreshing the internal documentation. - "maple_tree: lock checking and clean ups" (Liam Howlett) Syzbot reports incorrectly blame memory management exit paths for locking bugs, maple tree erase operations risk allocation failures without gfp flags and internal documentation lacks clarity. Improve lock error detection, update docs, fix race and allocation edge cases and optimize erase allocations using a fallback to GFP_KERNEL | GFP_NOFAIL. * tag 'mm-stable-2026-08-26-15-22' of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm: (172 commits) selftests/proc: make proc-maps-race work with READ_IMPLIES_EXEC memcg: move LRU size accounting on reparenting instead of copying it mm/vmscan: fix comment logic in balance_pgdat maple_tree: add helper mas_make_walkable() maple_tree: avoid extra gap calculation maple_tree: fix argument name in header maple_tree: change two GFP flags in tests maple_tree: document erase and allocations better maple_tree: avoid mas_erase() and mtree_erase() failures maple_tree: document that erase may use GFP_KERNEL for allocations maple_tree: catch race in mas_alloc_cyclic() maple_tree: add bulk parent set helper maple_tree: micro optimisation of mas_wr_store_type() maple_tree: optimise mas_wr_node_store() when not in rcu mode maple_tree: use prefetched value in mas_wr_store_type() maple_tree: clarify comments on mas_nomem() maple_tree: drop MAPLE_ALLOC_SLOTS maple_tree: drop dead code from mas_extend_spanning_null() maple_tree: documentation fix maple_tree: add write lock checking with lockdep sequence numbers ... |
||
|
Shakeel Butt
|
0e0ac326c5 |
memcg: move LRU size accounting on reparenting instead of copying it
When a memory cgroup is offlined its LRU folios are reparented to the parent. lruvec_reparent_lru() splices the child's lists into the parent's and credits the parent with the child's per-zone lru_zone_size[], but never clears the child's copy, so the size is copied rather than moved. lru_gen_reparent_memcg() does the same for MGLRU. The parent is left correct, credited with exactly the folios it took over. The stale value sits on the child and nothing will correct it: folio->memcg_data now resolves to the parent, so every later update_lru_size() for those folios goes there. Dying cgroups are not freed immediately and mem_cgroup_iter() still walks them, so shrink_lruvec() keeps being called on them. get_scan_count() reads the phantom counter through lruvec_lru_size() and the scan loop then grinds through nr[] in SWAP_CLUSTER_MAX steps against an empty list, for as long as the dead cgroup lives. Under MGLRU the MGLRU scanner runs instead, but count_shadow_nodes() sums all of NR_LRU_LISTS through lruvec_lru_size() and over-budgets the shadow node limit just the same. On one 251 GiB host a sweep of every mz->lru_zone_size[] found 380 counters describing folios on no list at all: 124777314 pages, 476 GiB, 1.89x the machine's RAM, across 57 cgroups. All were on memcgs with CSS_DYING set and CSS_ONLINE clear, and parent/child pairs reported byte-identical sizes. LRU_UNEVICTABLE needs its size moved too. Its list is deliberately not spliced because lruvec_init() poisons the head - the unevictable LRU is imaginary and folios are never threaded on it - but the size is kept by lruvec_add_folio()/lruvec_del_folio() and those folios account to the parent from here on. This depends on commit |
||
|
Enlin Mu
|
5d3fe91b70 |
mm/vmscan: fix comment logic in balance_pgdat
In balance_pgdat(), when the low watermark is met, processes sleeping on pfmemalloc_wait are woken up because they are able to safely make forward progress. However, the comment incorrectly states "they should not be able", which contradicts the actual code behavior. Fix this typo to accurately reflect the logic. Link: https://lore.kernel.org/20260821064057.4081-1-enlin.mu@linux.dev Signed-off-by: Enlin Mu <enlin.mu@unisoc.com> Signed-off-by: Enlin Mu <enlin.mu@linux.dev> Reviewed-by: Barry Song <baohua@kernel.org> Acked-by: Johannes Weiner <hannes@cmpxchg.org> Acked-by: Shakeel Butt <shakeel.butt@linux.dev> Cc: Axel Rasmussen <axelrasmussen@google.com> Cc: David Hildenbrand <david@kernel.org> Cc: Kairui Song <kasong@tencent.com> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Michal Hocko <mhocko@kernel.org> Cc: Wei Xu <weixugc@google.com> Cc: Yuanchu Xie <yuanchu@google.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Breno Leitao
|
f9dc428249 |
mm, swap: ratelimit bad swap entry reports
A corrupt page table hands the same bogus entry to get_swap_device() on
every access to the mapping, and every rejection is logged. One machine
logged 6185620 copies of the same line in a few hours.
swap_dup_entry_direct() prints the same message from the fork path, once
per call: the WARN_ON_ONCE() guarding it warns once, the pr_err() inside
does not.
Rate limit all three prints.
Link: https://lore.kernel.org/20260818-swap_part_one-v1-1-a4fc58119fc0@debian.org
Fixes:
|
||
|
Song Hu
|
48863da10b |
mm: memcg: release the css reference when a stock slot empties
consume_stock() can drive a stock slot's nr_pages to zero while its
cached[] pointer stays set, so the slot keeps pinning the css reference
that refill_stock() took. The offlining drain only flushes slots with
cached pages, so the reference is never released unless the slot happens
to be displaced by an unrelated charge or by CPU hotplug, and the memcg
lingers in the dying state - up to NR_MEMCG_STOCK (7) of them per CPU
under container churn.
Keeping the slot populated past the last page only saves a
css_get()/css_put() pair on the next charge of the same memcg, and costs
more than that: the offlining drain has to know about empty slots, and
refill_stock() cannot reuse them either, so a charge under a different
memcg evicts a live batch through the drain_idx rotation instead.
Drop the reference in consume_stock() when the slot empties. Empty slots
stop existing, so is_memcg_drain_needed() and the drain path stay as they
are, and refill_stock() reuses emptied slots directly. The cost is one
refcount pair per emptied slot, at most once per MEMCG_CHARGE_BATCH pages.
Link: https://lore.kernel.org/20260818130135.154315-1-husong@kylinos.cn
Fixes:
|
||
|
Hui Su
|
9add2cc22d |
mm/migrate_device: fix cache flush when replacing huge zero PMD
migrate_vma_insert_huge_pmd_page() calls flush_cache_page() before
replacing an existing huge zero PMD. However, the third argument to
flush_cache_page() is a PFN, while addr + HPAGE_PMD_SIZE is an end virtual
address.
More importantly, the mapping being invalidated is PMD-sized rather than
PAGE_SIZE-sized. Flush the whole PMD range with flush_cache_range(),
matching other huge PMD invalidation paths.
There is no userspace-visible effect today. The architectures that
currently enable ARCH_ENABLE_THP_MIGRATION use no-op implementations of
flush_cache_page()/flush_cache_range(). 32-bit ARM has non-trivial
implementations, but does not enable ARCH_ENABLE_THP_MIGRATION.
So this appears to be a latent API misuse rather than a currently
observable bug, and I don't think a stable backport is necessary.
Link: https://lore.kernel.org/20260817060845.377800-2-sh_def@163.com
Fixes:
|
||
|
Nathan Chancellor
|
f2b1cb39d5 |
arch_numa: avoid false positive fortify warning in setup_node_to_cpumask_map()
When building ARCH=riscv using clang with CONFIG_FORTIFY_SOURCE and
CONFIG_UBSAN_BOUNDS enabled, CONFIG_NR_CPUS > 64, and the default value of
2 for CONFIG_NODES_SHIFT, there is a compiletime warning from the fortify
routines.
In file included from mm/arch_numa.c:11:
In file included from include/linux/acpi.h:14:
In file included from include/linux/resource_ext.h:11:
In file included from include/linux/slab.h:17:
In file included from include/linux/gfp.h:7:
In file included from include/linux/mmzone.h:8:
In file included from include/linux/spinlock.h:60:
In file included from include/linux/interrupt_rc.h:17:
In file included from include/linux/smp.h:13:
In file included from include/linux/cpumask.h:11:
In file included from include/linux/bitmap.h:13:
In file included from include/linux/string.h:383:
include/linux/fortify-string.h:430:4: warning: call to '__write_overflow_field' declared with 'warning' attribute: detected write beyond size of field (1st parameter); maybe use struct_group()? [-Wattribue-warning]
430 | __write_overflow_field(p_size_field, size);
| ^
include/linux/fortify-string.h:430:4: note: called by function 'fortify_memset_chk(unsigned long, unsigned long, unsigned long)'
include/linux/bitmap.h:248:3: note: inlined by function 'setup_node_to_cpumask_map'
248 | memset(dst, 0, len);
| ^
include/linux/fortify-string.h:462:25: note: expanded from macro 'memset'
462 | #define memset(p, c, s) __fortify_memset_chk(p, c, s, \
| ^
include/linux/fortify-string.h:453:2: note: expanded from macro '__fortify_memset_chk'
453 | fortify_memset_chk(__fortify_size, p_size, p_size_field), \
| ^
include/linux/fortify-string.h:430:4: note: use '-gline-directives-only' (implied by '-g1') or higher for more accurate inlining chain locations
430 | __write_overflow_field(p_size_field, size);
| ^
1 warning generated.
In this configuration, MAX_NUMNODES is 4. clang unrolls the for loop in
setup_node_to_cpumask_map() past this, which triggers the fortify check
when accessing node_to_cpumask_map on the theoretical fifth loop iteration
because it would be an out of bounds write.
Make it clear to clang that nr_node_ids is bounded by MAX_NUMNODES due to
the logic in setup_nr_node_ids() by early returning in
setup_node_to_cpumask_map() should that condition be violated.
Link: https://lore.kernel.org/20260813-arch_numa-avoid-fortify-warning-v2-1-093ad97a78df@kernel.org
Signed-off-by: Nathan Chancellor <nathan@kernel.org>
Closes: https://github.com/ClangBuiltLinux/linux/issues/2174
Reviewed-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Cc: Kees Cook <kees@kernel.org>
Cc: Bill Wendling <morbo@google.com>
Cc: Justin Stitt <justinstitt@google.com>
Cc: Nathan Chancellor <nathan@kernel.org>
Cc: Nick Desaulniers <ndesaulniers@google.com>
Cc: <stable@vger.kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
||
|
Eric Kim
|
f52b3b89fa |
mm/rmap: synchronize lock and unlock target in anon_vma_clone
Currently, in anon_vma_clone(), src vma's anon_vma is assigned to active_anon_vma and is used when unlocking anon_vma after linking new AVCs. However, the anon_vma is locked using src->anon_vma, instead of active_anon_vma, making the lock and unlock target inconsistent. Use active_anon_vma for both locking and unlocking. Link: https://lore.kernel.org/OS7PR01MB139142FE16EC63B892559D40496DA2@OS7PR01MB13914.jpnprd01.prod.outlook.com Signed-off-by: Eric Kim <seohyun.kim@outlook.kr> Reviewed-by: Lorenzo Stoakes (ARM) <ljs@kernel.org> Reviewed-by: Lance Yang <lance.yang@linux.dev> Cc: David Hildenbrand <david@kernel.org> Cc: Harry Yoo <harry@kernel.org> Cc: Jann Horn <jannh@google.com> Cc: Liam R. Howlett <liam@infradead.org> Cc: Rik van Riel <riel@surriel.com> Cc: Vlastimil Babka <vbabka@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Andrew Morton
|
556147fc27 |
mm/hmm.c:hmm_do_fault(): suppress sparse warning
mm/hmm.c:673 hmm_do_fault() error: we previously assumed 'hmm_vma_walk->locked' could be null (see line 654)
Stanislav says this can't happen. Waste a few cycles to make the warning
go away.
[akpm@linux-foundation.org: WARN_ON_ONCE() if the handler didn't set ->locked, per Stanislav]
Link: https://lore.kernel.org/anu1N-DOnQwxO1kF@skinsburskii
Fixes:
|
||
|
Xie Yuanbin
|
a1b114b4ce |
mm/Kconfig: make MEMORY_FAILURE select MIGRATION
For embedded devices, lacking support for NUMA, memory hotplug/hotremove, CMA and huge pages is a quite common scenario. In this scenario, the demand for contiguous physical memory allocation is very low. To reduce the kernel image size, some devices disable the compaction. However, their SoCs do support DDR ECC, meaning that memory-failure may be needed. Migration is very useful for soft_offline_page() in memory-failure, which may be triggered by correctable memory errors. Most anonymous and file-mapped faulty pages can be migrated to other healthy pages. Currently, MEMORY_FAILURE does not explicitly select MIGRATION. When COMPACTION, MEMORY_HOTREMOVE, NUMA_MIGRATION and CMA are all disabled, MEMORY_FAILURE can be enabled, but MIGRATION cannot be selected. Make MEMORY_FAILURE select MIGRATION to handle this situation. Link: https://lore.kernel.org/20260813134916.292733-1-xieyuanbin1@huawei.com Signed-off-by: Xie Yuanbin <xieyuanbin1@huawei.com> Suggested-by: Mike Rapoport <rppt@kernel.org> Reviewed-by: Lorenzo Stoakes (ARM) <ljs@kernel.org> Reviewed-by: Mike Rapoport (Microsoft) <rppt@kernel.org> Acked-by: Zi Yan <ziy@nvidia.com> Acked-by: David Hildenbrand (Arm) <david@kernel.org> Acked-by: Miaohe Lin <linmiaohe@huawei.com> Cc: Alistair Popple <apopple@nvidia.com> Cc: "Borislav Petkov (AMD)" <bp@alien8.de> Cc: Byungchul Park <byungchul@sk.com> Cc: David Hildenbrand <david@kernel.org> Cc: Gregory Price <gourry@gourry.net> Cc: "Huang, Ying" <ying.huang@linux.alibaba.com> Cc: Joshua Hahn <joshua.hahnjy@gmail.com> Cc: Liam R. Howlett <liam@infradead.org> Cc: liaohua <liaohua4@huawei.com> Cc: "Luck, Tony" <tony.luck@intel.com> Cc: Matthew Brost <matthew.brost@intel.com> Cc: Michal Hocko <mhocko@suse.com> Cc: Naoya Horiguchi <nao.horiguchi@gmail.com> Cc: Rakie Kim <rakie.kim@sk.com> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Cc: Yuanbin Xie <xieyuanbin1@huawei.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Tejun Heo
|
f525001b33 |
percpu: drop CONFIG_DEBUG_FORCE_WEAK_PER_CPU
alpha requires percpu variables in modules to be defined as weak so that the compiler generates GOT based external references for them. This puts two extra restrictions on percpu variable definitions. The symbol must be globally unique even when static and a static percpu variable can't be defined inside a function. DEBUG_FORCE_WEAK_PER_CPU exists to give generic code build coverage for these restrictions without building for alpha. MEM_ALLOC_PROFILING defines a static percpu counter at each allocation call site and thus can't be built with weak percpu definitions, so it depends on !DEBUG_FORCE_WEAK_PER_CPU. As allmodconfig enables DEBUG_FORCE_WEAK_PER_CPU, this knocks MEM_ALLOC_PROFILING out of allmodconfig build coverage. allmodconfig coverage for MEM_ALLOC_PROFILING is worth more than build coverage for restrictions which only matter to alpha module builds. Drop DEBUG_FORCE_WEAK_PER_CPU. Restriction violations will now show up only on alpha builds. Link: https://lore.kernel.org/178656406317.2437052.7257990869957704195@slm.duckdns.org Signed-off-by: Tejun Heo <tj@kernel.org> Reported-by: Andrew Morton <akpm@linux-foundation.org> Reviewed-by: Suren Baghdasaryan <surenb@google.com> Acked-by: Gabriele Monaco <gmonaco@redhat.com> [include/rv/da_monitor.h] Cc: Dennis Zhou <dennis@kernel.org> Cc: Kent Overstreet <kent.overstreet@linux.dev> Cc: Steven Rostedt <rostedt@goodmis.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Kairui Song
|
f7e698e326 |
mm/mglru: fix and remove redundant unevictable folio handling
sort_folio() has a shortcut for moving folios that are no longer evictable
but are still sitting on a generation list. However, this shortcut is
buggy. It does not follow the PG_lru usage convention, and it has a more
serious issue.
Unevictable folios are not threaded on lists[LRU_UNEVICTABLE], so that
folio->lru can be reused to hold folio->mlock_count (see the comment in
lruvec_init()). Hence lruvec_add_folio() skips the list_add() for them,
and every other place that turns a folio unevictable initialises
mlock_count explicitly: lru_add() sets it to 0, __mlock_folio() and
__mlock_new_folio() set it to !!folio_test_mlocked(folio). sort_folio()
sets nothing, and the lru_gen_del_folio() right above it may have already
poisoned folio->lru via list_del(), so mlock_count ends up aliasing
LIST_POISON2, which reads as 0x122, i.e. 290. The result is user
visible. On munlock, __munlock_folio() decrements that bogus count, finds
it still non-zero and bails out before clearing PG_mlocked, so the folio
remains unevictable and the Mlocked accounting stays inflated until the
folio is freed.
The shortcut also touches the LRU flags in the wrong order. It calls
lru_gen_del_folio() while PG_lru is still set, so a concurrent
folio_test_clear_lru() (e.g. compaction, folio_isolate_lru()) can succeed
on a folio that has already been taken off the generation list, which may
lead to unexpected behavior.
So fix it by isolating them as common folios and letting the generic
shrink path cull them. This matches the classical LRU behavior, and there
should be no visible effect on the generic eviction or isolation behavior.
There is no performance concern either, such a folio goes through this
once, and then it is off the generation lists for good.
Link: https://lore.kernel.org/20260812-mglru-mlock-fix-v2-1-a3fec5853c08@tencent.com
Fixes:
|
||
|
Nico Pache (Red Hat)
|
27490db7ec |
mm/khugepaged: unmap pte before releasing vma write lock
We are currently dropping the anon_vma write lock before unmapping the PTE. Although this is safe, due to us still holding the mmap_write_lock, its safer and less confusing to switch the order of these two operations. Link: https://lore.kernel.org/20260811-khugepaged_pte_refactor-v4-6-ddac39d61c4a@linux.dev Signed-off-by: Nico Pache (Red Hat) <nico.pache@linux.dev> Suggested-by: David Hildenbrand <david@kernel.org> Acked-by: David Hildenbrand (Arm) <david@kernel.org> Reviewed-by: Zi Yan <ziy@nvidia.com> Reviewed-by: Baolin Wang <baolin.wang@linux.alibaba.com> Acked-by: Pedro Falcato <pfalcato@suse.de> Reviewed-by: Lorenzo Stoakes (ARM) <ljs@kernel.org> Reviewed-by: Lance Yang <lance.yang@linux.dev> Cc: Barry Song <baohua@kernel.org> Cc: Dev Jain <dev.jain@arm.com> Cc: Jonathan Corbet <corbet@lwn.net> Cc: Liam R. Howlett <liam@infradead.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Ryan Roberts <ryan.roberts@arm.com> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Usama Arif <usama.arif@linux.dev> Cc: Vlastimil Babka <vbabka@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Nico Pache (Red Hat)
|
cc044178ed |
mm/khugepaged: fix outdated comments
Fix comment in collapse_scan_pmd() that still described the old folio_mapcount() > folio_ref_count() check and a "512" false-positive scenario. The code now uses folio_expected_ref_count() != folio_ref_count() which doesn't suffer from the same limitation. Fix comment in collapse_huge_page() that referenced ptep_clear_flush, when the code actually uses pmdp_collapse_flush. Fix comment in __collapse_huge_page_swapin() that referenced the old function name khugepaged_scan_pmd, now collapse_scan_pmd. Also clean up some simple typos and stale terminology (mmap_sem -> mmap_lock, PG_lock -> folio lock, page -> folio, grammar). We also clarify a comment regarding where the max_ptes_none check is deferred to in mthp_collapse() from the original collapse_scan_pmd check. Update all comments that references a function to include parentheses. [nico.pache@linux.dev: fix outdated comments] Link: https://lore.kernel.org/1c96e2f3-802f-472b-81e6-4af17a721a3c@linux.dev Link: https://lore.kernel.org/20260811-khugepaged_pte_refactor-v4-4-ddac39d61c4a@linux.dev Signed-off-by: Nico Pache (Red Hat) <nico.pache@linux.dev> Acked-by: Usama Arif <usama.arif@linux.dev> Assisted-by: Cursor(claude-sonnet-4):4.6 Acked-by: David Hildenbrand (Arm) <david@kernel.org> Reviewed-by: Zi Yan <ziy@nvidia.com> Acked-by: Pedro Falcato <pfalcato@suse.de> Reviewed-by: Lorenzo Stoakes (ARM) <ljs@kernel.org> Reviewed-by: Lance Yang <lance.yang@linux.dev> Cc: Baolin Wang <baolin.wang@linux.alibaba.com> Cc: Barry Song <baohua@kernel.org> Cc: Dev Jain <dev.jain@arm.com> Cc: Jonathan Corbet <corbet@lwn.net> Cc: Liam R. Howlett <liam@infradead.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Ryan Roberts <ryan.roberts@arm.com> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Nico Pache (Red Hat)
|
948ec48e56 |
mm/khugepaged: introduce a count_collapse_event() helper
Provide a simple helper function to help reduce a often used, and duplicate pattern across the khugepaged code. When collapsing to a PMD we need to record a vm_event and the mTHP_stat event. When doing mTHP collapse we only update the mTHP stat. Link: https://lore.kernel.org/20260811-khugepaged_pte_refactor-v4-3-ddac39d61c4a@linux.dev Signed-off-by: Nico Pache (Red Hat) <nico.pache@linux.dev> Reviewed-by: Baolin Wang <baolin.wang@linux.alibaba.com> Acked-by: David Hildenbrand (Arm) <david@kernel.org> Acked-by: Usama Arif <usama.arif@linux.dev> Reviewed-by: Zi Yan <ziy@nvidia.com> Reviewed-by: Pedro Falcato <pfalcato@suse.de> Reviewed-by: Lorenzo Stoakes (ARM) <ljs@kernel.org> Reviewed-by: Lance Yang <lance.yang@linux.dev> Cc: Barry Song <baohua@kernel.org> Cc: Dev Jain <dev.jain@arm.com> Cc: Jonathan Corbet <corbet@lwn.net> Cc: Liam R. Howlett <liam@infradead.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Ryan Roberts <ryan.roberts@arm.com> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Nico Pache (Red Hat)
|
e8122742cf |
mm/khugepaged: extract reference check into folio_pte_referenced() helper
This change deduplicates the "is this PTE/folio referenced enough to be considered for a collapse" condition that was repeated in both __collapse_huge_page_isolate() and collapse_scan_pmd(), extracting it into a single inline helper function. Also move the comment and use it as the function header. While we are at it, updated the comment to clarify that a young pte is a recently accessed one. [nico.pache@linux.dev: drop the trivial helper kerneldoc and inline marker per review] Link: https://lore.kernel.org/9038f552-926b-4c4c-b023-69271f45e3d5@linux.dev Link: https://lore.kernel.org/20260811-khugepaged_pte_refactor-v4-2-ddac39d61c4a@linux.dev Signed-off-by: Nico Pache (Red Hat) <nico.pache@linux.dev> Acked-by: Usama Arif <usama.arif@linux.dev> Acked-by: David Hildenbrand (Arm) <david@kernel.org> Reviewed-by: Zi Yan <ziy@nvidia.com> Reviewed-by: Baolin Wang <baolin.wang@linux.alibaba.com> Reviewed-by: Lance Yang <lance.yang@linux.dev> Cc: Barry Song <baohua@kernel.org> Cc: Dev Jain <dev.jain@arm.com> Cc: Jonathan Corbet <corbet@lwn.net> Cc: Liam R. Howlett <liam@infradead.org> Cc: Lorenzo Stoakes (ARM) <ljs@kernel.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Ryan Roberts <ryan.roberts@arm.com> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Cc: Pedro Falcato <pfalcato@suse.de> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Nico Pache (Red Hat)
|
b86a7d03ea |
mm/khugepaged: refactor per-scan state clearing into collapse_control_init_scan()
Patch series "mm/khugepaged: several cleanups", v4. The following changes stem from a number of reviews during my khugepaged mTHP support series [1]. Some of these are minor code cleanups, issues or reviews that we decided to deferred to a followup series, or in the case of the more major patch of the series, changes [2] Lance Yang attempted while my series was in-flight and we decided to wait till later to try. The first 3 patches introduce helper functions to increase code reuse and readability. This includes a per-scan state clearing function, extracting the young page check into a helper, and a count_collapse_event() function to reduce a repetative pattern used across mTHP collapse. The 4th patch was the byproduct of me throwing Claude at all the comments in khugepaged verifying and looking for any outdated info. The 5th patch is based on Lance Yang's commit series [2] trying to extract the PTE state checking into a helper function. This required a bit of rewriting due to differences after mTHP collapse was introduced. I also took into account the changes requested during his patches review cycle. The remaining 2 patches were review points during my mTHP series that we agreed can be deferred to a later series. Thank you to those whos reviews and work I leveraged to achieve these cleanups. This patch (of 6): Extract the repeated clearing of node_load, alloc_nmask, and mthp_present_ptes into a helper to reduce duplication in collapse_scan_pmd() and collapse_scan_file(). Althought file scans do not current use the bitmap, they will in the future, and clearing it now is harmless. Link: https://lore.kernel.org/20260811-khugepaged_pte_refactor-v4-0-ddac39d61c4a@linux.dev Link: https://lore.kernel.org/20260811-khugepaged_pte_refactor-v4-1-ddac39d61c4a@linux.dev Link: https://lore.kernel.org/all/20260605161422.213817-1-npache@redhat.com/ [1] Link: https://lore.kernel.org/all/20251008043748.45554-1-lance.yang@linux.dev/ [2] Signed-off-by: Nico Pache (Red Hat) <nico.pache@linux.dev> Reviewed-by: Baolin Wang <baolin.wang@linux.alibaba.com> Acked-by: Usama Arif <usama.arif@linux.dev> Acked-by: David Hildenbrand (Arm) <david@kernel.org> Reviewed-by: Lorenzo Stoakes (ARM) <ljs@kernel.org> Reviewed-by: Zi Yan <ziy@nvidia.com> Reviewed-by: Pedro Falcato <pfalcato@suse.de> Reviewed-by: Lance Yang <lance.yang@linux.dev> Cc: Barry Song <baohua@kernel.org> Cc: Dev Jain <dev.jain@arm.com> Cc: Jonathan Corbet <corbet@lwn.net> Cc: Liam R. Howlett <liam@infradead.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Ryan Roberts <ryan.roberts@arm.com> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Hui Zhu
|
508537753b |
mm/mglru: fix young counter undercount for large folios
lru_gen_look_around() feeds its local 'young' counter into
suitable_to_scan(), which decides whether the current PMD is added to
the bloom filter and checked again on the next aging round.
The folio triggering the look-around is processed at function entry:
test_and_clear_young_ptes_notify() clears the accessed bits of the nr PTEs
it maps, and the function bails out if none of them is young. The loop
that follows therefore never recounts this folio, since its accessed bits
are already cleared. Every other young folio the loop finds is accounted
as a batch (young += nr), where nr is the number of consecutive PTEs it
maps. The triggering folio, however, still contributes a fixed young = 1
regardless of its size -- a leftover from before PTE batching. A large
triggering folio is thus accounted inconsistently with the rest of the
window.
Initialize young to nr so the triggering folio is accounted the same way
as any other young folio batch in the loop.
Note this is a deliberate overestimate, not a measured value. The
test-and-clear helper only reports whether any of the nr PTEs is young,
not how many were accessed, so the true number of accessed PTEs in a large
folio is unknown and can be smaller than nr. Counting the full batch is
intentional: the mm core tracks accessed/dirty state per folio, not per
page, so a per-page count is neither obtainable nor meaningful. The only
consumer is suitable_to_scan(), and the bloom filter it feeds tolerates
error. Overestimating is also the safe direction: at worst a PMD that saw
little access is rescanned, whereas underestimating could skip rescanning
a PMD whose folios are still hot and reclaim them incorrectly. (nr here
is the PTE batch size, not necessarily folio_nr_pages().)
Link: https://lore.kernel.org/20260813061019.49806-1-hui.zhu@linux.dev
Link: https://lore.kernel.org/20260812065933.103627-1-hui.zhu@linux.dev
Fixes:
|
||
|
Hui Su
|
bd1ad3cf07 |
kasan: fix quarantine_size accounting during cache removal
quarantine_size tracks the total number of bytes stored in
global_quarantine[]. It is incremented when per-CPU quarantine objects
are moved into the global quarantine and decremented when a global batch
is evicted by kasan_quarantine_reduce().
kasan_quarantine_remove_cache() also removes objects from the global
quarantine. qlist_move_cache() rebuilds the source batch and updates its
.bytes field, but quarantine_size is not adjusted accordingly.
As a result, quarantine_size remains over-counted by the size of the
removed objects. The stale accounting accumulates across cache removals.
Once the inflated value exceeds quarantine_max_size,
kasan_quarantine_reduce() can evict a batch even though the actual number
of bytes in global_quarantine[] is still below quarantine_max_size,
shortening the quarantine window.
Fix the accounting by recording each batch's size before
qlist_move_cache() and subtracting the number of bytes actually removed
from quarantine_size while holding quarantine_lock.
A KUnit reproducer used during testing observed the over-count grow by
4698864 bytes after one kasan_quarantine_remove_cache() call with the fix
reverted. With this change applied, the over-count did not grow.
Link: https://lore.kernel.org/20260811073332.1351893-1-sh_def@163.com
Fixes:
|
||
|
Christoph Hellwig
|
22779ae817 |
mm/swap: move swap_ops into file systems for file system-based swap
Currently swap to and from file systems goes through two indirect calls between the swap ops and the swap_rw method. Reduce this by directly providing the swap_ops from the file system. For this refactor swap_fs_submit into a swap_fs_prepare_rw helper that initializes the iov_iter on the callers stack so that file systems can call it directly, and use that to initialize file system specific ops in the NFS and SMB clients, which then get passed to swap_fs_activate. Link: https://lore.kernel.org/20260723054622.3460249-4-hch@lst.de Signed-off-by: Christoph Hellwig <hch@lst.de> Acked-by: Chris Li <chrisl@kernel.org> Cc: Baoquan He <baoquan.he@linux.dev> Cc: Kairui Song <kasong@tencent.com> Cc: Kairui Song <ryncsn@gmail.com> Cc: Kemeng Shi <shikemeng@huaweicloud.com> Cc: Nhat Pham <nphamcs@gmail.com> Cc: Steve French <sfrench@samba.org> Cc: Usama Arif <usama.arif@linux.dev> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Christoph Hellwig
|
52d85ca90c |
mm/swap: add a new swap_ops.h header to allow for pluggable swap ops
Add a new header to declare the swap_iocb, swap_ops and swap_ctx to allow for swap_ops implementations outside of mm/page_io.c. This will be used to remove the double indirection for file system-based swap. There is no functional change, just a move of the declarations. Note that there already is a swapops.h header, which is totally unrelated to struct swap_ops. The close naming is a bit unfortunate, but I could not think of a better name for this header. Link: https://lore.kernel.org/20260723054622.3460249-3-hch@lst.de Signed-off-by: Christoph Hellwig <hch@lst.de> Acked-by: Chris Li <chrisl@kernel.org> Cc: Baoquan He <baoquan.he@linux.dev> Cc: Kairui Song <kasong@tencent.com> Cc: Kairui Song <ryncsn@gmail.com> Cc: Kemeng Shi <shikemeng@huaweicloud.com> Cc: Nhat Pham <nphamcs@gmail.com> Cc: Steve French <sfrench@samba.org> Cc: Usama Arif <usama.arif@linux.dev> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Christoph Hellwig
|
3c5194eb2e |
mm/swap: revert to single-folio writes for synchronous swap devices
Patch series "swap_ops updates", v2. This series is a follow on to the swap ops series now in mm-unstable. The first patch reintroduces direct folio writes for synchronous swap files, the other two remove the double indirect for file system based swap. This patch (of 3): Kairui Song reported that zram benefits from submitting each folio directly instead of batching up I/O because the classic LRU scanning benefits from clearing the folio writeback bit in the scan loop. Accommodate that by kicking off writes for synchronous devices for each iteration. Link: https://lore.kernel.org/20260723054622.3460249-1-hch@lst.de Link: https://lore.kernel.org/20260723054622.3460249-2-hch@lst.de Signed-off-by: Christoph Hellwig <hch@lst.de> Acked-by: Usama Arif <usama.arif@linux.dev> Acked-by: Chris Li <chrisl@kernel.org> Cc: Baoquan He <baoquan.he@linux.dev> Cc: Kairui Song <kasong@tencent.com> Cc: Kairui Song <ryncsn@gmail.com> Cc: Kemeng Shi <shikemeng@huaweicloud.com> Cc: Nhat Pham <nphamcs@gmail.com> Cc: Steve French <sfrench@samba.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Longlong Xia
|
f7bf5cd5b5 |
zsmalloc: account for handle size in class lookup
zs_lookup_class_index() lets zram recompression decide whether a newly
compressed object would use a smaller size class. It currently classifies
the payload size directly, while zs_malloc() adds ZS_HANDLE_SIZE before
selecting the class.
This makes lookup disagree with allocation near size-class boundaries.
With 4 KiB pages, CONFIG_ZSMALLOC_CHAIN_SIZE=8, and 64-bit handles, a
1025-to-1024-byte recompression appears to move from class 64 to class 62
although both allocations use class 64. Conversely, a 1049-to-1025-byte
recompression appears to stay in class 64 although the allocations move
from class 65 to class 64.
As a result, zram can accept replacements with no allocation benefit or
reject ones that would save memory, potentially marking the object
incompressible.
Factor size-class selection into lookup_size_class(), account for the
handle there, and use the helper for both lookup and allocation.
Link: https://lore.kernel.org/20260809115518.3791787-1-xialonglong2025@163.com
Fixes:
|
||
|
Jann Horn
|
1dea8e081e |
mm/page-writeback: document folio_mark_dirty() locking more explicitly
We have had bugs where set_page_dirty() was used on a page from GUP without appropriate locking, leading to UAF, in: - KVM, see https://lore.kernel.org/r/20260810-x86-kvm-setpagedirty-v1-1-85f180892d4f@google.com - i915, see commit |
||
|
Breno Leitao
|
dd0dcfe8ef |
mm: kmemleak: default min_unref_scans to 2 for verbose auto-scan
Patch series "mm: kmemleak: default min_unref_scans to 2 for verbose kernels", v2. When CONFIG_DEBUG_KMEMLEAK_VERBOSE is set, which means the host is in auto scan mode, set min_unref_scans to 2, avoiding false positives. CONFIG_DEBUG_KMEMLEAK_VERBOSE depends on CONFIG_DEBUG_KMEMLEAK_AUTO_SCAN, so a kernel built with it already runs the scan thread periodically and the user has asked for detailed leak reports. The confirming second scan comes for free there, so default min_unref_scans to 2 in that case and keep it at 1 everywhere else. CONFIG_DEBUG_KMEMLEAK_VERBOSE defaults to n, so nothing changes for kernels that do not opt in. The other two patches bring the documentation and the selftest comments in line with the new conditional default. PS: A similar patch (v1 of this patchset) is applied to Meta's kernel, in real production hosts. This patch (of 3): min_unref_scans defers reporting an object as leaked until it has stayed unreferenced for that many consecutive scans, filtering out objects that are only transiently unreferenced during a scan. It defaults to 1, which reports on the first unreferenced scan. CONFIG_DEBUG_KMEMLEAK_VERBOSE depends on CONFIG_DEBUG_KMEMLEAK_AUTO_SCAN, so a kernel built with it runs the scan thread periodically and the user has opted into detailed leak reporting. A second confirming scan then happens on its own. Default min_unref_scans to 2 there to suppress transient false positives, and keep it at 1 otherwise, where a manually triggered scan is expected to report immediately. The value stays writable through the module parameter. CONFIG_DEBUG_KMEMLEAK_VERBOSE defaults to n, so this does not change the default for kernels that do not opt in. Link: https://lore.kernel.org/20260731-kmemleak_hardened-v2-0-7b9689ac77cb@debian.org Link: https://lore.kernel.org/20260731-kmemleak_hardened-v2-1-7b9689ac77cb@debian.org Signed-off-by: Breno Leitao <leitao@debian.org> Acked-by: Catalin Marinas <catalin.marinas@arm.com> Cc: David Hildenbrand <david@kernel.org> Cc: Jonathan Corbet <corbet@lwn.net> Cc: Liam R. Howlett <liam@infradead.org> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Shuah Khan <shuah@kernel.org> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Breno Leitao
|
70a964bafe |
mm: kmemleak: factor leak confirmation into a helper
The reporting loop in kmemleak_scan() decided whether to tag an object as a reported leak with a four-term compound condition whose last operand also had a side effect (++object->unref_scans). Mixing the candidate tests with the counter update made the check hard to read. Move the state transition into confirm_leak(): it returns true when a still-unreferenced suspect crosses min_unref_scans consecutive scans and is newly flagged OBJECT_REPORTED, leaving only the reporting bookkeeping in the caller. No functional change. Link: https://lore.kernel.org/20260713-catalin_pto-v1-3-5b93b1131089@debian.org Signed-off-by: Breno Leitao <leitao@debian.org> Reviewed-by: Catalin Marinas <catalin.marinas@arm.com> Cc: David Hildenbrand <david@kernel.org> Cc: Geert Uytterhoeven <geert@linux-m68k.org> Cc: Jonathan Corbet <corbet@lwn.net> Cc: Liam R. Howlett <liam@infradead.org> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Shuah Khan <shuah@kernel.org> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Breno Leitao
|
e776db8e71 |
mm: kmemleak: report leaks only after N consecutive unreferenced scans
kmemleak reports an object the first scan it is found unreferenced. Its mark phase runs without stopping the rest of the kernel and without a write barrier, so a live object whose only reference is briefly invisible during a concurrent RCU update -- e.g. a VMA moved between maple tree nodes, or a page-cache xa_node -- can be seen as unreferenced for that one scan. Because an object is flagged as reported only once, such a transient race turns into a permanent false positive. Track how many consecutive scans each object has been seen unreferenced and only report it once that reaches min_unref_scans, a new module parameter. It defaults to 1, leaving the behaviour unchanged; setting it higher (e.g. 2) still reports a genuine leak, one scan later, while an object referenced again before the threshold restarts its run and is never reported. min_unref_scans can be set at boot with kmemleak.min_unref_scans=<n> or at run-time via /sys/module/kmemleak/parameters/min_unref_scans. Link: https://lore.kernel.org/20260713-catalin_pto-v1-2-5b93b1131089@debian.org Signed-off-by: Breno Leitao <leitao@debian.org> Reviewed-by: Catalin Marinas <catalin.marinas@arm.com> Cc: David Hildenbrand <david@kernel.org> Cc: Geert Uytterhoeven <geert@linux-m68k.org> Cc: Jonathan Corbet <corbet@lwn.net> Cc: Liam R. Howlett <liam@infradead.org> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Shuah Khan <shuah@kernel.org> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Catalin Marinas
|
fb496eb062 |
mm: kmemleak: confirm suspected leaks with a second scan
Patch series "mm: kmemleak: reduce transient false positives by confirming leaks". This series combines two kmemleak enhancements that were originally submitted separately but both required rebasing after commit |
||
|
David Hildenbrand (Arm)
|
534b19bbb6 |
mm/gup_test: keep longterm pin state per file
The pin longterm test currently stores its data globally, shared among
multiple concurrent users of the interface (multiple open file descriptors
-> multiple "struct file"'s). That makes the gup_test interface
problematic to use concurrently: two users, such as concurrent selftest
runs, can interfere with the same longterm pin state.
While this has not been observed as a problem so far in practice, let's
just handle it cleanly. There could be a way to trigger selftest failures
by e.g., running the cow.c and gup_longerm.c selftests concurrently, but
we usually run them sequentially. Let's add a "Fixes" tag to be safe, but
not need to CC stable.
Link: https://lore.kernel.org/20260810-gup_test_data-v1-1-fb1d41be5bb4@kernel.org
Fixes:
|
||
|
Hui Su
|
8790303cba |
kasan: fix cache shrink race with CPU hotplug
kasan_quarantine_remove_cache() first invokes per_cpu_remove_cache() on
all online CPUs. Each callback moves objects belonging to the cache from
cpu_quarantine to the CPU's shrink_qlist, where they can later be freed
from task context.
kmem_cache_destroy() invokes the quarantine removal path while holding
cpus_read_lock(), but kmem_cache_shrink() does not. The latter can
therefore race with CPU offlining as follows:
kmem_cache_shrink() CPU hotplug
------------------- -----------
on_each_cpu()
CPU1 moves objects to
CPU1's shrink_qlist
on_each_cpu() returns
CPU1 goes offline
kasan_cpu_offline()
drains cpu_quarantine
leaves shrink_qlist untouched
for_each_online_cpu()
skips CPU1
The objects left on CPU1's shrink_qlist are not returned to the slab
allocator. This may prevent kmem_cache_shrink() from releasing slabs that
would otherwise become empty. If CPU1 remains offline, a later
kmem_cache_destroy() also skips the list and can report that the cache
still contains objects.
An intermittent occurrence was observed with a virtio-9p filesystem. The
mount and umount commands both returned 0, but the kernel logged the
following during the userspace-triggered teardown:
[ 2994.380134][ T111] BUG 9p-fcall-cache-1 (Tainted: G B ): Objects remaining on __kmem_cache_shutdown()
[ 2994.381140][ T111] Object 0xff11000004361118 @offset=4376
[ 2994.381607][ T111] Allocated in p9_fcall_init+0x201/0x400 age=19564 cpu=1 pid=104
[ 2994.382591][ T111] p9_fcall_init+0x201/0x400
[ 2994.382810][ T111] p9_tag_alloc+0x12f/0x700
[ 2994.382982][ T111] p9_client_prepare_req+0x102/0x3e0
[ 2994.383165][ T111] p9_client_rpc+0x1ab/0xa50
[ 2994.383334][ T111] p9_client_getattr_dotl+0xb0/0x1a0
[ 2994.383515][ T111] v9fs_vfs_getattr_dotl+0x115/0x360
[ 2994.383719][ T111] vfs_getattr_nosec+0x22c/0x3a0
[ 2994.383910][ T111] vfs_statx+0xd7/0x170
[ 2994.384062][ T111] vfs_fstatat+0x45/0x80
[ 2994.384215][ T111] __do_sys_newfstatat+0x84/0xe0
[ 2994.384386][ T111] do_syscall_64+0x115/0x6a0
[ 2994.384566][ T111] entry_SYSCALL_64_after_hwframe+0x77/0x7f
[ 2994.399720][ T111] WARNING: mm/slub.c:1244 at __kmem_cache_shutdown+0x363/0x500, CPU#0: busybox/111
[ 2994.405655][ T111] Call Trace:
[ 2994.406325][ T111] kmem_cache_destroy+0x73/0x1b0
[ 2994.406630][ T111] p9_client_destroy+0x271/0x3c0
[ 2994.407210][ T111] v9fs_session_close+0x3c/0x260
[ 2994.407409][ T111] v9fs_kill_super+0x48/0x90
[ 2994.407584][ T111] deactivate_locked_super+0xa3/0x160
[ 2994.407778][ T111] cleanup_mnt+0x1dd/0x3e0
Thus, a successful umount left objects in the 9p fcall cache and prevented
the cache from being destroyed cleanly.
Per-CPU shrink_qlist storage exists for every possible CPU, and each list
is protected by its own raw spinlock. Iterate over possible CPUs so that
a list populated before its CPU went offline is drained as well.
for_each_possible_cpu() can do more work than for_each_online_cpu(), but
this change only affects CONFIG_KASAN_GENERIC kernels. The extra work is
limited to cache shrink and cache destruction paths and does not affect
the normal allocation/free fast path. It adds one raw-spinlock-protected
scan of each possible CPU's shrink list. These lists are normally empty;
a non-empty list is traversed to remove objects belonging to the cache
being shrunk or destroyed.
Link: https://lore.kernel.org/20260808031459.3032812-1-sh_def@163.com
Fixes:
|
||
|
Hongfu Li
|
097492865f |
mm/cma: remove stray newline from auto-generated CMA area name
When no name is supplied, cma_new_area() generates names with format
"cma%d\n", introducing an unintended newline character ('\n') in the CMA
name.
Most CMA regions are created with explicit names, so this path is seldom
hit. The newline only creates cosmetic noise in debug logs, traces and
debugfs with no functional impact.
Link: https://lore.kernel.org/20260810093215.91419-1-hongfu.li@linux.dev
Signed-off-by: Hongfu Li <lihongfu@kylinos.cn>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Reviewed-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Reviewed-by: SJ Park <sj@kernel.org>
Reviewed-by: Anshuman Khandual <anshuman.khandual@arm.com>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
||
|
Yunzhao Li
|
1f2b4b28aa |
mm/zswap: use ratelimited stats flush in zswap_shrinker_count()
zswap_shrinker_count() calls mem_cgroup_flush_stats(), which takes the
global cgroup rstat lock synchronously. On machines with many CPUs and
NUMA nodes, this creates severe lock contention in the kswapd reclaim
path:
- Multiple kswapd threads (one per NUMA node) run concurrently.
- do_shrink_slab() invokes zswap_shrinker_count() for each
memcg-aware shrinker pass.
- Each call flushes the full cgroup rstat hierarchy under the global
lock.
On AMD EPYC 9684X machines (96 cores, 192 threads, 12 NUMA nodes) running
production workloads with zswap enabled, perf shows 2.88% of kernel cycles
in osq_lock contention from this path:
2.88% [k] osq_lock
--__mutex_lock.constprop.0
--__cgroup_rstat_lock
--cgroup_rstat_flush_locked
--cgroup_rstat_flush
--zswap_shrinker_count
do_shrink_slab
shrink_slab
shrink_node
balance_pgdat
kswapd
84% of kswapd kernel cycles are spent in
shrink_slab -> zswap_shrinker_count -> cgroup_rstat_flush, not in actual
page reclaim (shrink_lruvec).
Controlled A/B on identical hardware and workload:
shrinker=Y: 2.88% osq_lock, memory PSI 1.58%
shrinker=N: 0.00% osq_lock, memory PSI 0.57%
eBPF-based rstat lock wait measurement across 8 production metals
confirms the contention splits cleanly along shrinker enablement:
shrinker=Y: 50-250x more contended lock acquisitions (248/s vs 1.1/s)
shrinker=N: baseline lock wait (0.0017 s/s vs 1.04 s/s)
zswap_shrinker_count() only produces a heuristic estimate, scaled by
compression ratio via mult_frac(). The actual writeback happens in
zswap_shrinker_scan(). Slightly stale stats are acceptable here.
Switch to mem_cgroup_flush_stats_ratelimited(), which only flushes if
the periodic 2-second flusher is one full cycle late. This matches the
approach already used in prepare_scan_control() (mm/vmscan.c) for the
same reclaim path.
After applying this patch, rstat flush latency and lock wait time on
shrinker=Y machines dropped to the same level as shrinker=N controls,
while the zswap shrinker continues to function (pool size remains
bounded under the max_pool_percent cap).
Previously discussed:
- Chengming Zhou (Dec 2023): rstat contention from
zswap_shrinker_count [1]
- Shakeel Butt (Aug 2024): zswap_shrinker_count still uses sync
flush [2]
- Yosry Ahmed (Aug 2024): suggested eliminating in-kernel
flushers [3]
- Jesper Dangaard Brouer (Sep 2024): cgroup/rstat V11 patch [4]
Link: https://lore.kernel.org/20260702180908.150136-1-yunzhao@cloudflare.com
Link: https://lore.kernel.org/linux-mm/20231206103935.3440502-1-zhouchengming@bytedance.com/ [1]
Link: https://lore.kernel.org/linux-mm/CALvZod7LFxLCxVpOFH8b2Ppm8T40HPGMKQwX_=NPCWB_mFW+oQ@mail.gmail.com/ [2]
Link: https://lore.kernel.org/linux-mm/CAJD7tkYvFyOSX+rP_FKGBhxvZiCDxtpsNp-c5CGOA-4Bq9oXSg@mail.gmail.com/ [3]
Link: https://lore.kernel.org/linux-mm/172616070094.2055617.17676042522679701515.stgit@firesoul/ [4]
Suggested-by: Jesper Dangaard Brouer <hawk@kernel.org>
Signed-off-by: Jesper Dangaard Brouer <hawk@kernel.org>
Signed-off-by: Yunzhao Li <yunzhao@cloudflare.com>
Tested-by: Yunzhao Li <yunzhao@cloudflare.com>
Acked-by: Johannes Weiner <hannes@cmpxchg.org>
Acked-by: Jesper Dangaard Brouer <hawk@kernel.org>
Acked-by: Nhat Pham <nphamcs@gmail.com>
Cc: Chengming Zhou <chengming.zhou@linux.dev>
Cc: Shakeel Butt <shakeel.butt@linux.dev>
Cc: Yosry Ahmed <yosry@kernel.org>
Cc: Yunzhao Li <yunzhao@cloudflare.com>
Cc: Sourav Panda <souravpanda@google.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
||
|
Hongfu Li
|
b090524f77 |
mm/swap: fix swap_cluster_lock() !CONFIG_SWAP stub signature mismatch
The !CONFIG_SWAP stub for swap_cluster_lock() has mismatched prototype: it has an extra unused irq argument and uses pgoff_t instead of unsigned long for offset. All callers are under CONFIG_SWAP so the extra parameter is dead. Delete the unused stub function entirely. Link: https://lore.kernel.org/20260717071104.73467-1-hongfu.li@linux.dev Signed-off-by: Hongfu Li <lihongfu@kylinos.cn> Reviewed-by: Baoquan He <baoquan.he@linux.dev> Acked-by: Kairui Song <kasong@tencent.com> Cc: Barry Song <baohua@kernel.org> Cc: Chris Li <chrisl@kernel.org> Cc: Hongfu Li <lihongfu@kylinos.cn> Cc: Kemeng Shi <shikemeng@huaweicloud.com> Cc: Nhat Pham <nphamcs@gmail.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Christoph Hellwig
|
c01e6df60e |
mm/vmstat: add NRSWP{IN,OUT} counters
Count how many swap I/Os we cause. Due to batching this can be different than the current counter number of pages written/read, and tracking this information is useful to see how efficient the batching is. The counters are added at the end of enum vm_event_item and the vmstat_text array under the assumption that the order of fields in /proc/vmstat is an ABI. If that is not the case, they could be grouped with the other swap counters. Link: https://lore.kernel.org/20260713093350.2154226-8-hch@lst.de Signed-off-by: Christoph Hellwig <hch@lst.de> Reviewed-by: Nhat Pham <nphamcs@gmail.com> Cc: Baolin Wang <baolin.wang@linux.alibaba.com> Cc: Baoquan He <baoquan.he@linux.dev> Cc: Barry Song <baohua@kernel.org> Cc: Chris Li <chrisl@kernel.org> Cc: Kairui Song <kasong@tencent.com> Cc: Kemeng Shi <shikemeng@huaweicloud.com> Cc: Youngjun Park <youngjun.park@lge.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Christoph Hellwig
|
0df74c1158 |
mm/swap: remove SWP_FS_OPS
Provide a swap_fs_activate helper that directly sets up swap_fs_ops, and a flag in struct swap_ops to indicate of NOFS swapping is allowed. Link: https://lore.kernel.org/20260713093350.2154226-7-hch@lst.de Signed-off-by: Christoph Hellwig <hch@lst.de> Cc: Baolin Wang <baolin.wang@linux.alibaba.com> Cc: Baoquan He <baoquan.he@linux.dev> Cc: Barry Song <baohua@kernel.org> Cc: Chris Li <chrisl@kernel.org> Cc: Kairui Song <kasong@tencent.com> Cc: Kemeng Shi <shikemeng@huaweicloud.com> Cc: Nhat Pham <nphamcs@gmail.com> Cc: Youngjun Park <youngjun.park@lge.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Christoph Hellwig
|
563597895e |
mm/swap: use swap_ops to register swap device's methods
This simplifies codes and makes logic clearer. And also makes later any new swap device type being added easier to handle. Currently there are two types of swap devices: fs and bdev. [hch@lst.de: updated for the new submit and can_merge abstraction] Link: https://lore.kernel.org/20260713093350.2154226-6-hch@lst.de Signed-off-by: Baoquan He <baoquan.he@linux.dev> Signed-off-by: Christoph Hellwig <hch@lst.de> Suggested-by: Chris Li <chrisl@kernel.org> Reviewed-by: Nhat Pham <nphamcs@gmail.com> Cc: Baolin Wang <baolin.wang@linux.alibaba.com> Cc: Barry Song <baohua@kernel.org> Cc: Kairui Song <kasong@tencent.com> Cc: Kemeng Shi <shikemeng@huaweicloud.com> Cc: Youngjun Park <youngjun.park@lge.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Christoph Hellwig
|
4e915b16de |
mm/swap: remove count_swpout_vm_event
There is only one caller left, so merge it into that. Link: https://lore.kernel.org/20260713093350.2154226-5-hch@lst.de Signed-off-by: Christoph Hellwig <hch@lst.de> Reviewed-by: Baoquan He <baoquan.he@linux.dev> Reviewed-by: Nhat Pham <nphamcs@gmail.com> Cc: Baolin Wang <baolin.wang@linux.alibaba.com> Cc: Barry Song <baohua@kernel.org> Cc: Chris Li <chrisl@kernel.org> Cc: Kairui Song <kasong@tencent.com> Cc: Kemeng Shi <shikemeng@huaweicloud.com> Cc: Youngjun Park <youngjun.park@lge.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Christoph Hellwig
|
dda8fb68b5 |
mm/swap: also use struct swap_iocb for block I/O
Block I/O benefits from batching just as much as remote file systems. Extend struct swap_iocb to support building a bio on the fly as well, and rewrite the block based swap code for it. This especially benefits submit_bio based drivers that do not have the block plugging available, but also saves allocating extra bios for blk-mq drivers. Add a pre-allocated bio to struct swap_iocb in a union with kiocb used for file system based swap so that struct swap_iocb can be used for all swap I/O, and initialize the pool for it unconditionally. Various low-level bdev and fs functions are now replaced with a unified can_merge/add/submit scheme. Note that the block based swap code now uses the same memcg-based check previously added for file system based swap as well. Link: https://lore.kernel.org/20260713093350.2154226-4-hch@lst.de Signed-off-by: Christoph Hellwig <hch@lst.de> Cc: Baolin Wang <baolin.wang@linux.alibaba.com> Cc: Baoquan He <baoquan.he@linux.dev> Cc: Barry Song <baohua@kernel.org> Cc: Chris Li <chrisl@kernel.org> Cc: Kairui Song <kasong@tencent.com> Cc: Kemeng Shi <shikemeng@huaweicloud.com> Cc: Nhat Pham <nphamcs@gmail.com> Cc: Youngjun Park <youngjun.park@lge.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Christoph Hellwig
|
8f29aa226f |
mm/swap: introduce struct swap_io_ctx
Generalize the context currently provided by double pointers to struct swap_iocb to an on-stack context. This cleans up the code and prepares for adding more fields and supporting batching multiple folios into a single bio for block-based swap as well. This new swap_io_ctx is required for all functions using it, the old way of allowing a NULL iocb for some callers is removed to keep the interface consistent. To reduce code duplication caused by this, a new swap_cache_read_folio_sync helper is added to consolidate the code to call swap_cache_read_folio with a local swap_io_ctx. The unpug helpers are renamed to use the submit wording as they are generalized. Link: https://lore.kernel.org/20260713093350.2154226-3-hch@lst.de Signed-off-by: Christoph Hellwig <hch@lst.de> Reviewed-by: Nhat Pham <nphamcs@gmail.com> Cc: Baolin Wang <baolin.wang@linux.alibaba.com> Cc: Baoquan He <baoquan.he@linux.dev> Cc: Barry Song <baohua@kernel.org> Cc: Chris Li <chrisl@kernel.org> Cc: Kairui Song <kasong@tencent.com> Cc: Kemeng Shi <shikemeng@huaweicloud.com> Cc: Youngjun Park <youngjun.park@lge.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Christoph Hellwig
|
a8efc69a65 |
shmem: provide a shmem_write_folio wrapper
Patch series "better block swap batching and a different take on swap_ops v5". This series makes use of the swap_iocb for block as well so that it doesn't do inefficient single-bio I/O, and then rebases the swap_ops from Baoquan on top of the now very different method structure. When running doing kernels builds, which is a workload that doesn't really do much THP anonymous memory it still gets 2x clustering for writeout and 1.2x for reading back swap in. The overall times do not actually change, though. This patch (of 7): Provide a wrapper for the shmem abuses in drm to prepare for swap I/O refactoring by keeping swap_iocb handling entirely contained in mm/. Link: https://lore.kernel.org/20260713093350.2154226-1-hch@lst.de Link: https://lore.kernel.org/20260713093350.2154226-2-hch@lst.de Signed-off-by: Christoph Hellwig <hch@lst.de> Reviewed-by: Baoquan He <baoquan.he@linux.dev> Reviewed-by: Nhat Pham <nphamcs@gmail.com> Reviewed-by: Baolin Wang <baolin.wang@linux.alibaba.com> Acked-by: Chris Li <chrisl@kernel.org> Reviewed-by: Kairui Song <kasong@tencent.com> Cc: Kemeng Shi <shikemeng@huaweicloud.com> Cc: Barry Song <baohua@kernel.org> Cc: Youngjun Park <youngjun.park@lge.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Zhiling Zou
|
a44730dd05 |
mm: shmem: reject page-aligned fallocate end overflow
shmem_fallocate() validates offset + len with inode_newsize_ok(), but then
rounds that end offset up to a page boundary before entering the
preallocation loop.
For a valid request ending at MAX_LFS_FILESIZE, such as offset = 0 and len
= LLONG_MAX, adding PAGE_SIZE - 1 to the validated end can overflow the
signed loff_t used for the rounded end calculation. If that wrapped value
is then converted into a page index, shmem_fallocate() can enter the folio
allocation loop with an invalid range.
Use check_add_overflow() when calculating the page-aligned end, and fail
before entering the allocation loop if the rounded end cannot be
represented.
Link: https://lore.kernel.org/1929a466735dcbb9438936ff50b7a4fc2332a8a4.1785377919.git.zhilinz@nebusec.ai
Fixes:
|
||
|
Abhishek Bapat
|
33588e0b81 |
alloc_tag: add accuracy based filtering to ioctl
Extend the allocinfo filtering mechanism to allow users to filter tags based on their accuracy. [abhishekbapat@google.com: move `inaccurate` filtering criteria from `struct allocinfo_tag` to `struct allocinfo_filter`] Link: https://lore.kernel.org/e4e49ec4a5960292aeeb9e196526c18dc95228a2.1785867739.git.abhishekbapat@google.com Link: https://lore.kernel.org/396a5e4bc3b2990223ab355f2cd3ceb6aa15499e.1783532853.git.abhishekbapat@google.com Signed-off-by: Abhishek Bapat <abhishekbapat@google.com> Acked-by: Hao Ge <hao.ge@linux.dev> Acked-by: Suren Baghdasaryan <surenb@google.com> Cc: Jonathan Corbet <corbet@lwn.net> Cc: Kent Overstreet <kent.overstreet@linux.dev> Cc: Sourav Panda <souravpanda@google.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Abhishek Bapat
|
6f6769ea88 |
alloc_tag: add size-based filtering to ioctl
Extend the allocinfo filtering mechanism to allow users to filter tags based on the total number of bytes allocated [min_size, max_size]. The size range is inclusive. Filtering by size involves retrieving allocinfo per-CPU counters, which is an expensive operation. Hence, the performance of size-based filtering will be worse than other filters. Link: https://lore.kernel.org/0a7653b70ae0d64e967fbea0e933bc35f8ac656e.1783532853.git.abhishekbapat@google.com Signed-off-by: Abhishek Bapat <abhishekbapat@google.com> Acked-by: Hao Ge <hao.ge@linux.dev> Acked-by: Suren Baghdasaryan <surenb@google.com> Cc: Jonathan Corbet <corbet@lwn.net> Cc: Kent Overstreet <kent.overstreet@linux.dev> Cc: Sourav Panda <souravpanda@google.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Abhishek Bapat
|
5732a4e4c1 |
alloc_tag: add ioctl filters to /proc/allocinfo
Extend the capability of the IOCTL mechanism to filter allocations based on tag's module name, function name, file name and line number. Link: https://lore.kernel.org/6a6100c0c58cb2911f39126b9fe177a8c17db16f.1783532853.git.abhishekbapat@google.com Signed-off-by: Abhishek Bapat <abhishekbapat@google.com> Acked-by: Hao Ge <hao.ge@linux.dev> Acked-by: Suren Baghdasaryan <surenb@google.com> Cc: Jonathan Corbet <corbet@lwn.net> Cc: Kent Overstreet <kent.overstreet@linux.dev> Cc: Sourav Panda <souravpanda@google.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Suren Baghdasaryan
|
1d581ab234 |
alloc_tag: add ioctl to /proc/allocinfo
Patch series "alloc_tag: introduce IOCTL-based filtering for MAP", v8. Currently, memory allocation profiling data is primarily exposed through /proc/allocinfo. While useful for manual inspection, this text-based interface poses challenges for production monitoring and large-scale analysis: 1. Userspace must parse large amounts of text to extract specific fields. 2. To find specific tags, userspace must read the entire dataset, requiring many context switches and high data copying. 3. The kernel currently aggregates per-CPU counters for every allocation size, even those the user intends to filter out immediately. This series introduces a new IOCTL-based binary interface for allocinfo that supports kernel-side filtering. By allowing the user to specify a filter mask, we significantly reduce the work performed in-kernel and the amount of data transferred to userspace. The IOCTL mechanism was chosen for allocinfo to address the per-CPU counter aggregation bottleneck. A traditional read() operation must report the total allocation count and sizes for every code tag in the system. Doing so requires iterating across all CPUs to sum their per-CPU counters for thousands of tags, which introduces substantial runtime overhead. The IOCTL interface allows userspace to push selective filtering criteria directly into the kernel before the per-CPU counter aggregation. The kernel aggregates per-CPU counters only for a small subset of tags that match the filter. This results in significant performance improvement. Beyond fast filtered retrieval, the IOCTL foundation allows introducing a context capture mechanism in the future to capture the context for specific allocations. Performance measurements were conducted on an Intel Xeon Platinum 8481C (224 CPUs) with caches dropped before each run. The IOCTL mechanism shows a ~20x performance improvement for filtered queries. The kernel avoids the expensive per-CPU counter aggregation (alloc_tag_read) for any tags that fail the initial string or location filters. Scenario 1: Specific File Filtering (arch/x86/events/rapl.c) 1. Traditional (cat /proc/allocinfo | grep): 22ms (sys) 2. IOCTL Interface: 1ms (sys) Scenario 2: Compound Filtering (Filename + Size) 1. Traditional: (cat ... | grep | awk): 21ms (sys) 2. IOCTL Interface: 1ms (sys) Scenario 3: Size-Based Filtering (min_size = 1MB) 1. Traditional: (cat ... | awk): 21ms (sys) 2. IOCTL Interface: 14ms (sys) This patch (of 6): Add the following ioctl commands for /proc/allocinfo file: ALLOCINFO_IOC_CONTENT_ID - gets content identifier which can be used to check whether the file content has changed specifically due to module load/unload. Every time a module is loaded / unloaded, the returned value will be different. By comparing the identifier value at the beginning and at the end of the content retrieval operation, users can validate retrieved information for consistency. ALLOCINFO_IOC_GET_AT - gets the record at the specified position. This is the position of a record in /proc/allocinfo. ALLOCINFO_IOC_GET_NEXT - gets the record next to the last retrieved one. If no records were previously retrieved, returns the first record. Note, function file and module names often have the same prefixes, therefore when filtering for them, we compare the last 64 characters to minimize the chances of name collisions. [akpm@linux-foundation.org: include compat.h, per Suren] Closes: https://lore.kernel.org/oe-kbuild-all/202607091820.qbjlGhKK-lkp@intel.com/ Link: https://lore.kernel.org/cover.1783532853.git.abhishekbapat@google.com Link: https://lore.kernel.org/15596de2607ef13e7c77c6d74763f4ae992ec475.1783532853.git.abhishekbapat@google.com Signed-off-by: Suren Baghdasaryan <surenb@google.com> Signed-off-by: Abhishek Bapat <abhishekbapat@google.com> Acked-by: Hao Ge <hao.ge@linux.dev> Cc: Jonathan Corbet <corbet@lwn.net> Cc: Kent Overstreet <kent.overstreet@linux.dev> Cc: Sourav Panda <souravpanda@google.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Sourav Panda
|
34e0849142 |
mm/hugetlb_cma: support percentage-based hugetlb_cma reservation
Currently, hugetlb_cma reservation only supports absolute sizes (e.g., hugetlb_cma=2G or hugetlb_cma=0:1G,1:1G). This can be restrictive in heterogeneous environments or when deploying common kernel command lines across machines with different memory capacities. Add support for percentage-based hugetlb_cma reservation (e.g., hugetlb_cma=20% or hugetlb_cma=0:20%,1:10%). The percentage is calculated against the total memory (for global settings) or against the node-specific memory (for node-specific settings) using memblock APIs during early boot. Link: https://lore.kernel.org/20260807040003.2156630-1-souravpanda@google.com Signed-off-by: Sourav Panda <souravpanda@google.com> Acked-by: Usama Arif <usama.arif@linux.dev> Cc: David Hildenbrand <david@kernel.org> Cc: David Rientjes <rientjes@google.com> Cc: Frank van der Linden <fvdl@google.com> Cc: Greg Thelen <gthelen@google.com> Cc: Muchun Song <muchun.song@linux.dev> Cc: Oscar Salvador <osalvador@suse.de> Cc: Suren Baghdasaryan <surenb@google.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Mike Rapoport (Microsoft)
|
3774c56cc3 |
drivers/base, mm: move arch_numa.c to mm/
arch_numa.c implements boot time discovery and initialization of NUMA topology on architectures that select GENERIC_ARCH_NUMA (currently arm64 and riscv). Since this is step in the initialization of the memory management subsystem, it's logical to have arch_numa.c in mm/ alongside numa.c, numa_memblks.c and numa_emulation.c. Move arch_numa.c to mm/ and add its F: entry to "MEMBLOCK AND MEMORY MANAGEMENT INITIALIZATION" in MAINTAINERS. Link: https://lore.kernel.org/20260806-arch-numa-v1-1-968ec128121e@kernel.org Signed-off-by: Mike Rapoport (Microsoft) <rppt@kernel.org> Acked-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org> Acked-by: Lorenzo Stoakes (ARM) <ljs@kernel.org> Acked-by: Vlastimil Babka (SUSE) <vbabka@kernel.org> Acked-by: David Hildenbrand (Arm) <david@kernel.org> Acked-by: Danilo Krummrich <dakr@kernel.org> Cc: Albert Ou <aou@eecs.berkeley.edu> Cc: Alexandre Ghiti <alex@ghiti.fr> Cc: Catalin Marinas <catalin.marinas@arm.com> Cc: Liam R. Howlett <liam@infradead.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Palmer Dabbelt <palmer@dabbelt.com> Cc: "Rafael J. Wysocki" <rafael@kernel.org> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Will Deacon <will@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Hao Jia
|
6f34b4126b |
mm/zswap: support batch writeback in shrink_memcg()
Currently, shrink_memcg() writes back at most one entry per-node during
its traversal. This makes shrink_worker() inefficient, as it must
repeatedly re-enter shrink_memcg() to make any substantial progress.
Under high memory pressure, this can cause the writeback speed to be too
slow to keep up with refaults, leading to zswap store failures and forcing
pages to skip zswap and go directly to disk, which results in an LRU
inversion.
To address this, extend the per-node scan budget in shrink_memcg() from a
single entry to up to SWAP_CLUSTER_MAX pages, enabling batch writeback for
both the shrink_worker() and zswap_store() paths.
Test Setup:
- Total memory: 32 GB, 1 NUMA node.
- zswap settings: accept_threshold_percent=50, shrinker_enabled=N.
Test Case 1:
Set max_pool_percent=1, allocate 512MB of anonymous pages, and fill them
with random data (to avoid compression). Then, use cgroup memory.reclaim
to force a large amount of anonymous pages into zswap. At an interval of
2ms, allocate a 4K anonymous page where the first 4 bytes are random numbers
and the rest are zeros, and then trigger reclamation of this 4K page through
cgroup memory.reclaim. When the pool threshold is reached, shrink_memcg()
will be triggered.
The test data after running for 120s is as follows:
Baseline Patched
shrink_worker wakeups 5,363 169
shrink_memcg calls 11,373,201 350,703
written_back pages 40,212 40,241
zswap_store calls 161,190 163,753
store succeeded (ret=1) 102,743 117,183
store rejected (ret=0) 58,447 46,570
store reject rate ~36% ~28%
pool_limit_hit delta 55,826 33,760
pswpout 98,659 86,811
pswpin 2 0
Test Case 2:
We evaluated the following two sub-configurations using stress-ng inside
a cgroup capped at memory.max=1G for 120 seconds:
Test Case 2a (max_pool_percent=1): Continuously triggers the global
zswap pool limit, thereby waking up shrink_worker() to perform asynchronous
shrinking.
Test Case 2b (zswap.max=320M, max_pool_percent=50): Continuously triggers
the cgroup's zswap.max limit, thereby invoking synchronous shrinking.
Command executed for both setups:
bash -c 'echo $$ > /sys/fs/cgroup/zswaptest/cgroup.procs ; \
exec stress-ng --vm 4 --vm-bytes 4G --vm-keep --vm-method rand-set -t \
120s -q'
Test Case 2a (max_pool_percent=1):
Baseline Patched
shrink_worker wakeups 5,640 1,308
shrink_memcg calls 8,481,500 3,140,972
written_back pages 260 468,216
zswap_store calls 2,742,756 2,011,269
store succeeded (ret=1) 934,640 947,988
store rejected (ret=0) 1,808,116 1,063,281
store reject rate ~66% ~52%
pool_limit_hit delta 1,181,310 196,882
pswpout 1,808,376 1,531,497
pswpin 4,288,497 3,635,365
Test Case 2b (zswap.max=320M, max_pool_percent=50):
Baseline Patched
shrink_worker wakeups 0 0
shrink_memcg calls 687,608 54,002
written_back pages 639,176 846,663
zswap_store calls 1,224,222 1,228,548
store succeeded (ret=1) 992,816 1,208,123
store rejected (ret=0) 231,431 20,425
store reject rate ~19% ~2%
pool_limit_hit delta 0 0
pswpout 870,745 867,360
pswpin 1,707,823 1,216,814
Under identical workloads and runtimes, batched zswap shrinking exhibits a
significant reduction in both shrink_worker() wakeups and shrink_memcg()
calls. Furthermore, the sharp drop in both pswpin and zswap_store()
rejections demonstrates that batching zswap shrink operations effectively
mitigates zswap_store() failures caused by hitting the pool limit. This
significantly prevents pages from bypassing zswap and falling back
directly to disk, thereby reducing LRU inversion.
Link: https://lore.kernel.org/20260806070943.95542-3-jiahao.kernel@gmail.com
Signed-off-by: Hao Jia <jiahao1@lixiang.com>
Suggested-by: Yosry Ahmed <yosry@kernel.org>
Suggested-by: Johannes Weiner <hannes@cmpxchg.org>
Acked-by: Yosry Ahmed <yosry@kernel.org>
Acked-by: Nhat Pham <nphamcs@gmail.com>
Cc: Chengming Zhou <chengming.zhou@linux.dev>
Cc: Michal Hocko <mhocko@kernel.org>
Cc: Michal Koutný <mkoutny@suse.com>
Cc: Muchun Song <muchun.song@linux.dev>
Cc: Roman Gushchin <roman.gushchin@linux.dev>
Cc: Shakeel Butt <shakeel.butt@linux.dev>
Cc: Tejun Heo <tj@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
||
|
Hao Jia
|
dc8458f43f |
mm/zswap: fix global shrinker when memory cgroup is disabled
Patch series "mm/zswap: Fixes and improves the zswap shrink", v4.
This series fixes and improves the zswap global shrinker
(shrink_worker()): Patch 1: Fix missing global shrinker when memory cgroup
is disabled. Patch 2: Extend shrink_memcg() to support batch writeback
and thereby improving the writeback efficiency in the shrink_worker() and
zswap_store() paths.
This patch (of 2):
Zswap writeback when the global pool limit is hit fails when memory cgroup
is disabled. The pool remains full until it is organically drained by
swapins or memory freeing, leading to zswap store failures and pages
bypassing getting written directly to the backing swap device, causing LRU
inversion (hotter pages with higher fault latency).
This happens because mem_cgroup_iter() always returns NULL when memory
cgroups are disabled. As a result, the global shrinker shrink_worker()
repeatedly takes empty walks. After MAX_RECLAIM_RETRIES failed attempts,
the worker gives up without writing back any pages.
Therefore, when memory cgroup is disabled, fall through with the !memcg
branch and shrink the root memcg directly.
With memcg disabled, shrink_memcg() only returns -ENOENT when the root LRU
is empty, which means the total pages are already below thr. In the
absence of heavy concurrent zswap stores, the loop then safely bails out
via the zswap_total_pages() <= thr check; otherwise, it will resume
shrinking the memcg after processing the reschedule check. For any other
return value from shrink_memcg(), the loop is guaranteed to terminate,
either after MAX_RECLAIM_RETRIES failures or once the threshold is met.
This is a potential performance regression for people using zswap
without memcg that was introduced by the commit in "Fixes".
Link: https://lore.kernel.org/20260806070943.95542-1-jiahao.kernel@gmail.com
Link: https://lore.kernel.org/20260806070943.95542-2-jiahao.kernel@gmail.com
Fixes:
|
||
|
Ye Liu
|
54cc9b3847 |
mm: debug_page_alloc: fix NULL buf in debug_guardpage_minorder_setup
If the kernel command line includes "debug_guardpage_minorder" without an
equals sign (i.e., no value is provided), the early parameter parser
passes a NULL buf pointer to the setup function.
kstrtouint() does not perform a NULL check on its input and calls directly
into kstrtoull() which dereferences s[0] unconditionally, leading to a
NULL pointer dereference and early boot crash.
Additionally, the error path's pr_err("%s", buf) would also crash with a
NULL format argument.
Link: https://lore.kernel.org/20260806004556.2633049-1-ye.liu@linux.dev
Fixes:
|
||
|
Ye Liu
|
8be7c167be |
mm/show_mem: fix format string inconsistencies and type mismatches
Fix five format string issues in show_free_areas() and __show_mem(): 1-2. reserved_highatomic and free_highatomic: %luKB -> %lukB The uppercase "KB" is inconsistent with all other fields in the same output block and with /proc/meminfo convention. 3. local_pcp: %ukB -> %lukB with explicit (unsigned long) cast per_cpu_pages.count is int, so K(count) yields int. Using %u was a signed/unsigned mismatch. Cast to unsigned long and use %lu for consistency with all other K() usages in the file. 4. total pagecache pages: %ld -> %lu global_node_page_state() returns unsigned long. Using %ld is a signedness mismatch caught by gcc -Wformat-signedness. 5. hwpoisoned pages: %lu -> %ld atomic_long_read() returns long (signed). Using %lu is a signedness mismatch caught by gcc -Wformat-signedness. Verified with: make KCFLAGS="-Wformat -Wformat-signedness" mm/show_mem.o Link: https://lore.kernel.org/20260805021556.1908807-1-ye.liu@linux.dev Signed-off-by: Ye Liu <liuye@kylinos.cn> Acked-by: Vlastimil Babka (SUSE) <vbabka@kernel.org> Acked-by: Johannes Weiner <hannes@cmpxchg.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Muchun Song
|
e73aeb8a41 |
mm/sparse: keep mem_section_usage_size() internal
mem_section_usage_size() is only needed by sparsemem implementation code
after commit
|
||
|
Abhishek Bapat
|
afff109c2f |
alloc_tag: expose boot-time compression configuration
Currently, userspace has limited visibility into the exact active runtime
state of memory allocation profiling and its page extension compression
('sysctl.vm.mem_profiling={0|1|never}[,compressed]').
While reading the sysctl provides basic on/off status, it is currently
impossible for userspace to natively determine whether page-tag
compression was successfully enabled without scraping dmesg boot logs.
Add a new read-only sysctl representing how compression was configured
at boot time.
Link: https://lore.kernel.org/c795f8089f82841e8a6e00d7ca286da2b23aeb7b.1785950530.git.abhishekbapat@google.com
Signed-off-by: Abhishek Bapat <abhishekbapat@google.com>
Acked-by: Suren Baghdasaryan <surenb@google.com>
Cc: Hao Ge <hao.ge@linux.dev>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
||
|
Ye Liu
|
bb3e3c5c2d |
mm: debug_page_alloc: fix type mismatch for debug_guardpage_minorder
The debug_guardpage_minorder local variable is declared as unsigned int, but debug_guardpage_minorder_setup() uses unsigned long and kstrtoul() to parse the value. Use kstrtouint() with unsigned int local variable to match the actual type of _debug_guardpage_minorder. Also fix the format specifier from %lu to %u accordingly. Link: https://lore.kernel.org/20260805093108.2352900-1-ye.liu@linux.dev Signed-off-by: Ye Liu <liuye@kylinos.cn> Reviewed-by: Andrew Morton <akpm@linux-foundation.org> Cc: Johannes Weiner <hannes@cmpxchg.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Sang-Heon Jeon
|
0ddb8bb85b |
mm/page_ext: remove pgdat_page_ext_init()
pgdat_page_ext_init() sets pgdat->node_page_ext to NULL only on FLATMEM. FLATMEM depends on !NUMA, so the pgdat is always the zero-initialized contig_page_data and the store has no effect. So remove the call site, the unused function and its declaration. No functional change. Link: https://lore.kernel.org/20260804151145.3419768-3-ekffu200098@gmail.com Signed-off-by: Sang-Heon Jeon <ekffu200098@gmail.com> Acked-by: Zi Yan <ziy@nvidia.com> Acked-by: David Hildenbrand (Arm) <david@kernel.org> Cc: Johannes Weiner <hannes@cmpxchg.org> Cc: Liam R. Howlett <liam@infradead.org> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Sang-Heon Jeon
|
34568000f3 |
mm/Kconfig: make FLATMEM depend on !NUMA
Patch series "mm/page_ext: remove pgdat_page_ext_init()", v2. pgdat_page_ext_init() has no effect on FLATMEM. The pgdat is always the zero-initialized contig_page_data, because no architecture supports FLATMEM + NUMA. That constraint is only implicit in the arch Kconfig files. So patch 1 makes it explicit in mm/Kconfig, and patch 2 removes pgdat_page_ext_init(). No functional change. This patch (of 2): FLATMEM + NUMA is not supported by any architecture and fails to build. The constraint is only implicit in the arch Kconfig files. So make it explicit in mm/Kconfig. No functional change. Link: https://lore.kernel.org/20260804151145.3419768-1-ekffu200098@gmail.com Link: https://lore.kernel.org/20260804151145.3419768-2-ekffu200098@gmail.com Signed-off-by: Sang-Heon Jeon <ekffu200098@gmail.com> Suggested-by: Zi Yan <ziy@nvidia.com> Acked-by: Zi Yan <ziy@nvidia.com> Acked-by: David Hildenbrand (Arm) <david@kernel.org> Acked-by: Mike Rapoport (Microsoft) <rppt@kernel.org> Cc: Johannes Weiner <hannes@cmpxchg.org> Cc: Liam R. Howlett <liam@infradead.org> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Lorenzo Stoakes (ARM)
|
6fd3e592c0 |
mm: add some missing includes to mm-local headers
There are a number of internal headers local to mm/ which reference functions and data types without including the relevant headers. mm/vma.h is a special case that intentionally does not include additional headers, but the others are not. This breaks tooling like clangd (which is where I noticed this), though the build is OK due to the C files including the headers happening to include required dependencies. It's better to be explicit about dependencies anyway, so add the missing includes and fix clangd as a bonus. Link: https://lore.kernel.org/20260804-fix-some-local-headers-v1-1-a7beb173c116@kernel.org Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org> Acked-by: Vlastimil Babka (SUSE) <vbabka@kernel.org> Acked-by: David Hildenbrand (Arm) <david@kernel.org> Acked-by: Zi Yan <ziy@nvidia.com> Reviewed-by: Barry Song <baohua@kernel.org> Reviewed-by: Baoquan He <baoquan.he@linux.dev> Cc: Chris Li <chrisl@kernel.org> Cc: Johannes Weiner <hannes@cmpxchg.org> Cc: Kairui Song <kasong@tencent.com> Cc: Kemeng Shi <shikemeng@huaweicloud.com> Cc: Liam R. Howlett <liam@infradead.org> Cc: Michal Hocko <mhocko@kernel.org> Cc: Mike Rapoport <rppt@kernel.org> Cc: Muchun Song <muchun.song@linux.dev> Cc: Nhat Pham <nphamcs@gmail.com> Cc: Oscar Salvador <osalvador@suse.de> Cc: Roman Gushchin <roman.gushchin@linux.dev> Cc: Shakeel Butt <shakeel.butt@linux.dev> Cc: Suren Baghdasaryan <surenb@google.com> Cc: "Uladzislau Rezki (Sony)" <urezki@gmail.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Uladzislau Rezki (Sony)
|
b9183788a2 |
mm/vmalloc: do not warn on -ENOMEM from va_alloc()
Since vmalloc() accepts non-blocking GFP flags, allocation requests may fail when callers pass restrictive GFP masks. va_clip() may return -ENOMEM when its GFP_NOWAIT fallback allocation fails during NE_FIT_TYPE splitting. This is an expected failure, so va_alloc() should return the error without triggering a kernel splat. Link: https://lore.kernel.org/20260802104627.63892-1-urezki@gmail.com Signed-off-by: Uladzislau Rezki (Sony) <urezki@gmail.com> Reported-by: syzbot+61c997e6be1d9bb300ba@syzkaller.appspotmail.com Closes: https://lore.kernel.org/6a6d3cbd.6ce73036.24301b.000e.GAE@google.com Reviewed-by: Anshuman Khandual <anshuman.khandual@arm.com> Reviewed-by: Baoquan He <baoquan.he@linux.dev> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Hongfu Li
|
a44ab4bd1e |
ksm: update comments and docs to reference folio->mapping
The KSM code already stores and checks the stable node key via folio->mapping, but the comment in ksm_get_folio() and the reverse mapping documentation in ksm.rst still refer to page->mapping. This is a pure wording update to match the folio-based implementation. No functional change is intended. Link: https://lore.kernel.org/20260805105927.41987-1-hongfu.li@linux.dev Signed-off-by: Hongfu Li <lihongfu@kylinos.cn> Acked-by: David Hildenbrand (Arm) <david@kernel.org> Reviewed-by: Xu Xin <xu.xin16@zte.com.cn> Reviewed-by: Dongliang Mu <dzm91@hust.edu.cn> Cc: Alex Shi <alexs@kernel.org> Cc: Chengming Zhou <chengming.zhou@linux.dev> Cc: David Hildenbrand <david@kernel.org> Cc: Jonathan Corbet <corbet@lwn.net> Cc: Liam R. Howlett <liam@infradead.org> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Cc: Yanteng Si <si.yanteng@linux.dev> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Longlong Xia
|
3c37cac718 |
mm/ksm: avoid missing ksmd wakeups in ksm_enter
__ksm_enter() decides whether ksmd needs a wakeup by checking if the mm
slot list is empty before inserting the new slot.
The empty check is currently outside ksm_mmlist_lock. Another CPU can
remove the last slot and let ksmd go back to sleep after the unlocked
check, while this CPU inserts a new slot and skips the wakeup based on the
stale result.
Take ksm_mmlist_lock before checking the list so the empty-to-nonempty
transition and the insertion are observed as one critical section.
Link: https://lore.kernel.org/20260805132736.1063408-1-xialonglong2025@163.com
Fixes:
|
||
|
Jianlin Shi
|
8282cb36d0 |
mm/page_alloc: only update lowmem_reserve_ratio on sysctl write
lowmem_reserve_ratio_sysctl_handler() ignores the return value of proc_dointvec_minmax() and always calls setup_per_zone_lowmem_reserve(), even for read operations. Fix three issues: 1. Propagate errors from proc_dointvec_minmax() instead of always returning success. For example, writing non-integer garbage to the sysctl now returns an error instead of silently succeeding with unchanged values. 2. Only call setup_per_zone_lowmem_reserve() when the sysctl is actually written, matching the write-only refresh pattern of min_free_kbytes and watermark_scale_factor handlers. 3. On write, parse into a temporary ratio[] array and only copy into sysctl_lowmem_reserve_ratio[] and refresh derived state after the full vector is validated. This avoids leaving the ratio array partially updated while skipping setup when proc_dointvec_minmax() returns an error on a later element (suggested by Andrew Morton). Drop the manual "< 1 -> 0" sanitization loop and set .extra1 = SYSCTL_ZERO on the ctl_table entry so proc_dointvec_minmax() enforces the minimum on write; negative values now return -EINVAL instead of being silently coerced to 0 (suggested by Vlastimil Babka). [akpm@linux-foundation.org: add comment, per hannes] Link: https://lore.kernel.org/anSRGASe5FIrqwlg@cmpxchg.org Link: https://lore.kernel.org/linux-mm/tencent_FFD4F4D728AAE8A8AE0AF277A59854A29A06@qq.com/ Link: https://lore.kernel.org/tencent_A860C873956A52E26AD8D309A308A241BA08@qq.com Signed-off-by: Jianlin Shi <shijianlin11@foxmail.com> Acked-by: Vlastimil Babka (SUSE) <vbabka@kernel.org> Cc: Brendan Jackman <jackmanb@google.com> Cc: Johannes Weiner <hannes@cmpxchg.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Zi Yan <ziy@nvidia.com> Cc: Joel Granados <joel.granados@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Eric Biggers
|
c310a8932a |
mm/swap: reject swapon() on filesystem-level encrypted files
ext4 and f2fs don't prevent filesystem-level encrypted files from being set up directly as swap files. In this case, encryption is bypassed. No one should be doing this, vs. the methods of encrypted swap that actually do work (such as swapping to a dm-crypt device, or swapping to a loopback device on top of a filesystem-level encrypted file). Nevertheless, to prevent user error, make swapon() explicitly reject this case. Document this behavior in fscrypt.rst as well. Link: https://lore.kernel.org/20260803180426.3123-1-ebiggers@kernel.org Fixes: |
||
|
Artem Lytkin
|
272b0d84b1 |
mm/vmalloc: make vm_struct.nr_pages an unsigned long
vm_struct::nr_pages is an unsigned int, and the file keeps deriving byte counts from it as nr_pages << PAGE_SHIFT. A shift is evaluated in the type of its promoted left operand, so those are 32-bit arithmetic and wrap at 4 GiB of bytes, which is 2^20 pages. Every site depends on a cast being remembered; vmap() has one, two recent commits did not. vread_iter() then computes a size of zero for a 4 GiB VM_ALLOC area and /proc/kcore returns it as zeros while reporting a successful read, which drgn, crash or gdb cannot tell from real memory, and the vrealloc() grow-in-place check declines a request that would have fit. Widen the field so the class of bug goes away instead of one site at a time. Everything feeding or consuming it widens too: vm_area_alloc_pages() and its accumulators, nr_small_pages, new_nr_pages and old_nr_pages, the index range of vm_area_free_pages(), and three page indexes that were plain int. Five casts go. Two prints needed fixing as well, %u in vmalloc_dump_obj() and %d for the unsigned field in vmalloc_info_show(). No bug report behind this, I found it reading the code. The 4 GiB wrap needs only a machine with over 4 GiB of memory. Neither larger threshold is a practical concern: 2^32 pages, where the field itself truncates, is 16 TiB and beyond what hardware can populate, and 2^31, where the plain int indexes break, is 8 TiB and larger than anything in the tree asks for. The int *nr cursor in the mapping path is unchanged and is separate work. Users outside mm/vmalloc.c need no change either. Those handing the count to a narrower parameter cannot drive it near 2^31, and kho_preserve_vmalloc() stores it into a 32-bit ABI field that still receives the same low bits; above 2^32 pages the truncation just moves out of vm_struct into that store. sizeof(struct vm_struct) on x86-64 stays 72 bytes with CONFIG_HAVE_ARCH_HUGE_VMALLOC=n and goes from 72 to 80 with it enabled, both inside the kmalloc-96 bucket it already comes from. Link: https://lore.kernel.org/20260801114915.115224-1-iprintercanon@gmail.com Fixes: |
||
|
David Hildenbrand (Arm)
|
fc6415a384 |
mm/gup: factor out LRU cache draining for folio into lru_cache_drain_for_folio()
KVM with guest_memfd wants to remove any folio references due to LRU caches, as it really must only allow to convert folios from shared to private when there are no unexpected folio references (e.g., from GUP references). So, to drive the refcount down, it needs a way to flush the LRU caches. Let's factor out what we have in lru_cache_drain_for_folio(). Document it, and also mention that concurrent folio (un)mapping might, in theory, miss detecting LRU cache references. Keep obtaining the expected refcount twice to minimize the possibility. For the current and future user that should work, and we don't really have a better alternative: we could detect if the mapcount changed, but it would still be racy and add more complexity with questionable benefit. Maybe there is a chance to avoid the draining entirely in the future, by avoiding extra references from the LRU cache: Hugh thinks there might be a way. But for the time being, this handling is unfortunately required. Make folio_may_be_lru_cached() accept a const pointer so lru_cache_drain_for_folio() can accept a const pointer as well. Link: https://lore.kernel.org/20260806-lru_cache_drain_for_folio-v1-1-c6287d295e99@kernel.org Signed-off-by: David Hildenbrand (Arm) <david@kernel.org> Reviewed-by: Fuad Tabba <fuad.tabba@linux.dev> Cc: Ackerley Tng <ackerleytng@google.com> Cc: Baoquan He <baoquan.he@linux.dev> Cc: Barry Song <baohua@kernel.org> Cc: Chris Li <chrisl@kernel.org> Cc: Jason Gunthorpe <jgg@ziepe.ca> Cc: John Hubbard <jhubbard@nvidia.com> Cc: Kairui Song <kasong@tencent.com> Cc: Kemeng Shi <shikemeng@huaweicloud.com> Cc: Liam R. Howlett <liam@infradead.org> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Nhat Pham <nphamcs@gmail.com> Cc: Peter Xu <peterx@redhat.com> Cc: Sean Christopherson <seanjc@google.com> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
David Hildenbrand (Arm)
|
078e1a0fc4 |
mm/gup: fix always draining LRU caches in collect_longterm_unpinnable_folios()
folio_may_be_lru_cached() is currently only true for small folios, and for small folios FOLL_PIN adds GUP_PIN_COUNTING_BIAS references instead of 1 in try_grab_folio()/try_grab_folio_fast(). Consequently, our folio_ref_count(folio) != folio_expected_ref_count(folio) + 1 check in collect_longterm_unpinnable_folios() will currently always identify "reference mismatch" and first drain the local LRU cache to then drain the LRU cache on all CPUs, as collect_longterm_unpinnable_folios() is really called after pinning the folios with FOLL_PIN. Add a comment because the current code is not quite intuitive: we used to drain only to make sure the folio_isolate_lru() would succeed. But then we also started draining to make later migration more reliable. We'll refactor that code soon a bit, to also make it usable in other context where we really want to remove any references from LRU caches. Let's add CC stable, because having an easy way for excessive LRU cache draining on all CPUs does not sound right. In common scenarios we don't expect to ever have to drain. Link: https://lore.kernel.org/20260731-check_and_migrate_movable_folios-v1-1-e0002d7b791e@kernel.org Fixes: |
||
|
Zi Yan
|
c299a2285d |
mm/huge_memory: use folio's memcg inside __folio_split()
Patch series "Honor XA_FLAGS_ACCOUNT in xas_split_alloc() and charge to folio's memcg", v3. __GFP_ACCOUNT is needed for xarray node allocation accounting when XA_FLAGS_ACCOUNT is set. Commit |
||
|
Arvind Yadav
|
34a00895d0 |
mm/migrate_device: clear stale mapping after freeing swapcache
__migrate_device_pages() reads the folio mapping before calling
folio_free_swap(). When folio_free_swap() succeeds, the folio is removed
from the swap cache, but the saved mapping still points to swap_space.
Passing the stale mapping to folio_migrate_mapping() makes it use the
mapped-folio path for a folio that is no longer in swapcache. It can then
operate on swap_space.i_pages with invalid reference accounting,
eventually triggering a folio reference count BUG.
After a successful split, nr still contains the number of pages in the
original large folio, although each resulting page is now a separate
order-0 folio. Reset nr to 1 so each split folio is processed separately,
including its own swapcache removal and mapping lookup.
Refresh the saved mapping after folio_free_swap() so the current folio
state is used during migration.
Link: https://lore.kernel.org/20260728062832.1107127-1-arvind.yadav@intel.com
Fixes:
|
||
|
Wilson Felipe Pereira
|
7822fa5f4f |
mm: zswap: drop list_lru param from zswap_lru_add() and _del()
Since zswap_lru_add() and zswap_lru_del() are only called with the global zswap_list_lru, remove the redundant list_lru argument and use zswap_list_lru directly. Link: https://lore.kernel.org/20260731053721.1412304-1-wfelipe@google.com Signed-off-by: Wilson Felipe Pereira <wfelipe@google.com> Acked-by: Johannes Weiner <hannes@cmpxchg.org> Acked-by: Yosry Ahmed <yosry@kernel.org> Reviewed-by: SJ Park <sj@kernel.org> Acked-by: Nhat Pham <nphamcs@gmail.com> Cc: Chengming Zhou <chengming.zhou@linux.dev> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Sang-Heon Jeon
|
8380671909 |
mm/sparse: correct init section annotations
The !SPARSEMEM_EXTREME stub of sparse_index_init() has no annotation but the SPARSEMEM_EXTREME variant is __meminit. So mark the stub __meminit too. mminit_validate_memmodel_limits() is only called by memory_present(), which is __init. So mark it __init. sparse_usagebuf and sparse_usagebuf_end are only used by sparse_init_early_section(), sparse_usage_init() and sparse_usage_fini(), which are all __init. So mark them __initdata. Link: https://lore.kernel.org/20260731164758.1210668-1-ekffu200098@gmail.com Signed-off-by: Sang-Heon Jeon <ekffu200098@gmail.com> Reviewed-by: Andrew Morton <akpm@linux-foundation.org> Acked-by: David Hildenbrand (Arm) <david@kernel.org> Acked-by: Mike Rapoport (Microsoft) <rppt@kernel.org> Reviewed-by: Anshuman Khandual <anshuman.khandual@arm.com> Cc: Liam R. Howlett <liam@infradead.org> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Pratyush Mallick
|
7a39f03bc9 |
mm/page_reporting: add page_reporting_delay_ms module parameter
Free page reporting currently hardcodes a 2-second interval between reports. This rigid delay cannot accommodate diverse guest workloads. This patch introduces a module parameter, page_reporting_delay_ms (default: 2000), allowing users to tune the reporting rate: - Lower values enable aggressive memory reclamation by returning unused pages to the host immediately. - Higher values help batch pages during spiky allocation/free churn, reducing hypercalls and nested page fault overheads. Setting the delay to 0 is safe and execution is strictly gated by: - reporting is only triggered by high-order page frees. - expensive hypercalls are bounded by a slot capacity watermark check before proceeding. Link: https://lore.kernel.org/20260731193705.2902728-1-pratmal@google.com Signed-off-by: Pratyush Mallick <pratmal@google.com> Reviewed-by: SJ Park <sj@kernel.org> Acked-by: Lorenzo Stoakes (ARM) <ljs@kernel.org> Acked-by: David Hildenbrand (Arm) <david@kernel.org> Cc: Anshuman Khandual <anshuman.khandual@arm.com> Cc: Brendan Jackman <jackmanb@google.com> Cc: Greg Thelen <gthelen@google.com> Cc: Johannes Weiner <hannes@cmpxchg.org> Cc: Jonathan Corbet <corbet@lwn.net> Cc: Liam R. Howlett <liam@infradead.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: SeongJae Park <sj@kernel.org> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Guillaume Morin
|
5120b1e048 |
hugetlb: only adjust reservation during unmapping if mapcount is 0
Since |
||
|
Alexander Gordeev
|
62e39381b7 |
mm: use proper PTE accessor in move_ptes()
Follow the pattern established by commit |
||
|
Shakeel Butt
|
6b0d108336 |
memcg: bypass the reclaim and oom killer for dying tasks once oom_reaper is done
At Meta, we are seeing instances where an OOM killed job is stuck in the
exit path for several hours. In one particular case, the job was stuck
for more than 8 hours and I had to manually remove the memory.max limits
to allow the process to exit.
The job was a single process job and had ~55 GiB memory.max and zswap
enabled. It had almost 0 anon in memory and ~111 GiB in zswap compressed
to ~51 GiB zswap pool (i.e. almost all of memory.current was zswap).
Nothing was left on the LRUs to reclaim.
On further inspection, I observed ~20k threads of that process stuck with
the following stack:
[<0>] mem_cgroup_out_of_memory+0x4e/0xa0
[<0>] charge_memcg+0x8bf/0x990
[<0>] mem_cgroup_swapin_charge_folio+0x4e/0x80
[<0>] __read_swap_cache_async+0x10c/0x260
[<0>] swapin_readahead+0x116/0x3f0
[<0>] do_swap_page+0x13c/0x1ce0
[<0>] handle_mm_fault+0x61d/0x11f0
[<0>] do_user_addr_fault+0x3e7/0x6d0
[<0>] exc_page_fault+0x8f/0x110
[<0>] asm_exc_page_fault+0x22/0x30
[<0>] __get_user_8+0x14/0x20
[<0>] futex_cleanup+0x27/0x1c0
[<0>] futex_exit_release+0x47/0x60
[<0>] do_exit+0x107/0x940
[<0>] do_group_exit+0x81/0xa0
[<0>] get_signal+0x2b1/0x6e0
[<0>] arch_do_signal_or_restart+0x1a/0x1c0
[<0>] exit_to_user_mode_loop+0xa8/0x1c0
[<0>] do_syscall_64+0x152/0x250
[<0>] entry_SYSCALL_64_after_hwframe+0x4b/0x53
In addition the dmesg was filled with "Out of memory and no killable
processes..." messages.
I have no idea why oom reaper was not able to reap/unmap the process. My
guess is that since oom reaper tries to acquire mmap_lock in read mode
limited number of times and then gives up, there might be a thread of that
process which had mmap_lock in write mode at that time.
My initial suspicion was the futex_cleanup and kernel page fault causing
infinite fault and charge retries but that was put to rest in previous
discussions happened on similar problem [1].
My current theory is that it is just a simple slow serialization behind
the oom_lock. Unlike page allocator, memcg charge code takes the oom_lock
without the "try". Though memcg oom code uses mutex_lock_killable(), note
that in the call stack get_signal() consumes SIGKILL (or
sigdelset(SIGKILL)) before calling do_group_exit(). So this
mutex_lock_killable() is just a mutex_lock() here. Therefore 10s of
thousands of threads are waiting on oom_lock and one by one they get
-EFAULT from get_user() in the futex cleanup code and bails out.
Discussion from [1] led to commit
|
||
|
Guopeng Zhang
|
9477820c63 |
mm: memcg: stop reclaim when a limit update is superseded
kernfs serializes file operations only per open file, so separate open files can update the same memory.high or memory.max file concurrently. Both handlers store the new limit before synchronous reclaim, but continue to use the writer's local target in the reclaim loop. If another writer raises or removes the limit, the first writer can continue reclaiming toward a stale target. For memory.max, this can leave the writer looping indefinitely once reclaim retries are exhausted. The OOM path sees sufficient margin under the current limit and returns true without killing, while the writer still compares usage against its stale target and records another OOM event. Check the current limit at the start of each reclaim iteration and stop if it no longer matches the writer's target. Reproducer: Populate a cgroup with anonymous memory and disable swapping. Lower memory.max from one open file, then restore it to "max" through another open file after the new limit becomes visible. Without the patch, the first writer remains blocked and repeatedly increments the OOM event counter. With the patch, it returns normally. This was not motivated by a reported production workload. We found it through automated randomized testing for our cgroup observability work and reduced it to the reproducer above. Link: https://lore.kernel.org/20260724021805.1234583-1-guopeng.zhang@linux.dev Fixes: |
||
|
Usama Arif
|
7b9f4e5f81 |
mm/vmscan: reduce lru_lock contention via vmstat-derived scan-balance cost
The anon/file scan balance in get_scan_count() is driven by two scalars in
struct lruvec, anon_cost and file_cost, accumulated by every reclaim
producer under lruvec->lru_lock. The acquisition sites for cost work
specifically are:
- shrink_inactive_list() re-takes lru_lock at function exit purely
to call lru_note_cost_unlock_irq() with (nr_pageout, nr_scanned -
nr_reclaimed). One acquisition per inactive shrink.
- shrink_active_list() does the same with (0, nr_rotated). One
acquisition per active shrink.
- workingset_refault() takes the lock via folio_lruvec_lock_irq()
purely to record the refault cost. One acquisition per refault.
- prepare_scan_control() takes lru_lock just to snapshot the two
scalars into sc->{anon,file}_cost.
- lru_note_cost_unlock_irq() itself walks parent_lruvec and
re-acquires lru_lock on each ancestor to propagate the update,
adding O(memcg-depth) acquisitions per producer call.
This hurts because lru_lock is already a heavy contention point on
memory-heavy workloads: every isolate_lru_folios(), move_folios_to_lru()
and folio_add_lru() takes it. The cost work itself is trivial (two scalar
bumps and one comparison), but it contends with and causes contention for
actual LRU manipulation. The parent_lruvec() walk also multiplies
cost-update overhead by memcg hierarchy depth.
The balance formula for anon and file, respectively, is this:
cost = nr_io * SWAP_CLUSTER_MAX + nr_rotated
Instead of recording cost and running averaging logic directly when these
events occur, snapshot running vmstat counters once per reclaim cycle and
derive the balance from event deltas since the last run.
Use PGROTATE_* from the preceding patch for the rotation input.
WORKINGSET_RESTORE_* and NR_VMSCAN_WRITE provide the remaining event
counters. Charge NR_VMSCAN_WRITE through lruvec stats so all inputs can
be sampled per lruvec and aggregated through the memcg hierarchy. This is
overall cheaper and has fewer lock acquisition sites.
Moving accumulation and decay to the reclaim side also improves the cost
model across reclaim gaps. With producer-side decay, events that happen
while reclaim is idle still age each other before reclaim ever samples the
costs. If a workload refaults a large anon set and then a smaller file
set before reclaim runs again, the later file activity can age the earlier
anon activity out of the cost model. The new scheme observes the whole
between-reclaim delta and decays anon and file proportionally, so the
scan-balance history better represents what happened since the last
reclaim pass.
A dedicated per-lruvec spinlock, cost_lock, serialises the delta
extraction, the cost->count update and the halving loop against concurrent
reclaimers in the same memcg+node.
NR_VMSCAN_WRITE is accounted at writeout(), so reclaim_stat.nr_pageout is
no longer needed and is removed.
memcg-v1's memory.stat anon_cost/file_cost is now sourced from
cost[].count instead of the removed lruvec anon_cost/file_cost fields.
The reported values only refresh when prepare_scan_control() runs and are
bounded at ~lrusize/4 by the halving loop; the scan-balance signal they
express is unchanged.
Under pure MGLRU the scan-balance signal itself is not consumed (both
prepare_scan_control() and get_scan_count() are short-circuited on the
MGLRU paths, and MGLRU's own type/tier selection comes from
read_ctrl_pos() on lrugen->{avg_refaulted,avg_total,refaulted,evicted},
not from anon_cost/file_cost). NR_VMSCAN_WRITE naturally covers writeout
from either reclaim implementation. The preceding patch also bumps
PGROTATE_{ANON,FILE} from evict_folios(), so rotation-driven reclaim work
is accounted consistently across both implementations.
Link: https://lore.kernel.org/20260727162550.2032-4-usama.arif@linux.dev
Signed-off-by: Usama Arif <usama.arif@linux.dev>
Acked-by: Shakeel Butt <shakeel.butt@linux.dev>
Acked-by: Johannes Weiner <hannes@cmpxchg.org>
Acked-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Cc: Axel Rasmussen <axelrasmussen@google.com>
Cc: Baoquan He <baoquan.he@linux.dev>
Cc: Chris Li <chrisl@kernel.org>
Cc: David Hildenbrand <david@kernel.org>
Cc: David Rientjes <rientjes@google.com>
Cc: Kairui Song <kasong@tencent.com>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Muchun Song <muchun.song@linux.dev>
Cc: Nhat Pham <nphamcs@gmail.com>
Cc: Roman Gushchin <roman.gushchin@linux.dev>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Wei Xu <weixugc@google.com>
Cc: Yuanchu Xie <yuanchu@google.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
||
|
Usama Arif
|
1b089def0f |
mm/vmscan: add pgrotate_anon and pgrotate_file vmstat counters
Reclaim can spend substantial work on an LRU type without immediately
reclaiming or demoting a corresponding amount of memory. Record this work
in PGROTATE_ANON and PGROTATE_FILE.
For classic LRU reclaim:
- Inactive-list reclaim adds nr_scanned - nr_reclaimed to the
corresponding anon/file counter when isolation succeeds.
- Active-list reclaim adds referenced executable file folios that
are retained on the active list to PGROTATE_FILE. Active anon
reclaim does not contribute this component.
For MGLRU, add the number of initially isolated pages that remain
unreclaimed after both the initial and retry passes to the counter for the
selected anon/file type.
These counters are distinct from the existing pgrotated vm event.
pgrotated records an actual move to the inactive-list tail, primarily
after reclaim-marked writeback completes or failed invalidation leaves a
folio for accelerated reclaim. PGROTATE_ANON and PGROTATE_FILE measure
reclaim cost and do not imply that a folio moved to an LRU tail.
A subsequent patch will consume these counters for anon/file scan
balancing.
Link: https://lore.kernel.org/20260727162550.2032-3-usama.arif@linux.dev
Signed-off-by: Usama Arif <usama.arif@linux.dev>
Acked-by: Shakeel Butt <shakeel.butt@linux.dev>
Acked-by: Johannes Weiner <hannes@cmpxchg.org>
Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Cc: Axel Rasmussen <axelrasmussen@google.com>
Cc: Baoquan He <baoquan.he@linux.dev>
Cc: Chris Li <chrisl@kernel.org>
Cc: David Hildenbrand <david@kernel.org>
Cc: David Rientjes <rientjes@google.com>
Cc: Kairui Song <kasong@tencent.com>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Muchun Song <muchun.song@linux.dev>
Cc: Nhat Pham <nphamcs@gmail.com>
Cc: Roman Gushchin <roman.gushchin@linux.dev>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Wei Xu <weixugc@google.com>
Cc: Yuanchu Xie <yuanchu@google.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
||
|
Usama Arif
|
a69797fb36 |
mm/vmstat, mm/memcontrol: add _monotonic vmstat readers
Patch series "mm/vmscan: reduce lru_lock contention via vmstat-derived
scan-balance cost", v5.
The anon/file scan balance heuristic in get_scan_count() is fed by two
scalars in struct lruvec (anon_cost, file_cost) that every reclaim
producer updates under lruvec->lru_lock. The cost-recording work itself
is trivial, but it both contends for and contributes to contention on
lru_lock - which is often a contention point on memory-pressured
workloads. Specifically:
- shrink_inactive_list() re-acquires lru_lock at function exit just
to call lru_note_cost_unlock_irq().
- shrink_active_list() does the same after rotation accounting.
- workingset_refault() takes folio_lruvec_lock_irq() purely to
record the refault cost.
- prepare_scan_control() snapshots anon_cost/file_cost under
lru_lock.
- lru_note_cost_unlock_irq() itself walks parent_lruvec() and
re-acquires lru_lock on every ancestor, multiplying the cost
of every update by memcg-hierarchy depth.
This series removes those producer-side acquisitions entirely. The
rotation inputs become per-LRU PGROTATE_{ANON,FILE} vmstat counters.
NR_VMSCAN_WRITE already captures reclaim-driven pageout at writeout();
charge it through lruvec_stat_mod_folio() so it is available per lruvec
and aggregated through the memcg hierarchy. Reclaim does not submit
filesystem folios for writeback from this path, so pageout contributes
only to anon cost. WORKINGSET_RESTORE_* already captures the refault
input.
PGROTATE_* are also useful independently of scan balancing. They are
cumulative base-page events, not unique-page counts. Classic inactive
reclaim records scan work that does not produce immediate reclaim or
demotion, while active reclaim records referenced executable file folios
retained on the active list. MGLRU records initially isolated pages that
remain unreclaimed after its retry passes. Read alongside pgscan_* and
pgsteal_*, their deltas identify which LRU type is consuming reclaim CPU
without producing immediate yield. Unlike the existing pgrotated event,
they do not imply a move to the inactive-list tail.
prepare_scan_control() reads the raw cost signals without lru_lock:
anon = PGROTATE_ANON +
(NR_VMSCAN_WRITE + WORKINGSET_RESTORE_ANON) * SWAP_CLUSTER_MAX
file = PGROTATE_FILE +
WORKINGSET_RESTORE_FILE * SWAP_CLUSTER_MAX
It folds the deltas into a per-lruvec accumulator. A dedicated per-lruvec
cost_lock, not touched by isolate_lru_folios(), move_folios_to_lru(), or
folio_add_lru(), serialises the accumulator RMW and the lrusize/4 halving
check. Hierarchy aggregation is implicit in rstat propagation, so the
parent_lruvec() walk and the lru_reparent_memcg() cost-splice both
disappear.
Moving accumulation and decay to the reclaim side also improves the cost
model across reclaim gaps. With producer-side decay, events that happen
while reclaim is idle still age each other before reclaim ever samples the
costs. If a workload refaults a large anon set and then a smaller file
set before reclaim runs again, the later file activity can age the earlier
anon activity out of the cost model. The new scheme observes the whole
between-reclaim delta and decays anon and file proportionally, so the
scan-balance history better represents what happened since the last
reclaim pass.
Trade-offs:
- Cost reads see rstat-aggregated values that can lag until periodic /
reader-triggered flushing.
- Per-lruvec footprint grows by 4 unsigned longs + a spinlock (a
struct lru_cost { count, last_rotated, last_io } per side), which
is a small cost.
- NR_VMSCAN_WRITE now also updates the folio's lruvec/memcg stat,
adding memcg stat accounting to the reclaim writeout path while
preserving the existing node-level total.
== Numbers ==
Tested on a 176-core, 256 GB host. The benchmark drives sustained
swap-out/refault inside a tight memcg using vm-scalability/usemem:
usemem -n 16 --prealloc --prefault --random $((256*1024*1024))
run inside a two-level memcg with memory.max=512M on the leaf (4 GB anon
working set has to fit in 512 MB -> continuous shrink_inactive_list +
workingset_refault). A 16 GB swap file is used. Measurement is a 30 s
`perf lock record -a` window over otherwise-idle hardware.
Workload rates are identical on both kernels (the bench drives the same
memory pressure):
baseline patched delta
pgscan_direct / s 172,662 171,817 ~0%
pgsteal_direct / s 67,162 66,306 ~0%
workingset_refault_anon / s
40,696 39,830 ~0%
perf lock contention (total wait per 30 s window):
Lock Name Before After % change
shrink_lruvec+0x770 722.84 ms 0 -100% (eliminated)
(= lru_note_cost_unlock_irq)
workingset_refault+0x167 385.26 ms 0 -100% (eliminated)
(= lru_note_cost_refault)
shrink_node+0x4ad 689.43 ms 26.95 ms -96%
shrink_active_list 208.34 ms 15.97 ms -92%
lru_add_drain_cpu+0x34 1.96 s 917.71 ms -53%
Total LRU lock wait ~4.23 s ~1.66 s -61%
The two specific contention sites the patch removes (shrink_lruvec+0x770 =
lru_note_cost_unlock_irq; workingset_refault+0x167 =
lru_note_cost_refault) are completely absent from the patched
perf-lock-contention output. Secondary reductions in shrink_node,
shrink_active_list, lru_add_drain_cpu and pgrefill/pgactivate look like
knock-on effects from removing the cost-recording overhead and the
parent_lruvec walk.
The remaining ~1.66 s of LRU lock wait on the patched kernel is dominated
by the per-CPU pagevec drain (lru_add_drain_cpu) and the main reclaim path
in shrink_lruvec.
The numbers above can be reproduced using the script in [1].
This patch (of 3):
lruvec_page_state(), node_page_state(), and global_node_page_state() all
clamp negative reads to zero on CONFIG_SMP so that a transient per-CPU
delta skew presents as zero pages rather than as a garbage unsigned value.
This is the right behaviour for non-monotonic page-count readers.
It is however incorrect for callers that snapshot a monotonically-
incremented event counter and compute a delta from two samples. Once the
underlying signed long wraps past LONG_MAX, the clamped read drops to zero
while the previously-recorded snapshot still holds the pre-wrap value; the
unsigned subtraction then underflows into a ~2^31 spurious delta for
32-bit architecture and corrupts the caller's accumulator.
Add non-clamping siblings that return the underlying state value cast to
unsigned long:
global_node_page_state_monotonic()
node_page_state_monotonic()
lruvec_page_state_monotonic()
With both samples read via the _monotonic variant, unsigned modular
subtraction stays correct across a signed-long wraparound as long as the
true growth between two samples fits in unsigned long (< 2^32 on 32-bit, <
2^64 on 64-bit); the 32-bit bound is the practically-reachable one that
motivates this helper.
The variants are only safe for monotonically-incremented counters.
Non-monotonic page-count readers must keep using the existing clamped
helpers so transient negative reads still present as zero.
This is a prerequisite for a later patch which replaces the producer-side
anon_cost/file_cost accumulators with a read-side accumulator in
prepare_scan_control() that samples monotonic per-LRU vmstat counters
(PGROTATE_*, NR_VMSCAN_WRITE, WORKINGSET_RESTORE_*) via
lruvec_page_state_monotonic() and folds their unsigned modular deltas into
lruvec->cost[].count.
Link: https://lore.kernel.org/20260727162550.2032-1-usama.arif@linux.dev
Link: https://lore.kernel.org/20260727162550.2032-2-usama.arif@linux.dev
Link: https://gist.github.com/uarif1/a4eb33a86c5b2d7bbc55b42f0956e884 [1]
Signed-off-by: Usama Arif <usama.arif@linux.dev>
Acked-by: Johannes Weiner <hannes@cmpxchg.org>
Acked-by: Shakeel Butt <shakeel.butt@linux.dev>
Acked-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Cc: Axel Rasmussen <axelrasmussen@google.com>
Cc: Baoquan He <baoquan.he@linux.dev>
Cc: Chris Li <chrisl@kernel.org>
Cc: David Hildenbrand <david@kernel.org>
Cc: David Rientjes <rientjes@google.com>
Cc: Kairui Song <kasong@tencent.com>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Muchun Song <muchun.song@linux.dev>
Cc: Nhat Pham <nphamcs@gmail.com>
Cc: Roman Gushchin <roman.gushchin@linux.dev>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Wei Xu <weixugc@google.com>
Cc: Yuanchu Xie <yuanchu@google.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
||
|
Ridong Chen
|
8a90519585 |
mm: vmscan: fix node reclaim ignoring swappiness parameter
sc_swappiness() had two separate definitions depending on CONFIG_MEMCG.
The !CONFIG_MEMCG variant simply returned vm_swappiness, ignoring the
proactive_swappiness value passed through scan_control. This caused the
swappiness parameter written to /sys/devices/system/node/nodeX/reclaim to
have no effect when CONFIG_MEMCG is disabled.
Fix this by consolidating sc_swappiness() into a single definition that
checks sc->proactive_swappiness first, then falls back to
mem_cgroup_swappiness() which already handles both CONFIG_MEMCG and
!CONFIG_MEMCG.
Before fix (swappiness=max ignored, mostly file pages reclaimed):
# cat /proc/sys/vm/swappiness
60
# cat /proc/vmstat | grep pgsteal
pgsteal_kswapd 0
pgsteal_direct 0
pgsteal_khugepaged 0
pgsteal_proactive 1840
pgsteal_anon 25
pgsteal_file 1815
# echo "64M swappiness=max" > /sys/devices/system/node/node0/reclaim
# cat /proc/vmstat | grep pgsteal
pgsteal_kswapd 0
pgsteal_direct 0
pgsteal_khugepaged 0
pgsteal_proactive 18013
pgsteal_anon 337
pgsteal_file 17676
After fix (swappiness=max honored, anon pages reclaimed as expected):
# cat /proc/vmstat | grep pgsteal
pgsteal_kswapd 0
pgsteal_direct 0
pgsteal_khugepaged 0
pgsteal_proactive 0
pgsteal_anon 0
pgsteal_file 0
# echo "64M swappiness=max" > /sys/devices/system/node/node0/reclaim
# cat /proc/vmstat | grep pgsteal
pgsteal_kswapd 0
pgsteal_direct 0
pgsteal_khugepaged 0
pgsteal_proactive 16283
pgsteal_anon 16283
pgsteal_file 0
Link: https://lore.kernel.org/20260723032434.2016749-3-ridong.chen@linux.dev
Fixes:
|
||
|
Ridong Chen
|
1b7c8fe294 |
memcg: move mem_cgroup_swappiness and vm_swappiness to mm/swap.h
Patch series "mm: vmscan: fix node reclaim ignoring swappiness parameter", v4. The per-node proactive reclaim interface (/sys/devices/system/node/nodeX/reclaim) accepts a swappiness parameter, but it is silently ignored when CONFIG_MEMCG is disabled. The root cause is that sc_swappiness() has separate implementations for CONFIG_MEMCG and !CONFIG_MEMCG, and the latter never checks proactive_swappiness. Patch 1 moves mem_cgroup_swappiness() and vm_swappiness out of the public include/linux/swap.h into the mm-private mm/swap.h, and makes the helper handle both CONFIG_MEMCG and !CONFIG_MEMCG in a single inline function. This is a prerequisite for unifying sc_swappiness(). Patch 2 consolidates sc_swappiness() into a single definition that works regardless of CONFIG_MEMCG, fixing the node reclaim swappiness bug. This patch (of 2): The per-memcg swappiness knob is v1-only; v2 always uses global vm_swappiness and ignores the per-cgroup field. Both mem_cgroup_swappiness() and vm_swappiness are only used within mm/ (memcontrol.c, memcontrol-v1.c, vmscan.c), so move them out of the public include/linux/swap.h into the mm-private mm/swap.h. This keeps unrelated declarations out of include/linux/swap.h. Guard memcg->swappiness with CONFIG_MEMCG_V1 as well, so v2-only kernels drop the unused field. No functional change for v1; v2-only kernels drop the unused field. Link: https://lore.kernel.org/20260723032434.2016749-1-ridong.chen@linux.dev Link: https://lore.kernel.org/20260723032434.2016749-2-ridong.chen@linux.dev Signed-off-by: Ridong Chen <chenridong@xiaomi.com> Acked-by: Johannes Weiner <hannes@cmpxchg.org> Reviewed-by: Barry Song <baohua@kernel.org> Reviewed-by: Song Hu <husong@kylinos.cn> Acked-by: Shakeel Butt <shakeel.butt@linux.dev> Cc: Axel Rasmussen <axelrasmussen@google.com> Cc: Baoquan He <baoquan.he@linux.dev> Cc: Chris Li <chrisl@kernel.org> Cc: David Hildenbrand <david@kernel.org> Cc: Davidlohr Bueso <dave@stgolabs.net> Cc: Kairui Song <kasong@tencent.com> Cc: Kemeng Shi <shikemeng@huaweicloud.com> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Michal Hocko <mhocko@kernel.org> Cc: Muchun Song <muchun.song@linux.dev> Cc: Nhat Pham <nphamcs@gmail.com> Cc: Roman Gushchin <roman.gushchin@linux.dev> Cc: Wei Xu <weixugc@google.com> Cc: Yuanchu Xie <yuanchu@google.com> Cc: Qi Zheng <qi.zheng@linux.dev> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Baolin Wang
|
0ee06ee38a |
mm: mglru: promote mapped executable folios after first usage
Classical LRU protects mapped executable file folios through commit |
||
|
Baolin Wang
|
b64727d264 |
mm: vmscan: add a helper to identify file-backed executable folios
Add a helper to identify file-backed executable folios to avoid duplicate code. No functional changes. Link: https://lore.kernel.org/2ee74f9f98ac45a2f0db0ceb018e086bdb671d17.1784509721.git.baolin.wang@linux.alibaba.com Signed-off-by: Baolin Wang <baolin.wang@linux.alibaba.com> Acked-by: Johannes Weiner <hannes@cmpxchg.org> Reviewed-by: Axel Rasmussen <axelrasmussen@google.com> Reviewed-by: David Hildenbrand (Arm) <david@kernel.org> Reviewed-by: Kairui Song <kasong@tencent.com> Reviewed-by: Barry Song <baohua@kernel.org> Cc: Harry Yoo <harry@kernel.org> Cc: Jann Horn <jannh@google.com> Cc: Lance Yang <lance.yang@linux.dev> Cc: Liam R. Howlett <liam@infradead.org> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Michal Hocko <mhocko@kernel.org> Cc: Rik van Riel <riel@surriel.com> Cc: Shakeel Butt <shakeel.butt@linux.dev> Cc: Vlastimil Babka <vbabka@kernel.org> Cc: Wei Xu <weixugc@google.com> Cc: Yuanchu Xie <yuanchu@google.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Baolin Wang
|
3bf07ce805 |
mm: vmscan: convert folio_referenced() to use vma_flags_t
Patch series "promote mapped executable folios after first usage for MGLRU", v4. Now MGLRU's protection of mapped executable file folios is less reliable. Follow the classical LRU's logic, promoting mapped executable file folios after their first usage to give executable code a better chance to stay in memory and improve workload performance (See patch 2 for more details). This patch (of 3): Replace use of the legacy vm_flags_t flags with vma_flags_t values for folio_referenced() and related logic. This is also a preparation for the following changes. No functional changes. Link: https://lore.kernel.org/cover.1784509721.git.baolin.wang@linux.alibaba.com Link: https://lore.kernel.org/2bd39e16ec19e3e3c4716aa9a1a25775c26cac57.1784509721.git.baolin.wang@linux.alibaba.com Signed-off-by: Baolin Wang <baolin.wang@linux.alibaba.com> Acked-by: David Hildenbrand (Arm) <david@kernel.org> Acked-by: Johannes Weiner <hannes@cmpxchg.org> Reviewed-by: Barry Song <baohua@kernel.org> Reviewed-by: Kairui Song <kasong@tencent.com> Reviewed-by: Axel Rasmussen <axelrasmussen@google.com> Cc: Harry Yoo <harry@kernel.org> Cc: Jann Horn <jannh@google.com> Cc: Lance Yang <lance.yang@linux.dev> Cc: Liam R. Howlett <liam@infradead.org> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Michal Hocko <mhocko@kernel.org> Cc: Rik van Riel <riel@surriel.com> Cc: Shakeel Butt <shakeel.butt@linux.dev> Cc: Vlastimil Babka <vbabka@kernel.org> Cc: Wei Xu <weixugc@google.com> Cc: Yuanchu Xie <yuanchu@google.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Breno Leitao
|
3541a2b06e |
mm/kmemleak: report RCU-tasks quiescent states during the scan
kmemleak_scan() can run for ages on large debug kernels. It was
causing some soft-lockups which I got fixed with commit
3175fcfec8b16baeb ("mm/kmemleak: avoid soft lockup when scanning task
stacks") with our beloved cond_resched().
I've got the fix above deployed in the Meta fleet, and now I am seeing:
INFO: rcu_tasks detected stalls on tasks:
task:kmemleak state:R ... nvcsw: 274/274 holdout: 1 idle_cpu: -1/3
scan_block
scan_gray_list
kmemleak_scan
and, worse, blocks the callers waiting on that grace period. Here a BPF
struct_ops map free, which waits via synchronize_rcu_mult(call_rcu,
call_rcu_tasks), is stuck long enough to also trip the hung task check:
INFO: task kworker/...:bpf_map_free_deferred blocked for 122 seconds
__wait_rcu_gp
bpf_struct_ops_map_free
Then I've learned that cond_resched() is not an RCU-tasks quiescent
state, so, we need to use stronger primitives.
Use cond_resched_tasks_rcu_qs() at the scan reschedule points so the scan
reports an RCU-tasks quiescent state as it proceeds.
Inspired by commit
|
||
|
Lorenzo Stoakes (ARM)
|
93c0c8dc87 |
mm/rmap: use anon pgoff to track MAP_PRIVATE file-backed anon folios
Currently anonymous folios belonging to CoW'd MAP_PRIVATE file-backed
mappings are indexed by their page offset within the file in which they
were originally mapped.
This differs from anonymous folios belonging to pure anon mappings which
are indexed by their anonymous page offset (the address at which they'd
belong in the VMA when first faulted).
This change fixes this inconsistency, always indexing anonymous folios by
their anonymous page offset regardless of the VMA to which they belong.
The foundations have been laid such that we need only switch this
functionality on such by:
* Using linear_anon_page_index() in __folio_set_anon() to assign the
folio's index to the anonymous linear index rather than the file-backed
one.
* Otherwise using linear_anon_page_index() in all instances where
anonymous folios are being referenced or manipulated.
* Replacing vma_address() with vma_filebacked_address() or
vma_anon_address() as appropriate.
* Updating the merging logic to check that anonymous page offsets are
aligned as well as filebacked ones for MAP_PRIVATE file-backed VMAs,
introducing needs_adjacent_anon_pgoff() to figure out when this is
required.
* Updating linear_folio_page_index() to invoke linear_anon_page_index()
if the folio is anonymous.
* Updating vma_address_end() to use the VMA's anonymous page offset when
pvmw->pgoff is anonymous.
* Correcting folio_within_range() to use anonymous page offset for
anonymous folios.
This will have no impact on merging of anonymous VMAs, whose page offset
and anonymous page offset are identical, nor will it impact shared
file-backed VMAs, which will continue to be merged based on the
file-backed page offset.
However, MAP_PRIVATE file-backed mappings must now be aligned on anonymous
page offset as well.
In most instances this should have no impact on merging of file-backed
mappings, which are usually not merged all that often, let alone
MAP_PRIVATE mapped ones, and rarely remapped and faulted before being
moved back in place (the case in which a merge may now fail).
One subtle impact of this change is in NUMA interleaving - since commit
|
||
|
Lorenzo Stoakes (ARM)
|
5f653f8b7a |
mm/migrate: calculate large folio page index using PFN
Rather than having to figure out the page index to use using linear_page_index(), calculate it using PFN. This is a more natural fit as the linear page index is immaterial to determining the folio page index. Derive the page index from the offset between migration entry PFN and folio PFN - pvmw.pfn (set via DEFINE_FOLIO_VMA_WALK() which uses folio_pfn() to obtain it). Additionally remove a not so useful comment and clean the code layout up. No functional change intended. Link: https://lore.kernel.org/20260813-b4-scalable-cow-virt-pgoff-v5-13-c21581c0c3c8@kernel.org Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org> Suggested-by: David Hildenbrand (Arm) <david@kernel.org> Acked-by: David Hildenbrand (Arm) <david@kernel.org> Cc: Adrian Hunter <adrian.hunter@intel.com> Cc: Alexander Deucher <alexander.deucher@amd.com> Cc: Alexander Gordeev <agordeev@linux.ibm.com> Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com> Cc: Alistair Popple <apopple@nvidia.com> Cc: Arnaldo Carvalho de Melo <acme@kernel.org> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Baolin Wang <baolin.wang@linux.alibaba.com> Cc: Baoquan He <baoquan.he@linux.dev> Cc: Barry Song <baohua@kernel.org> Cc: Boris Brezillon <boris.brezillon@collabora.com> Cc: Byungchul Park <byungchul@sk.com> Cc: Chengming Zhou <chengming.zhou@linux.dev> Cc: Chris Li <chrisl@kernel.org> Cc: Christan König <christian.koenig@amd.com> Cc: Christian Borntraeger <borntraeger@linux.ibm.com> Cc: Claudio Imbrenda <imbrenda@linux.ibm.com> Cc: Dave Airlie <airlied@gmail.com> Cc: Dev Jain <dev.jain@arm.com> Cc: Gerald Schaefer <gerald.schaefer@linux.ibm.com> Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org> Cc: Gregory Price (Meta) <gourry@gourry.net> Cc: Harry Yoo <harry@kernel.org> Cc: Heiko Carstens <hca@linux.ibm.com> Cc: Huang Ray <Ray.Huang@amd.com> Cc: "Huang, Ying" <ying.huang@linux.alibaba.com> Cc: Ian Rogers <irogers@google.com> Cc: Ingo Molnar <mingo@redhat.com> Cc: James Clark <james.clark@linaro.org> Cc: Jan Kara <jack@suse.cz> Cc: Jann Horn <jannh@google.com> Cc: Janosch Frank <frankja@linux.ibm.com> Cc: Jason Gunthorpe <jgg@ziepe.ca> Cc: Jiri Olsa <jolsa@kernel.org> Cc: John Hubbard <jhubbard@nvidia.com> Cc: Joshua Hahn <joshua.hahnjy@gmail.com> Cc: Kairui Song <kasong@tencent.com> Cc: Kees Cook <kees@kernel.org> Cc: Kemeng Shi <shikemeng@huaweicloud.com> Cc: Lance Yang <lance.yang@linux.dev> Cc: Liam R. Howlett <liam@infradead.org> Cc: Liviu Dudau <liviu.dudau@arm.com> Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com> Cc: Marc Rutland <mark.rutland@arm.com> Cc: "Masami Hiramatsu (Google)" <mhiramat@kernel.org> Cc: Matthew Auld <matthew.auld@intel.com> Cc: Matthew Brost <matthew.brost@intel.com> Cc: Matthew Wilcox (Oracle) <willy@infradead.org> Cc: Maxime Ripard <mripard@kernel.org> Cc: Miaohe Lin <linmiaohe@huawei.com> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Muchun Song <muchun.song@linux.dev> Cc: Namhyung kim <namhyung@kernel.org> Cc: Naoya Horiguchi <nao.horiguchi@gmail.com> Cc: Nhat Pham <nphamcs@gmail.com> Cc: Nico Pache <npache@redhat.com> Cc: Oleg Nesterov <oleg@redhat.com> Cc: Oscar Salvador <osalvador@suse.de> Cc: Pedro Falcato <pfalcato@suse.de> Cc: Peter Xu <peterx@redhat.com> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Rakie Kim <rakie.kim@sk.com> Cc: Rik van Riel <riel@surriel.com> Cc: Rodrigo Vivi <rodrigo.vivi@intel.com> Cc: Ryan Roberts <ryan.roberts@arm.com> Cc: Steven Price <steven.price@arm.com> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Sven Schnelle <svens@linux.ibm.com> Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com> Cc: Thomas Zimemrmann <tzimmermann@suse.de> Cc: Vasily Gorbik <gor@linux.ibm.com> Cc: Vlastimil Babka <vbabka@kernel.org> Cc: xu xin <xu.xin16@zte.com.cn> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Lorenzo Stoakes (ARM)
|
8e658ecc3b |
mm/huge_memory: update remove_migration_pmd() to accept a folio
This function does not need to accept a page and requiring it to is unnecessary and misleading. make_[writable, readable]_device_private_entry() must be passed a PMD-aligned PFN as they immediately used to obtain a softleaf PMD entry and the same argument applies to folio_add_[anon, file]_rmap_pmd(). While we are here, update a VM_BUG_ON() to a VM_WARN_ON_ONCE(). No functional change intended. Link: https://lore.kernel.org/20260813-b4-scalable-cow-virt-pgoff-v5-12-c21581c0c3c8@kernel.org Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org> Acked-by: David Hildenbrand (Arm) <david@kernel.org> Cc: Adrian Hunter <adrian.hunter@intel.com> Cc: Alexander Deucher <alexander.deucher@amd.com> Cc: Alexander Gordeev <agordeev@linux.ibm.com> Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com> Cc: Alistair Popple <apopple@nvidia.com> Cc: Arnaldo Carvalho de Melo <acme@kernel.org> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Baolin Wang <baolin.wang@linux.alibaba.com> Cc: Baoquan He <baoquan.he@linux.dev> Cc: Barry Song <baohua@kernel.org> Cc: Boris Brezillon <boris.brezillon@collabora.com> Cc: Byungchul Park <byungchul@sk.com> Cc: Chengming Zhou <chengming.zhou@linux.dev> Cc: Chris Li <chrisl@kernel.org> Cc: Christan König <christian.koenig@amd.com> Cc: Christian Borntraeger <borntraeger@linux.ibm.com> Cc: Claudio Imbrenda <imbrenda@linux.ibm.com> Cc: Dave Airlie <airlied@gmail.com> Cc: Dev Jain <dev.jain@arm.com> Cc: Gerald Schaefer <gerald.schaefer@linux.ibm.com> Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org> Cc: Gregory Price (Meta) <gourry@gourry.net> Cc: Harry Yoo <harry@kernel.org> Cc: Heiko Carstens <hca@linux.ibm.com> Cc: Huang Ray <Ray.Huang@amd.com> Cc: "Huang, Ying" <ying.huang@linux.alibaba.com> Cc: Ian Rogers <irogers@google.com> Cc: Ingo Molnar <mingo@redhat.com> Cc: James Clark <james.clark@linaro.org> Cc: Jan Kara <jack@suse.cz> Cc: Jann Horn <jannh@google.com> Cc: Janosch Frank <frankja@linux.ibm.com> Cc: Jason Gunthorpe <jgg@ziepe.ca> Cc: Jiri Olsa <jolsa@kernel.org> Cc: John Hubbard <jhubbard@nvidia.com> Cc: Joshua Hahn <joshua.hahnjy@gmail.com> Cc: Kairui Song <kasong@tencent.com> Cc: Kees Cook <kees@kernel.org> Cc: Kemeng Shi <shikemeng@huaweicloud.com> Cc: Lance Yang <lance.yang@linux.dev> Cc: Liam R. Howlett <liam@infradead.org> Cc: Liviu Dudau <liviu.dudau@arm.com> Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com> Cc: Marc Rutland <mark.rutland@arm.com> Cc: "Masami Hiramatsu (Google)" <mhiramat@kernel.org> Cc: Matthew Auld <matthew.auld@intel.com> Cc: Matthew Brost <matthew.brost@intel.com> Cc: Matthew Wilcox (Oracle) <willy@infradead.org> Cc: Maxime Ripard <mripard@kernel.org> Cc: Miaohe Lin <linmiaohe@huawei.com> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Muchun Song <muchun.song@linux.dev> Cc: Namhyung kim <namhyung@kernel.org> Cc: Naoya Horiguchi <nao.horiguchi@gmail.com> Cc: Nhat Pham <nphamcs@gmail.com> Cc: Nico Pache <npache@redhat.com> Cc: Oleg Nesterov <oleg@redhat.com> Cc: Oscar Salvador <osalvador@suse.de> Cc: Pedro Falcato <pfalcato@suse.de> Cc: Peter Xu <peterx@redhat.com> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Rakie Kim <rakie.kim@sk.com> Cc: Rik van Riel <riel@surriel.com> Cc: Rodrigo Vivi <rodrigo.vivi@intel.com> Cc: Ryan Roberts <ryan.roberts@arm.com> Cc: Steven Price <steven.price@arm.com> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Sven Schnelle <svens@linux.ibm.com> Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com> Cc: Thomas Zimemrmann <tzimmermann@suse.de> Cc: Vasily Gorbik <gor@linux.ibm.com> Cc: Vlastimil Babka <vbabka@kernel.org> Cc: xu xin <xu.xin16@zte.com.cn> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Lorenzo Stoakes (ARM)
|
e7006fbd60 |
mm: clean up vma_address_end()
vma_address_end() is a confusing function with a lot of moving parts so clean it up prior to extending it for anon page indexed mappings. Const-ify some variables and establish pgoff_vma_start and pgoff_end variables to clearly identify the page offset for the start of the VMA and the end of the page offset range specified by the page walk. This simplifies the function significantly and lays the groundwork for a future change to update this function to account for anonymously page indexed folios. No functional change intended. Link: https://lore.kernel.org/20260813-b4-scalable-cow-virt-pgoff-v5-11-c21581c0c3c8@kernel.org Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org> Acked-by: David Hildenbrand (Arm) <david@kernel.org> Cc: Adrian Hunter <adrian.hunter@intel.com> Cc: Alexander Deucher <alexander.deucher@amd.com> Cc: Alexander Gordeev <agordeev@linux.ibm.com> Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com> Cc: Alistair Popple <apopple@nvidia.com> Cc: Arnaldo Carvalho de Melo <acme@kernel.org> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Baolin Wang <baolin.wang@linux.alibaba.com> Cc: Baoquan He <baoquan.he@linux.dev> Cc: Barry Song <baohua@kernel.org> Cc: Boris Brezillon <boris.brezillon@collabora.com> Cc: Byungchul Park <byungchul@sk.com> Cc: Chengming Zhou <chengming.zhou@linux.dev> Cc: Chris Li <chrisl@kernel.org> Cc: Christan König <christian.koenig@amd.com> Cc: Christian Borntraeger <borntraeger@linux.ibm.com> Cc: Claudio Imbrenda <imbrenda@linux.ibm.com> Cc: Dave Airlie <airlied@gmail.com> Cc: Dev Jain <dev.jain@arm.com> Cc: Gerald Schaefer <gerald.schaefer@linux.ibm.com> Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org> Cc: Gregory Price (Meta) <gourry@gourry.net> Cc: Harry Yoo <harry@kernel.org> Cc: Heiko Carstens <hca@linux.ibm.com> Cc: Huang Ray <Ray.Huang@amd.com> Cc: "Huang, Ying" <ying.huang@linux.alibaba.com> Cc: Ian Rogers <irogers@google.com> Cc: Ingo Molnar <mingo@redhat.com> Cc: James Clark <james.clark@linaro.org> Cc: Jan Kara <jack@suse.cz> Cc: Jann Horn <jannh@google.com> Cc: Janosch Frank <frankja@linux.ibm.com> Cc: Jason Gunthorpe <jgg@ziepe.ca> Cc: Jiri Olsa <jolsa@kernel.org> Cc: John Hubbard <jhubbard@nvidia.com> Cc: Joshua Hahn <joshua.hahnjy@gmail.com> Cc: Kairui Song <kasong@tencent.com> Cc: Kees Cook <kees@kernel.org> Cc: Kemeng Shi <shikemeng@huaweicloud.com> Cc: Lance Yang <lance.yang@linux.dev> Cc: Liam R. Howlett <liam@infradead.org> Cc: Liviu Dudau <liviu.dudau@arm.com> Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com> Cc: Marc Rutland <mark.rutland@arm.com> Cc: "Masami Hiramatsu (Google)" <mhiramat@kernel.org> Cc: Matthew Auld <matthew.auld@intel.com> Cc: Matthew Brost <matthew.brost@intel.com> Cc: Matthew Wilcox (Oracle) <willy@infradead.org> Cc: Maxime Ripard <mripard@kernel.org> Cc: Miaohe Lin <linmiaohe@huawei.com> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Muchun Song <muchun.song@linux.dev> Cc: Namhyung kim <namhyung@kernel.org> Cc: Naoya Horiguchi <nao.horiguchi@gmail.com> Cc: Nhat Pham <nphamcs@gmail.com> Cc: Nico Pache <npache@redhat.com> Cc: Oleg Nesterov <oleg@redhat.com> Cc: Oscar Salvador <osalvador@suse.de> Cc: Pedro Falcato <pfalcato@suse.de> Cc: Peter Xu <peterx@redhat.com> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Rakie Kim <rakie.kim@sk.com> Cc: Rik van Riel <riel@surriel.com> Cc: Rodrigo Vivi <rodrigo.vivi@intel.com> Cc: Ryan Roberts <ryan.roberts@arm.com> Cc: Steven Price <steven.price@arm.com> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Sven Schnelle <svens@linux.ibm.com> Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com> Cc: Thomas Zimemrmann <tzimmermann@suse.de> Cc: Vasily Gorbik <gor@linux.ibm.com> Cc: Vlastimil Babka <vbabka@kernel.org> Cc: xu xin <xu.xin16@zte.com.cn> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Lorenzo Stoakes (ARM)
|
50c5f35a64 |
mm/rmap: track whether the page VMA mapped pgoff is anonymous
Update the page_vma_mapped_walk structure to track whether the pgoff being tracked is an anonymous pgoff or not and update the comments to reflect this. This is necessary in order to determine the correct VMA page offset in vma_address_end() when pvmw->nr_pages > 1. Also document that pvmw->pgoff is meaningless for pvmw->nr_pages == 1 and for KSM. Do not set this field where pgoff is not specified. This is laying the groundwork for eventually using anonymous page offsets as the index for all anonymous folios. No functional change intended. Link: https://lore.kernel.org/20260813-b4-scalable-cow-virt-pgoff-v5-10-c21581c0c3c8@kernel.org Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org> Acked-by: David Hildenbrand (Arm) <david@kernel.org> Cc: Adrian Hunter <adrian.hunter@intel.com> Cc: Alexander Deucher <alexander.deucher@amd.com> Cc: Alexander Gordeev <agordeev@linux.ibm.com> Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com> Cc: Alistair Popple <apopple@nvidia.com> Cc: Arnaldo Carvalho de Melo <acme@kernel.org> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Baolin Wang <baolin.wang@linux.alibaba.com> Cc: Baoquan He <baoquan.he@linux.dev> Cc: Barry Song <baohua@kernel.org> Cc: Boris Brezillon <boris.brezillon@collabora.com> Cc: Byungchul Park <byungchul@sk.com> Cc: Chengming Zhou <chengming.zhou@linux.dev> Cc: Chris Li <chrisl@kernel.org> Cc: Christan König <christian.koenig@amd.com> Cc: Christian Borntraeger <borntraeger@linux.ibm.com> Cc: Claudio Imbrenda <imbrenda@linux.ibm.com> Cc: Dave Airlie <airlied@gmail.com> Cc: Dev Jain <dev.jain@arm.com> Cc: Gerald Schaefer <gerald.schaefer@linux.ibm.com> Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org> Cc: Gregory Price (Meta) <gourry@gourry.net> Cc: Harry Yoo <harry@kernel.org> Cc: Heiko Carstens <hca@linux.ibm.com> Cc: Huang Ray <Ray.Huang@amd.com> Cc: "Huang, Ying" <ying.huang@linux.alibaba.com> Cc: Ian Rogers <irogers@google.com> Cc: Ingo Molnar <mingo@redhat.com> Cc: James Clark <james.clark@linaro.org> Cc: Jan Kara <jack@suse.cz> Cc: Jann Horn <jannh@google.com> Cc: Janosch Frank <frankja@linux.ibm.com> Cc: Jason Gunthorpe <jgg@ziepe.ca> Cc: Jiri Olsa <jolsa@kernel.org> Cc: John Hubbard <jhubbard@nvidia.com> Cc: Joshua Hahn <joshua.hahnjy@gmail.com> Cc: Kairui Song <kasong@tencent.com> Cc: Kees Cook <kees@kernel.org> Cc: Kemeng Shi <shikemeng@huaweicloud.com> Cc: Lance Yang <lance.yang@linux.dev> Cc: Liam R. Howlett <liam@infradead.org> Cc: Liviu Dudau <liviu.dudau@arm.com> Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com> Cc: Marc Rutland <mark.rutland@arm.com> Cc: "Masami Hiramatsu (Google)" <mhiramat@kernel.org> Cc: Matthew Auld <matthew.auld@intel.com> Cc: Matthew Brost <matthew.brost@intel.com> Cc: Matthew Wilcox (Oracle) <willy@infradead.org> Cc: Maxime Ripard <mripard@kernel.org> Cc: Miaohe Lin <linmiaohe@huawei.com> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Muchun Song <muchun.song@linux.dev> Cc: Namhyung kim <namhyung@kernel.org> Cc: Naoya Horiguchi <nao.horiguchi@gmail.com> Cc: Nhat Pham <nphamcs@gmail.com> Cc: Nico Pache <npache@redhat.com> Cc: Oleg Nesterov <oleg@redhat.com> Cc: Oscar Salvador <osalvador@suse.de> Cc: Pedro Falcato <pfalcato@suse.de> Cc: Peter Xu <peterx@redhat.com> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Rakie Kim <rakie.kim@sk.com> Cc: Rik van Riel <riel@surriel.com> Cc: Rodrigo Vivi <rodrigo.vivi@intel.com> Cc: Ryan Roberts <ryan.roberts@arm.com> Cc: Steven Price <steven.price@arm.com> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Sven Schnelle <svens@linux.ibm.com> Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com> Cc: Thomas Zimemrmann <tzimmermann@suse.de> Cc: Vasily Gorbik <gor@linux.ibm.com> Cc: Vlastimil Babka <vbabka@kernel.org> Cc: xu xin <xu.xin16@zte.com.cn> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Lorenzo Stoakes (ARM)
|
6a993c7fbc |
mm: propagate VMA anonymous page offset on map, remap, split + merge
We must correctly update VMA anonymous page offset state on all VMA operations that would result in it changing, with special attention given to remapping. We cover most cases by simply updating vma_set_range() to do so (with a new anonymous page offset parameter), but also notably must update the merging and mapping logic to propagate this parameter correctly. The remap logic remains the same - we may update the anonymous page offset if the VMA is unfaulted, but now this applies to MAP_PRIVATE file-backed mappings too, so we update the code to reflect this. Note that we use __linear_anon_page_index() upon remap as the VMA may be shared, in order that we update the field consistently regardless of VMA type. Similarly, pass through anon page offset to the merge logic, updating the vma_merge_struct struct to propagate it, and also use __linear_anon_page_index() to obtain the anonymous page index so it can be safely used for both shared and MAP_PRIVATE file-backed mappings. In copy_vma(), the anonymous page offset is updated regardless of whether the mapping is a CoW mapping or not. This is both to keep the anonymous page offset consistent even for non-CoW mappings (it is set so should at least remain correct) and makes the logic cleaner. A self-merge however remains permitted only for mappings which can have a populated vma->anon_vma and do not require alignment on a separate file offset - that is pure anonymous VMAs, so only set can_self_merge if vma_is_anonymous(). Finally, we update insert_vm_struct() to correctly set the anonymous page offset on insertion of a VMA. We simply ensure state is correctly propagated here, so no functional changes are intended. Also update VMA userland tests to reflect this change. Link: https://lore.kernel.org/20260813-b4-scalable-cow-virt-pgoff-v5-9-c21581c0c3c8@kernel.org Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org> Acked-by: David Hildenbrand (Arm) <david@kernel.org> Cc: Adrian Hunter <adrian.hunter@intel.com> Cc: Alexander Deucher <alexander.deucher@amd.com> Cc: Alexander Gordeev <agordeev@linux.ibm.com> Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com> Cc: Alistair Popple <apopple@nvidia.com> Cc: Arnaldo Carvalho de Melo <acme@kernel.org> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Baolin Wang <baolin.wang@linux.alibaba.com> Cc: Baoquan He <baoquan.he@linux.dev> Cc: Barry Song <baohua@kernel.org> Cc: Boris Brezillon <boris.brezillon@collabora.com> Cc: Byungchul Park <byungchul@sk.com> Cc: Chengming Zhou <chengming.zhou@linux.dev> Cc: Chris Li <chrisl@kernel.org> Cc: Christan König <christian.koenig@amd.com> Cc: Christian Borntraeger <borntraeger@linux.ibm.com> Cc: Claudio Imbrenda <imbrenda@linux.ibm.com> Cc: Dave Airlie <airlied@gmail.com> Cc: Dev Jain <dev.jain@arm.com> Cc: Gerald Schaefer <gerald.schaefer@linux.ibm.com> Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org> Cc: Gregory Price (Meta) <gourry@gourry.net> Cc: Harry Yoo <harry@kernel.org> Cc: Heiko Carstens <hca@linux.ibm.com> Cc: Huang Ray <Ray.Huang@amd.com> Cc: "Huang, Ying" <ying.huang@linux.alibaba.com> Cc: Ian Rogers <irogers@google.com> Cc: Ingo Molnar <mingo@redhat.com> Cc: James Clark <james.clark@linaro.org> Cc: Jan Kara <jack@suse.cz> Cc: Jann Horn <jannh@google.com> Cc: Janosch Frank <frankja@linux.ibm.com> Cc: Jason Gunthorpe <jgg@ziepe.ca> Cc: Jiri Olsa <jolsa@kernel.org> Cc: John Hubbard <jhubbard@nvidia.com> Cc: Joshua Hahn <joshua.hahnjy@gmail.com> Cc: Kairui Song <kasong@tencent.com> Cc: Kees Cook <kees@kernel.org> Cc: Kemeng Shi <shikemeng@huaweicloud.com> Cc: Lance Yang <lance.yang@linux.dev> Cc: Liam R. Howlett <liam@infradead.org> Cc: Liviu Dudau <liviu.dudau@arm.com> Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com> Cc: Marc Rutland <mark.rutland@arm.com> Cc: "Masami Hiramatsu (Google)" <mhiramat@kernel.org> Cc: Matthew Auld <matthew.auld@intel.com> Cc: Matthew Brost <matthew.brost@intel.com> Cc: Matthew Wilcox (Oracle) <willy@infradead.org> Cc: Maxime Ripard <mripard@kernel.org> Cc: Miaohe Lin <linmiaohe@huawei.com> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Muchun Song <muchun.song@linux.dev> Cc: Namhyung kim <namhyung@kernel.org> Cc: Naoya Horiguchi <nao.horiguchi@gmail.com> Cc: Nhat Pham <nphamcs@gmail.com> Cc: Nico Pache <npache@redhat.com> Cc: Oleg Nesterov <oleg@redhat.com> Cc: Oscar Salvador <osalvador@suse.de> Cc: Pedro Falcato <pfalcato@suse.de> Cc: Peter Xu <peterx@redhat.com> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Rakie Kim <rakie.kim@sk.com> Cc: Rik van Riel <riel@surriel.com> Cc: Rodrigo Vivi <rodrigo.vivi@intel.com> Cc: Ryan Roberts <ryan.roberts@arm.com> Cc: Steven Price <steven.price@arm.com> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Sven Schnelle <svens@linux.ibm.com> Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com> Cc: Thomas Zimemrmann <tzimmermann@suse.de> Cc: Vasily Gorbik <gor@linux.ibm.com> Cc: Vlastimil Babka <vbabka@kernel.org> Cc: xu xin <xu.xin16@zte.com.cn> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Lorenzo Stoakes (ARM)
|
9998bc06d7 |
mm/vma: fix self-merge check in copy_vma()
The existing logic is very confusing so improve things. Firstly rename the confusing faulted_in_anon_vma variable to can_self_merge and update this when the page offset is updated. What is being checked for is a 'self-merge' - that is between the VMA being remapped and its prior VMA (remember that this is copy_vma() - if a non-MREMAP_DONTUNMAP remap the original VMA is only removed afterwards). This can happen if the VMA is moved immediately adjacent to itself, either before or after it: |----------------|----------------| | | | v | v |...............||---------------||...............| | new || old || new | |...............||---------------||---------------| In these cases the old VMA is simply expanded to cover the new range. It is also possible for the move to both self-merge and merge with a prior VMA if it is placed between a preceding VMA and its old self: |---------------| | | v | |---------------||...............||---------------| | prev || new || old | |---------------||...............||---------------| In this case, the old VMA is removed and 'prev' is expanded and replaces it. Since copy_vma_and_data() which calls copy_vma() intends to reference the old VMA after the merge, it must have this pointer updated. This kind of self-merge is not possible with a succeeding merge, as the merge always prefers to expand the preceding VMA if possible. copy_vma() accounts for this by explicitly checking to see if a self-merge occurred and updating the vmap pointer if so. However it incorrect did so even for a subsequent merge (this is simply a noop so it had no impact). So change this to only check for the case which matters - a backwards merge - and rearrange the parameters to make it clearer we're doing that - i.e. check new_vma->vm_start < old_vma_start (having already renamed vma_start to old_vma_start to make it clear this is the previous VMA). Also update the existing wall-of-text comment to be a lot clearer. While we're here, replace the VM_BUG_ON_VMA() with a VM_WARN_ON_ONCE_VMA() and update the VMA userland tests accordingly. No functional change intended. Link: https://lore.kernel.org/20260813-b4-scalable-cow-virt-pgoff-v5-7-c21581c0c3c8@kernel.org Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org> Acked-by: David Hildenbrand (Arm) <david@kernel.org> Cc: Adrian Hunter <adrian.hunter@intel.com> Cc: Alexander Deucher <alexander.deucher@amd.com> Cc: Alexander Gordeev <agordeev@linux.ibm.com> Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com> Cc: Alistair Popple <apopple@nvidia.com> Cc: Arnaldo Carvalho de Melo <acme@kernel.org> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Baolin Wang <baolin.wang@linux.alibaba.com> Cc: Baoquan He <baoquan.he@linux.dev> Cc: Barry Song <baohua@kernel.org> Cc: Boris Brezillon <boris.brezillon@collabora.com> Cc: Byungchul Park <byungchul@sk.com> Cc: Chengming Zhou <chengming.zhou@linux.dev> Cc: Chris Li <chrisl@kernel.org> Cc: Christan König <christian.koenig@amd.com> Cc: Christian Borntraeger <borntraeger@linux.ibm.com> Cc: Claudio Imbrenda <imbrenda@linux.ibm.com> Cc: Dave Airlie <airlied@gmail.com> Cc: Dev Jain <dev.jain@arm.com> Cc: Gerald Schaefer <gerald.schaefer@linux.ibm.com> Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org> Cc: Gregory Price (Meta) <gourry@gourry.net> Cc: Harry Yoo <harry@kernel.org> Cc: Heiko Carstens <hca@linux.ibm.com> Cc: Huang Ray <Ray.Huang@amd.com> Cc: "Huang, Ying" <ying.huang@linux.alibaba.com> Cc: Ian Rogers <irogers@google.com> Cc: Ingo Molnar <mingo@redhat.com> Cc: James Clark <james.clark@linaro.org> Cc: Jan Kara <jack@suse.cz> Cc: Jann Horn <jannh@google.com> Cc: Janosch Frank <frankja@linux.ibm.com> Cc: Jason Gunthorpe <jgg@ziepe.ca> Cc: Jiri Olsa <jolsa@kernel.org> Cc: John Hubbard <jhubbard@nvidia.com> Cc: Joshua Hahn <joshua.hahnjy@gmail.com> Cc: Kairui Song <kasong@tencent.com> Cc: Kees Cook <kees@kernel.org> Cc: Kemeng Shi <shikemeng@huaweicloud.com> Cc: Lance Yang <lance.yang@linux.dev> Cc: Liam R. Howlett <liam@infradead.org> Cc: Liviu Dudau <liviu.dudau@arm.com> Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com> Cc: Marc Rutland <mark.rutland@arm.com> Cc: "Masami Hiramatsu (Google)" <mhiramat@kernel.org> Cc: Matthew Auld <matthew.auld@intel.com> Cc: Matthew Brost <matthew.brost@intel.com> Cc: Matthew Wilcox (Oracle) <willy@infradead.org> Cc: Maxime Ripard <mripard@kernel.org> Cc: Miaohe Lin <linmiaohe@huawei.com> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Muchun Song <muchun.song@linux.dev> Cc: Namhyung kim <namhyung@kernel.org> Cc: Naoya Horiguchi <nao.horiguchi@gmail.com> Cc: Nhat Pham <nphamcs@gmail.com> Cc: Nico Pache <npache@redhat.com> Cc: Oleg Nesterov <oleg@redhat.com> Cc: Oscar Salvador <osalvador@suse.de> Cc: Pedro Falcato <pfalcato@suse.de> Cc: Peter Xu <peterx@redhat.com> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Rakie Kim <rakie.kim@sk.com> Cc: Rik van Riel <riel@surriel.com> Cc: Rodrigo Vivi <rodrigo.vivi@intel.com> Cc: Ryan Roberts <ryan.roberts@arm.com> Cc: Steven Price <steven.price@arm.com> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Sven Schnelle <svens@linux.ibm.com> Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com> Cc: Thomas Zimemrmann <tzimmermann@suse.de> Cc: Vasily Gorbik <gor@linux.ibm.com> Cc: Vlastimil Babka <vbabka@kernel.org> Cc: xu xin <xu.xin16@zte.com.cn> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Lorenzo Stoakes (ARM)
|
dba10745e9 |
mm: introduce and use vma_filebacked_address()
In cases where we know that the VMA is file-backed, use vma_filebacked_address() rather than vma_address(). This lays the foundation for using the anonymous page offset via vma_anon_address(). Also add an assert to ensure that the VMA whose address is required is not anonymous. No functional change intended. Link: https://lore.kernel.org/20260813-b4-scalable-cow-virt-pgoff-v5-6-c21581c0c3c8@kernel.org Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org> Tested-by: syzbot@syzkaller.appspotmail.com Acked-by: David Hildenbrand (Arm) <david@kernel.org> Cc: Adrian Hunter <adrian.hunter@intel.com> Cc: Alexander Deucher <alexander.deucher@amd.com> Cc: Alexander Gordeev <agordeev@linux.ibm.com> Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com> Cc: Alistair Popple <apopple@nvidia.com> Cc: Arnaldo Carvalho de Melo <acme@kernel.org> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Baolin Wang <baolin.wang@linux.alibaba.com> Cc: Baoquan He <baoquan.he@linux.dev> Cc: Barry Song <baohua@kernel.org> Cc: Boris Brezillon <boris.brezillon@collabora.com> Cc: Byungchul Park <byungchul@sk.com> Cc: Chengming Zhou <chengming.zhou@linux.dev> Cc: Chris Li <chrisl@kernel.org> Cc: Christan König <christian.koenig@amd.com> Cc: Christian Borntraeger <borntraeger@linux.ibm.com> Cc: Claudio Imbrenda <imbrenda@linux.ibm.com> Cc: Dave Airlie <airlied@gmail.com> Cc: Dev Jain <dev.jain@arm.com> Cc: Gerald Schaefer <gerald.schaefer@linux.ibm.com> Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org> Cc: Gregory Price (Meta) <gourry@gourry.net> Cc: Harry Yoo <harry@kernel.org> Cc: Heiko Carstens <hca@linux.ibm.com> Cc: Huang Ray <Ray.Huang@amd.com> Cc: "Huang, Ying" <ying.huang@linux.alibaba.com> Cc: Ian Rogers <irogers@google.com> Cc: Ingo Molnar <mingo@redhat.com> Cc: James Clark <james.clark@linaro.org> Cc: Jan Kara <jack@suse.cz> Cc: Jann Horn <jannh@google.com> Cc: Janosch Frank <frankja@linux.ibm.com> Cc: Jason Gunthorpe <jgg@ziepe.ca> Cc: Jiri Olsa <jolsa@kernel.org> Cc: John Hubbard <jhubbard@nvidia.com> Cc: Joshua Hahn <joshua.hahnjy@gmail.com> Cc: Kairui Song <kasong@tencent.com> Cc: Kees Cook <kees@kernel.org> Cc: Kemeng Shi <shikemeng@huaweicloud.com> Cc: Lance Yang <lance.yang@linux.dev> Cc: Liam R. Howlett <liam@infradead.org> Cc: Liviu Dudau <liviu.dudau@arm.com> Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com> Cc: Marc Rutland <mark.rutland@arm.com> Cc: "Masami Hiramatsu (Google)" <mhiramat@kernel.org> Cc: Matthew Auld <matthew.auld@intel.com> Cc: Matthew Brost <matthew.brost@intel.com> Cc: Matthew Wilcox (Oracle) <willy@infradead.org> Cc: Maxime Ripard <mripard@kernel.org> Cc: Miaohe Lin <linmiaohe@huawei.com> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Muchun Song <muchun.song@linux.dev> Cc: Namhyung kim <namhyung@kernel.org> Cc: Naoya Horiguchi <nao.horiguchi@gmail.com> Cc: Nhat Pham <nphamcs@gmail.com> Cc: Nico Pache <npache@redhat.com> Cc: Oleg Nesterov <oleg@redhat.com> Cc: Oscar Salvador <osalvador@suse.de> Cc: Pedro Falcato <pfalcato@suse.de> Cc: Peter Xu <peterx@redhat.com> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Rakie Kim <rakie.kim@sk.com> Cc: Rik van Riel <riel@surriel.com> Cc: Rodrigo Vivi <rodrigo.vivi@intel.com> Cc: Ryan Roberts <ryan.roberts@arm.com> Cc: Steven Price <steven.price@arm.com> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Sven Schnelle <svens@linux.ibm.com> Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com> Cc: Thomas Zimemrmann <tzimmermann@suse.de> Cc: Vasily Gorbik <gor@linux.ibm.com> Cc: Vlastimil Babka <vbabka@kernel.org> Cc: xu xin <xu.xin16@zte.com.cn> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Lorenzo Stoakes (ARM)
|
2a8de2d1d6 |
mm: update print_bad_page_map() to show anon index if appropriate
If the VMA is a CoW mapping page offset may differ from anon page offset, indicating different positions in the relevant rmap trees. Update print_bad_page_map() to reflect that - if the mapping is non-CoW or the indexes match, then output only one index as before, otherwise output both with (file) or (anon) suffixes to reflect which is which. It's not possible to give only one index as there is no folio available to perform folio_test_anon() upon (the page table entry is bad so this is unavailable). This is potentially useful debugging information and matches the existing page offset provided. Use the raw __linear_anon_page_index() function so as to always output this value regardless of whether the mapping is file-backed or not and to avoid asserts that shouldn't apply here. Link: https://lore.kernel.org/20260813-b4-scalable-cow-virt-pgoff-v5-5-c21581c0c3c8@kernel.org Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org> Acked-by: David Hildenbrand (Arm) <david@kernel.org> Reviewed-by: Gregory Price (Meta) <gourry@gourry.net> Cc: Adrian Hunter <adrian.hunter@intel.com> Cc: Alexander Deucher <alexander.deucher@amd.com> Cc: Alexander Gordeev <agordeev@linux.ibm.com> Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com> Cc: Alistair Popple <apopple@nvidia.com> Cc: Arnaldo Carvalho de Melo <acme@kernel.org> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Baolin Wang <baolin.wang@linux.alibaba.com> Cc: Baoquan He <baoquan.he@linux.dev> Cc: Barry Song <baohua@kernel.org> Cc: Boris Brezillon <boris.brezillon@collabora.com> Cc: Byungchul Park <byungchul@sk.com> Cc: Chengming Zhou <chengming.zhou@linux.dev> Cc: Chris Li <chrisl@kernel.org> Cc: Christan König <christian.koenig@amd.com> Cc: Christian Borntraeger <borntraeger@linux.ibm.com> Cc: Claudio Imbrenda <imbrenda@linux.ibm.com> Cc: Dave Airlie <airlied@gmail.com> Cc: Dev Jain <dev.jain@arm.com> Cc: Gerald Schaefer <gerald.schaefer@linux.ibm.com> Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org> Cc: Harry Yoo <harry@kernel.org> Cc: Heiko Carstens <hca@linux.ibm.com> Cc: Huang Ray <Ray.Huang@amd.com> Cc: "Huang, Ying" <ying.huang@linux.alibaba.com> Cc: Ian Rogers <irogers@google.com> Cc: Ingo Molnar <mingo@redhat.com> Cc: James Clark <james.clark@linaro.org> Cc: Jan Kara <jack@suse.cz> Cc: Jann Horn <jannh@google.com> Cc: Janosch Frank <frankja@linux.ibm.com> Cc: Jason Gunthorpe <jgg@ziepe.ca> Cc: Jiri Olsa <jolsa@kernel.org> Cc: John Hubbard <jhubbard@nvidia.com> Cc: Joshua Hahn <joshua.hahnjy@gmail.com> Cc: Kairui Song <kasong@tencent.com> Cc: Kees Cook <kees@kernel.org> Cc: Kemeng Shi <shikemeng@huaweicloud.com> Cc: Lance Yang <lance.yang@linux.dev> Cc: Liam R. Howlett <liam@infradead.org> Cc: Liviu Dudau <liviu.dudau@arm.com> Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com> Cc: Marc Rutland <mark.rutland@arm.com> Cc: "Masami Hiramatsu (Google)" <mhiramat@kernel.org> Cc: Matthew Auld <matthew.auld@intel.com> Cc: Matthew Brost <matthew.brost@intel.com> Cc: Matthew Wilcox (Oracle) <willy@infradead.org> Cc: Maxime Ripard <mripard@kernel.org> Cc: Miaohe Lin <linmiaohe@huawei.com> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Muchun Song <muchun.song@linux.dev> Cc: Namhyung kim <namhyung@kernel.org> Cc: Naoya Horiguchi <nao.horiguchi@gmail.com> Cc: Nhat Pham <nphamcs@gmail.com> Cc: Nico Pache <npache@redhat.com> Cc: Oleg Nesterov <oleg@redhat.com> Cc: Oscar Salvador <osalvador@suse.de> Cc: Pedro Falcato <pfalcato@suse.de> Cc: Peter Xu <peterx@redhat.com> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Rakie Kim <rakie.kim@sk.com> Cc: Rik van Riel <riel@surriel.com> Cc: Rodrigo Vivi <rodrigo.vivi@intel.com> Cc: Ryan Roberts <ryan.roberts@arm.com> Cc: Steven Price <steven.price@arm.com> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Sven Schnelle <svens@linux.ibm.com> Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com> Cc: Thomas Zimemrmann <tzimmermann@suse.de> Cc: Vasily Gorbik <gor@linux.ibm.com> Cc: Vlastimil Babka <vbabka@kernel.org> Cc: xu xin <xu.xin16@zte.com.cn> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Lorenzo Stoakes (ARM)
|
c02fe674fe |
mm: abstract vma_address() and introduce vma_anon_address()
Introduce __vma_address() which abstracts the VMA start page offset field as pgoff_start, then update vma_address() to use it. Then introduce vma_anon_address() which does the equivalent of vma_address(), only using the anonymous page offset of the VMA rather than the file-backed one. Also add an assert to ensure that the function is not called for mappings which are file-backed but not MAP_PRIVATE to ensure it is only used in the correct places. This will be necessary for determining the address of a folio's index within a VMA when the folio belongs to a MAP_PRIVATE file-backed VMA but has been CoW'd, and thus is anonymous, once the anonymous VMA page offset field is used for the reverse mapping. No callers are updated, so no functional change intended. Link: https://lore.kernel.org/20260813-b4-scalable-cow-virt-pgoff-v5-4-c21581c0c3c8@kernel.org Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org> Acked-by: David Hildenbrand (Arm) <david@kernel.org> Cc: Adrian Hunter <adrian.hunter@intel.com> Cc: Alexander Deucher <alexander.deucher@amd.com> Cc: Alexander Gordeev <agordeev@linux.ibm.com> Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com> Cc: Alistair Popple <apopple@nvidia.com> Cc: Arnaldo Carvalho de Melo <acme@kernel.org> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Baolin Wang <baolin.wang@linux.alibaba.com> Cc: Baoquan He <baoquan.he@linux.dev> Cc: Barry Song <baohua@kernel.org> Cc: Boris Brezillon <boris.brezillon@collabora.com> Cc: Byungchul Park <byungchul@sk.com> Cc: Chengming Zhou <chengming.zhou@linux.dev> Cc: Chris Li <chrisl@kernel.org> Cc: Christan König <christian.koenig@amd.com> Cc: Christian Borntraeger <borntraeger@linux.ibm.com> Cc: Claudio Imbrenda <imbrenda@linux.ibm.com> Cc: Dave Airlie <airlied@gmail.com> Cc: Dev Jain <dev.jain@arm.com> Cc: Gerald Schaefer <gerald.schaefer@linux.ibm.com> Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org> Cc: Gregory Price (Meta) <gourry@gourry.net> Cc: Harry Yoo <harry@kernel.org> Cc: Heiko Carstens <hca@linux.ibm.com> Cc: Huang Ray <Ray.Huang@amd.com> Cc: "Huang, Ying" <ying.huang@linux.alibaba.com> Cc: Ian Rogers <irogers@google.com> Cc: Ingo Molnar <mingo@redhat.com> Cc: James Clark <james.clark@linaro.org> Cc: Jan Kara <jack@suse.cz> Cc: Jann Horn <jannh@google.com> Cc: Janosch Frank <frankja@linux.ibm.com> Cc: Jason Gunthorpe <jgg@ziepe.ca> Cc: Jiri Olsa <jolsa@kernel.org> Cc: John Hubbard <jhubbard@nvidia.com> Cc: Joshua Hahn <joshua.hahnjy@gmail.com> Cc: Kairui Song <kasong@tencent.com> Cc: Kees Cook <kees@kernel.org> Cc: Kemeng Shi <shikemeng@huaweicloud.com> Cc: Lance Yang <lance.yang@linux.dev> Cc: Liam R. Howlett <liam@infradead.org> Cc: Liviu Dudau <liviu.dudau@arm.com> Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com> Cc: Marc Rutland <mark.rutland@arm.com> Cc: "Masami Hiramatsu (Google)" <mhiramat@kernel.org> Cc: Matthew Auld <matthew.auld@intel.com> Cc: Matthew Brost <matthew.brost@intel.com> Cc: Matthew Wilcox (Oracle) <willy@infradead.org> Cc: Maxime Ripard <mripard@kernel.org> Cc: Miaohe Lin <linmiaohe@huawei.com> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Muchun Song <muchun.song@linux.dev> Cc: Namhyung kim <namhyung@kernel.org> Cc: Naoya Horiguchi <nao.horiguchi@gmail.com> Cc: Nhat Pham <nphamcs@gmail.com> Cc: Nico Pache <npache@redhat.com> Cc: Oleg Nesterov <oleg@redhat.com> Cc: Oscar Salvador <osalvador@suse.de> Cc: Pedro Falcato <pfalcato@suse.de> Cc: Peter Xu <peterx@redhat.com> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Rakie Kim <rakie.kim@sk.com> Cc: Rik van Riel <riel@surriel.com> Cc: Rodrigo Vivi <rodrigo.vivi@intel.com> Cc: Ryan Roberts <ryan.roberts@arm.com> Cc: Steven Price <steven.price@arm.com> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Sven Schnelle <svens@linux.ibm.com> Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com> Cc: Thomas Zimemrmann <tzimmermann@suse.de> Cc: Vasily Gorbik <gor@linux.ibm.com> Cc: Vlastimil Babka <vbabka@kernel.org> Cc: xu xin <xu.xin16@zte.com.cn> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Lorenzo Stoakes (ARM)
|
51943a18ad |
mm: provide vma_[flags_]is_cow_mapping() and remove is_cow_mapping()
All remaining callers of is_cow_mapping() are invoking it in the form of is_cow_mapping(vma->vm_flags) or an indirected version of this. Therefore, provide a helper - vma_is_cow_mapping() to directly test the VMA. Additionally provide a new helper vma_flags_is_cow_mapping() which performs the check using the new vma_flags_t type, and share this logic between vma_is_cow_mapping() and vma_desc_is_cow_mapping(). With these changes, no callers of is_cow_mapping() remain, so remove it. Also update the userland VMA tests to reflect the change. No functional change intended. [akpm@linux-foundation.org: fix kerneldoc comment typo, per Lorenzo] Link: https://lore.kernel.org/aob1goSSPH6sTN9y@gremlin Link: https://lore.kernel.org/20260813-b4-scalable-cow-virt-pgoff-v5-2-c21581c0c3c8@kernel.org Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org> Acked-by: David Hildenbrand (Arm) <david@kernel.org> Cc: Adrian Hunter <adrian.hunter@intel.com> Cc: Alexander Deucher <alexander.deucher@amd.com> Cc: Alexander Gordeev <agordeev@linux.ibm.com> Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com> Cc: Alistair Popple <apopple@nvidia.com> Cc: Arnaldo Carvalho de Melo <acme@kernel.org> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Baolin Wang <baolin.wang@linux.alibaba.com> Cc: Baoquan He <baoquan.he@linux.dev> Cc: Barry Song <baohua@kernel.org> Cc: Boris Brezillon <boris.brezillon@collabora.com> Cc: Byungchul Park <byungchul@sk.com> Cc: Chengming Zhou <chengming.zhou@linux.dev> Cc: Chris Li <chrisl@kernel.org> Cc: Christan König <christian.koenig@amd.com> Cc: Christian Borntraeger <borntraeger@linux.ibm.com> Cc: Claudio Imbrenda <imbrenda@linux.ibm.com> Cc: Dave Airlie <airlied@gmail.com> Cc: Dev Jain <dev.jain@arm.com> Cc: Gerald Schaefer <gerald.schaefer@linux.ibm.com> Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org> Cc: Gregory Price (Meta) <gourry@gourry.net> Cc: Harry Yoo <harry@kernel.org> Cc: Heiko Carstens <hca@linux.ibm.com> Cc: Huang Ray <Ray.Huang@amd.com> Cc: "Huang, Ying" <ying.huang@linux.alibaba.com> Cc: Ian Rogers <irogers@google.com> Cc: Ingo Molnar <mingo@redhat.com> Cc: James Clark <james.clark@linaro.org> Cc: Jan Kara <jack@suse.cz> Cc: Jann Horn <jannh@google.com> Cc: Janosch Frank <frankja@linux.ibm.com> Cc: Jason Gunthorpe <jgg@ziepe.ca> Cc: Jiri Olsa <jolsa@kernel.org> Cc: John Hubbard <jhubbard@nvidia.com> Cc: Joshua Hahn <joshua.hahnjy@gmail.com> Cc: Kairui Song <kasong@tencent.com> Cc: Kees Cook <kees@kernel.org> Cc: Kemeng Shi <shikemeng@huaweicloud.com> Cc: Lance Yang <lance.yang@linux.dev> Cc: Liam R. Howlett <liam@infradead.org> Cc: Liviu Dudau <liviu.dudau@arm.com> Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com> Cc: Marc Rutland <mark.rutland@arm.com> Cc: "Masami Hiramatsu (Google)" <mhiramat@kernel.org> Cc: Matthew Auld <matthew.auld@intel.com> Cc: Matthew Brost <matthew.brost@intel.com> Cc: Matthew Wilcox (Oracle) <willy@infradead.org> Cc: Maxime Ripard <mripard@kernel.org> Cc: Miaohe Lin <linmiaohe@huawei.com> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Muchun Song <muchun.song@linux.dev> Cc: Namhyung kim <namhyung@kernel.org> Cc: Naoya Horiguchi <nao.horiguchi@gmail.com> Cc: Nhat Pham <nphamcs@gmail.com> Cc: Nico Pache <npache@redhat.com> Cc: Oleg Nesterov <oleg@redhat.com> Cc: Oscar Salvador <osalvador@suse.de> Cc: Pedro Falcato <pfalcato@suse.de> Cc: Peter Xu <peterx@redhat.com> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Rakie Kim <rakie.kim@sk.com> Cc: Rik van Riel <riel@surriel.com> Cc: Rodrigo Vivi <rodrigo.vivi@intel.com> Cc: Ryan Roberts <ryan.roberts@arm.com> Cc: Steven Price <steven.price@arm.com> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Sven Schnelle <svens@linux.ibm.com> Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com> Cc: Thomas Zimemrmann <tzimmermann@suse.de> Cc: Vasily Gorbik <gor@linux.ibm.com> Cc: Vlastimil Babka <vbabka@kernel.org> Cc: xu xin <xu.xin16@zte.com.cn> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Lorenzo Stoakes (ARM)
|
0bd14001eb |
mm/vma: introduce VMA anon page offset field and add helpers
Patch series "mm/rmap: index MAP_PRIVATE file-backed folios by anonymous pgoff", v5. In memory management we've managed to manufacture a great deal of confusion around the concept of anonymous memory. We have: 1. 'Pure anon' memory - anonymous VMAs whose folios are anonymous and swap-backed (thus for reclaim purposes, treated as anonymous). These are simple enough. 2. shmem - file-backed VMAs, file-backed folios (from rmap perspective) so present in the page cache and mapped by an address_space object, but whose folios are also swap-backed (thus treated as anonymous for reclaim purposes). 3. MAP_PRIVATE-mapped /dev/zero - a strange beast whose VMAs have vma->vm_file set, but which clears vma->vm_ops to satisfy vma_is_anonymous(), resulting in VMAs that were mmap()'d referencing a file, but are in every other sense anonymous, including the folios. 4. Other MAP_PRIVATE-file backed mappings - These possess file-backed VMAs and have file-backed folios until CoW'd, at which point those CoW'd folios are anonymous. This series fixes issue 3. In order for us to traverse VMAs using the reverse mapping, we require two fields - folio->mapping and folio->index. The first tells the rmap code where to look for VMAs, and the second tells it at which offset the folio starts within the referenced object. For anonymous folios, folio->mapping points at an anon_vma object. For file-backed folios, it points at an address_space. And: * For file-backed folios folio->index is simply the page offset of the start of the folio within the file. * For anonymous folios belonging to pure anon mappings, folio->index is equal to the anonymous page offset of the folio. * For anonymous folios belonging to file-backed mappings (i.e. CoW'd folios of a MAP_PRIVATE file-backed mapping), folio->index is equal to the file page offset. This series establishes a new anonymous page offset property of VMAs to allow us to map anonymous folios at their anonymous page offset, consistent with pure anon. The purpose of doing so is to lay the foundations for the scalable CoW work. This is necessary because scalable CoW looks in the maple tree for the VMA located at folio->index << PAGE_SHIFT, before falling back to looking up tracked remaps if necessary. The MAP_PRIVATE file-backed case means that folio indices will very often conflict with one another and this remap tracking becomes substantially more contended, and of course the fast path can never be used. This also makes it possible, in future, to unshare anonymously mapped folios with deep fork hierarchies on remap, eliminating the need for remap tracking in the vast majority of cases. Similar to page offset of pure anonymous VMAs, we update the anonymous page offset of unfaulted file-backed VMAs on remap, but do not once CoW'd (i.e. vma->anon_vma is non-NULL). Overall, there is little impact on mergeability, which remains exactly the same for pure anonymous and shared file-backed mappings, with the only impact being on MAP_PRIVATE-mapped file-backed mappings, which must now match on anonymous page offset as well as file page offset to be merged. To fail to merge like this would require CoW'ing the mapping, then finding another VMA with identical file and compatible page offset to remap next to. This is therefore very much an edge case that should have very little impact (and which scalable CoW may very well address in any case). This patch (of 16): Establish fields in vm_area_struct to store the anonymous page offset of VMAs. Initially, the anonymous page offset of a VMA is vma->vm_start >> PAGE_SHIFT. When a VMA is remapped to new_address its anonymous page offset is either updated to new_address >> PAGE_SHIFT if unfaulted or, if faulted, remains equal to the anonymous page offset it had when first faulted. Currently, anonymous folios belonging to CoW'd MAP_PRIVATE-mapped file-backed VMAs are tracked by their file offsets. By adding anonymous offset as a property of VMAs, we can now track them by their anonymous page offset instead. By tracking this, we provide the means by which to eliminate this inconsistency, and more importantly lay the foundations for future work for the scalable CoW anonymous rmap rework. This patch simply adds the fields and some simple helpers. Subsequent patches will update mm code to make use of these fields correctly. The fields chosen are packed in the VMA such that, for 64-bit kernel builds, no additional space is taken up. The first field is present on cacheline 0 containing key VMA fields, and the second on cacheline 3, which contains file-backed reverse mapping fields. Given the relative time spent accessing reverse mapping fields as well as updating them, there shouldn't be any performance impact here from false sharing. Update the VMA userland tests to account for this change. No callsites are updated yet, so no functional change intended. Link: https://lore.kernel.org/20260813-b4-scalable-cow-virt-pgoff-v5-0-c21581c0c3c8@kernel.org Link: https://lore.kernel.org/20260813-b4-scalable-cow-virt-pgoff-v5-1-c21581c0c3c8@kernel.org Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org> Acked-by: David Hildenbrand (Arm) <david@kernel.org> Reviewed-by: Gregory Price (Meta) <gourry@gourry.net> Reviewed-by: Xu Xin <xu.xin16@zte.com.cn> Cc: Adrian Hunter <adrian.hunter@intel.com> Cc: Alexander Deucher <alexander.deucher@amd.com> Cc: Alexander Gordeev <agordeev@linux.ibm.com> Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com> Cc: Alistair Popple <apopple@nvidia.com> Cc: Arnaldo Carvalho de Melo <acme@kernel.org> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Baolin Wang <baolin.wang@linux.alibaba.com> Cc: Baoquan He <baoquan.he@linux.dev> Cc: Barry Song <baohua@kernel.org> Cc: Boris Brezillon <boris.brezillon@collabora.com> Cc: Byungchul Park <byungchul@sk.com> Cc: Chengming Zhou <chengming.zhou@linux.dev> Cc: Chris Li <chrisl@kernel.org> Cc: Christan König <christian.koenig@amd.com> Cc: Christian Borntraeger <borntraeger@linux.ibm.com> Cc: Claudio Imbrenda <imbrenda@linux.ibm.com> Cc: Dave Airlie <airlied@gmail.com> Cc: Dev Jain <dev.jain@arm.com> Cc: Gerald Schaefer <gerald.schaefer@linux.ibm.com> Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org> Cc: Harry Yoo <harry@kernel.org> Cc: Heiko Carstens <hca@linux.ibm.com> Cc: Huang Ray <Ray.Huang@amd.com> Cc: "Huang, Ying" <ying.huang@linux.alibaba.com> Cc: Ian Rogers <irogers@google.com> Cc: Ingo Molnar <mingo@redhat.com> Cc: James Clark <james.clark@linaro.org> Cc: Jan Kara <jack@suse.cz> Cc: Jann Horn <jannh@google.com> Cc: Janosch Frank <frankja@linux.ibm.com> Cc: Jason Gunthorpe <jgg@ziepe.ca> Cc: Jiri Olsa <jolsa@kernel.org> Cc: John Hubbard <jhubbard@nvidia.com> Cc: Joshua Hahn <joshua.hahnjy@gmail.com> Cc: Kairui Song <kasong@tencent.com> Cc: Kees Cook <kees@kernel.org> Cc: Kemeng Shi <shikemeng@huaweicloud.com> Cc: Lance Yang <lance.yang@linux.dev> Cc: Liam R. Howlett <liam@infradead.org> Cc: Liviu Dudau <liviu.dudau@arm.com> Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com> Cc: Marc Rutland <mark.rutland@arm.com> Cc: "Masami Hiramatsu (Google)" <mhiramat@kernel.org> Cc: Matthew Auld <matthew.auld@intel.com> Cc: Matthew Brost <matthew.brost@intel.com> Cc: Matthew Wilcox (Oracle) <willy@infradead.org> Cc: Maxime Ripard <mripard@kernel.org> Cc: Miaohe Lin <linmiaohe@huawei.com> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Muchun Song <muchun.song@linux.dev> Cc: Namhyung kim <namhyung@kernel.org> Cc: Naoya Horiguchi <nao.horiguchi@gmail.com> Cc: Nhat Pham <nphamcs@gmail.com> Cc: Nico Pache <npache@redhat.com> Cc: Oleg Nesterov <oleg@redhat.com> Cc: Oscar Salvador <osalvador@suse.de> Cc: Pedro Falcato <pfalcato@suse.de> Cc: Peter Xu <peterx@redhat.com> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Rakie Kim <rakie.kim@sk.com> Cc: Rik van Riel <riel@surriel.com> Cc: Rodrigo Vivi <rodrigo.vivi@intel.com> Cc: Ryan Roberts <ryan.roberts@arm.com> Cc: Steven Price <steven.price@arm.com> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Sven Schnelle <svens@linux.ibm.com> Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com> Cc: Thomas Zimemrmann <tzimmermann@suse.de> Cc: Vasily Gorbik <gor@linux.ibm.com> Cc: Vlastimil Babka <vbabka@kernel.org> Cc: xu xin <xu.xin16@zte.com.cn> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Kaitao Cheng
|
0b4268ac77 |
mm/kconfig: drop redundant dependency wrappers
Some mm Kconfig entries repeat dependencies that are already expressed by their surrounding blocks or menus. The zsmalloc allocator options menu already depends on ZSMALLOC, so the outer if ZSMALLOC block does not add any extra constraint. MEMORY_HOTREMOVE and MHP_MEMMAP_ON_MEMORY are both inside the if MEMORY_HOTPLUG block, so their local depends on MEMORY_HOTPLUG entries are redundant. PTE_MARKER_UFFD_WP is the only entry under if USERFAULTFD. Move the USERFAULTFD dependency into the symbol itself and combine it with the architecture support dependency. This keeps the same visibility and defaults while avoiding duplicate dependency expressions. Link: https://lore.kernel.org/20260712093326.8313-1-kaitao.cheng@linux.dev Signed-off-by: Kaitao Cheng <chengkaitao@kylinos.cn> Suggested-by: Julian Braha <julianbraha@gmail.com> Reviewed-by: Lorenzo Stoakes (ARM) <ljs@kernel.org> Acked-by: David Hildenbrand (Arm) <david@kernel.org> Cc: Liam R. Howlett <liam@infradead.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Muchun Song <muchun.song@linux.dev> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
David Hildenbrand (Arm)
|
092836fedd |
mm: standardize printing for pgtable entries
Bad page map reporting currently stores page table entry values in an unsigned long long and prints them with fixed 64-bit-oriented format strings. This is inconsistent across call sites and does not work well for architectures where page table entry values are not naturally represented as 64-bit values, such as 32-bit or 128-bit entries. Introduce a common helper to convert raw page table entry values into a fixed-width hexadecimal string based on the actual entry size. Use it for bad page map reporting and for dumping the page table walk in __print_bad_page_map_pgtable(). Pass page table entry values to the reporting path as raw bytes together with their size, instead of forcing them through an unsigned long long. It keeps the printed output consistent and avoids truncation or misleading formatting for non-64-bit page table entries. Link: https://lore.kernel.org/20260709044334.1741263-1-anshuman.khandual@arm.com Signed-off-by: David Hildenbrand (Arm) <david@kernel.org> Co-developed-by: Anshuman Khandual <anshuman.khandual@arm.com> Signed-off-by: Anshuman Khandual <anshuman.khandual@arm.com> Cc: Andriy Shevchenko <andriy.shevchenko@linux.intel.com> Cc: David Hildenbrand <david@kernel.org> Cc: Hugh Dickins <hughd@google.com> Cc: Matthew Wilcox (Oracle) <willy@infradead.org> Cc: Ryan Roberts <ryan.roberts@arm.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Kaitao Cheng
|
1eba458a54 |
mm/kconfig: drop redundant memory hotplug dependencies
MHP_MEMMAP_ON_MEMORY is defined inside the MEMORY_HOTPLUG block, and MEMORY_HOTPLUG already depends on SPARSEMEM_VMEMMAP. Keep the explicit MEMORY_HOTPLUG dependency for local readability, but drop the redundant SPARSEMEM_VMEMMAP dependency. ZONE_DEVICE depends on MEMORY_HOTREMOVE, which depends on MEMORY_HOTPLUG. MEMORY_HOTPLUG in turn depends on SPARSEMEM_VMEMMAP. Drop the direct MEMORY_HOTPLUG and SPARSEMEM_VMEMMAP dependencies from ZONE_DEVICE. This does not change the set of valid configurations. Link: https://lore.kernel.org/20260707090331.52971-1-kaitao.cheng@linux.dev Signed-off-by: Kaitao Cheng <chengkaitao@kylinos.cn> Acked-by: David Hildenbrand (Arm) <david@kernel.org> Acked-by: Muchun Song <muchun.song@linux.dev> Reviewed-by: Lorenzo Stoakes <ljs@kernel.org> Cc: Liam R. Howlett <liam@infradead.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Hajime Tazaki
|
644ad84870 |
mm: nommu: point to the write iterator upon split_vma
When a user invokes munmap(2) on a partial region allocated by mmap(), the
kernel may split the original region if necessary and shrink it to the
correct size. At the beginning of vmi_shrink_vma(), the unused part is
cleared; however, an assertion is triggered if the shrink occurs after
split_vma().
This commit fixes the issue by correctly configuring the pointer to the
iterator at the end of split_vma().
This bug was detected using the Linux Test Project (LTP) test linked
below, running on a nommu UML (User-Mode Linux) environment (via an
out-of-tree extension to UML).
Here is a minimal reproducible chunk of code for this issue:
void *addr;
size_t pagesize = getpagesize();
addr = mmap(NULL, pagesize * 4, PROT_READ | PROT_WRITE,
MAP_ANONYMOUS | MAP_PRIVATE, -1, 0);
munmap(addr + pagesize * 1, pagesize);
This is the console output with CONFIG_DEBUG_MAPLE_TREE=y.
nommu: WARN at __mas_set_range:791 (1)
MAS: tree=0000000091c23b08 enode=0000000065057663
(ma_active)
Store Type:
node_store
[9/9] index=70af8000 last=ffffffffffffffff
min=0 max=ffffffffffffffff sheaf=0000000000000000, request 0
depth=0, flags=0
maple_tree(0000000091c23b08) flags 307, height 1 root 0000000083394c06
0-ffffffffffffffff: node 0000000010c90bd6 depth 0 type 1 parent
0000000050e1ddf8 contents: 0000000000000000 707A
7FFF 00000000eb0ac2b5 707AFFFF 0000000000000000 7093FFFF
0000000045ead616 7095FFFF 0000000000000000 7096CFFF 000
00000681c7151 7096FFFF 0000000000000000 70AF3FFF 000000006c78b9e9
70AF4FFF 000000001914ab0b 70AF7FFF 00000000000
00000 FFFFFFFFFFFFFFFF 0000000000000000 0 0000000000000000 0
0000000000000000 0 0000000000000000 0 0000000000000
000 0 00000000bca8be4f
0-707a7fff: 0000000000000000
707a8000-707affff: 00000000eb0ac2b5
707b0000-7093ffff: 0000000000000000
70940000-7095ffff: 0000000045ead616
70960000-7096cfff: 0000000000000000
7096d000-7096ffff: 00000000681c7151
70970000-70af3fff: 0000000000000000
70af4000-70af4fff: 000000006c78b9e9
70af5000-70af7fff: 000000001914ab0b
70af8000-ffffffffffffffff: 0000000000000000
nommu: Pass: 796 Run:797
Link: https://github.com/linux-test-project/ltp/blob/master/testcases/kernel/syscalls/mseal/mseal02.c
Link: https://lore.kernel.org/20260702012546.665383-1-thehajime@gmail.com
Signed-off-by: Hajime Tazaki <thehajime@gmail.com>
Cc: Jann Horn <jannh@google.com>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Pedro Falcato <pfalcato@suse.de>
Cc: Vlastimil Babka <vbabka@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
||
|
Johannes Weiner
|
d230991493 |
mm: mempolicy: fix automatic numa balancing for shmem
Neha reports that mapped shmem aren't considered for NUMA balancing, noting convergence problems and bandwidth bottlenecking for cachelib based workloads on tiered memory systems. Looking at the code and going through the git history, this doesn't actually seem intentional: Commit |