github-mirror/linux
1
0
Fork 0
mirror of https://github.com/torvalds/linux.git synced 2026-11-13 13:38:14 +01:00
Code Issues Actions Packages Projects Releases Wiki Activity Graph
54fdcbfe1c
linux/mm
History
Frank van der Linden b480cbb071 mm/page_alloc: don't increase highatomic reserve after pcp alloc 
Higher order GFP_ATOMIC allocations can be served through a PCP list with
ALLOC_HIGHATOMIC set.  Such an allocation can e.g.  happen if a zone is
between the low and min watermarks, and get_page_from_freelist is retried
after the alloc_flags are relaxed.

The call to reserve_highatomic_pageblock() after such a PCP allocation
will result in an increase every single time: the page from the
(unmovable) PCP list will never have migrate type MIGRATE_HIGHATOMIC,
since MIGRATE_HIGHATOMIC pages do not appear on the unmovable PCP list. 
So a new pageblock is converted to MIGRATE_HIGHATOMIC.

Eventually that leads to the maximum of 1% of the zone being used up by
(often mostly free) MIGRATE_HIGHATOMIC pageblocks, for no good reason. 
Since this space is not available for normal allocations, this wastes
memory and will push things in to reclaim too soon.

This was observed on a system that ran a test with bursts of memory
activity, paired with GFP_ATOMIC SLUB activity.  These would lead to a new
slab being allocated with GFP_ATOMIC, sometimes hitting the
get_page_from_freelist retry path by being below the low watermark.  While
the frequency of those allocations was low, it kept adding up over time,
and the number of MIGRATE_ATOMIC pageblocks kept increasing.

If a higher order atomic allocation can be served by the unmovable PCP
list, there is probably no need yet to extend the reserves.  So, move the
check and possible extension of the highatomic reserves to the buddy case
only, and do not refill the PCP list for ALLOC_HIGHATOMIC if it's empty. 
This way, the PCP list is tried for ALLOC_HIGHATOMIC for a fast atomic
allocation.  But it will immediately fall back to rmqueue_buddy() if it's
empty.  In rmqueue_buddy(), the MIGRATE_HIGHATOMIC buddy lists are tried
first (as before), and the reserves are extended only if that fails.

With this change, the test was stable.  Highatomic reserves were built up,
but to a normal level.  No highatomic failures were seen.

This is similar to the patch proposed in [1] by Zhiguo Jiang, but
re-arranged a bit.

Link: https://lkml.kernel.org/r/20260320173426.1831267-1-fvdl@google.com
Link: https://lore.kernel.org/all/20231122013925.1507-1-justinjiang@vivo.com/ [1]
Fixes: 44042b4498 ("mm/page_alloc: allow high-order pages to be stored on the per-cpu lists")
Signed-off-by: Zhiguo Jiang <justinjiang@vivo.com>
Signed-off-by: Frank van der Linden <fvdl@google.com>
Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Cc: Brendan Jackman <jackmanb@google.com>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Michal Hocko <mhocko@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Zhiguo Jiang <justinjiang@vivo.com>
Cc: Zi Yan <ziy@nvidia.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
2026-04-05 13:53:36 -07:00
..
damon mm/damon/core: document damos_commit_dests() failure semantics 2026-04-05 13:53:34 -07:00
kasan kasan: fix bug type classification for SW_TAGS mode 2026-04-05 13:53:18 -07:00
kfence kfence: add kfence.fault parameter 2026-04-05 13:53:06 -07:00
kmsan Convert 'alloc_obj' family to use the new default GFP_KERNEL argument 2026-02-21 17:09:51 -08:00
tests sparc/mm: export symbols for lazy_mmu_mode KUnit tests 2026-01-31 14:22:40 -08:00
backing-dev.c treewide: Replace kmalloc with kmalloc_obj for non-scalar types 2026-02-21 01:02:28 -08:00
balloon.c mm: rename CONFIG_BALLOON_COMPACTION to CONFIG_BALLOON_MIGRATION 2026-01-31 14:22:36 -08:00
bootmem_info.c mm/bootmem_info: avoid using sparse_decode_mem_map() 2026-04-05 13:53:32 -07:00
bpf_memcontrol.c bpf: Revert "bpf: drop KF_ACQUIRE flag on BPF kfunc bpf_get_root_mem_cgroup()" 2026-01-21 09:38:16 -08:00
cma_debug.c Convert 'alloc_obj' family to use the new default GFP_KERNEL argument 2026-02-21 17:09:51 -08:00
cma_sysfs.c Convert 'alloc_obj' family to use the new default GFP_KERNEL argument 2026-02-21 17:09:51 -08:00
cma.c mm/cma: move put_page_testzero() out of VM_WARN_ON in cma_release() 2026-03-04 09:44:22 -08:00
cma.h
compaction.c mm/compaction: fix low_pfn advance on isolating hugetlb 2025-09-28 11:51:29 -07:00
debug_page_alloc.c
debug_page_ref.c
debug_vm_pgtable.c mm/debug_vm_pgtable: replace WRITE_ONCE() with pxd_clear() 2026-04-05 13:53:11 -07:00
debug.c mm: constify __dump_folio() arguments 2025-11-20 13:43:57 -08:00
dmapool_test.c Convert 'alloc_obj' family to use the new default GFP_KERNEL argument 2026-02-21 17:09:51 -08:00
dmapool.c
early_ioremap.c mm/early_ioremap: clean up the use of WARN() for debugging 2026-01-26 20:02:26 -08:00
execmem.c mm/execmem: make the populate and alloc atomic 2026-04-05 13:53:34 -07:00
fadvise.c mm/fadvise: validate offset in generic_fadvise 2026-04-05 13:52:53 -07:00
fail_page_alloc.c
failslab.c
filemap.c folio_batch: rename pagevec.h to folio_batch.h 2026-04-05 13:53:07 -07:00
folio-compat.c mm: add SPDX id lines to some mm source files 2026-02-06 15:47:16 -08:00
gup_test.c mm: add SPDX id lines to some mm source files 2026-02-06 15:47:16 -08:00
gup_test.h
gup.c folio_batch: rename pagevec.h to folio_batch.h 2026-04-05 13:53:07 -07:00
highmem.c mm/highmem: fix __kmap_to_page() build error 2026-01-31 14:22:38 -08:00
hmm.c treewide: Replace kmalloc with kmalloc_obj for non-scalar types 2026-02-21 01:02:28 -08:00
huge_memory.c mm: introduce is_pmd_order helper 2026-04-05 13:53:29 -07:00
hugetlb_cgroup.c Convert 'alloc_flex' family to use the new default GFP_KERNEL argument 2026-02-21 17:09:51 -08:00
hugetlb_cma.c mm: hugetlb_cma: mark hugetlb_cma{_only} as __ro_after_init 2026-01-31 14:22:43 -08:00
hugetlb_cma.h mm: hugetlb: allocate frozen pages for gigantic allocation 2026-01-26 20:02:28 -08:00
hugetlb_internal.h mm/hugetlb: extract sysctl into hugetlb_sysctl.c 2025-11-20 13:43:57 -08:00
hugetlb_sysctl.c mm, hugetlb: implement movable_gigantic_pages sysctl 2026-01-20 19:24:50 -08:00
hugetlb_sysfs.c mm/hugetlb: extract sysfs into hugetlb_sysfs.c 2025-11-20 13:43:57 -08:00
hugetlb_vmemmap.c mm/hugetlb: remove hugetlb_optimize_vmemmap_key static key 2026-04-05 13:53:09 -07:00
hugetlb_vmemmap.h
hugetlb.c mm: move vma_mmu_pagesize() from hugetlb to vma.c 2026-04-05 13:53:23 -07:00
hwpoison-inject.c
init-mm.c mm: rename cpu_bitmap field to flexible_array 2026-01-19 12:30:00 -08:00
internal.h mm: change to return bool for the MMU notifier's young flag check 2026-04-05 13:53:35 -07:00
interval_tree.c mm/memory: simplify calculation in unmap_mapping_range_tree() 2026-04-05 13:53:13 -07:00
ioremap.c
Kconfig mm: introduce CONFIG_NUMA_MIGRATION and simplify CONFIG_MIGRATION 2026-04-05 13:53:33 -07:00
Kconfig.debug mm: fix DEBUG_RODATA_TEST indentation in Kconfig 2025-11-29 10:41:09 -08:00
khugepaged.c mm/khugepaged: unify khugepaged and madv_collapse with collapse_single_pmd() 2026-04-05 13:53:30 -07:00
kmemleak.c mm/kmemleak: use PF_KTHREAD flag to detect kernel threads 2026-04-05 13:53:02 -07:00
ksm.c ksm: initialize the addr only once in rmap_walk_ksm 2026-04-05 13:52:35 -07:00
list_lru.c Convert 'alloc_obj' family to use the new default GFP_KERNEL argument 2026-02-21 17:09:51 -08:00
maccess.c
madvise.c mm: rename zap_page_range_single() to zap_vma_range() 2026-04-05 13:53:15 -07:00
Makefile mm.git review status for linus..mm-nonmm-stable 2026-02-12 12:13:01 -08:00
mapping_dirty_helpers.c mm/dirty: replace READ_ONCE() with pudp_get() 2025-11-16 17:27:58 -08:00
memblock.c memblock: updates for 7.0-rc1 2026-02-14 12:39:34 -08:00
memcontrol-v1.c memcg: consolidate private id refcount get/put helpers 2026-04-05 13:53:00 -07:00
memcontrol-v1.h memcg: consolidate private id refcount get/put helpers 2026-04-05 13:53:00 -07:00
memcontrol.c mm/memcontrol: fix reclaim_options leak in try_charge_memcg() 2026-04-05 13:53:36 -07:00
memfd_luo.c mm: memfd_luo: preserve file seals 2026-04-05 13:53:00 -07:00
memfd.c memfd: export memfd_{add,get}_seals() 2026-04-05 13:53:00 -07:00
memory_hotplug.c mm/memory_hotplug: simplify check_pfn_span() 2026-04-05 13:53:31 -07:00
memory-failure.c treewide: Replace kmalloc with kmalloc_obj for non-scalar types 2026-02-21 01:02:28 -08:00
memory-tiers.c mm: introduce CONFIG_NUMA_MIGRATION and simplify CONFIG_MIGRATION 2026-04-05 13:53:33 -07:00
memory.c mm: introduce is_pmd_order helper 2026-04-05 13:53:29 -07:00
mempolicy.c mm: introduce CONFIG_NUMA_MIGRATION and simplify CONFIG_MIGRATION 2026-04-05 13:53:33 -07:00
mempool.c Convert 'alloc_obj' family to use the new default GFP_KERNEL argument 2026-02-21 17:09:51 -08:00
memremap.c mm/zone_device: reinitialize large zone device private folios 2026-01-26 19:03:48 -08:00
memtest.c mm/memtest: add underflow detection for size calculation 2026-01-09 11:53:51 +02:00
migrate_device.c mm/migrate_device: document folio_get requirement before frozen PMD split 2026-04-05 13:53:19 -07:00
migrate.c mm: introduce CONFIG_NUMA_MIGRATION and simplify CONFIG_MIGRATION 2026-04-05 13:53:33 -07:00
mincore.c mm: replace remaining pte_to_swp_entry() with softleaf_from_pte() 2025-11-24 15:08:52 -08:00
mlock.c mm: prevent droppable mappings from being locked 2026-04-05 13:53:25 -07:00
mm_init.c mm: mark early-init static variables with __meminitdata 2026-04-05 13:53:34 -07:00
mm_slot.h
mmap_lock.c mm/vma: improve and document __is_vma_write_locked() 2026-01-31 14:22:51 -08:00
mmap.c mm: abstract reading sysctl_max_map_count, and READ_ONCE() 2026-04-05 13:53:28 -07:00
mmu_gather.c mm/mmu_gather: replace IPI with synchronize_rcu() when batch allocation fails 2026-04-05 13:53:05 -07:00
mmu_notifier.c mm: change to return bool for the MMU notifier's young flag check 2026-04-05 13:53:35 -07:00
mmzone.c
mprotect.c mm/pkeys: remove unused tsk parameter from arch_set_user_pkey_access() 2026-04-05 13:52:57 -07:00
mremap.c mm/khugepaged: rename hpage_collapse_* to collapse_* 2026-04-05 13:53:30 -07:00
mseal.c mm: fix minor spelling mistakes in comments 2026-01-20 19:24:48 -08:00
msync.c
nommu.c mm: abstract reading sysctl_max_map_count, and READ_ONCE() 2026-04-05 13:53:28 -07:00
numa_emulation.c
numa_memblks.c memblock: numa_memblks: fix detection of NUMA node for CXL windows 2026-02-21 09:58:22 -08:00
numa.c
oom_kill.c mm/oom_kill.c: simpilfy rcu call with guard(rcu) 2026-04-05 13:53:17 -07:00
page_alloc.c mm/page_alloc: don't increase highatomic reserve after pcp alloc 2026-04-05 13:53:36 -07:00
page_counter.c
page_ext.c mm/page_ext: Add page_ext_get_from_phys() 2026-01-21 12:51:48 +01:00
page_frag_cache.c
page_idle.c mm/page_idle.c: remove redundant mmu notifier in aging code 2026-04-05 13:53:02 -07:00
page_io.c mm: update outdated comments for removed scan_swap_map_slots() 2026-04-05 13:53:35 -07:00
page_isolation.c mm: page_isolation: introduce page_is_unmovable() 2026-01-31 14:22:42 -08:00
page_owner.c treewide: Replace kmalloc with kmalloc_obj for non-scalar types 2026-02-21 01:02:28 -08:00
page_poison.c
page_reporting.c mm/page_reporting: change page_reporting_order to PAGE_REPORTING_ORDER_UNSPECIFIED 2026-04-05 13:53:17 -07:00
page_reporting.h
page_table_check.c mm: provide address parameter to p{te,md,ud}_user_accessible_page() 2026-01-26 20:02:35 -08:00
page_vma_mapped.c mm: centralize+fix comments about compound_mapcount() in new sync_with_folio_pmd_zap() 2026-04-05 13:53:03 -07:00
page-writeback.c mm: rename zap_vma_pages() to zap_vma() 2026-04-05 13:53:14 -07:00
pagewalk.c mm/pagewalk: drop FW_MIGRATION 2026-04-05 13:53:10 -07:00
percpu-internal.h
percpu-km.c
percpu-stats.c
percpu-vm.c kmsan: remove hard-coded GFP_KERNEL flags 2025-11-16 17:27:54 -08:00
percpu.c percpu: add double free check to pcpu_free_area() 2026-01-31 14:22:52 -08:00
pgalloc-track.h
pgtable-generic.c mm: change to return bool for pmdp_clear_flush_young() 2026-04-05 13:53:35 -07:00
process_vm_access.c
ptdump.c mm/ptdump: replace READ_ONCE() with standard page table accessors 2025-11-16 17:27:52 -08:00
readahead.c mm.git review status for linus..mm-stable 2026-02-12 11:32:37 -08:00
rmap.c mm/mglru: fix cgroup OOM during MGLRU state switching 2026-04-05 13:53:33 -07:00
rodata_test.c
secretmem.c mm: rename VMA flag helpers to be more readable 2026-04-05 13:53:18 -07:00
shmem_quota.c treewide: Replace kmalloc with kmalloc_obj for non-scalar types 2026-02-21 01:02:28 -08:00
shmem.c mm: introduce is_pmd_order helper 2026-04-05 13:53:29 -07:00
show_mem.c mm/vmscan: add tracepoint and reason for kswapd_failures reset 2026-01-31 14:22:38 -08:00
shrinker_debug.c memcg: rename mem_cgroup_ino() to mem_cgroup_id() 2026-01-26 20:02:25 -08:00
shrinker.c mm: do not allocate shrinker info with cgroup.memory=nokmem 2026-04-05 13:53:05 -07:00
shuffle.c
shuffle.h
slab_common.c Merge branch 'slab/for-7.0/sheaves' into slab/for-next 2026-02-10 09:10:00 +01:00
slab.h mm/slab: use compound_head() in page_slab() 2026-04-05 13:53:10 -07:00
slub.c slab: fix memory leak when refill_sheaf() fails 2026-03-11 17:55:26 +01:00
sparse-vmemmap.c mm: mark early-init static variables with __meminitdata 2026-04-05 13:53:34 -07:00
sparse.c mm/sparse: move memory hotplug bits to sparse-vmemmap.c 2026-04-05 13:53:33 -07:00
swap_cgroup.c
swap_state.c folio_batch: rename PAGEVEC_SIZE to FOLIO_BATCH_SIZE 2026-04-05 13:53:07 -07:00
swap_table.h mm, swap: use the swap table to track the swap count 2026-04-05 13:52:59 -07:00
swap.c folio_batch: rename PAGEVEC_SIZE to FOLIO_BATCH_SIZE 2026-04-05 13:53:07 -07:00
swap.h mm, swap: no need to clear the shadow explicitly 2026-04-05 13:52:59 -07:00
swapfile.c mm/swapfile: remove duplicate include of swap_table.h 2026-04-05 13:53:31 -07:00
truncate.c folio_batch: rename PAGEVEC_SIZE to FOLIO_BATCH_SIZE 2026-04-05 13:53:07 -07:00
usercopy.c usercopy: Remove folio references from check_heap_object() 2025-11-13 11:01:08 +01:00
userfaultfd.c mm/userfaultfd: fix hugetlb fault mutex hash calculation 2026-04-05 13:53:29 -07:00
util.c mm: rework compound_head() for power-of-2 sizeof(struct page) 2026-04-05 13:53:08 -07:00
vma_exec.c mm: softdirty: add pgtable_supports_soft_dirty() 2025-11-24 15:08:54 -08:00
vma_init.c Summary of significant series in this pull request: 2025-10-02 18:18:33 -07:00
vma_internal.h mm: relocate the page table ceiling and floor definitions 2026-02-12 15:42:53 -08:00
vma.c mm: abstract reading sysctl_max_map_count, and READ_ONCE() 2026-04-05 13:53:28 -07:00
vma.h mm: make vm_area_desc utilise vma_flags_t only 2026-02-12 15:42:59 -08:00
vmalloc.c mm: vmalloc: update outdated comment for renamed vread() 2026-04-05 13:53:34 -07:00
vmpressure.c Convert 'alloc_obj' family to use the new default GFP_KERNEL argument 2026-02-21 17:09:51 -08:00
vmscan.c mm: update outdated comments for removed scan_swap_map_slots() 2026-04-05 13:53:35 -07:00
vmstat.c mm: vmalloc: streamline vmalloc memory accounting 2026-04-05 13:53:04 -07:00
workingset.c mm/workingset: leave highest bits empty for anon shadow 2026-04-05 13:52:58 -07:00
zpdesc.h
zsmalloc.c mm/zsmalloc: copy KMSAN metadata in zs_page_migrate() 2026-04-05 13:53:34 -07:00
zswap.c mm/zswap: remove SWP_SYNCHRONOUS_IO swapcache bypass workaround 2026-04-05 13:53:02 -07:00