github-mirror/linux
1
0
Fork 0
mirror of https://github.com/torvalds/linux.git synced 2026-09-14 16:10:02 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity Graph
0e0ac326c5
linux/mm
History
Shakeel Butt 0e0ac326c5 memcg: move LRU size accounting on reparenting instead of copying it 
When a memory cgroup is offlined its LRU folios are reparented to the
parent.  lruvec_reparent_lru() splices the child's lists into the
parent's and credits the parent with the child's per-zone
lru_zone_size[], but never clears the child's copy, so the size is
copied rather than moved.  lru_gen_reparent_memcg() does the same for
MGLRU.

The parent is left correct, credited with exactly the folios it took
over.  The stale value sits on the child and nothing will correct it:
folio->memcg_data now resolves to the parent, so every later
update_lru_size() for those folios goes there.

Dying cgroups are not freed immediately and mem_cgroup_iter() still
walks them, so shrink_lruvec() keeps being called on them.
get_scan_count() reads the phantom counter through lruvec_lru_size() and
the scan loop then grinds through nr[] in SWAP_CLUSTER_MAX steps against
an empty list, for as long as the dead cgroup lives.  Under MGLRU the
MGLRU scanner runs instead, but count_shadow_nodes() sums all of
NR_LRU_LISTS through lruvec_lru_size() and over-budgets the shadow node
limit just the same.

On one 251 GiB host a sweep of every mz->lru_zone_size[] found 380
counters describing folios on no list at all: 124777314 pages, 476 GiB,
1.89x the machine's RAM, across 57 cgroups.  All were on memcgs with
CSS_DYING set and CSS_ONLINE clear, and parent/child pairs reported
byte-identical sizes.

LRU_UNEVICTABLE needs its size moved too.  Its list is deliberately not
spliced because lruvec_init() poisons the head - the unevictable LRU is
imaginary and folios are never threaded on it - but the size is kept by
lruvec_add_folio()/lruvec_del_folio() and those folios account to the
parent from here on.

This depends on commit bf4ade7dbd ("memcg: keep folio's objcg same as
its node") and must not be backported ahead of it.  Without that
invariant a folio's objcg can belong to another node, so a folio already
spliced onto the parent's list can still resolve to the child's lruvec
until the objcg's node is reparented in a later iteration of
memcg_reparent_objcgs(); clearing the child's counter early then lets
lruvec_del_folio() underflow it and trip the WARN_ONCE()/VM_BUG_ON() in
mem_cgroup_update_lru_size().

Link: https://lore.kernel.org/20260822024707.77192-1-shakeel.butt@linux.dev
Fixes: 07a6e9a2c1 ("mm: vmscan: prepare for reparenting traditional LRU folios")
Fixes: f304652609 ("mm: vmscan: prepare for reparenting MGLRU folios")
Signed-off-by: Shakeel Butt <shakeel.butt@linux.dev>
Acked-by: Michal Hocko <mhocko@suse.com>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Roman Gushchin <roman.gushchin@linux.dev>
Cc: Muchun Song <muchun.song@linux.dev>
Cc: <stable@vger.kernel.org> # After: bf4ade7dbd76: memcg: keep folio's objcg same as its node
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
2026-08-24 18:43:33 -07:00
..
damon Merge branch 'mm-hotfixes-stable' into mm-stable to pick up 2026-08-24 18:40:27 -07:00
kasan kasan: fix quarantine_size accounting during cache removal 2026-08-24 18:43:21 -07:00
kfence slab changes for 7.2 - part 2 2026-06-22 08:28:48 -07:00
kmsan mm: split out vmalloc declarations from internal.h 2026-08-04 19:18:46 -07:00
tests sparc/mm: export symbols for lazy_mmu_mode KUnit tests 2026-01-31 14:22:40 -08:00
alloc_tag.c alloc_tag: add accuracy based filtering to ioctl 2026-08-24 18:43:13 -07:00
arch_numa.c arch_numa: avoid false positive fortify warning in setup_node_to_cpumask_map() 2026-08-24 18:43:25 -07:00
backing-dev.c mm: blk-cgroup: fix use-after-free in cgwb_release_workfn() 2026-04-18 23:24:27 -07:00
balloon.c mm: rename CONFIG_BALLOON_COMPACTION to CONFIG_BALLOON_MIGRATION 2026-01-31 14:22:36 -08:00
bpf_memcontrol.c
cma_debug.c mm/cma_debug: fix invalid accesses for inactive CMA areas 2026-05-28 20:50:33 -07:00
cma_sysfs.c mm/cma_sysfs: skip inactive CMA areas in sysfs 2026-06-03 16:25:49 -07:00
cma.c mm/cma: remove stray newline from auto-generated CMA area name 2026-08-24 18:43:16 -07:00
cma.h mm: add some missing includes to mm-local headers 2026-08-24 18:43:03 -07:00
compaction.c mm: page_alloc: fix non-movable reclaim storm in defrag_mode 2026-08-06 18:57:24 -07:00
debug_page_alloc.c mm: debug_page_alloc: fix NULL buf in debug_guardpage_minorder_setup 2026-08-24 18:43:08 -07:00
debug_page_ref.c
debug_vm_pgtable.c mm: decouple protnone helpers from CONFIG_NUMA_BALANCING 2026-08-04 19:18:36 -07:00
debug.c mm: prefer mm->def_vma_flags in mm logic 2026-08-06 18:57:02 -07:00
dmapool_test.c Convert 'alloc_obj' family to use the new default GFP_KERNEL argument 2026-02-21 17:09:51 -08:00
dmapool.c
early_ioremap.c mm/early_ioremap: clarify early_ioremap_reset() semantics 2026-08-06 18:57:21 -07:00
execmem.c mm: split out vmalloc declarations from internal.h 2026-08-04 19:18:46 -07:00
fadvise.c mm/fadvise: validate offset in generic_fadvise 2026-04-05 13:52:53 -07:00
fail_page_alloc.c
failslab.c
filemap.c Merge branch 'mm-hotfixes-stable' into mm-stable to pick up 2026-08-24 18:40:27 -07:00
folio-compat.c mm/page-writeback: document folio_mark_dirty() locking more explicitly 2026-08-24 18:43:19 -07:00
folio.c memcg: move LRU size accounting on reparenting instead of copying it 2026-08-24 18:43:33 -07:00
gup_test.c mm/gup_test: keep longterm pin state per file 2026-08-24 18:43:17 -07:00
gup_test.h
gup.c mm/gup: factor out LRU cache draining for folio into lru_cache_drain_for_folio() 2026-08-24 18:43:00 -07:00
highmem.c mm/highmem: fix __kmap_to_page() build error 2026-01-31 14:22:38 -08:00
hmm.c mm/hmm.c:hmm_do_fault(): suppress sparse warning 2026-08-24 18:43:24 -07:00
huge_memory.c mm/huge_memory: use folio's memcg inside __folio_split() 2026-08-24 18:42:59 -07:00
hugetlb_cgroup.c Convert 'alloc_flex' family to use the new default GFP_KERNEL argument 2026-02-21 17:09:51 -08:00
hugetlb_cma.c mm/hugetlb_cma: support percentage-based hugetlb_cma reservation 2026-08-24 18:43:11 -07:00
hugetlb_cma.h mm: add some missing includes to mm-local headers 2026-08-24 18:43:03 -07:00
hugetlb_internal.h
hugetlb_sysctl.c
hugetlb_sysfs.c
hugetlb_vmemmap.c mm/hugetlb_vmemmap: remove bootmem_info leftovers 2026-08-04 19:18:42 -07:00
hugetlb_vmemmap.h mm/sparse-vmemmap: remove sparse_vmemmap_init_nid_late() 2026-07-28 21:12:00 -07:00
hugetlb.c hugetlb: only adjust reservation during unmapping if mapcount is 0 2026-08-24 18:42:58 -07:00
hwpoison-inject.c
init-mm.c exec_state: relocate dumpable information 2026-05-26 11:02:01 +02:00
internal.h mm/gup: factor out LRU cache draining for folio into lru_cache_drain_for_folio() 2026-08-24 18:43:00 -07:00
interval_tree.c mm/rmap: use anon pgoff to track MAP_PRIVATE file-backed anon folios 2026-08-24 18:42:53 -07:00
ioremap.c
Kconfig mm/Kconfig: make MEMORY_FAILURE select MIGRATION 2026-08-24 18:43:24 -07:00
Kconfig.debug percpu: drop CONFIG_DEBUG_FORCE_WEAK_PER_CPU 2026-08-24 18:43:24 -07:00
khugepaged.c mm/khugepaged: unmap pte before releasing vma write lock 2026-08-24 18:43:23 -07:00
kmemleak.c mm: kmemleak: default min_unref_scans to 2 for verbose auto-scan 2026-08-24 18:43:18 -07:00
ksm.c ksm: update comments and docs to reference folio->mapping 2026-08-24 18:43:02 -07:00
list_lru.c mm.git review status for mm-hotfixes-stable..mm-stable 2026-06-19 10:14:34 -07:00
maccess.c
madvise.c mm/swap: add a new swap_ops.h header to allow for pluggable swap ops 2026-08-24 18:43:20 -07:00
Makefile drivers/base, mm: move arch_numa.c to mm/ 2026-08-24 18:43:10 -07:00
mapping_dirty_helpers.c mm/vma: use vma_start_pgoff(), linear_page_index() in mm code 2026-08-04 19:19:00 -07:00
memblock.c mm: split out mm_init and memblock declarations from internal.h 2026-08-04 19:18:45 -07:00
memcontrol-v1.c mm/vmscan: reduce lru_lock contention via vmstat-derived scan-balance cost 2026-08-24 18:42:56 -07:00
memcontrol-v1.h mm: add some missing includes to mm-local headers 2026-08-24 18:43:03 -07:00
memcontrol.c mm: memcg: release the css reference when a stock slot empties 2026-08-24 18:43:26 -07:00
memfd_luo.c mm/memfd_luo: document preservation of file seals 2026-05-04 14:03:16 +00:00
memfd.c mm/vma: update do_mmap() to use vma_flags_t 2026-08-06 18:57:01 -07:00
memory_hotplug.c mm/memory_hotplug: add offline_and_remove_memory_ranges() 2026-08-06 18:56:59 -07:00
memory-failure.c mm: introduce and use vma_filebacked_address() 2026-08-24 18:42:51 -07:00
memory-tiers.c mm: introduce CONFIG_NUMA_MIGRATION and simplify CONFIG_MIGRATION 2026-04-05 13:53:33 -07:00
memory.c mm: update print_bad_page_map() to show anon index if appropriate 2026-08-24 18:42:51 -07:00
mempolicy.c mm: provide vma_[flags_]is_cow_mapping() and remove is_cow_mapping() 2026-08-24 18:42:50 -07:00
mempool.c mm: simplify the mempool_alloc_bulk API 2026-06-03 18:20:47 +02:00
memremap.c mm: decrement MTHP_STAT_NR_ANON in free_zone_device_folio() 2026-07-28 17:37:31 -07:00
memtest.c
migrate_device.c mm/migrate_device: fix cache flush when replacing huge zero PMD 2026-08-24 18:43:26 -07:00
migrate.c mm/migrate: calculate large folio page index using PFN 2026-08-24 18:42:53 -07:00
mincore.c mm: mincore: refactor mincore_page() 2026-08-06 18:57:15 -07:00
mlock.c mm/mlock: convert mlock code to use vma_flags_t 2026-08-06 18:57:03 -07:00
mm_init.c mm/page_ext: remove pgdat_page_ext_init() 2026-08-24 18:43:03 -07:00
mm_init.h mm: split out mm_init and memblock declarations from internal.h 2026-08-04 19:18:45 -07:00
mm_slot.h mm/mm_slot.h: add comments for mm_slot_lookup/insert 2026-08-06 18:57:04 -07:00
mmap_lock.c mm/vma: improve and document __is_vma_write_locked() 2026-01-31 14:22:51 -08:00
mmap.c mm/vma: convert miscellaneous uses of VMA flags in core mm 2026-08-06 18:57:03 -07:00
mmu_gather.c mm/mmu_gather: replace IPI with synchronize_rcu() when batch allocation fails 2026-04-05 13:53:05 -07:00
mmu_notifier.c mm/vma: correct incorrect vma.h inclusion 2026-08-04 19:19:04 -07:00
mmzone.c mm/vmscan: reduce lru_lock contention via vmstat-derived scan-balance cost 2026-08-24 18:42:56 -07:00
mprotect.c mm/mprotect: convert mprotect code to use vma_flags_t 2026-08-06 18:57:04 -07:00
mremap.c mm: use proper PTE accessor in move_ptes() 2026-08-24 18:42:58 -07:00
mseal.c mm/mseal: remove further superfluous comments, do_mseal() 2026-08-06 18:57:12 -07:00
msync.c mm/vma: use vma_start_pgoff(), linear_page_index() in mm code 2026-08-04 19:19:00 -07:00
nommu.c mm: nommu: point to the write iterator upon split_vma 2026-08-24 18:42:48 -07:00
numa_emulation.c mm/fake-numa: fix under-allocation detection in uniform split 2026-06-02 08:34:03 +03:00
numa_memblks.c memblock: numa_memblks: fix detection of NUMA node for CXL windows 2026-02-21 09:58:22 -08:00
numa.c
oom_kill.c mm/oom_kill.c: simpilfy rcu call with guard(rcu) 2026-04-05 13:53:17 -07:00
page_alloc.c mm/page_alloc: only update lowmem_reserve_ratio on sysctl write 2026-08-24 18:43:01 -07:00
page_alloc.h mm: replace __GFP_NO_CODETAG with ALLOC_NO_CODETAG 2026-07-30 19:40:44 -07:00
page_counter.c
page_ext.c mm/page_ext: remove pgdat_page_ext_init() 2026-08-24 18:43:03 -07:00
page_frag_cache.c mm: replace __GFP_NO_CODETAG with ALLOC_NO_CODETAG 2026-07-30 19:40:44 -07:00
page_idle.c mm/page_idle.c: remove redundant mmu notifier in aging code 2026-04-05 13:53:02 -07:00
page_io.c mm/swap: move swap_ops into file systems for file system-based swap 2026-08-24 18:43:20 -07:00
page_isolation.c mm: split out internal page_alloc.h 2026-07-30 19:40:41 -07:00
page_owner.c mm: split out internal page_alloc.h 2026-07-30 19:40:41 -07:00
page_poison.c
page_reporting.c mm/page_reporting: add page_reporting_delay_ms module parameter 2026-08-24 18:42:58 -07:00
page_reporting.h
page_table_check.c Merge branch 'mm-hotfixes-stable' into mm-stable to pick up 2026-08-24 18:40:27 -07:00
page_vma_mapped.c mm/rmap: use anon pgoff to track MAP_PRIVATE file-backed anon folios 2026-08-24 18:42:53 -07:00
page-writeback.c mm/page-writeback: document folio_mark_dirty() locking more explicitly 2026-08-24 18:43:19 -07:00
pagewalk.c Merge branch 'mm-hotfixes-stable' into mm-stable to pick up 2026-08-24 18:40:27 -07:00
percpu-internal.h mm/percpu-internal.h: optimise pcpu_chunk struct to save memory 2026-06-02 15:22:13 -07:00
percpu-km.c mm/percpu-km: clear page->private before free them 2026-08-04 19:18:32 -07:00
percpu-stats.c
percpu-vm.c mm: split out vmalloc declarations from internal.h 2026-08-04 19:18:46 -07:00
percpu.c mm/percpu: avoid IO/FS reclaim in backing allocations 2026-07-28 21:11:47 -07:00
pgalloc-track.h mm: add some missing includes to mm-local headers 2026-08-24 18:43:03 -07:00
pgtable-generic.c mm: change to return bool for pmdp_clear_flush_young() 2026-04-05 13:53:35 -07:00
process_vm_access.c
ptdump.c mm/ptdump: always stabilise against page table freeing using init_mm 2026-08-04 20:02:00 -07:00
readahead.c Revert "mm: limit filemap_fault readahead to VMA boundaries" 2026-06-21 11:37:38 -07:00
rmap.c mm/rmap: synchronize lock and unlock target in anon_vma_clone 2026-08-24 18:43:25 -07:00
rodata_test.c
secretmem.c mm/secretmem: don't allow highmem folios 2026-08-06 18:57:09 -07:00
shmem_quota.c treewide: Replace kmalloc with kmalloc_obj for non-scalar types 2026-02-21 01:02:28 -08:00
shmem.c mm/swap: add a new swap_ops.h header to allow for pluggable swap ops 2026-08-24 18:43:20 -07:00
show_mem.c mm/show_mem: fix format string inconsistencies and type mismatches 2026-08-24 18:43:06 -07:00
shrinker_debug.c mm: shrinker: fix NULL pointer dereference in debugfs 2026-07-01 19:02:52 -07:00
shrinker.c mm: shrinker: fix shrinker_info teardown race with expansion 2026-07-01 19:02:52 -07:00
shuffle.c mm: split out internal page_alloc.h 2026-07-30 19:40:41 -07:00
shuffle.h mm: add some missing includes to mm-local headers 2026-08-24 18:43:03 -07:00
slab_common.c mm/slab: prevent unbounded recursion in free path with new kmalloc type 2026-07-14 17:22:40 +02:00
slab.h mm: fix ASSERT_EXCLUSIVE_BITS by passing memdesc_flags_t by pointer 2026-07-30 19:40:31 -07:00
slub.c mm: factor out can_spin_trylock() 2026-07-30 19:40:44 -07:00
sparse-vmemmap.c mm: split out sparse declarations from internal.h 2026-08-04 19:18:46 -07:00
sparse.c mm/sparse: keep mem_section_usage_size() internal 2026-08-24 18:43:06 -07:00
sparse.h mm/sparse: keep mem_section_usage_size() internal 2026-08-24 18:43:06 -07:00
swap_state.c mm/swap: add a new swap_ops.h header to allow for pluggable swap ops 2026-08-24 18:43:20 -07:00
swap_table.h mm, swap: merge zeromap into swap table 2026-06-02 15:22:23 -07:00
swap.h mm/swap: add a new swap_ops.h header to allow for pluggable swap ops 2026-08-24 18:43:20 -07:00
swapfile.c mm, swap: ratelimit bad swap entry reports 2026-08-24 18:43:27 -07:00
truncate.c mm/truncate: use folio_split() in truncate_inode_partial_folio() 2026-06-21 11:37:16 -07:00
usercopy.c
userfaultfd.c mm/rmap: use anon pgoff to track MAP_PRIVATE file-backed anon folios 2026-08-24 18:42:53 -07:00
util.c mm/vma: update do_mmap() to use vma_flags_t 2026-08-06 18:57:01 -07:00
vma_exec.c mm: propagate VMA anonymous page offset on map, remap, split + merge 2026-08-24 18:42:52 -07:00
vma_init.c mm/vma: introduce VMA anon page offset field and add helpers 2026-08-24 18:42:49 -07:00
vma_internal.h mm/vma: correct incorrect vma.h inclusion 2026-08-04 19:19:04 -07:00
vma.c mm/rmap: use anon pgoff to track MAP_PRIVATE file-backed anon folios 2026-08-24 18:42:53 -07:00
vma.h mm: propagate VMA anonymous page offset on map, remap, split + merge 2026-08-24 18:42:52 -07:00
vmalloc.c mm/vmalloc: do not warn on -ENOMEM from va_alloc() 2026-08-24 18:43:02 -07:00
vmalloc.h mm: add some missing includes to mm-local headers 2026-08-24 18:43:03 -07:00
vmpressure.c mm/vmpressure: move v1 userspace eventfd code into memcontrol-v1.c 2026-07-30 19:40:28 -07:00
vmscan.c memcg: move LRU size accounting on reparenting instead of copying it 2026-08-24 18:43:33 -07:00
vmstat.c mm/vmstat: add NRSWP{IN,OUT} counters 2026-08-24 18:43:15 -07:00
workingset.c mm/vmscan: reduce lru_lock contention via vmstat-derived scan-balance cost 2026-08-24 18:42:56 -07:00
zpdesc.h
zsmalloc.c zsmalloc: account for handle size in class lookup 2026-08-24 18:43:19 -07:00
zswap.c mm/swap: add a new swap_ops.h header to allow for pluggable swap ops 2026-08-24 18:43:20 -07:00