From c7a467fbcbbc31e57a5077bfc9456d9f29cb1571 Mon Sep 17 00:00:00 2001 From: "Harry Yoo (Meta)" Date: Tue, 13 Oct 2026 20:34:39 +0100 Subject: [PATCH] mm/percpu: account full object size for percpu memory allocations When memory allocation profiling is enabled, the pages backing percpu memory chunks are attributed to func:pcpu_alloc_pages. However, the accounting of individual percpu memory allocations via alloc_percpu() and friends record only the size of a single object rather than its full size across all CPUs. As a result, it is often not straightforward to identify which kernel code is actually responsible for percpu memory, because the size is under-accounted by a factor of num_possible_cpus(). Account full object size with pcpu_obj_full_size(size) instead, which multiplies the object size by num_possible_cpus() and adds the objcg pointer overhead. This is consistent with what memcg already does for charging percpu allocations. Verified with 64-CPU VM by spawning 400 processes in a memory cgroup and measuring mm_init()'s rss_stat counters. Delta of kernel/fork.c:1142 func:mm_init in /proc/allocinfo: bytes calls per allocation unpatched 6320 395 16 patched 418176 396 1056 'percpu' field in memory.stat: 422400 (1056 bytes per process) Memory allocation profiling now accounts the same amount of memory as what memcg charges. Link: https://lore.kernel.org/20261013-fix-alloc-percpu-accounting-v1-1-03e788a1e41f@kernel.org Fixes: 60fa4a9e2323 ("mm: percpu: add codetag reference into pcpuobj_ext") Signed-off-by: Harry Yoo (Meta) Signed-off-by: Andrew Morton Reviewed-by: Joshua Hahn Assisted-by: LLM # Verification Cc: Dennis Zhou Cc: Tejun Heo Cc: Christoph Lameter Cc: Suren Baghdasaryan Cc: Hao Ge Cc: --- mm/percpu.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/mm/percpu.c b/mm/percpu.c index 3eff382e565a..d7883507dc15 100644 --- a/mm/percpu.c +++ b/mm/percpu.c @@ -1696,14 +1696,15 @@ static void pcpu_alloc_tag_alloc_hook(struct pcpu_chunk *chunk, int off, { if (mem_alloc_profiling_enabled() && likely(chunk->obj_exts)) { alloc_tag_add(&chunk->obj_exts[off >> PCPU_MIN_ALLOC_SHIFT].tag, - current->alloc_tag, size); + current->alloc_tag, pcpu_obj_full_size(size)); } } static void pcpu_alloc_tag_free_hook(struct pcpu_chunk *chunk, int off, size_t size) { if (mem_alloc_profiling_enabled() && likely(chunk->obj_exts)) - alloc_tag_sub(&chunk->obj_exts[off >> PCPU_MIN_ALLOC_SHIFT].tag, size); + alloc_tag_sub(&chunk->obj_exts[off >> PCPU_MIN_ALLOC_SHIFT].tag, + pcpu_obj_full_size(size)); } #else static void pcpu_alloc_tag_alloc_hook(struct pcpu_chunk *chunk, int off,