From 1b4bbae111f13f01732c04a6663af209772e259a Mon Sep 17 00:00:00 2001 From: Breno Leitao Date: Mon, 21 Sep 2026 05:30:26 -0700 Subject: [PATCH] mm: kmemleak: move the struct page scan into a helper Patch series "mm: kmemleak: batch the struct page scan". kmemleak walks the struct page array as a scan root and hands it to scan_block() one page at a time. scan_block() takes kmemleak_lock with interrupts disabled for the duration of the call, so the scanner acquires the lock once per online PFN in order to look at a single 64-byte struct page. Batching adjacent pages up to that size keeps exactly the same pages under the same maximum lock hold time, with MAX_SCAN_SIZE / pagesize fewer acquisitions. Patch 1 lifts the loop out of __kmemleak_scan() into scan_zone_pages() with no functional change. Patch 2 does the batching, which is then contained in that one function. This improves the performance due to less atomic operations, which is not a big deal on a regular machine, but, given kmemleak usually comes with extra debug options, such as PROVE_LOCKING, DEBUG_SPINLOCK, etc. For instance, measuring Meta's "debug kernel flavor" on an arm64 hosts, this improve the scan time by 20%. It is safe to get more work into scan_block(), given it has the protections, added by commit eb11f56eeca560 ("mm/kmemleak: stop the task stack scan early when interrupted") MAX_SCAN_SIZE is also not a new maximum for a single scan_block() call. kmemleak_scan_task_stacks() already hands it a whole task stack in one go. On arm64 and x86_64 THREAD_SIZE is never below 16 KiB, four times MAX_SCAN_SIZE, and it is 64 KiB on arm64 with 64K pages. This patch (of 2): The struct page scanning loop sits inline in __kmemleak_scan(), nested three levels deep and sharing the caller's "stop" variable with the zone walk around it. Move it into scan_zone_pages(), which scans one zone and returns 1 if scanning should stop. No functional change; this only makes room for changing how the pages are handed to scan_block(). Link: https://lore.kernel.org/20260921-b4-kmemleak-page-scan-v1-1-fb97d4801b3a@debian.org Signed-off-by: Breno Leitao Signed-off-by: Andrew Morton Reviewed-by: Catalin Marinas --- mm/kmemleak.c | 57 ++++++++++++++++++++++++++++++--------------------- 1 file changed, 34 insertions(+), 23 deletions(-) diff --git a/mm/kmemleak.c b/mm/kmemleak.c index 5d0daea93c47..fe34c5650f42 100644 --- a/mm/kmemleak.c +++ b/mm/kmemleak.c @@ -1855,6 +1855,39 @@ static void dedup_flush(struct xarray *dedup) } } +/* + * Scan the struct pages of a zone, skipping memory holes, pages that belong to + * another zone and pages that are not in use. Returns 1 if the scan should be + * stopped. + */ +static int scan_zone_pages(struct zone *zone) +{ + unsigned long start_pfn = zone->zone_start_pfn; + unsigned long end_pfn = zone_end_pfn(zone); + unsigned long pfn; + + for (pfn = start_pfn; pfn < end_pfn; pfn++) { + struct page *page = pfn_to_online_page(pfn); + + if (!(pfn & 63)) + cond_resched_tasks_rcu_qs(); + + if (!page) + continue; + + /* only scan pages belonging to this zone */ + if (page_zone(page) != zone) + continue; + /* only scan if page is in use */ + if (page_count(page) == 0) + continue; + if (scan_block(page, page + 1, NULL)) + return 1; + } + + return 0; +} + /* * Scan data sections and all the referenced memory blocks allocated via the * kernel's standard allocators. This function must be called with the @@ -1928,29 +1961,7 @@ static int __kmemleak_scan(bool full) */ get_online_mems(); for_each_populated_zone(zone) { - unsigned long start_pfn = zone->zone_start_pfn; - unsigned long end_pfn = zone_end_pfn(zone); - unsigned long pfn; - - for (pfn = start_pfn; pfn < end_pfn; pfn++) { - struct page *page = pfn_to_online_page(pfn); - - if (!(pfn & 63)) - cond_resched_tasks_rcu_qs(); - - if (!page) - continue; - - /* only scan pages belonging to this zone */ - if (page_zone(page) != zone) - continue; - /* only scan if page is in use */ - if (page_count(page) == 0) - continue; - stop = scan_block(page, page + 1, NULL); - if (stop) - break; - } + stop = scan_zone_pages(zone); if (stop) break; }