github-mirror/linux
1
0
Fork 0
mirror of https://github.com/torvalds/linux.git synced 2026-11-07 06:17:02 +01:00
Code Issues Actions Packages Projects Releases Wiki Activity Graph

mm: kmemleak: scan the struct page array in MAX_SCAN_SIZE batches

Browse Source 
scan_zone_pages() scans the struct page array one page per scan_block()
call:

	if (scan_block(page, page + 1, NULL))

scan_block() takes kmemleak_lock with interrupts disabled for the
duration of the call, so this acquires the lock once per online PFN to
scan a single struct page.

Gather runs of adjacent eligible struct pages and pass each run to
scan_block() in one call, capped at MAX_SCAN_SIZE.

The longest kmemleak_lock hold duration is now MAX_SCAN_SIZE worth of
bytes.  The same bound scan_large_block() already applies to the data
sections and the per-CPU areas.

This can make the scan faster. On an arm64 VM with 24 GiB and ~17 GiB
in use, the struct page phase goes from 4390k scan_block() calls down
to 69k for the same 4390k pages scanned, and the whole scan about 20%
faster (on debug kernel). The win is in the per-acquisition cost, so on
a kernel built without the lock debugging options the scan time is
unchanged. I don't think it will be a problem doing more on
scan_block(), given we have the scan_should_stop() protection.

Coverage is unchanged: a temporary assertion comparing the number of
eligible pages against the number actually passed to scan_block()
matched on every zone of every scan, including while memory was being
freed underneath the scan.

Link: https://lore.kernel.org/20260921-b4-kmemleak-page-scan-v1-2-fb97d4801b3a@debian.org
Signed-off-by: Breno Leitao <leitao@debian.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Reviewed-by: Catalin Marinas <catalin.marinas@arm.com>
This commit is contained in:
Breno Leitao 2026-09-21 05:30:27 -07:00 committed by Andrew Morton
parent 1b4bbae111
commit 213753c71e
1 changed files with 21 additions and 8 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

29
mm/kmemleak.c
View File

@ -1862,8 +1862,11 @@ static void dedup_flush(struct xarray *dedup)
*/
static int scan_zone_pages(struct zone *zone)
{
const unsigned int max_batch = MAX_SCAN_SIZE / sizeof(struct page);
unsigned long start_pfn = zone->zone_start_pfn;
unsigned long end_pfn = zone_end_pfn(zone);
struct page *first = NULL, *last = NULL;
unsigned int batch = 0;
unsigned long pfn;
for (pfn = start_pfn; pfn < end_pfn; pfn++) {
@ -1872,19 +1875,29 @@ static int scan_zone_pages(struct zone *zone)
if (!(pfn & 63))
cond_resched_tasks_rcu_qs();
if (!page)
continue;
/* only scan in-use pages belonging to this zone */
if (page && (page_zone(page) != zone ||
page_count(page) == 0))
page = NULL;
/* only scan pages belonging to this zone */
if (page_zone(page) != zone)
if (page && first && page == last + 1 &&
batch < max_batch) {
last = page;
batch++;
continue;
/* only scan if page is in use */
if (page_count(page) == 0)
continue;
if (scan_block(page, page + 1, NULL))
}
if (first && scan_block(first, last + 1, NULL))
return 1;
first = page;
last = page;
batch = page ? 1 : 0;
}
if (first && scan_block(first, last + 1, NULL))
return 1;
return 0;
}