github-mirror/linux
1
0
Fork 0
mirror of https://github.com/torvalds/linux.git synced 2026-11-07 06:17:02 +01:00
Code Issues Actions Packages Projects Releases Wiki Activity Graph

mm: filemap: move lruvec accounting outside the xarray lock

Browse Source 
__filemap_add_folio() inserts a folio and updates mapping->nrpages while
holding mapping->i_pages.xa_lock with interrupts disabled.  The XArray
insertion and nrpages update require the lock, but the lruvec statistic
updates do not.  With CONFIG_MEMCG, those calls also update per-CPU memcg
and lruvec counters and notify cgroup rstat, extending the critical
section.

Move the lruvec accounting after a successful XArray insertion and after
xas_unlock_irq().  The page-cache references pin the folio, while the
folio lock keeps folio->mapping stable and prevents removal until
accounting is complete.  This moves one lruvec update for ordinary folios
and a second for PMD-mappable folios out of the serialized section.

In a 30-second system-wide perf lock contention -ab capture on a
production host, the hottest caller-stack record attributed to
__filemap_add_folio() had 20,867 contentions and 557.930 ms total wait. 
That was 14% of the 3.998 seconds of aggregate lock wait in the capture. 
Moving lruvec accuting outside of critical section should help optimize
it.

Link: https://lore.kernel.org/20260916125122.2696271-1-usama.arif@linux.dev
Signed-off-by: Usama Arif <usama.arif@linux.dev>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Reviewed-by: Shakeel Butt <shakeel.butt@linux.dev>
Acked-by: Muchun Song <muchun.song@linux.dev>
Reviewed-by: Vishal Moola (Fractile) <vishal.moola@gmail.com>
Reviewed-by: Jan Kara <jack@suse.cz>
Cc: David Hildenbrand <david@kernel.org>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Matthew Wilcox (Oracle) <willy@infradead.org>
Cc: Michal Hocko <mhocko@kernel.org>
Cc: Rik van Riel <riel@surriel.com>
Cc: Roman Gushchin <roman.gushchin@linux.dev>
This commit is contained in:
Usama Arif 2026-09-16 05:51:22 -07:00 committed by Andrew Morton
parent de6be06f4a
commit 103142868a
1 changed files with 7 additions and 8 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

15
mm/filemap.c
View File

@ -918,14 +918,6 @@ noinline int __filemap_add_folio(struct address_space *mapping,
mapping->nrpages += nr;
/* hugetlb pages do not participate in page cache accounting */
if (!huge) {
lruvec_stat_mod_folio(folio, NR_FILE_PAGES, nr);
if (folio_test_pmd_mappable(folio))
lruvec_stat_mod_folio(folio,
NR_FILE_THPS, nr);
}
unlock:
xas_unlock_irq(&xas);
@ -942,6 +934,13 @@ noinline int __filemap_add_folio(struct address_space *mapping,
if (xas_error(&xas))
goto error;
/* hugetlb pages do not participate in page cache accounting */
if (!huge) {
lruvec_stat_mod_folio(folio, NR_FILE_PAGES, nr);
if (folio_test_pmd_mappable(folio))
lruvec_stat_mod_folio(folio, NR_FILE_THPS, nr);
}
trace_mm_filemap_add_to_page_cache(folio);
return 0;
error: