github-mirror/linux
1
0
Fork 0
mirror of https://github.com/torvalds/linux.git synced 2026-05-31 02:24:24 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity Graph

x86/clear_page: introduce clear_pages()

Browse Source 
Performance when clearing with string instructions (x86-64-stosq and
similar) can vary significantly based on the chunk-size used.

  $ perf bench mem memset -k 4KB -s 4GB -f x86-64-stosq
  # Running 'mem/memset' benchmark:
  # function 'x86-64-stosq' (movsq-based memset() in arch/x86/lib/memset_64.S)
  # Copying 4GB bytes ...

      13.748208 GB/sec

  $ perf bench mem memset -k 2MB -s 4GB -f x86-64-stosq
  # Running 'mem/memset' benchmark:
  # function 'x86-64-stosq' (movsq-based memset() in
  # arch/x86/lib/memset_64.S)
  # Copying 4GB bytes ...

      15.067900 GB/sec

  $ perf bench mem memset -k 1GB -s 4GB -f x86-64-stosq
  # Running 'mem/memset' benchmark:
  # function 'x86-64-stosq' (movsq-based memset() in arch/x86/lib/memset_64.S)
  # Copying 4GB bytes ...

      38.104311 GB/sec

(Both on AMD Milan.)

With a change in chunk-size from 4KB to 1GB, we see the performance go
from 13.7 GB/sec to 38.1 GB/sec.  For the chunk-size of 2MB the change
isn't quite as drastic but it is worth adding a clear_page() variant that
can handle contiguous page-extents.

Link: https://lkml.kernel.org/r/20260107072009.1615991-6-ankur.a.arora@oracle.com
Signed-off-by: Ankur Arora <ankur.a.arora@oracle.com>
Tested-by: Raghavendra K T <raghavendra.kt@amd.com>
Reviewed-by: David Hildenbrand (Red Hat) <david@kernel.org>
Cc: Andy Lutomirski <luto@kernel.org>
Cc: "Borislav Petkov (AMD)" <bp@alien8.de>
Cc: Boris Ostrovsky <boris.ostrovsky@oracle.com>
Cc: David Hildenbrand <david@redhat.com>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Konrad Rzessutek Wilk <konrad.wilk@oracle.com>
Cc: Lance Yang <ioworker0@gmail.com>
Cc: "Liam R. Howlett" <Liam.Howlett@oracle.com>
Cc: Li Zhe <lizhe.67@bytedance.com>
Cc: Lorenzo Stoakes <lorenzo.stoakes@oracle.com>
Cc: Mateusz Guzik <mjguzik@gmail.com>
Cc: Matthew Wilcox (Oracle) <willy@infradead.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Vlastimil Babka <vbabka@suse.cz>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
This commit is contained in:
Ankur Arora 2026-01-06 23:20:06 -08:00 committed by Andrew Morton
parent 54a6b89a3d
commit cb431accb3
1 changed files with 12 additions and 5 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

17
arch/x86/include/asm/page_64.h
View File

@ -52,8 +52,9 @@ void __clear_pages_unrolled(void *page);
KCFI_REFERENCE(__clear_pages_unrolled);
/**
* clear_page() - clear a page using a kernel virtual address.
* @addr: address of kernel page
* clear_pages() - clear a page range using a kernel virtual address.
* @addr: start address of kernel page range
* @npages: number of pages
*
* Switch between three implementations of page clearing based on CPU
* capabilities:
@ -81,11 +82,11 @@ KCFI_REFERENCE(__clear_pages_unrolled);
*
* Does absolutely no exception handling.
*/
static inline void clear_page(void *addr)
static inline void clear_pages(void *addr, unsigned int npages)
{
u64 len = PAGE_SIZE;
u64 len = npages * PAGE_SIZE;
/*
* Clean up KMSAN metadata for the page being cleared. The assembly call
* Clean up KMSAN metadata for the pages being cleared. The assembly call
* below clobbers @addr, so perform unpoisoning before it.
*/
kmsan_unpoison_memory(addr, len);
@ -106,6 +107,12 @@ static inline void clear_page(void *addr)
: "a" (0)
: "cc", "memory");
}
#define clear_pages clear_pages
static inline void clear_page(void *addr)
{
clear_pages(addr, 1);
}
void copy_page(void *to, void *from);
KCFI_REFERENCE(copy_page);