github-mirror/linux
1
0
Fork 0
mirror of https://github.com/torvalds/linux.git synced 2026-11-10 17:11:02 +01:00
Code Issues Actions Packages Projects Releases Wiki Activity Graph

x86/string: extend memcpy_flushcache() fixed-size fastpaths

Browse Source 
The x86 memcpy_nontemporal() helper maps to memcpy_flushcache(), and the
ZONE_DEVICE template-copy path uses it to copy one struct page at a time.

The relevant copy size is sizeof(struct page).  On x86_64, the base struct
page layout is 64 bytes.  Adding either the KMSAN metadata pointers or an
out-of-flags last_cpupid field can make it 80 bytes after alignment, and
enabling both can make it 96 bytes.

memcpy_flushcache() currently only has inline fixed-size cases for 4, 8,
and 16 bytes.  As a result, these constant-sized struct page copies fall
through to __memcpy_flushcache() even though the compiler knows the copy
size at the call site.

Add fixed-size MOVNTI cases up to 96 bytes so the ZONE_DEVICE
template-copy path can keep these struct page copies in the inline
memcpy_flushcache() path.

This matters for ZONE_DEVICE memmap initialization because the copy
happens once per initialized struct page.  For a 100 GB fsdax namespace
with map=dev, this is about 25 million struct page copies during nd_pmem
binding or rebinding.

Tested in a VM with a 100 GB fsdax namespace device configured with
map=dev and a 100 GB devdax namespace (align=2097152) on Intel Ice Lake
server.

Test procedure:
Rebind the nd_pmem and dax_pmem drivers 30 times and collect the memmap
initialization time from the pr_debug() output of
memmap_init_zone_device().

With memcpy_nontemporal() used by the ZONE_DEVICE template-copy path:
  Average of rebinds for nd_pmem driver: 150.40 ms
  Average of rebinds for dax_pmem driver: 161.83 ms

With this x86 fixed-size fastpath patch applied:
  Average of rebinds for nd_pmem driver: 71.93 ms
  Average of rebinds for dax_pmem driver: 87.37 ms

This further reduces the average memmap initialization time measured
during rebind by about 52.2% for nd_pmem and 46.0% for dax_pmem.

Link: https://lore.kernel.org/20260831111638.76012-8-lizhe.67@bytedance.com
Signed-off-by: Li Zhe <lizhe.67@bytedance.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Suggested-by: Borislav Petkov <bp@alien8.de>
Acked-by: Borislav Petkov (AMD) <bp@alien8.de>
Acked-by: Dave Hansen <dave.hansen@linux.intel.com>
Cc: Alistair Popple <apopple@nvidia.com>
Cc: Arnd Bergmann <arnd@arndb.de>
Cc: Balbir Singh <balbirs@nvidia.com>
Cc: David Hildenbrand (Arm) <david@kernel.org>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Kees Cook <kees@kernel.org>
Cc: Mike Rapoport (Microsoft) <rppt@kernel.org>
Cc: Muchun Song <muchun.song@linux.dev>
This commit is contained in:
Li Zhe 2026-08-31 19:16:38 +08:00 committed by Andrew Morton
parent e52dcd69fb
commit 11bb36dbdf
1 changed files with 56 additions and 15 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

71
arch/x86/include/asm/string_64.h
View File

@ -82,23 +82,64 @@ int strcmp(const char *cs, const char *ct);
#ifdef CONFIG_ARCH_HAS_UACCESS_FLUSHCACHE
#define __HAVE_ARCH_MEMCPY_FLUSHCACHE 1
void __memcpy_flushcache(void *dst, const void *src, size_t cnt);
static __always_inline void memcpy_flushcache(void *dst, const void *src, size_t cnt)
static __always_inline void movnti_4(void *dst, const void *src)
{
if (__builtin_constant_p(cnt)) {
switch (cnt) {
case 4:
asm ("movntil %1, %0" : "=m"(*(u32 *)dst) : "r"(*(u32 *)src));
return;
case 8:
asm ("movntiq %1, %0" : "=m"(*(u64 *)dst) : "r"(*(u64 *)src));
return;
case 16:
asm ("movntiq %1, %0" : "=m"(*(u64 *)dst) : "r"(*(u64 *)src));
asm ("movntiq %1, %0" : "=m"(*(u64 *)(dst + 8)) : "r"(*(u64 *)(src + 8)));
return;
}
asm volatile("movntil %1, %0"
: "=m"(*(u32 *)dst)
: "r"(*(const u32 *)src)
: "memory");
}
static __always_inline void movnti_8(void *dst, const void *src)
{
asm volatile("movntiq %1, %0"
: "=m"(*(u64 *)dst)
: "r"(*(const u64 *)src)
: "memory");
}
static __always_inline void movnti_16(void *dst, const void *src)
{
movnti_8(dst, src);
movnti_8(dst + 8, src + 8);
}
static __always_inline void movnti_32(void *dst, const void *src)
{
movnti_16(dst, src);
movnti_16(dst + 16, src + 16);
}
static __always_inline void movnti_64(void *dst, const void *src)
{
movnti_32(dst, src);
movnti_32(dst + 32, src + 32);
}
static __always_inline void memcpy_flushcache(void *dst, const void *src,
size_t cnt)
{
if (!__builtin_constant_p(cnt))
return __memcpy_flushcache(dst, src, cnt);
/*
* The relevant fixed-size copies here are the x86_64 struct page sizes:
* 64, 80, and 96 bytes. Keep 32-byte and 48-byte copies inline as well
* instead of sending those nearby fixed-size cases back to
* __memcpy_flushcache().
*/
switch (cnt) {
case 4: movnti_4(dst, src); break;
case 8: movnti_8(dst, src); break;
case 16: movnti_16(dst, src); break;
case 32: movnti_32(dst, src); break;
case 48: movnti_32(dst, src); movnti_16(dst + 32, src + 32); break;
case 64: movnti_64(dst, src); break;
case 80: movnti_64(dst, src); movnti_16(dst + 64, src + 64); break;
case 96: movnti_64(dst, src); movnti_32(dst + 64, src + 64); break;
default: __memcpy_flushcache(dst, src, cnt); break;
}
__memcpy_flushcache(dst, src, cnt);
}
#define memcpy_nontemporal memcpy_nontemporal