github-mirror/linux
1
0
Fork 0
mirror of https://github.com/torvalds/linux.git synced 2026-07-27 01:32:21 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity Graph

mm.git review status for linus..mm-stable

Browse Source 
Everything:
 
 Total patches:       118
 Reviews/patch:       1.57
 Reviewed rate:       62%
 
 Excluding DAMON:
 
 Total patches:       117
 Reviews/patch:       1.58
 Reviewed rate:       63%
 
 Excluding DAMON and selftests:
 
 Total patches:       42
 Reviews/patch:       3.14
 Reviewed rate:       92%
 
 Patch series in this merge:
 
 - The 14 patch series "khugepaged: add mTHP collapse support" from Nico
   Pache provides khugepaged with the capability to collapse anonymous
   memory regions to mTHPs.
 
 - The 14 patch series "Remove CONFIG_READ_ONLY_THP_FOR_FS and enable
   file THP for writable files" from Zi Yan ends up removing the
   READ_ONLY_THP_FOR_FS check in file_thp_enabled(), so that khugepaged and
   MADV_COLLAPSE can run on FSes with PMD THP pagecache support even
   without READ_ONLY_THP_FOR_FS enabled.
 
 - The 55 patch series "make MM selftests more CI friendly" from Mike
   Rapoport provides general fixes and cleanups to the MM selftests.  It
   also moves more MM selftests uder the kselftest framework, making them
   more amenable to ongoing CI testing.
 
 - The 13 patch series "selftests/mm: fix failures and robustness
   improvements" from Sayali Patil fixes several issues in MM selftests
   which were revealed by powerpc 64k pagesize.
 
 - The 3 patch series "selftests/mm: assorted fixes for hmm-tests" from
   Sayali Patil fixes more 64k pagesize issues in the MM selftests.
 -----BEGIN PGP SIGNATURE-----
 
 iHUEABYKAB0WIQTTMBEPP41GrTpTJgfdBJ7gKXxAjgUCajqsgwAKCRDdBJ7gKXxA
 jlNoAP0bM125zRNpqy13hUYfofQQg8lC9k7cfEHqVtvD/dpaKgD/Xbjd8AUGbwjs
 qoj3nNmUOfGFPubPZsU6YSFyVsYfzwM=
 =T44n
 -----END PGP SIGNATURE-----

Merge tag 'mm-stable-2026-06-23-08-55' of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm

Pull more MM updates from Andrew Morton:

 - "khugepaged: add mTHP collapse support" (Nico Pache)

   Provide khugepaged with the capability to collapse anonymous memory
   regions to mTHPs

 - "Remove CONFIG_READ_ONLY_THP_FOR_FS and enable file THP for writable
   files" (Zi Yan)

   Remove the READ_ONLY_THP_FOR_FS check in file_thp_enabled(), so that
   khugepaged and MADV_COLLAPSE can run on filesystems with PMD THP
   pagecache support even without READ_ONLY_THP_FOR_FS enabled

 - "make MM selftests more CI friendly" (Mike Rapoport)

   General fixes and cleanups to the MM selftests. Also move more MM
   selftests under the kselftest framework, making them more amenable to
   ongoing CI testing

 - "selftests/mm: fix failures and robustness improvements" and
   "selftests/mm: assorted fixes for hmm-tests" (Sayali Patil)

   Fix several issues in MM selftests which were revealed by powerpc 64k
   pagesize

* tag 'mm-stable-2026-06-23-08-55' of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm: (118 commits)
  Revert "mm: limit filemap_fault readahead to VMA boundaries"
  mm/vmscan: pass NULL to trace vmscan node reclaim
  mm: use mapping_mapped to simplify the code
  selftests/mm: fix exclusive_cow test fork() handling
  selftests/mm: remove hardcoded THP sizing assumptions in hmm tests
  selftests/mm: allow PUD-level entries in compound testcase of hmm tests
  mm/gup_test: reject wrapped user ranges
  mm/page_frag: reject invalid CPUs in page_frag_test
  mm/damon/core: always put unsuccessfully committed target pids
  mm: page_isolation: avoid unsafe folio reads while scanning compound pages
  mm/shrinker: do not hold RCU lock in shrinker_debugfs_count_show()
  selftests: mm: fix and speedup "droppable" test
  mm: merge writeout into pageout
  MAINTAINERS: add Hao Ge as reviewer for codetag and alloc_tag
  selftests/mm: clarify alternate unmapping in compaction_test
  selftests/mm: move hwpoison setup into run_test() and silence modprobe output for memory-failure category
  selftests/mm: skip uffd-stress test when nr_pages_per_cpu is zero
  selftests/mm: skip uffd-wp-mremap if UFFD write-protect is unsupported
  selftests/mm: ensure destination is hugetlb-backed in hugetlb-mremap
  selftest/mm: register existing mapping with userfaultfd in hugetlb-mremap
  ...
This commit is contained in:
Linus Torvalds 2026-06-23 12:03:44 -07:00
commit a1a8bab741
77 changed files with 2840 additions and 2314 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

15
Documentation/admin-guide/mm/hugetlbpage.rst
View File

@ -455,7 +455,7 @@ used to change the file attributes on hugetlbfs.
Also, it is important to note that no such mount command is required if
applications are going to use only shmat/shmget system calls or mmap with
MAP_HUGETLB. For an example of how to use mmap with MAP_HUGETLB see
:ref:`map_hugetlb <map_hugetlb>` below.
:ref:`examples <examples>` below.
Users who wish to use hugetlb memory via shared memory segment should be
members of a supplementary group and system admin needs to configure that gid
@ -473,16 +473,13 @@ a hugetlb page and the length is smaller than the hugepage size.
Examples
========
.. _map_hugetlb:
.. _examples:
``map_hugetlb``
see tools/testing/selftests/mm/map_hugetlb.c
``hugetlb-shm``
see tools/testing/selftests/mm/hugetlb-shm.c
``hugepage-shm``
see tools/testing/selftests/mm/hugepage-shm.c
``hugepage-mmap``
see tools/testing/selftests/mm/hugepage-mmap.c
``hugetlb-mmap``
see tools/testing/selftests/mm/hugetlb-mmap.c
The `libhugetlbfs`_ library provides a wide range of userspace tools
to help with huge page usability, environment setup, and control.

71
Documentation/admin-guide/mm/transhuge.rst
View File

@ -63,7 +63,8 @@ often.
THP can be enabled system wide or restricted to certain tasks or even
memory ranges inside task's address space. Unless THP is completely
disabled, there is ``khugepaged`` daemon that scans memory and
collapses sequences of basic pages into PMD-sized huge pages.
collapses sequences of basic pages into huge pages of either PMD size
or mTHP sizes, if the system is configured to do so.
The THP behaviour is controlled via :ref:`sysfs <thp_sysfs>`
interface and using madvise(2) and prctl(2) system calls.
@ -219,10 +220,10 @@ this behaviour by writing 0 to shrink_underused, and enable it by writing
echo 0 > /sys/kernel/mm/transparent_hugepage/shrink_underused
echo 1 > /sys/kernel/mm/transparent_hugepage/shrink_underused
khugepaged will be automatically started when PMD-sized THP is enabled
khugepaged will be automatically started when any THP size is enabled
(either of the per-size anon control or the top-level control are set
to "always" or "madvise"), and it'll be automatically shutdown when
PMD-sized THP is disabled (when both the per-size anon control and the
all THP sizes are disabled (when both the per-size anon control and the
top-level control are "never")
process THP controls
@ -265,8 +266,8 @@ Khugepaged controls
-------------------
.. note::
khugepaged currently only searches for opportunities to collapse to
PMD-sized THP and no attempt is made to collapse to other THP
khugepaged currently only searches for opportunities to collapse file/shmem
to PMD-sized THP. Only anonymous memory will attempt to collapse to other THP
sizes.
khugepaged runs usually at low frequency so while one may not want to
@ -296,11 +297,11 @@ allocation failure to throttle the next allocation attempt::
The khugepaged progress can be seen in the number of pages collapsed (note
that this counter may not be an exact count of the number of pages
collapsed, since "collapsed" could mean multiple things: (1) A PTE mapping
being replaced by a PMD mapping, or (2) All 4K physical pages replaced by
one 2M hugepage. Each may happen independently, or together, depending on
the type of memory and the failures that occur. As such, this value should
be interpreted roughly as a sign of progress, and counters in /proc/vmstat
consulted for more accurate accounting)::
being replaced by a PMD mapping, or (2) physical pages replaced by one
hugepage of various sizes (PMD-sized or mTHP). Each may happen independently,
or together, depending on the type of memory and the failures that occur.
As such, this value should be interpreted roughly as a sign of progress,
and counters in /proc/vmstat consulted for more accurate accounting)::
/sys/kernel/mm/transparent_hugepage/khugepaged/pages_collapsed
@ -308,16 +309,21 @@ for each pass::
/sys/kernel/mm/transparent_hugepage/khugepaged/full_scans
``max_ptes_none`` specifies how many extra small pages (that are
not already mapped) can be allocated when collapsing a group
of small pages into one large page::
``max_ptes_none`` specifies how many empty (none/zero) pages are allowed
when collapsing a group of small pages into one large page::
/sys/kernel/mm/transparent_hugepage/khugepaged/max_ptes_none
A higher value leads to use additional memory for programs.
A lower value leads to gain less thp performance. Value of
max_ptes_none can waste cpu time very little, you can
ignore it.
For PMD-sized THP collapse, this directly limits the number of empty pages
allowed in the 2MB region.
For mTHP collapse, only 0 or (HPAGE_PMD_NR - 1) are supported. At
HPAGE_PMD_NR - 1, we collapse to the highest possible order. Any intermediate
value will emit a warning and mTHP collapse will default to max_ptes_none=0.
A higher value allows more empty pages, potentially leading to more memory
usage but better THP performance. A lower value is more conservative and
may result in fewer THP collapses.
``max_ptes_swap`` specifies how many pages can be brought in from
swap when collapsing a group of pages into a transparent huge page::
@ -337,6 +343,15 @@ that THP is shared. Exceeding the number would block the collapse::
A higher value may increase memory footprint for some workloads.
.. note::
For mTHP collapse, khugepaged does not support collapsing regions that
contain shared or swapped out pages, as this could lead to continuous
promotion to higher orders. The collapse will fail if any shared or
swapped PTEs are encountered during the scan.
Currently, madvise_collapse only supports collapsing to PMD-sized THPs
and does not attempt mTHP collapses.
Boot parameters
===============
@ -639,6 +654,14 @@ anon_fault_fallback_charge
instead falls back to using huge pages with lower orders or
small pages even though the allocation was successful.
collapse_alloc
is incremented every time a huge page is successfully allocated for a
khugepaged collapse.
collapse_alloc_failed
is incremented every time a huge page allocation fails during a
khugepaged collapse.
zswpout
is incremented every time a huge page is swapped out to zswap in one
piece without splitting.
@ -706,6 +729,20 @@ nr_anon_partially_mapped
an anonymous THP as "partially mapped" and count it here, even though it
is not actually partially mapped anymore.
collapse_exceed_none_pte
The number of collapse attempts that failed due to exceeding the
max_ptes_none threshold.
collapse_exceed_swap_pte
The number of collapse attempts that failed due to exceeding the
max_ptes_swap threshold. For non-PMD orders this occurs if a mTHP range
contains at least one swap PTE.
collapse_exceed_shared_pte
The number of collapse attempts that failed due to exceeding the
max_ptes_shared threshold. For non-PMD orders this occurs if a mTHP range
contains at least one shared PTE.
As the system ages, allocating huge pages may be expensive as the
system uses memory compaction to copy data around memory to free a
huge page for use. There are some counters in ``/proc/vmstat`` to help

2
MAINTAINERS
View File

@ -6421,6 +6421,7 @@ F: Documentation/process/code-of-conduct.rst
CODE TAGGING
M: Suren Baghdasaryan <surenb@google.com>
M: Kent Overstreet <kent.overstreet@linux.dev>
R: Hao Ge <hao.ge@linux.dev>
S: Maintained
F: include/asm-generic/codetag.lds.h
F: include/linux/codetag.h
@ -16863,6 +16864,7 @@ F: tools/testing/memblock/
MEMORY ALLOCATION PROFILING
M: Suren Baghdasaryan <surenb@google.com>
M: Kent Overstreet <kent.overstreet@linux.dev>
R: Hao Ge <hao.ge@linux.dev>
L: linux-mm@kvack.org
S: Maintained
F: Documentation/mm/allocation-profiling.rst

4
fs/hugetlbfs/inode.c
View File

@ -614,7 +614,7 @@ static void hugetlb_vmtruncate(struct inode *inode, loff_t offset)
i_size_write(inode, offset);
i_mmap_lock_write(mapping);
if (!RB_EMPTY_ROOT(&mapping->i_mmap.rb_root))
if (mapping_mapped(mapping))
hugetlb_vmdelete_list(&mapping->i_mmap, pgoff, 0,
ZAP_FLAG_DROP_MARKER);
i_mmap_unlock_write(mapping);
@ -675,7 +675,7 @@ static long hugetlbfs_punch_hole(struct inode *inode, loff_t offset, loff_t len)
/* Unmap users of full pages in the hole. */
if (hole_end > hole_start) {
if (!RB_EMPTY_ROOT(&mapping->i_mmap.rb_root))
if (mapping_mapped(mapping))
hugetlb_vmdelete_list(&mapping->i_mmap,
hole_start >> PAGE_SHIFT,
hole_end >> PAGE_SHIFT, 0);

3
fs/inode.c
View File

@ -275,9 +275,6 @@ int inode_init_always_gfp(struct super_block *sb, struct inode *inode, gfp_t gfp
mapping->flags = 0;
mapping->wb_err = 0;
atomic_set(&mapping->i_mmap_writable, 0);
#ifdef CONFIG_READ_ONLY_THP_FOR_FS
atomic_set(&mapping->nr_thps, 0);
#endif
mapping_set_gfp_mask(mapping, GFP_HIGHUSER_MOVABLE);
mapping->writeback_index = 0;
init_rwsem(&mapping->invalidate_lock);

27
fs/open.c
View File

@ -968,33 +968,6 @@ static int do_dentry_open(struct file *f,
if ((f->f_flags & O_DIRECT) && !(f->f_mode & FMODE_CAN_ODIRECT))
return -EINVAL;
/*
* XXX: Huge page cache doesn't support writing yet. Drop all page
* cache for this file before processing writes.
*/
if (f->f_mode & FMODE_WRITE) {
/*
* Depends on full fence from get_write_access() to synchronize
* against collapse_file() regarding i_writecount and nr_thps
* updates. Ensures subsequent insertion of THPs into the page
* cache will fail.
*/
if (filemap_nr_thps(inode->i_mapping)) {
struct address_space *mapping = inode->i_mapping;
filemap_invalidate_lock(inode->i_mapping);
/*
* unmap_mapping_range just need to be called once
* here, because the private pages is not need to be
* unmapped mapping (e.g. data segment of dynamic
* shared libraries here).
*/
unmap_mapping_range(mapping, 0, 0, 0);
truncate_inode_pages(mapping, 0);
filemap_invalidate_unlock(inode->i_mapping);
}
}
return 0;
cleanup_all:

4
include/linux/alloc_tag.h
View File

@ -163,11 +163,11 @@ static inline void alloc_tag_sub_check(union codetag_ref *ref)
{
WARN_ONCE(ref && !ref->ct, "alloc_tag was not set\n");
}
void alloc_tag_add_early_pfn(unsigned long pfn);
void alloc_tag_add_early_pfn(unsigned long pfn, gfp_t gfp_flags);
#else
static inline void alloc_tag_add_check(union codetag_ref *ref, struct alloc_tag *tag) {}
static inline void alloc_tag_sub_check(union codetag_ref *ref) {}
static inline void alloc_tag_add_early_pfn(unsigned long pfn) {}
static inline void alloc_tag_add_early_pfn(unsigned long pfn, gfp_t gfp_flags) {}
#endif
/* Caller should verify both ref and tag to be valid */

3
include/linux/codetag.h
View File

@ -74,8 +74,9 @@ struct codetag_iterator {
.flags = 0, \
}
void codetag_lock_module_list(struct codetag_type *cttype, bool lock);
void codetag_lock_module_list(struct codetag_type *cttype);
bool codetag_trylock_module_list(struct codetag_type *cttype);
void codetag_unlock_module_list(struct codetag_type *cttype);
struct codetag_iterator codetag_get_ct_iter(struct codetag_type *cttype);
struct codetag *codetag_next_ct(struct codetag_iterator *iter);

5
include/linux/fs.h
View File

@ -460,7 +460,6 @@ struct mapping_metadata_bhs {
* memory mappings.
* @gfp_mask: Memory allocation flags to use for allocating pages.
* @i_mmap_writable: Number of VM_SHARED, VM_MAYWRITE mappings.
* @nr_thps: Number of THPs in the pagecache (non-shmem only).
* @i_mmap: Tree of private and shared mappings.
* @i_mmap_rwsem: Protects @i_mmap and @i_mmap_writable.
* @nrpages: Number of page entries, protected by the i_pages lock.
@ -476,10 +475,6 @@ struct address_space {
struct rw_semaphore invalidate_lock;
gfp_t gfp_mask;
atomic_t i_mmap_writable;
#ifdef CONFIG_READ_ONLY_THP_FOR_FS
/* number of thp, only for non-shmem files */
atomic_t nr_thps;
#endif
struct rb_root_cached i_mmap;
unsigned long nrpages;
pgoff_t writeback_index;

30
include/linux/huge_mm.h
View File

@ -128,6 +128,8 @@ enum mthp_stat_item {
MTHP_STAT_ANON_FAULT_ALLOC,
MTHP_STAT_ANON_FAULT_FALLBACK,
MTHP_STAT_ANON_FAULT_FALLBACK_CHARGE,
MTHP_STAT_COLLAPSE_ALLOC,
MTHP_STAT_COLLAPSE_ALLOC_FAILED,
MTHP_STAT_ZSWPOUT,
MTHP_STAT_SWPIN,
MTHP_STAT_SWPIN_FALLBACK,
@ -142,6 +144,9 @@ enum mthp_stat_item {
MTHP_STAT_SPLIT_DEFERRED,
MTHP_STAT_NR_ANON,
MTHP_STAT_NR_ANON_PARTIALLY_MAPPED,
MTHP_STAT_COLLAPSE_EXCEED_SWAP,
MTHP_STAT_COLLAPSE_EXCEED_NONE,
MTHP_STAT_COLLAPSE_EXCEED_SHARED,
__MTHP_STAT_COUNT
};
@ -414,27 +419,6 @@ static inline int split_huge_page_to_order(struct page *page, unsigned int new_o
return split_huge_page_to_list_to_order(page, NULL, new_order);
}
/**
* try_folio_split_to_order() - try to split a @folio at @page to @new_order
* using non uniform split.
* @folio: folio to be split
* @page: split to @new_order at the given page
* @new_order: the target split order
*
* Try to split a @folio at @page using non uniform split to @new_order, if
* non uniform split is not supported, fall back to uniform split. After-split
* folios are put back to LRU list. Use min_order_for_split() to get the lower
* bound of @new_order.
*
* Return: 0 - split is successful, otherwise split failed.
*/
static inline int try_folio_split_to_order(struct folio *folio,
struct page *page, unsigned int new_order)
{
if (folio_check_splittable(folio, new_order, SPLIT_TYPE_NON_UNIFORM))
return split_huge_page_to_order(&folio->page, new_order);
return folio_split(folio, new_order, page, NULL);
}
static inline int split_huge_page(struct page *page)
{
return split_huge_page_to_list_to_order(page, NULL, 0);
@ -672,8 +656,8 @@ static inline int split_folio_to_list(struct folio *folio, struct list_head *lis
return -EINVAL;
}
static inline int try_folio_split_to_order(struct folio *folio,
struct page *page, unsigned int new_order)
static inline int folio_split(struct folio *folio, unsigned int new_order,
struct page *page, struct list_head *list)
{
VM_WARN_ON_ONCE_FOLIO(1, folio);
return -EINVAL;

58
include/linux/pagemap.h
View File

@ -513,41 +513,39 @@ static inline bool mapping_large_folio_support(const struct address_space *mappi
return mapping_max_folio_order(mapping) > 0;
}
/**
* mapping_pmd_folio_support() - Check if a mapping supports PMD-sized folio
* @mapping: The address_space
*
* While some mappings support large folios, they might not support PMD-sized
* folios. This function checks whether a mapping supports PMD-sized folios.
* For example, khugepaged needs this information before attempting to
* collapsing THPs.
*
* Return: True if PMD-sized folios are supported, otherwise false.
*/
#ifdef CONFIG_TRANSPARENT_HUGEPAGE
static inline bool mapping_pmd_folio_support(const struct address_space *mapping)
{
/* AS_FOLIO_ORDER is only reasonable for pagecache folios */
VM_WARN_ON_ONCE((unsigned long)mapping & FOLIO_MAPPING_ANON);
return mapping_min_folio_order(mapping) <= PMD_ORDER &&
mapping_max_folio_order(mapping) >= PMD_ORDER;
}
#else
static inline bool mapping_pmd_folio_support(const struct address_space *mapping)
{
return false;
}
#endif
/* Return the maximum folio size for this pagecache mapping, in bytes. */
static inline size_t mapping_max_folio_size(const struct address_space *mapping)
{
return PAGE_SIZE << mapping_max_folio_order(mapping);
}
static inline int filemap_nr_thps(const struct address_space *mapping)
{
#ifdef CONFIG_READ_ONLY_THP_FOR_FS
return atomic_read(&mapping->nr_thps);
#else
return 0;
#endif
}
static inline void filemap_nr_thps_inc(struct address_space *mapping)
{
#ifdef CONFIG_READ_ONLY_THP_FOR_FS
if (!mapping_large_folio_support(mapping))
atomic_inc(&mapping->nr_thps);
#else
WARN_ON_ONCE(mapping_large_folio_support(mapping) == 0);
#endif
}
static inline void filemap_nr_thps_dec(struct address_space *mapping)
{
#ifdef CONFIG_READ_ONLY_THP_FOR_FS
if (!mapping_large_folio_support(mapping))
atomic_dec(&mapping->nr_thps);
#else
WARN_ON_ONCE(mapping_large_folio_support(mapping) == 0);
#endif
}
struct address_space *folio_mapping(const struct folio *folio);
/**
@ -1350,7 +1348,6 @@ struct readahead_control {
struct file_ra_state *ra;
/* private: use the readahead_* accessors instead */
pgoff_t _index;
pgoff_t _max_index; /* limit readahead to _max_index, inclusive */
unsigned int _nr_pages;
unsigned int _batch_count;
bool dropbehind;
@ -1364,7 +1361,6 @@ struct readahead_control {
.mapping = m, \
.ra = r, \
._index = i, \
._max_index = ULONG_MAX, \
}
#define VM_READAHEAD_PAGES (SZ_128K / PAGE_SIZE)

2
include/linux/shmem_fs.h
View File

@ -128,7 +128,7 @@ int shmem_writeout(struct folio *folio, struct swap_iocb **plug,
void shmem_truncate_range(struct inode *inode, loff_t start, uoff_t end);
int shmem_unuse(unsigned int type);
#ifdef CONFIG_TRANSPARENT_HUGEPAGE
#if defined(CONFIG_TRANSPARENT_HUGEPAGE) && defined(CONFIG_SHMEM)
unsigned long shmem_allowable_huge_orders(struct inode *inode,
struct vm_area_struct *vma, pgoff_t index,
loff_t write_end, bool shmem_huge_force);

34
include/trace/events/huge_memory.h
View File

@ -89,40 +89,44 @@ TRACE_EVENT(mm_khugepaged_scan_pmd,
TRACE_EVENT(mm_collapse_huge_page,
TP_PROTO(struct mm_struct *mm, int isolated, int status),
TP_PROTO(struct mm_struct *mm, int isolated, int status, unsigned int order),
TP_ARGS(mm, isolated, status),
TP_ARGS(mm, isolated, status, order),
TP_STRUCT__entry(
__field(struct mm_struct *, mm)
__field(int, isolated)
__field(int, status)
__field(unsigned int, order)
),
TP_fast_assign(
__entry->mm = mm;
__entry->isolated = isolated;
__entry->status = status;
__entry->order = order;
),
TP_printk("mm=%p, isolated=%d, status=%s",
TP_printk("mm=%p, isolated=%d, status=%s, order=%u",
__entry->mm,
__entry->isolated,
__print_symbolic(__entry->status, SCAN_STATUS))
__print_symbolic(__entry->status, SCAN_STATUS),
__entry->order)
);
TRACE_EVENT(mm_collapse_huge_page_isolate,
TP_PROTO(struct folio *folio, int none_or_zero,
int referenced, int status),
int referenced, int status, unsigned int order),
TP_ARGS(folio, none_or_zero, referenced, status),
TP_ARGS(folio, none_or_zero, referenced, status, order),
TP_STRUCT__entry(
__field(unsigned long, pfn)
__field(int, none_or_zero)
__field(int, referenced)
__field(int, status)
__field(unsigned int, order)
),
TP_fast_assign(
@ -130,26 +134,30 @@ TRACE_EVENT(mm_collapse_huge_page_isolate,
__entry->none_or_zero = none_or_zero;
__entry->referenced = referenced;
__entry->status = status;
__entry->order = order;
),
TP_printk("scan_pfn=0x%lx, none_or_zero=%d, referenced=%d, status=%s",
TP_printk("scan_pfn=0x%lx, none_or_zero=%d, referenced=%d, status=%s, order=%u",
__entry->pfn,
__entry->none_or_zero,
__entry->referenced,
__print_symbolic(__entry->status, SCAN_STATUS))
__print_symbolic(__entry->status, SCAN_STATUS),
__entry->order)
);
TRACE_EVENT(mm_collapse_huge_page_swapin,
TP_PROTO(struct mm_struct *mm, int swapped_in, int referenced, int ret),
TP_PROTO(struct mm_struct *mm, int swapped_in, int referenced, int ret,
unsigned int order),
TP_ARGS(mm, swapped_in, referenced, ret),
TP_ARGS(mm, swapped_in, referenced, ret, order),
TP_STRUCT__entry(
__field(struct mm_struct *, mm)
__field(int, swapped_in)
__field(int, referenced)
__field(int, ret)
__field(unsigned int, order)
),
TP_fast_assign(
@ -157,13 +165,15 @@ TRACE_EVENT(mm_collapse_huge_page_swapin,
__entry->swapped_in = swapped_in;
__entry->referenced = referenced;
__entry->ret = ret;
__entry->order = order;
),
TP_printk("mm=%p, swapped_in=%d, referenced=%d, ret=%d",
TP_printk("mm=%p, swapped_in=%d, referenced=%d, ret=%d, order=%u",
__entry->mm,
__entry->swapped_in,
__entry->referenced,
__entry->ret)
__entry->ret,
__entry->order)
);
TRACE_EVENT(mm_khugepaged_scan_file,

157
lib/alloc_tag.c
View File

@ -45,6 +45,7 @@ int alloc_tag_ref_offs;
struct allocinfo_private {
struct codetag_iterator iter;
struct codetag_iterator reported_iter;
bool print_header;
};
@ -54,20 +55,24 @@ static void *allocinfo_start(struct seq_file *m, loff_t *pos)
loff_t node = *pos;
priv = (struct allocinfo_private *)m->private;
codetag_lock_module_list(alloc_tag_cttype, true);
codetag_lock_module_list(alloc_tag_cttype);
if (node == 0) {
priv->print_header = true;
priv->iter = codetag_get_ct_iter(alloc_tag_cttype);
codetag_next_ct(&priv->iter);
} else {
priv->iter = priv->reported_iter;
}
codetag_next_ct(&priv->iter);
return priv->iter.ct ? priv : NULL;
}
static void *allocinfo_next(struct seq_file *m, void *arg, loff_t *pos)
{
struct allocinfo_private *priv = (struct allocinfo_private *)arg;
struct codetag *ct = codetag_next_ct(&priv->iter);
struct codetag *ct;
priv->reported_iter = priv->iter;
ct = codetag_next_ct(&priv->iter);
(*pos)++;
if (!ct)
return NULL;
@ -77,7 +82,7 @@ static void *allocinfo_next(struct seq_file *m, void *arg, loff_t *pos)
static void allocinfo_stop(struct seq_file *m, void *arg)
{
codetag_lock_module_list(alloc_tag_cttype, false);
codetag_unlock_module_list(alloc_tag_cttype);
}
static void print_allocinfo_header(struct seq_buf *buf)
@ -136,7 +141,7 @@ size_t alloc_tag_top_users(struct codetag_bytes *tags, size_t count, bool can_sl
return 0;
if (can_sleep)
codetag_lock_module_list(alloc_tag_cttype, true);
codetag_lock_module_list(alloc_tag_cttype);
else if (!codetag_trylock_module_list(alloc_tag_cttype))
return 0;
@ -161,7 +166,7 @@ size_t alloc_tag_top_users(struct codetag_bytes *tags, size_t count, bool can_sl
}
}
codetag_lock_module_list(alloc_tag_cttype, false);
codetag_unlock_module_list(alloc_tag_cttype);
return nr;
}
@ -767,50 +772,82 @@ static __init bool need_page_alloc_tagging(void)
* their codetag uninitialized. Track these early PFNs so we can clear
* their codetag refs later to avoid warnings when they are freed.
*
* Early allocations include:
* - Base allocations independent of CPU count
* - Per-CPU allocations (e.g., CPU hotplug callbacks during smp_init,
* such as trace ring buffers, scheduler per-cpu data)
*
* For simplicity, we fix the size to 8192.
* If insufficient, a warning will be triggered to alert the user.
*
* TODO: Replace fixed-size array with dynamic allocation using
* a GFP flag similar to ___GFP_NO_OBJ_EXT to avoid recursion.
* Each page is cast to a pfn_pool: the first few bytes hold metadata
* (next pointer and slot count), the remainder stores PFNs.
*/
#define EARLY_ALLOC_PFN_MAX 8192
struct pfn_pool {
struct pfn_pool *next;
atomic_t count;
unsigned long pfns[];
};
static unsigned long early_pfns[EARLY_ALLOC_PFN_MAX] __initdata;
static atomic_t early_pfn_count __initdata = ATOMIC_INIT(0);
#define PFN_POOL_SIZE ((PAGE_SIZE - offsetof(struct pfn_pool, pfns)) / \
sizeof(unsigned long))
/*
* Skip early PFN recording for a page allocation. Reuses the
* %__GFP_NO_OBJ_EXT bit. Used by __alloc_tag_add_early_pfn() to avoid
* recursion when allocating pages for the early PFN tracking list
* itself.
*
* Codetags of the pages allocated with __GFP_NO_CODETAG should be
* cleared (via clear_page_tag_ref()) before freeing the pages to prevent
* alloc_tag_sub_check() from triggering a warning.
*/
#define __GFP_NO_CODETAG __GFP_NO_OBJ_EXT
static struct pfn_pool *current_pfn_pool __initdata;
static void __init __alloc_tag_add_early_pfn(unsigned long pfn)
{
int old_idx, new_idx;
struct pfn_pool *pool;
int idx;
do {
old_idx = atomic_read(&early_pfn_count);
if (old_idx >= EARLY_ALLOC_PFN_MAX) {
pr_warn_once("Early page allocations before page_ext init exceeded EARLY_ALLOC_PFN_MAX (%d)\n",
EARLY_ALLOC_PFN_MAX);
return;
}
new_idx = old_idx + 1;
} while (!atomic_try_cmpxchg(&early_pfn_count, &old_idx, new_idx));
pool = READ_ONCE(current_pfn_pool);
if (!pool || atomic_read(&pool->count) >= PFN_POOL_SIZE) {
struct page *new_page = alloc_page(__GFP_HIGH | __GFP_NO_CODETAG);
struct pfn_pool *new;
early_pfns[old_idx] = pfn;
if (!new_page) {
pr_warn_once("early PFN tracking page allocation failed\n");
return;
}
new = page_address(new_page);
new->next = pool;
atomic_set(&new->count, 0);
if (cmpxchg(&current_pfn_pool, pool, new) != pool) {
clear_page_tag_ref(new_page);
__free_page(new_page);
continue;
}
pool = new;
}
idx = atomic_read(&pool->count);
if (idx >= PFN_POOL_SIZE)
continue;
if (atomic_cmpxchg(&pool->count, idx, idx + 1) == idx)
break;
} while (1);
pool->pfns[idx] = pfn;
}
typedef void alloc_tag_add_func(unsigned long pfn);
static alloc_tag_add_func __rcu *alloc_tag_add_early_pfn_ptr __refdata =
RCU_INITIALIZER(__alloc_tag_add_early_pfn);
void alloc_tag_add_early_pfn(unsigned long pfn)
void alloc_tag_add_early_pfn(unsigned long pfn, gfp_t gfp_flags)
{
alloc_tag_add_func *alloc_tag_add;
if (static_key_enabled(&mem_profiling_compressed))
return;
/* Skip allocations for the tracking list itself to avoid recursion. */
if (gfp_flags & __GFP_NO_CODETAG)
return;
rcu_read_lock();
alloc_tag_add = rcu_dereference(alloc_tag_add_early_pfn_ptr);
if (alloc_tag_add)
@ -820,7 +857,9 @@ void alloc_tag_add_early_pfn(unsigned long pfn)
static void __init clear_early_alloc_pfn_tag_refs(void)
{
unsigned int i;
struct pfn_pool *pool, *next;
struct page *page;
int i;
if (static_key_enabled(&mem_profiling_compressed))
return;
@ -829,37 +868,45 @@ static void __init clear_early_alloc_pfn_tag_refs(void)
/* Make sure we are not racing with __alloc_tag_add_early_pfn() */
synchronize_rcu();
for (i = 0; i < atomic_read(&early_pfn_count); i++) {
unsigned long pfn = early_pfns[i];
for (pool = current_pfn_pool; pool; pool = next) {
int nr_pfns = atomic_read(&pool->count);
if (pfn_valid(pfn)) {
struct page *page = pfn_to_page(pfn);
union pgtag_ref_handle handle;
union codetag_ref ref;
for (i = 0; i < nr_pfns; i++) {
unsigned long pfn = pool->pfns[i];
if (get_page_tag_ref(page, &ref, &handle)) {
/*
* An early-allocated page could be freed and reallocated
* after its page_ext is initialized but before we clear it.
* In that case, it already has a valid tag set.
* We should not overwrite that valid tag with CODETAG_EMPTY.
*
* Note: there is still a small race window between checking
* ref.ct and calling set_codetag_empty(). We accept this
* race as it's unlikely and the extra complexity of atomic
* cmpxchg is not worth it for this debug-only code path.
*/
if (ref.ct) {
if (pfn_valid(pfn)) {
union pgtag_ref_handle handle;
union codetag_ref ref;
if (get_page_tag_ref(pfn_to_page(pfn), &ref, &handle)) {
/*
* An early-allocated page could be freed and reallocated
* after its page_ext is initialized but before we clear it.
* In that case, it already has a valid tag set.
* We should not overwrite that valid tag
* with CODETAG_EMPTY.
*
* Note: there is still a small race window between checking
* ref.ct and calling set_codetag_empty(). We accept this
* race as it's unlikely and the extra complexity of atomic
* cmpxchg is not worth it for this debug-only code path.
*/
if (ref.ct) {
put_page_tag_ref(handle);
continue;
}
set_codetag_empty(&ref);
update_page_tag_ref(handle, &ref);
put_page_tag_ref(handle);
continue;
}
set_codetag_empty(&ref);
update_page_tag_ref(handle, &ref);
put_page_tag_ref(handle);
}
}
next = pool->next;
page = virt_to_page(pool);
clear_page_tag_ref(page);
__free_page(page);
}
}
#else /* !CONFIG_MEM_ALLOC_PROFILING_DEBUG */

12
lib/codetag.c
View File

@ -35,12 +35,9 @@ struct codetag_module {
static DEFINE_MUTEX(codetag_lock);
static LIST_HEAD(codetag_types);
void codetag_lock_module_list(struct codetag_type *cttype, bool lock)
void codetag_lock_module_list(struct codetag_type *cttype)
{
if (lock)
down_read(&cttype->mod_lock);
else
up_read(&cttype->mod_lock);
down_read(&cttype->mod_lock);
}
bool codetag_trylock_module_list(struct codetag_type *cttype)
@ -48,6 +45,11 @@ bool codetag_trylock_module_list(struct codetag_type *cttype)
return down_read_trylock(&cttype->mod_lock) != 0;
}
void codetag_unlock_module_list(struct codetag_type *cttype)
{
up_read(&cttype->mod_lock);
}
struct codetag_iterator codetag_get_ct_iter(struct codetag_type *cttype)
{
struct codetag_iterator iter = {

11
mm/Kconfig
View File

@ -989,17 +989,6 @@ config THP_SWAP
For selection by architectures with reasonable THP sizes.
config READ_ONLY_THP_FOR_FS
bool "Read-only THP for filesystems (EXPERIMENTAL)"
depends on TRANSPARENT_HUGEPAGE
help
Allow khugepaged to put read-only file-backed pages in THP.
This is marked experimental because it is a new feature. Write
support of file THPs will be developed in the next few release
cycles.
config NO_PAGE_MAPCOUNT
bool "No per-page mapcount (EXPERIMENTAL)"
help

55
mm/damon/core.c
View File

@ -1387,10 +1387,36 @@ static int damon_commit_target(
return 0;
}
/*
* damon_revert_target_commits() - revert unsuccessful target commits.
* @dst: Commit destination context
* @failed: Commit failed destination target
* @src: Commit source context
*
* Revert target states that changed by damon_commit_target(), and cannot be
* cleaned up by the destination context's ops.cleanup_target().
*/
static void damon_revert_target_commits(struct damon_ctx *dst,
struct damon_target *failed, struct damon_ctx *src)
{
struct damon_target *target;
if (!damon_target_has_pid(src))
return;
if (dst->ops.cleanup_target)
return;
damon_for_each_target(target, dst) {
if (target == failed)
return;
put_pid(target->pid);
}
}
static int damon_commit_targets(
struct damon_ctx *dst, struct damon_ctx *src)
{
struct damon_target *dst_target, *next, *src_target, *new_target;
struct damon_target *failed;
int i = 0, j = 0, err;
damon_for_each_target_safe(dst_target, next, dst) {
@ -1404,8 +1430,10 @@ static int damon_commit_targets(
dst_target, damon_target_has_pid(dst),
src_target, damon_target_has_pid(src),
src->min_region_sz);
if (err)
return err;
if (err) {
failed = dst_target;
goto out;
}
} else {
struct damos *s;
@ -1419,25 +1447,34 @@ static int damon_commit_targets(
}
}
failed = NULL;
damon_for_each_target_safe(src_target, next, src) {
if (j++ < i)
continue;
/* target to remove has no matching dst */
if (src_target->obsolete)
return -EINVAL;
if (src_target->obsolete) {
err = -EINVAL;
goto out;
}
new_target = damon_new_target();
if (!new_target)
return -ENOMEM;
if (!new_target) {
err = -ENOMEM;
goto out;
}
err = damon_commit_target(new_target, false,
src_target, damon_target_has_pid(src),
src->min_region_sz);
if (err) {
damon_destroy_target(new_target, NULL);
return err;
goto out;
}
damon_add_target(dst, new_target);
}
return 0;
out:
damon_revert_target_commits(dst, failed, src);
return err;
}
static void damon_commit_filter(struct damon_filter *dst,
@ -1571,8 +1608,10 @@ int damon_commit_ctx(struct damon_ctx *dst, struct damon_ctx *src)
*/
if (!damon_attrs_equals(&dst->attrs, &src->attrs)) {
err = damon_set_attrs(dst, &src->attrs);
if (err)
if (err) {
damon_revert_target_commits(dst, NULL, src);
return err;
}
}
dst->pause = src->pause;
dst->ops = src->ops;

5
mm/filemap.c
View File

@ -189,7 +189,6 @@ static void filemap_unaccount_folio(struct address_space *mapping,
lruvec_stat_mod_folio(folio, NR_SHMEM_THPS, -nr);
} else if (folio_test_pmd_mappable(folio)) {
lruvec_stat_mod_folio(folio, NR_FILE_THPS, -nr);
filemap_nr_thps_dec(mapping);
}
if (test_bit(AS_KERNEL_FILE, &folio->mapping->flags))
mod_node_page_state(folio_pgdat(folio),
@ -3324,8 +3323,6 @@ static struct file *do_sync_mmap_readahead(struct vm_fault *vmf)
unsigned int thp_order = 0;
unsigned short mmap_miss;
ractl._max_index = vmf->vma->vm_pgoff + vma_pages(vmf->vma) - 1;
/* Use the readahead code, even if readahead is disabled */
if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE) && (vm_flags & VM_HUGEPAGE)) {
/*
@ -3421,7 +3418,6 @@ static struct file *do_sync_mmap_readahead(struct vm_fault *vmf)
* mmap read-around
*/
ra->start = max_t(long, 0, vmf->pgoff - ra->ra_pages / 2);
ra->start = max(ra->start, vmf->vma->vm_pgoff);
ra->size = ra->ra_pages;
ra->async_size = ra->ra_pages / 4;
ra->order = 0;
@ -3469,7 +3465,6 @@ static struct file *do_async_mmap_readahead(struct vm_fault *vmf,
}
if (folio_test_readahead(folio)) {
ractl._max_index = vmf->vma->vm_pgoff + vma_pages(vmf->vma) - 1;
fpin = maybe_unlock_mmap_for_io(vmf, fpin);
page_cache_async_ra(&ractl, folio, ra->ra_pages);
}

12
mm/gup_test.c
View File

@ -105,11 +105,15 @@ static int __gup_test_ioctl(unsigned int cmd,
unsigned long i, nr_pages, addr, next;
long nr;
struct page **pages;
unsigned long end;
int ret = 0;
bool needs_mmap_lock =
cmd != GUP_FAST_BENCHMARK && cmd != PIN_FAST_BENCHMARK;
if (gup->size > ULONG_MAX)
if (gup->addr > ULONG_MAX || gup->size > ULONG_MAX)
return -EINVAL;
if (check_add_overflow((unsigned long)gup->addr,
(unsigned long)gup->size, &end))
return -EINVAL;
nr_pages = gup->size / PAGE_SIZE;
@ -125,13 +129,13 @@ static int __gup_test_ioctl(unsigned int cmd,
i = 0;
nr = gup->nr_pages_per_call;
start_time = ktime_get();
for (addr = gup->addr; addr < gup->addr + gup->size; addr = next) {
for (addr = gup->addr; addr < end; addr = next) {
if (nr != gup->nr_pages_per_call)
break;
next = addr + nr * PAGE_SIZE;
if (next > gup->addr + gup->size) {
next = gup->addr + gup->size;
if (next > end) {
next = end;
nr = (next - addr) / PAGE_SIZE;
}

50
mm/huge_memory.c
View File

@ -89,9 +89,6 @@ static inline bool file_thp_enabled(struct vm_area_struct *vma)
{
struct inode *inode;
if (!IS_ENABLED(CONFIG_READ_ONLY_THP_FOR_FS))
return false;
if (!vma->vm_file)
return false;
@ -100,7 +97,10 @@ static inline bool file_thp_enabled(struct vm_area_struct *vma)
if (IS_ANON_FILE(inode))
return false;
return !inode_is_open_for_write(inode) && S_ISREG(inode->i_mode);
if (!mapping_pmd_folio_support(vma->vm_file->f_mapping))
return false;
return S_ISREG(inode->i_mode);
}
/* If returns true, we are unable to access the VMA's folios. */
@ -702,6 +702,8 @@ static struct kobj_attribute _name##_attr = __ATTR_RO(_name)
DEFINE_MTHP_STAT_ATTR(anon_fault_alloc, MTHP_STAT_ANON_FAULT_ALLOC);
DEFINE_MTHP_STAT_ATTR(anon_fault_fallback, MTHP_STAT_ANON_FAULT_FALLBACK);
DEFINE_MTHP_STAT_ATTR(anon_fault_fallback_charge, MTHP_STAT_ANON_FAULT_FALLBACK_CHARGE);
DEFINE_MTHP_STAT_ATTR(collapse_alloc, MTHP_STAT_COLLAPSE_ALLOC);
DEFINE_MTHP_STAT_ATTR(collapse_alloc_failed, MTHP_STAT_COLLAPSE_ALLOC_FAILED);
DEFINE_MTHP_STAT_ATTR(zswpout, MTHP_STAT_ZSWPOUT);
DEFINE_MTHP_STAT_ATTR(swpin, MTHP_STAT_SWPIN);
DEFINE_MTHP_STAT_ATTR(swpin_fallback, MTHP_STAT_SWPIN_FALLBACK);
@ -718,6 +720,10 @@ DEFINE_MTHP_STAT_ATTR(split_failed, MTHP_STAT_SPLIT_FAILED);
DEFINE_MTHP_STAT_ATTR(split_deferred, MTHP_STAT_SPLIT_DEFERRED);
DEFINE_MTHP_STAT_ATTR(nr_anon, MTHP_STAT_NR_ANON);
DEFINE_MTHP_STAT_ATTR(nr_anon_partially_mapped, MTHP_STAT_NR_ANON_PARTIALLY_MAPPED);
DEFINE_MTHP_STAT_ATTR(collapse_exceed_swap_pte, MTHP_STAT_COLLAPSE_EXCEED_SWAP);
DEFINE_MTHP_STAT_ATTR(collapse_exceed_none_pte, MTHP_STAT_COLLAPSE_EXCEED_NONE);
DEFINE_MTHP_STAT_ATTR(collapse_exceed_shared_pte, MTHP_STAT_COLLAPSE_EXCEED_SHARED);
static struct attribute *anon_stats_attrs[] = {
&anon_fault_alloc_attr.attr,
@ -734,6 +740,9 @@ static struct attribute *anon_stats_attrs[] = {
&split_deferred_attr.attr,
&nr_anon_attr.attr,
&nr_anon_partially_mapped_attr.attr,
&collapse_exceed_swap_pte_attr.attr,
&collapse_exceed_none_pte_attr.attr,
&collapse_exceed_shared_pte_attr.attr,
NULL,
};
@ -767,6 +776,8 @@ static struct attribute *any_stats_attrs[] = {
#endif
&split_attr.attr,
&split_failed_attr.attr,
&collapse_alloc_attr.attr,
&collapse_alloc_failed_attr.attr,
NULL,
};
@ -3753,33 +3764,9 @@ int folio_check_splittable(struct folio *folio, unsigned int new_order,
if (!folio->mapping && !folio_test_anon(folio))
return -EBUSY;
if (folio_test_anon(folio)) {
/* order-1 is not supported for anonymous THP. */
if (new_order == 1)
return -EINVAL;
} else if (split_type == SPLIT_TYPE_NON_UNIFORM || new_order) {
if (IS_ENABLED(CONFIG_READ_ONLY_THP_FOR_FS) &&
!mapping_large_folio_support(folio->mapping)) {
/*
* We can always split a folio down to a single page
* (new_order == 0) uniformly.
*
* For any other scenario
* a) uniform split targeting a large folio
* (new_order > 0)
* b) any non-uniform split
* we must confirm that the file system supports large
* folios.
*
* Note that we might still have THPs in such
* mappings, which is created from khugepaged when
* CONFIG_READ_ONLY_THP_FOR_FS is enabled. But in that
* case, the mapping does not actually support large
* folios properly.
*/
return -EINVAL;
}
}
/* order-1 is not supported for anonymous THP. */
if (folio_test_anon(folio) && new_order == 1)
return -EINVAL;
/*
* swapcache folio could only be split to order 0
@ -3867,7 +3854,6 @@ static int __folio_freeze_and_split_unmapped(struct folio *folio, unsigned int n
} else {
lruvec_stat_mod_folio(folio,
NR_FILE_THPS, -nr);
filemap_nr_thps_dec(mapping);
}
}
}

734
mm/khugepaged.c
View File

File diff suppressed because it is too large Load Diff

4
mm/memory.c
View File

@ -4386,7 +4386,7 @@ void unmap_mapping_folio(struct folio *folio)
details.zap_flags = ZAP_FLAG_DROP_MARKER;
i_mmap_lock_read(mapping);
if (unlikely(!RB_EMPTY_ROOT(&mapping->i_mmap.rb_root)))
if (unlikely(mapping_mapped(mapping)))
unmap_mapping_range_tree(&mapping->i_mmap, first_index,
last_index, &details);
i_mmap_unlock_read(mapping);
@ -4416,7 +4416,7 @@ void unmap_mapping_pages(struct address_space *mapping, pgoff_t start,
last_index = ULONG_MAX;
i_mmap_lock_read(mapping);
if (unlikely(!RB_EMPTY_ROOT(&mapping->i_mmap.rb_root)))
if (unlikely(mapping_mapped(mapping)))
unmap_mapping_range_tree(&mapping->i_mmap, first_index,
last_index, &details);
i_mmap_unlock_read(mapping);

18
mm/page_alloc.c
View File

@ -1246,7 +1246,7 @@ void __clear_page_tag_ref(struct page *page)
/* Should be called only if mem_alloc_profiling_enabled() */
static noinline
void __pgalloc_tag_add(struct page *page, struct task_struct *task,
unsigned int nr)
unsigned int nr, gfp_t gfp_flags)
{
union pgtag_ref_handle handle;
union codetag_ref ref;
@ -1260,17 +1260,17 @@ void __pgalloc_tag_add(struct page *page, struct task_struct *task,
* page_ext is not available yet, record the pfn so we can
* clear the tag ref later when page_ext is initialized.
*/
alloc_tag_add_early_pfn(page_to_pfn(page));
alloc_tag_add_early_pfn(page_to_pfn(page), gfp_flags);
if (task->alloc_tag)
alloc_tag_set_inaccurate(task->alloc_tag);
}
}
static inline void pgalloc_tag_add(struct page *page, struct task_struct *task,
unsigned int nr)
unsigned int nr, gfp_t gfp_flags)
{
if (mem_alloc_profiling_enabled())
__pgalloc_tag_add(page, task, nr);
__pgalloc_tag_add(page, task, nr, gfp_flags);
}
/* Should be called only if mem_alloc_profiling_enabled() */
@ -1303,7 +1303,7 @@ static inline void pgalloc_tag_sub_pages(struct alloc_tag *tag, unsigned int nr)
#else /* CONFIG_MEM_ALLOC_PROFILING */
static inline void pgalloc_tag_add(struct page *page, struct task_struct *task,
unsigned int nr) {}
unsigned int nr, gfp_t gfp_flags) {}
static inline void pgalloc_tag_sub(struct page *page, unsigned int nr) {}
static inline void pgalloc_tag_sub_pages(struct alloc_tag *tag, unsigned int nr) {}
@ -1858,7 +1858,7 @@ inline void post_alloc_hook(struct page *page, unsigned int order,
set_page_owner(page, order, gfp_flags);
page_table_check_alloc(page, order);
pgalloc_tag_add(page, current, 1 << order);
pgalloc_tag_add(page, current, 1 << order, gfp_flags);
}
static void prep_new_page(struct page *page, unsigned int order, gfp_t gfp_flags,
@ -6653,7 +6653,8 @@ static int sysctl_min_unmapped_ratio_sysctl_handler(const struct ctl_table *tabl
if (rc)
return rc;
setup_min_unmapped_ratio();
if (write)
setup_min_unmapped_ratio();
return 0;
}
@ -6680,7 +6681,8 @@ static int sysctl_min_slab_ratio_sysctl_handler(const struct ctl_table *table, i
if (rc)
return rc;
setup_min_slab_ratio();
if (write)
setup_min_slab_ratio();
return 0;
}

19
mm/page_isolation.c
View File

@ -41,8 +41,14 @@ bool page_is_unmovable(struct zone *zone, struct page *page,
* We need not scan over tail pages because we don't
* handle each tail page individually in migration.
*/
if (PageHuge(page) || PageCompound(page)) {
if (PageCompound(page)) {
struct folio *folio = page_folio(page);
unsigned long nr_pages, pfn;
unsigned int order;
order = compound_order(&folio->page);
if (order > MAX_FOLIO_ORDER)
return true;
if (folio_test_hugetlb(folio)) {
struct hstate *h;
@ -54,15 +60,16 @@ bool page_is_unmovable(struct zone *zone, struct page *page,
* The huge page may be freed so can not
* use folio_hstate() directly.
*/
h = size_to_hstate(folio_size(folio));
if (h && !hugepage_migration_supported(h))
h = size_to_hstate(PAGE_SIZE << order);
if (!h || !hugepage_migration_supported(h))
return true;
} else if (!folio_test_lru(folio)) {
} else if (!PageLRU(page)) {
return true;
}
*step = folio_nr_pages(folio) - folio_page_idx(folio, page);
nr_pages = 1UL << order;
pfn = page_to_pfn(page);
*step = (pfn | (nr_pages - 1)) + 1 - pfn;
return false;
}

6
mm/readahead.c
View File

@ -335,8 +335,6 @@ static void do_page_cache_ra(struct readahead_control *ractl,
return;
end_index = (isize - 1) >> PAGE_SHIFT;
if (end_index > ractl->_max_index)
end_index = ractl->_max_index;
if (index > end_index)
return;
/* Don't read past the page containing the last byte of the file */
@ -487,7 +485,7 @@ void page_cache_ra_order(struct readahead_control *ractl,
pgoff_t start = readahead_index(ractl);
pgoff_t index = start;
unsigned int min_order = mapping_min_folio_order(mapping);
pgoff_t limit;
pgoff_t limit = (i_size_read(mapping->host) - 1) >> PAGE_SHIFT;
pgoff_t mark;
unsigned int nofs;
int err = 0;
@ -500,8 +498,6 @@ void page_cache_ra_order(struct readahead_control *ractl,
goto fallback;
}
limit = (i_size_read(mapping->host) - 1) >> PAGE_SHIFT;
limit = min(limit, ractl->_max_index);
if (limit > index + ra->size - 1) {
limit = index + ra->size - 1;
mark = index + ra->size - ra->async_size;

4
mm/shrinker_debug.c
View File

@ -57,8 +57,6 @@ static int shrinker_debugfs_count_show(struct seq_file *m, void *v)
if (!count_per_node)
return -ENOMEM;
rcu_read_lock();
memcg_aware = shrinker->flags & SHRINKER_MEMCG_AWARE;
memcg = mem_cgroup_iter(NULL, NULL, NULL);
@ -88,8 +86,6 @@ static int shrinker_debugfs_count_show(struct seq_file *m, void *v)
}
} while ((memcg = mem_cgroup_iter(NULL, memcg, NULL)) != NULL);
rcu_read_unlock();
kfree(count_per_node);
return ret;
}

8
mm/truncate.c
View File

@ -177,7 +177,7 @@ int truncate_inode_folio(struct address_space *mapping, struct folio *folio)
return 0;
}
static int try_folio_split_or_unmap(struct folio *folio, struct page *split_at,
static int folio_split_or_unmap(struct folio *folio, struct page *split_at,
unsigned long min_order)
{
enum ttu_flags ttu_flags =
@ -186,7 +186,7 @@ static int try_folio_split_or_unmap(struct folio *folio, struct page *split_at,
TTU_IGNORE_MLOCK;
int ret;
ret = try_folio_split_to_order(folio, split_at, min_order);
ret = folio_split(folio, min_order, split_at, NULL);
/*
* If the split fails, unmap the folio, so it will be refaulted
@ -252,7 +252,7 @@ bool truncate_inode_partial_folio(struct folio *folio, loff_t start, loff_t end)
min_order = mapping_min_folio_order(folio->mapping);
split_at = folio_page(folio, PAGE_ALIGN_DOWN(offset) / PAGE_SIZE);
if (!try_folio_split_or_unmap(folio, split_at, min_order)) {
if (!folio_split_or_unmap(folio, split_at, min_order)) {
/*
* try to split at offset + length to make sure folios within
* the range can be dropped, especially to avoid memory waste
@ -279,7 +279,7 @@ bool truncate_inode_partial_folio(struct folio *folio, loff_t start, loff_t end)
/* make sure folio2 is large and does not change its mapping */
if (folio_test_large(folio2) &&
folio2->mapping == folio->mapping)
try_folio_split_or_unmap(folio2, split_at2, min_order);
folio_split_or_unmap(folio2, split_at2, min_order);
folio_unlock(folio2);
out:

69
mm/vmscan.c
View File

@ -612,11 +612,38 @@ typedef enum {
PAGE_CLEAN,
} pageout_t;
static pageout_t writeout(struct folio *folio, struct address_space *mapping,
struct swap_iocb **plug, struct list_head *folio_list)
/*
* pageout is called by shrink_folio_list() for each dirty folio.
*/
static pageout_t pageout(struct folio *folio, struct address_space *mapping,
struct swap_iocb **plug, struct list_head *folio_list)
{
int res;
/*
* We no longer attempt to writeback filesystem folios here, other
* than tmpfs/shmem. That's taken care of in page-writeback.
* If we find a dirty filesystem folio at the end of the LRU list,
* typically that means the filesystem is saturating the storage
* with contiguous writes and telling it to write a folio here
* would only make the situation worse by injecting an element
* of random access.
*
* If the folio is swapcache, write it back even if that would
* block, for some throttling. This happens by accident, because
* swap_backing_dev_info is bust: it doesn't reflect the
* congestion state of the swapdevs. Easy to fix, if needed.
*
* A freeable shmem or swapcache folio is referenced only by the
* caller that isolated the folio and the page cache.
*/
if (folio_ref_count(folio) != 1 + folio_nr_pages(folio) || !mapping)
return PAGE_KEEP;
if (!shmem_mapping(mapping) && !folio_test_anon(folio))
return PAGE_ACTIVATE;
if (!folio_clear_dirty_for_io(folio))
return PAGE_CLEAN;
folio_set_reclaim(folio);
/*
@ -645,38 +672,6 @@ static pageout_t writeout(struct folio *folio, struct address_space *mapping,
return PAGE_SUCCESS;
}
/*
* pageout is called by shrink_folio_list() for each dirty folio.
*/
static pageout_t pageout(struct folio *folio, struct address_space *mapping,
struct swap_iocb **plug, struct list_head *folio_list)
{
/*
* We no longer attempt to writeback filesystem folios here, other
* than tmpfs/shmem. That's taken care of in page-writeback.
* If we find a dirty filesystem folio at the end of the LRU list,
* typically that means the filesystem is saturating the storage
* with contiguous writes and telling it to write a folio here
* would only make the situation worse by injecting an element
* of random access.
*
* If the folio is swapcache, write it back even if that would
* block, for some throttling. This happens by accident, because
* swap_backing_dev_info is bust: it doesn't reflect the
* congestion state of the swapdevs. Easy to fix, if needed.
*
* A freeable shmem or swapcache folio is referenced only by the
* caller that isolated the folio and the page cache.
*/
if (folio_ref_count(folio) != 1 + folio_nr_pages(folio) || !mapping)
return PAGE_KEEP;
if (!shmem_mapping(mapping) && !folio_test_anon(folio))
return PAGE_ACTIVATE;
if (!folio_clear_dirty_for_io(folio))
return PAGE_CLEAN;
return writeout(folio, mapping, plug, folio_list);
}
/*
* Same as remove_mapping, but if the folio is removed from the mapping, it
* gets returned with a refcount of 0.
@ -6711,11 +6706,11 @@ unsigned long try_to_free_pages(struct zonelist *zonelist, int order,
return 1;
set_task_reclaim_state(current, &sc.reclaim_state);
trace_mm_vmscan_direct_reclaim_begin(sc.gfp_mask, order, 0);
trace_mm_vmscan_direct_reclaim_begin(sc.gfp_mask, order, NULL);
nr_reclaimed = do_try_to_free_pages(zonelist, &sc);
trace_mm_vmscan_direct_reclaim_end(nr_reclaimed, 0);
trace_mm_vmscan_direct_reclaim_end(nr_reclaimed, NULL);
set_task_reclaim_state(current, NULL);
return nr_reclaimed;
@ -7781,7 +7776,7 @@ static unsigned long __node_reclaim(struct pglist_data *pgdat, gfp_t gfp_mask,
delayacct_freepages_end();
psi_memstall_leave(&pflags);
trace_mm_vmscan_node_reclaim_end(sc->nr_reclaimed, 0);
trace_mm_vmscan_node_reclaim_end(sc->nr_reclaimed, NULL);
return sc->nr_reclaimed;
}

3
mm/zsmalloc.c
View File

@ -565,8 +565,7 @@ static int zs_stats_size_show(struct seq_file *s, void *v)
total_freeable += freeable;
}
seq_puts(s, "\n");
seq_printf(s, " %5s %5s ", "Total", "");
seq_printf(s, "\n %5s %5s ", "Total", "");
for (fg = ZS_INUSE_RATIO_10; fg < NR_FULLNESS_GROUPS; fg++)
seq_printf(s, "%9lu ", inuse_totals[fg]);

4
tools/testing/selftests/mm/.gitignore vendored
View File

@ -4,6 +4,10 @@ hugepage-mmap
hugepage-mremap
hugepage-shm
hugepage-vmemmap
hugetlb-mmap
hugetlb-mremap
hugetlb-shm
hugetlb-vmemmap
hugetlb-madvise
hugetlb-read-hwpoison
hugetlb-soft-offline

13
tools/testing/selftests/mm/Makefile
View File

@ -61,16 +61,15 @@ TEST_GEN_FILES += gup_longterm
TEST_GEN_FILES += gup_test
TEST_GEN_FILES += hmm-tests
TEST_GEN_FILES += hugetlb-madvise
TEST_GEN_FILES += hugetlb-mmap
TEST_GEN_FILES += hugetlb-mremap
TEST_GEN_FILES += hugetlb-read-hwpoison
TEST_GEN_FILES += hugetlb-shm
TEST_GEN_FILES += hugetlb-soft-offline
TEST_GEN_FILES += hugepage-mmap
TEST_GEN_FILES += hugepage-mremap
TEST_GEN_FILES += hugepage-shm
TEST_GEN_FILES += hugepage-vmemmap
TEST_GEN_FILES += hugetlb-vmemmap
TEST_GEN_FILES += khugepaged
TEST_GEN_FILES += madv_populate
TEST_GEN_FILES += map_fixed_noreplace
TEST_GEN_FILES += map_hugetlb
TEST_GEN_FILES += map_populate
ifneq (,$(filter $(ARCH),arm64 riscv riscv64 x86 x86_64 loongarch32 loongarch64))
TEST_GEN_FILES += memfd_secret
@ -188,8 +187,8 @@ TEST_FILES += write_hugetlb_memory.sh
include ../lib.mk
$(TEST_GEN_PROGS): vm_util.c thp_settings.c
$(TEST_GEN_FILES): vm_util.c thp_settings.c
$(TEST_GEN_PROGS): vm_util.c hugepage_settings.c
$(TEST_GEN_FILES): vm_util.c hugepage_settings.c
$(OUTPUT)/uffd-stress: uffd-common.c
$(OUTPUT)/uffd-unit-tests: uffd-common.c

45
tools/testing/selftests/mm/charge_reserved_hugetlb.sh
View File

@ -17,6 +17,7 @@ if ! command -v killall >/dev/null 2>&1; then
fi
nr_hugepgs=$(cat /proc/sys/vm/nr_hugepages)
trap 'echo "$nr_hugepgs" > /proc/sys/vm/nr_hugepages' EXIT INT TERM
fault_limit_file=limit_in_bytes
reservation_limit_file=rsvd.limit_in_bytes
@ -70,7 +71,6 @@ function cleanup() {
if [[ -e $cgroup_path/hugetlb_cgroup_test2 ]]; then
rmdir $cgroup_path/hugetlb_cgroup_test2
fi
echo 0 >/proc/sys/vm/nr_hugepages
echo CLEANUP DONE
}
@ -94,6 +94,15 @@ function get_machine_hugepage_size() {
}
MB=$(get_machine_hugepage_size)
if (( MB >= 1024 )); then
# For 1GB hugepages
UNIT="GB"
MB_DISPLAY=$((MB / 1024))
else
# For 2MB hugepages
UNIT="MB"
MB_DISPLAY=$MB
fi
function setup_cgroup() {
local name="$1"
@ -103,11 +112,12 @@ function setup_cgroup() {
mkdir $cgroup_path/$name
echo writing cgroup limit: "$cgroup_limit"
echo "$cgroup_limit" >$cgroup_path/$name/hugetlb.${MB}MB.$fault_limit_file
echo "$cgroup_limit" > \
$cgroup_path/$name/hugetlb.${MB_DISPLAY}${UNIT}.$fault_limit_file
echo writing reservation limit: "$reservation_limit"
echo "$reservation_limit" > \
$cgroup_path/$name/hugetlb.${MB}MB.$reservation_limit_file
$cgroup_path/$name/hugetlb.${MB_DISPLAY}${UNIT}.$reservation_limit_file
if [ -e "$cgroup_path/$name/cpuset.cpus" ]; then
echo 0 >$cgroup_path/$name/cpuset.cpus
@ -142,7 +152,7 @@ function wait_for_file_value() {
function wait_for_hugetlb_memory_to_get_depleted() {
local cgroup="$1"
local path="$cgroup_path/$cgroup/hugetlb.${MB}MB.$reservation_usage_file"
local path="$cgroup_path/$cgroup/hugetlb.${MB_DISPLAY}${UNIT}.$reservation_usage_file"
wait_for_file_value "$path" "0"
}
@ -150,7 +160,7 @@ function wait_for_hugetlb_memory_to_get_depleted() {
function wait_for_hugetlb_memory_to_get_reserved() {
local cgroup="$1"
local size="$2"
local path="$cgroup_path/$cgroup/hugetlb.${MB}MB.$reservation_usage_file"
local path="$cgroup_path/$cgroup/hugetlb.${MB_DISPLAY}${UNIT}.$reservation_usage_file"
wait_for_file_value "$path" "$size"
}
@ -158,7 +168,7 @@ function wait_for_hugetlb_memory_to_get_reserved() {
function wait_for_hugetlb_memory_to_get_written() {
local cgroup="$1"
local size="$2"
local path="$cgroup_path/$cgroup/hugetlb.${MB}MB.$fault_usage_file"
local path="$cgroup_path/$cgroup/hugetlb.${MB_DISPLAY}${UNIT}.$fault_usage_file"
wait_for_file_value "$path" "$size"
}
@ -180,8 +190,8 @@ function write_hugetlbfs_and_get_usage() {
hugetlb_difference=0
reserved_difference=0
local hugetlb_usage=$cgroup_path/$cgroup/hugetlb.${MB}MB.$fault_usage_file
local reserved_usage=$cgroup_path/$cgroup/hugetlb.${MB}MB.$reservation_usage_file
local hugetlb_usage=$cgroup_path/$cgroup/hugetlb.${MB_DISPLAY}${UNIT}.$fault_usage_file
local reserved_usage=$cgroup_path/$cgroup/hugetlb.${MB_DISPLAY}${UNIT}.$reservation_usage_file
local hugetlb_before=$(cat $hugetlb_usage)
local reserved_before=$(cat $reserved_usage)
@ -312,8 +322,10 @@ function run_test() {
cleanup_hugetlb_memory "hugetlb_cgroup_test"
local final_hugetlb=$(cat $cgroup_path/hugetlb_cgroup_test/hugetlb.${MB}MB.$fault_usage_file)
local final_reservation=$(cat $cgroup_path/hugetlb_cgroup_test/hugetlb.${MB}MB.$reservation_usage_file)
local final_hugetlb=$(cat \
$cgroup_path/hugetlb_cgroup_test/hugetlb.${MB_DISPLAY}${UNIT}.$fault_usage_file)
local final_reservation=$(cat \
$cgroup_path/hugetlb_cgroup_test/hugetlb.${MB_DISPLAY}${UNIT}.$reservation_usage_file)
echo $hugetlb_difference
echo $reserved_difference
@ -369,10 +381,14 @@ function run_multiple_cgroup_test() {
reservation_failed1=$reservation_failed
oom_killed1=$oom_killed
local cgroup1_hugetlb_usage=$cgroup_path/hugetlb_cgroup_test1/hugetlb.${MB}MB.$fault_usage_file
local cgroup1_reservation_usage=$cgroup_path/hugetlb_cgroup_test1/hugetlb.${MB}MB.$reservation_usage_file
local cgroup2_hugetlb_usage=$cgroup_path/hugetlb_cgroup_test2/hugetlb.${MB}MB.$fault_usage_file
local cgroup2_reservation_usage=$cgroup_path/hugetlb_cgroup_test2/hugetlb.${MB}MB.$reservation_usage_file
local cgroup1_hugetlb_usage=\
$cgroup_path/hugetlb_cgroup_test1/hugetlb.${MB_DISPLAY}${UNIT}.$fault_usage_file
local cgroup1_reservation_usage=\
$cgroup_path/hugetlb_cgroup_test1/hugetlb.${MB_DISPLAY}${UNIT}.$reservation_usage_file
local cgroup2_hugetlb_usage=\
$cgroup_path/hugetlb_cgroup_test2/hugetlb.${MB_DISPLAY}${UNIT}.$fault_usage_file
local cgroup2_reservation_usage=\
$cgroup_path/hugetlb_cgroup_test2/hugetlb.${MB_DISPLAY}${UNIT}.$reservation_usage_file
local usage_before_second_write=$(cat $cgroup1_hugetlb_usage)
local reservation_usage_before_second_write=$(cat $cgroup1_reservation_usage)
@ -599,4 +615,3 @@ if [[ $do_umount ]]; then
rmdir $cgroup_path
fi
echo "$nr_hugepgs" > /proc/sys/vm/nr_hugepages

118
tools/testing/selftests/mm/compaction_test.c
View File

@ -17,6 +17,7 @@
#include <string.h>
#include "kselftest.h"
#include "hugepage_settings.h"
#define MAP_SIZE_MB 100
#define MAP_SIZE (MAP_SIZE_MB * 1024 * 1024)
@ -82,124 +83,44 @@ int prereq(void)
return -1;
}
int check_compaction(unsigned long mem_free, unsigned long hugepage_size,
unsigned long initial_nr_hugepages)
int check_compaction(unsigned long mem_free, unsigned long hugepage_size)
{
unsigned long nr_hugepages_ul;
int fd, ret = -1;
unsigned long nr_hugepages;
int compaction_index = 0;
char nr_hugepages[20] = {0};
char init_nr_hugepages[24] = {0};
char target_nr_hugepages[24] = {0};
int slen;
snprintf(init_nr_hugepages, sizeof(init_nr_hugepages),
"%lu", initial_nr_hugepages);
int ret = -1;
/* We want to test with 80% of available memory. Else, OOM killer comes
in to play */
mem_free = mem_free * 0.8;
fd = open("/proc/sys/vm/nr_hugepages", O_RDWR | O_NONBLOCK);
if (fd < 0) {
ksft_print_msg("Failed to open /proc/sys/vm/nr_hugepages: %s\n",
strerror(errno));
ret = -1;
goto out;
}
/*
* Request huge pages for about half of the free memory. The Kernel
* will allocate as much as it can, and we expect it will get at least 1/3
*/
nr_hugepages_ul = mem_free / hugepage_size / 2;
snprintf(target_nr_hugepages, sizeof(target_nr_hugepages),
"%lu", nr_hugepages_ul);
slen = strlen(target_nr_hugepages);
if (write(fd, target_nr_hugepages, slen) != slen) {
ksft_print_msg("Failed to write %lu to /proc/sys/vm/nr_hugepages: %s\n",
nr_hugepages_ul, strerror(errno));
goto close_fd;
}
lseek(fd, 0, SEEK_SET);
if (read(fd, nr_hugepages, sizeof(nr_hugepages)) <= 0) {
ksft_print_msg("Failed to re-read from /proc/sys/vm/nr_hugepages: %s\n",
strerror(errno));
goto close_fd;
}
nr_hugepages = mem_free / hugepage_size / 2;
hugetlb_set_nr_default_pages(nr_hugepages);
/* We should have been able to request at least 1/3 rd of the memory in
huge pages */
nr_hugepages_ul = strtoul(nr_hugepages, NULL, 10);
if (!nr_hugepages_ul) {
nr_hugepages = hugetlb_nr_default_pages();
if (!nr_hugepages) {
ksft_print_msg("ERROR: No memory is available as huge pages\n");
goto close_fd;
goto out;
}
compaction_index = mem_free/(nr_hugepages_ul * hugepage_size);
compaction_index = mem_free/(nr_hugepages * hugepage_size);
lseek(fd, 0, SEEK_SET);
if (write(fd, init_nr_hugepages, strlen(init_nr_hugepages))
!= strlen(init_nr_hugepages)) {
ksft_print_msg("Failed to write value to /proc/sys/vm/nr_hugepages: %s\n",
strerror(errno));
goto close_fd;
}
ksft_print_msg("Number of huge pages allocated = %lu\n",
nr_hugepages_ul);
ksft_print_msg("Number of huge pages allocated = %lu\n", nr_hugepages);
if (compaction_index > 3) {
ksft_print_msg("ERROR: Less than 1/%d of memory is available\n"
"as huge pages\n", compaction_index);
goto close_fd;
}
ret = 0;
close_fd:
close(fd);
out:
ksft_test_result(ret == 0, "check_compaction\n");
return ret;
}
int set_zero_hugepages(unsigned long *initial_nr_hugepages)
{
int fd, ret = -1;
char nr_hugepages[20] = {0};
fd = open("/proc/sys/vm/nr_hugepages", O_RDWR | O_NONBLOCK);
if (fd < 0) {
ksft_print_msg("Failed to open /proc/sys/vm/nr_hugepages: %s\n",
strerror(errno));
goto out;
}
if (read(fd, nr_hugepages, sizeof(nr_hugepages)) <= 0) {
ksft_print_msg("Failed to read from /proc/sys/vm/nr_hugepages: %s\n",
strerror(errno));
goto close_fd;
}
lseek(fd, 0, SEEK_SET);
/* Start with the initial condition of 0 huge pages */
if (write(fd, "0", sizeof(char)) != sizeof(char)) {
ksft_print_msg("Failed to write 0 to /proc/sys/vm/nr_hugepages: %s\n",
strerror(errno));
goto close_fd;
}
*initial_nr_hugepages = strtoul(nr_hugepages, NULL, 10);
ret = 0;
close_fd:
close(fd);
out:
ksft_test_result(ret == 0, "check_compaction\n");
return ret;
}
@ -212,18 +133,17 @@ int main(int argc, char **argv)
unsigned long mem_free = 0;
unsigned long hugepage_size = 0;
long mem_fragmentable_MB = 0;
unsigned long initial_nr_hugepages;
ksft_print_header();
if (prereq() || geteuid())
ksft_exit_skip("Prerequisites unsatisfied\n");
ksft_set_plan(1);
/* Start the test without hugepages reducing mem_free */
if (set_zero_hugepages(&initial_nr_hugepages))
ksft_exit_fail();
if (!hugetlb_setup_default_exact(0))
ksft_exit_skip("Could not reset nr_hugepages\n");
ksft_set_plan(1);
lim.rlim_cur = RLIM_INFINITY;
lim.rlim_max = RLIM_INFINITY;
@ -261,6 +181,9 @@ int main(int argc, char **argv)
mem_fragmentable_MB -= MAP_SIZE_MB;
}
/* Unmap every other entry in the list to create fragmentation with
* locked pages before invoking check_compaction().
*/
for (entry = list; entry != NULL; entry = entry->next) {
munmap(entry->map, MAP_SIZE);
if (!entry->next)
@ -268,8 +191,7 @@ int main(int argc, char **argv)
entry = entry->next;
}
if (check_compaction(mem_free, hugepage_size,
initial_nr_hugepages) == 0)
if (check_compaction(mem_free, hugepage_size) == 0)
ksft_exit_pass();
ksft_exit_fail();

31
tools/testing/selftests/mm/cow.c
View File

@ -29,7 +29,7 @@
#include "../../../../mm/gup_test.h"
#include "kselftest.h"
#include "vm_util.h"
#include "thp_settings.h"
#include "hugepage_settings.h"
static size_t pagesize;
static int pagemap_fd;
@ -37,7 +37,7 @@ static size_t pmdsize;
static int nr_thpsizes;
static size_t thpsizes[20];
static int nr_hugetlbsizes;
static size_t hugetlbsizes[10];
static unsigned long hugetlbsizes[10];
static int gup_fd;
static bool has_huge_zeropage;
@ -202,7 +202,7 @@ static void do_test_cow_in_parent(char *mem, size_t size, bool do_mprotect,
log_test_result(KSFT_FAIL);
goto close_comm_pipes;
} else if (!ret) {
exit(fn(mem, size, &comm_pipes));
_exit(fn(mem, size, &comm_pipes));
}
while (read(comm_pipes.child_ready[0], &buf, 1) != 1)
@ -333,7 +333,7 @@ static void do_test_vmsplice_in_parent(char *mem, size_t size,
;
/* Modify page content in the child. */
memset(mem, 0xff, size);
exit(0);
_exit(0);
}
if (!before_fork) {
@ -480,7 +480,7 @@ static void do_test_iouring(char *mem, size_t size, bool use_fork)
write(comm_pipes.child_ready[1], "0", 1);
while (read(comm_pipes.parent_ready[0], &buf, 1) != 1)
;
exit(0);
_exit(0);
}
while (read(comm_pipes.child_ready[0], &buf, 1) != 1)
@ -645,7 +645,7 @@ static void do_test_ro_pin(char *mem, size_t size, enum ro_pin_test test,
write(comm_pipes.child_ready[1], "0", 1);
while (read(comm_pipes.parent_ready[0], &buf, 1) != 1)
;
exit(0);
_exit(0);
}
/* Wait until our child is ready. */
@ -956,7 +956,7 @@ static void do_run_with_thp(test_fn fn, enum thp_run thp_run, size_t thpsize)
log_test_result(KSFT_FAIL);
goto munmap;
} else if (!ret) {
exit(0);
_exit(0);
}
wait(&ret);
/* Allow for sharing all pages again. */
@ -1347,13 +1347,13 @@ static void do_test_anon_thp_collapse(char *mem, size_t size,
switch (test) {
case ANON_THP_COLLAPSE_UNSHARED:
case ANON_THP_COLLAPSE_FULLY_SHARED:
exit(child_memcmp_fn(mem, size, &comm_pipes));
_exit(child_memcmp_fn(mem, size, &comm_pipes));
break;
case ANON_THP_COLLAPSE_LOWER_SHARED:
exit(child_memcmp_fn(mem, size / 2, &comm_pipes));
_exit(child_memcmp_fn(mem, size / 2, &comm_pipes));
break;
case ANON_THP_COLLAPSE_UPPER_SHARED:
exit(child_memcmp_fn(mem + size / 2, size / 2,
_exit(child_memcmp_fn(mem + size / 2, size / 2,
&comm_pipes));
break;
default:
@ -1881,21 +1881,21 @@ int main(int argc, char **argv)
ksft_print_header();
thp_save_settings();
pagesize = getpagesize();
pmdsize = read_pmd_pagesize();
if (pmdsize) {
/* Only if THP is supported. */
thp_read_settings(&default_settings);
default_settings.hugepages[sz2ord(pmdsize, pagesize)].enabled = THP_INHERIT;
thp_save_settings();
thp_push_settings(&default_settings);
ksft_print_msg("[INFO] detected PMD size: %zu KiB\n",
pmdsize / 1024);
nr_thpsizes = detect_thp_sizes(thpsizes, ARRAY_SIZE(thpsizes));
}
nr_hugetlbsizes = detect_hugetlb_page_sizes(hugetlbsizes,
ARRAY_SIZE(hugetlbsizes));
nr_hugetlbsizes = hugetlb_setup(2, hugetlbsizes, ARRAY_SIZE(hugetlbsizes));
has_huge_zeropage = detect_huge_zeropage();
ksft_set_plan(ARRAY_SIZE(anon_test_cases) * tests_per_anon_test_case() +
@ -1911,10 +1911,5 @@ int main(int argc, char **argv)
run_anon_thp_test_cases();
run_non_anon_test_cases();
if (pmdsize) {
/* Only if THP is supported. */
thp_restore_settings();
}
ksft_finished();
}

48
tools/testing/selftests/mm/droppable.c
View File

@ -17,10 +17,10 @@
int main(int argc, char *argv[])
{
size_t alloc_size = 134217728;
size_t page_size = getpagesize();
const size_t alloc_size = 2 * 1024 * 1024;
int retry_count = 10;
bool dropped;
void *alloc;
pid_t child;
ksft_print_header();
ksft_set_plan(1);
@ -35,26 +35,32 @@ int main(int argc, char *argv[])
exit(KSFT_FAIL);
}
memset(alloc, 'A', alloc_size);
for (size_t i = 0; i < alloc_size; i += page_size)
assert(*(uint8_t *)(alloc + i));
child = fork();
assert(child >= 0);
if (!child) {
for (;;)
*(char *)malloc(page_size) = 'B';
}
for (bool done = false; !done;) {
for (size_t i = 0; i < alloc_size; i += page_size) {
if (!*(uint8_t *)(alloc + i)) {
done = true;
break;
while (retry_count--) {
if (madvise(alloc, alloc_size, MADV_PAGEOUT)) {
if (errno == EINVAL) {
ksft_test_result_skip("madvise(MADV_PAGEOUT) not supported\n");
exit(KSFT_SKIP);
}
ksft_test_result_fail("madvise(MADV_PAGEOUT) error: %s\n", strerror(errno));
exit(KSFT_FAIL);
}
}
kill(child, SIGTERM);
ksft_test_result_pass("MAP_DROPPABLE: PASS\n");
exit(KSFT_PASS);
dropped = memchr(alloc, 'A', alloc_size) == NULL;
/*
* Speculative reference can temporarily prevent some
* pages from getting dropped. So sleep and retry.
*
* If a page is not droppable for 10s, something
* is seriously messed up and we want to fail.
*/
if (dropped)
break;
sleep(1);
}
ksft_test_result(dropped, "madvise(MADV_PAGEOUT) behavior\n");
ksft_finished();
}

25
tools/testing/selftests/mm/folio_split_race_test.c
View File

@ -25,7 +25,7 @@
#include <unistd.h>
#include "vm_util.h"
#include "kselftest.h"
#include "thp_settings.h"
#include "hugepage_settings.h"
uint64_t page_size;
uint64_t pmd_pagesize;
@ -226,23 +226,6 @@ static uint64_t run_iteration(void)
return reader_failures;
}
static void thp_cleanup_handler(int signum)
{
thp_restore_settings();
/*
* Restore default handler and re-raise the signal to exit.
* This is to ensure the test process exits with the correct
* status code corresponding to the signal.
*/
signal(signum, SIG_DFL);
raise(signum);
}
static void thp_settings_cleanup(void)
{
thp_restore_settings();
}
int main(void)
{
struct thp_settings current_settings;
@ -261,12 +244,6 @@ int main(void)
ksft_exit_skip("Please run the test as root\n");
thp_save_settings();
/* make sure thp settings are restored */
if (atexit(thp_settings_cleanup) != 0)
ksft_exit_fail_msg("atexit failed\n");
signal(SIGINT, thp_cleanup_handler);
signal(SIGTERM, thp_cleanup_handler);
thp_read_settings(&current_settings);
current_settings.shmem_enabled = SHMEM_ADVISE;

20
tools/testing/selftests/mm/guard-regions.c
View File

@ -21,7 +21,7 @@
#include <sys/uio.h>
#include <unistd.h>
#include "vm_util.h"
#include "thp_settings.h"
#include "hugepage_settings.h"
#include "../pidfd/pidfd.h"
@ -2203,17 +2203,6 @@ TEST_F(guard_regions, collapse)
if (variant->backing != ANON_BACKED)
ASSERT_EQ(ftruncate(self->fd, size), 0);
/*
* We must close and re-open local-file backed as read-only for
* CONFIG_READ_ONLY_THP_FOR_FS to work.
*/
if (variant->backing == LOCAL_FILE_BACKED) {
ASSERT_EQ(close(self->fd), 0);
self->fd = open(self->path, O_RDONLY);
ASSERT_GE(self->fd, 0);
}
ptr = mmap_(self, variant, NULL, size, PROT_READ, 0, 0);
ASSERT_NE(ptr, MAP_FAILED);
@ -2237,9 +2226,10 @@ TEST_F(guard_regions, collapse)
/*
* Now collapse the entire region. This should fail in all cases.
*
* The madvise() call will also fail if CONFIG_READ_ONLY_THP_FOR_FS is
* not set for the local file case, but we can't differentiate whether
* this occurred or if the collapse was rightly rejected.
* The madvise() call will also fail if the file system does not support
* large folio or the supported orders do not include PMD_ORDER for the
* local file case, but we can't differentiate whether this occurred or
* if the collapse was rightly rejected.
*/
EXPECT_NE(madvise(ptr, size, MADV_COLLAPSE), 0);

5
tools/testing/selftests/mm/gup_longterm.c
View File

@ -29,10 +29,11 @@
#include "../../../../mm/gup_test.h"
#include "kselftest.h"
#include "vm_util.h"
#include "hugepage_settings.h"
static size_t pagesize;
static int nr_hugetlbsizes;
static size_t hugetlbsizes[10];
static unsigned long hugetlbsizes[10];
static int gup_fd;
static __fsword_t get_fs_type(int fd)
@ -509,7 +510,7 @@ int main(int argc, char **argv)
int i;
pagesize = getpagesize();
nr_hugetlbsizes = detect_hugetlb_page_sizes(hugetlbsizes,
nr_hugetlbsizes = hugetlb_setup(2, hugetlbsizes,
ARRAY_SIZE(hugetlbsizes));
ksft_print_header();

15
tools/testing/selftests/mm/gup_test.c
View File

@ -14,6 +14,7 @@
#include <mm/gup_test.h>
#include "kselftest.h"
#include "vm_util.h"
#include "hugepage_settings.h"
#define MB (1UL << 20)
@ -94,6 +95,7 @@ int main(int argc, char **argv)
int filed, i, opt, nr_pages = 1, thp = -1, write = 1, nthreads = 1, ret;
int flags = MAP_PRIVATE;
char *file = "/dev/zero";
bool hugetlb = false;
pthread_t *tid;
char *p;
@ -168,6 +170,7 @@ int main(int argc, char **argv)
break;
case 'H':
flags |= (MAP_HUGETLB | MAP_ANONYMOUS);
hugetlb = true;
break;
default:
ksft_exit_fail_msg("Wrong argument\n");
@ -199,6 +202,18 @@ int main(int argc, char **argv)
}
ksft_print_header();
if (hugetlb) {
unsigned long hp_size = default_huge_page_size();
if (!hp_size)
ksft_exit_skip("HugeTLB is unavailable\n");
size = (size + hp_size - 1) & ~(hp_size - 1);
if (!hugetlb_setup_default(size / hp_size))
ksft_exit_skip("Not enough huge pages\n");
}
ksft_set_plan(nthreads);
filed = open(file, O_RDWR|O_CREAT, 0664);

149
tools/testing/selftests/mm/hmm-tests.c
View File

@ -11,6 +11,7 @@
*/
#include "kselftest_harness.h"
#include "hugepage_settings.h"
#include <errno.h>
#include <fcntl.h>
@ -21,13 +22,13 @@
#include <strings.h>
#include <time.h>
#include <pthread.h>
#include <limits.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <sys/mman.h>
#include <sys/ioctl.h>
#include <sys/time.h>
/*
* This is a private UAPI to the kernel test module so it isn't exported
* in the usual include/uapi/... directory.
@ -69,6 +70,9 @@ enum {
#ifndef FOLL_LONGTERM
#define FOLL_LONGTERM 0x100 /* mapping lifetime is indefinite */
#endif
HUGETLB_SETUP_DEFAULT_PAGES(1)
FIXTURE(hmm)
{
int fd;
@ -632,7 +636,7 @@ TEST_F(hmm, anon_write_child)
}
close(child_fd);
exit(0);
_exit(0);
}
}
}
@ -712,7 +716,7 @@ TEST_F(hmm, anon_write_child_shared)
ASSERT_EQ(ptr[i], -i);
close(child_fd);
exit(0);
_exit(0);
}
/*
@ -784,8 +788,8 @@ TEST_F(hmm, anon_write_hugetlbfs)
int *ptr;
int ret;
if (!default_hsize)
SKIP(return, "Huge page size could not be determined");
if (!hugetlb_free_default_pages())
SKIP(return, "Not enough huge pages");
size = ALIGN(TWOMEG, default_hsize);
npages = size >> self->page_shift;
@ -1599,8 +1603,8 @@ TEST_F(hmm2, snapshot)
}
/*
* Test the hmm_range_fault() HMM_PFN_PMD flag for large pages that
* should be mapped by a large page table entry.
* Test the hmm_range_fault() handling of large pages (PMD or PUD)
* that should be mapped by a large page table entry.
*/
TEST_F(hmm, compound)
{
@ -1610,13 +1614,13 @@ TEST_F(hmm, compound)
unsigned long default_hsize = default_huge_page_size();
int *ptr;
unsigned char *m;
unsigned char prot;
int ret;
unsigned long i;
/* Skip test if we can't allocate a hugetlbfs page. */
if (!default_hsize)
SKIP(return, "Huge page size could not be determined");
if (!hugetlb_free_default_pages())
SKIP(return, "Not enough huge pages");
size = ALIGN(TWOMEG, default_hsize);
npages = size >> self->page_shift;
@ -1646,11 +1650,20 @@ TEST_F(hmm, compound)
ASSERT_EQ(ret, 0);
ASSERT_EQ(buffer->cpages, npages);
/* Check what the device saw. */
/*
* Check what the device saw. The region is backed by a single huge
* page that the device reports either at PMD or at PUD level depending
* on the configured default hugepage size. Determine that level from
* the first page and require every page in the range to match it
* exactly, so that a fragmented mapping mixing levels (or a missing
* large-page bit) is still caught and reported with its actual value.
*/
m = buffer->mirror;
prot = HMM_DMIRROR_PROT_WRITE |
((m[0] & HMM_DMIRROR_PROT_PUD) ? HMM_DMIRROR_PROT_PUD :
HMM_DMIRROR_PROT_PMD);
for (i = 0; i < npages; ++i)
ASSERT_EQ(m[i], HMM_DMIRROR_PROT_WRITE |
HMM_DMIRROR_PROT_PMD);
ASSERT_EQ(m[i], prot);
/* Make the region read-only. */
ret = mprotect(buffer->ptr, size, PROT_READ);
@ -1661,11 +1674,17 @@ TEST_F(hmm, compound)
ASSERT_EQ(ret, 0);
ASSERT_EQ(buffer->cpages, npages);
/* Check what the device saw. */
/*
* Check what the device saw after mprotect(PROT_READ). Same
* approach as above: determine the mapping level from the first
* page and require every page to match it exactly.
*/
m = buffer->mirror;
prot = HMM_DMIRROR_PROT_READ |
((m[0] & HMM_DMIRROR_PROT_PUD) ? HMM_DMIRROR_PROT_PUD :
HMM_DMIRROR_PROT_PMD);
for (i = 0; i < npages; ++i)
ASSERT_EQ(m[i], HMM_DMIRROR_PROT_READ |
HMM_DMIRROR_PROT_PMD);
ASSERT_EQ(m[i], prot);
munmap(buffer->ptr, buffer->size);
buffer->ptr = NULL;
@ -1865,6 +1884,8 @@ TEST_F(hmm, exclusive_cow)
unsigned long i;
int *ptr;
int ret;
pid_t pid;
int status;
npages = ALIGN(HMM_BUFFER_SIZE, self->page_size) >> self->page_shift;
ASSERT_NE(npages, 0);
@ -1893,14 +1914,37 @@ TEST_F(hmm, exclusive_cow)
ASSERT_EQ(ret, 0);
ASSERT_EQ(buffer->cpages, npages);
fork();
pid = fork();
if (pid == -1)
ASSERT_EQ(pid, 0);
/* Fault pages back to system memory and check them. */
if (pid == 0) {
/*
* Child verifies COW independently, then _exit(0)s so it does
* not run the test teardown. A failed ASSERT_* here makes the
* harness abort() the child, so the parent sees
* !WIFEXITED(status) below and fails in turn.
*/
for (i = 0, ptr = buffer->ptr; i < size / sizeof(*ptr); ++i)
ASSERT_EQ(ptr[i]++, i);
for (i = 0, ptr = buffer->ptr; i < size / sizeof(*ptr); ++i)
ASSERT_EQ(ptr[i], i + 1);
_exit(0);
}
/* Parent: also increment to verify COW works for both processes. */
for (i = 0, ptr = buffer->ptr; i < size / sizeof(*ptr); ++i)
ASSERT_EQ(ptr[i]++, i);
for (i = 0, ptr = buffer->ptr; i < size / sizeof(*ptr); ++i)
ASSERT_EQ(ptr[i], i+1);
ASSERT_EQ(ptr[i], i + 1);
/* Parent: wait for child and then free the buffer. */
ASSERT_EQ(waitpid(pid, &status, 0), pid);
ASSERT_TRUE(WIFEXITED(status));
ASSERT_EQ(WEXITSTATUS(status), 0);
hmm_buffer_free(buffer);
}
@ -2062,7 +2106,7 @@ TEST_F(hmm, hmm_cow_in_device)
if (pid == -1)
ASSERT_EQ(pid, 0);
if (!pid) {
/* Child process waits for SIGTERM from the parent. */
/* Child process waits for SIGKILL from the parent. */
while (1) {
}
/* Should not reach this */
@ -2075,10 +2119,10 @@ TEST_F(hmm, hmm_cow_in_device)
ptr[i] = i;
/* Terminate child and wait */
EXPECT_EQ(0, kill(pid, SIGTERM));
EXPECT_EQ(0, kill(pid, SIGKILL));
EXPECT_EQ(pid, waitpid(pid, &status, 0));
EXPECT_NE(0, WIFSIGNALED(status));
EXPECT_EQ(SIGTERM, WTERMSIG(status));
EXPECT_EQ(SIGKILL, WTERMSIG(status));
/* Take snapshot to CPU pagetables */
ret = hmm_dmirror_cmd(self->fd, HMM_DMIRROR_SNAPSHOT, buffer, npages);
@ -2362,12 +2406,21 @@ TEST_F(hmm, migrate_partial_unmap_fault)
struct hmm_buffer *buffer;
unsigned long npages;
unsigned long size = read_pmd_pagesize();
unsigned long unmap_size;
unsigned long offsets[3];
unsigned long i;
void *old_ptr;
void *map;
int *ptr;
int ret, j, use_thp;
int offsets[] = { 0, 512 * ONEKB, ONEMEG };
if (!size)
size = TWOMEG;
unmap_size = size / 2;
offsets[0] = 0;
offsets[1] = size / 4;
offsets[2] = size / 2;
for (use_thp = 0; use_thp < 2; ++use_thp) {
for (j = 0; j < ARRAY_SIZE(offsets); ++j) {
@ -2409,12 +2462,12 @@ TEST_F(hmm, migrate_partial_unmap_fault)
for (i = 0, ptr = buffer->mirror; i < size / sizeof(*ptr); ++i)
ASSERT_EQ(ptr[i], i);
munmap(buffer->ptr + offsets[j], ONEMEG);
munmap(buffer->ptr + offsets[j], unmap_size);
/* Fault pages back to system memory and check them. */
for (i = 0, ptr = buffer->ptr; i < size / sizeof(*ptr); ++i)
if (i * sizeof(int) < offsets[j] ||
i * sizeof(int) >= offsets[j] + ONEMEG)
i * sizeof(int) >= offsets[j] + unmap_size)
ASSERT_EQ(ptr[i], i);
buffer->ptr = old_ptr;
@ -2428,12 +2481,19 @@ TEST_F(hmm, migrate_remap_fault)
struct hmm_buffer *buffer;
unsigned long npages;
unsigned long size = read_pmd_pagesize();
unsigned long offsets[3];
unsigned long i;
void *old_ptr, *new_ptr = NULL;
void *map;
int *ptr;
int ret, j, use_thp, dont_unmap, before;
int offsets[] = { 0, 512 * ONEKB, ONEMEG };
if (!size)
size = TWOMEG;
offsets[0] = 0;
offsets[1] = size / 4;
offsets[2] = size / 2;
for (before = 0; before < 2; ++before) {
for (dont_unmap = 0; dont_unmap < 2; ++dont_unmap) {
@ -2836,38 +2896,45 @@ static inline int run_migration_benchmark(int fd, int use_thp, size_t buffer_siz
TEST_F_TIMEOUT(hmm, benchmark_thp_migration, 120)
{
struct benchmark_results thp_results, regular_results;
size_t thp_size = 2 * 1024 * 1024; /* 2MB - typical THP size */
size_t thp_size = read_pmd_pagesize();
int iterations = 5;
if (!thp_size)
thp_size = TWOMEG;
printf("\nHMM THP Migration Benchmark\n");
printf("---------------------------\n");
printf("System page size: %ld bytes\n", sysconf(_SC_PAGESIZE));
/* Test different buffer sizes */
size_t test_sizes[] = {
thp_size / 4, /* 512KB - smaller than THP */
thp_size / 2, /* 1MB - half THP */
thp_size, /* 2MB - single THP */
thp_size * 2, /* 4MB - two THPs */
thp_size * 4, /* 8MB - four THPs */
thp_size * 8, /* 16MB - eight THPs */
thp_size * 128, /* 256MB - one twenty eight THPs */
thp_size / 4, /* quarter THP */
thp_size / 2, /* half THP */
thp_size, /* single THP */
thp_size * 2, /* two THPs */
thp_size * 4, /* four THPs */
thp_size * 8, /* eight THPs */
thp_size * 128, /* one twenty eight THPs */
};
static const char *const test_names[] = {
"Small Buffer (512KB)",
"Half THP Size (1MB)",
"Single THP Size (2MB)",
"Two THP Size (4MB)",
"Four THP Size (8MB)",
"Eight THP Size (16MB)",
"One twenty eight THP Size (256MB)"
"Small Buffer",
"Half THP Size",
"Single THP Size",
"Two THP Size",
"Four THP Size",
"Eight THP Size",
"One twenty eight THP Size"
};
int num_tests = ARRAY_SIZE(test_sizes);
/* Run all tests */
for (int i = 0; i < num_tests; i++) {
/* Skip test sizes exceeding INT_MAX to avoid overflow */
if (test_sizes[i] > INT_MAX)
break;
/* Test with THP */
ASSERT_EQ(run_migration_benchmark(self->fd, 1, test_sizes[i],
iterations, &thp_results), 0);

78
tools/testing/selftests/mm/hugepage-mmap.c
View File

@ -1,78 +0,0 @@
// SPDX-License-Identifier: GPL-2.0
/*
* hugepage-mmap:
*
* Example of using huge page memory in a user application using the mmap
* system call. Before running this application, make sure that the
* administrator has mounted the hugetlbfs filesystem (on some directory
* like /mnt) using the command mount -t hugetlbfs nodev /mnt. In this
* example, the app is requesting memory of size 256MB that is backed by
* huge pages.
*/
#define _GNU_SOURCE
#include <stdlib.h>
#include <stdio.h>
#include <unistd.h>
#include <sys/mman.h>
#include <fcntl.h>
#include "kselftest.h"
#define LENGTH (256UL*1024*1024)
#define PROTECTION (PROT_READ | PROT_WRITE)
static void check_bytes(char *addr)
{
ksft_print_msg("First hex is %x\n", *((unsigned int *)addr));
}
static void write_bytes(char *addr)
{
unsigned long i;
for (i = 0; i < LENGTH; i++)
*(addr + i) = (char)i;
}
static int read_bytes(char *addr)
{
unsigned long i;
check_bytes(addr);
for (i = 0; i < LENGTH; i++)
if (*(addr + i) != (char)i) {
ksft_print_msg("Error: Mismatch at %lu\n", i);
return 1;
}
return 0;
}
int main(void)
{
void *addr;
int fd, ret;
ksft_print_header();
ksft_set_plan(1);
fd = memfd_create("hugepage-mmap", MFD_HUGETLB);
if (fd < 0)
ksft_exit_fail_msg("memfd_create() failed: %s\n", strerror(errno));
addr = mmap(NULL, LENGTH, PROTECTION, MAP_SHARED, fd, 0);
if (addr == MAP_FAILED) {
close(fd);
ksft_exit_fail_msg("mmap(): %s\n", strerror(errno));
}
ksft_print_msg("Returned address is %p\n", addr);
check_bytes(addr);
write_bytes(addr);
ret = read_bytes(addr);
munmap(addr, LENGTH);
close(fd);
ksft_test_result(!ret, "Read same data\n");
ksft_exit(!ret);
}

316
tools/testing/selftests/mm/thp_settings.c → tools/testing/selftests/mm/hugepage_settings.c
View File

@ -1,13 +1,16 @@
// SPDX-License-Identifier: GPL-2.0
#include <dirent.h>
#include <fcntl.h>
#include <limits.h>
#include <signal.h>
#include <stddef.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include "vm_util.h"
#include "thp_settings.h"
#include "hugepage_settings.h"
#define THP_SYSFS "/sys/kernel/mm/transparent_hugepage/"
#define MAX_SETTINGS_DEPTH 4
@ -15,6 +18,7 @@ static struct thp_settings settings_stack[MAX_SETTINGS_DEPTH];
static int settings_index;
static struct thp_settings saved_settings;
static char dev_queue_read_ahead_path[PATH_MAX];
static bool thp_settings_saved;
static const char * const thp_enabled_strings[] = {
"never",
@ -44,47 +48,6 @@ static const char * const shmem_enabled_strings[] = {
NULL
};
int read_file(const char *path, char *buf, size_t buflen)
{
int fd;
ssize_t numread;
fd = open(path, O_RDONLY);
if (fd == -1)
return 0;
numread = read(fd, buf, buflen - 1);
if (numread < 1) {
close(fd);
return 0;
}
buf[numread] = '\0';
close(fd);
return (unsigned int) numread;
}
unsigned long read_num(const char *path)
{
char buf[21];
if (read_file(path, buf, sizeof(buf)) < 0) {
perror("read_file()");
exit(EXIT_FAILURE);
}
return strtoul(buf, NULL, 10);
}
void write_num(const char *path, unsigned long num)
{
char buf[21];
sprintf(buf, "%ld", num);
write_file(path, buf, strlen(buf) + 1);
}
int thp_read_string(const char *name, const char * const strings[])
{
char path[PATH_MAX];
@ -298,12 +261,20 @@ void thp_pop_settings(void)
void thp_restore_settings(void)
{
thp_write_settings(&saved_settings);
if (thp_settings_saved)
thp_write_settings(&saved_settings);
}
void thp_save_settings(void)
static void __thp_save_settings(void)
{
if (!thp_available())
return;
if (thp_settings_saved)
return;
thp_read_settings(&saved_settings);
thp_settings_saved = true;
}
void thp_set_read_ahead_path(char *path)
@ -370,3 +341,260 @@ bool thp_is_enabled(void)
/* THP is considered enabled if it's either "always" or "madvise" */
return mode == 1 || mode == 3;
}
#define HUGETLB_MAX_NR_PAGESIZES 10
struct hugetlb_settings {
unsigned long nr_hugepages[HUGETLB_MAX_NR_PAGESIZES];
unsigned long sizes[HUGETLB_MAX_NR_PAGESIZES];
unsigned long default_size;
int nr_sizes;
};
static struct hugetlb_settings hugetlb_saved_settings;
static bool hugetlb_settings_saved;
int detect_hugetlb_page_sizes(unsigned long sizes[], int max)
{
static struct hugetlb_settings *settings = &hugetlb_saved_settings;
DIR *dir;
int count = 0;
if (settings->nr_sizes) {
if (settings->nr_sizes < max)
max = settings->nr_sizes;
for (count = 0; count < max; count++)
sizes[count] = settings->sizes[count];
return count;
}
dir = opendir("/sys/kernel/mm/hugepages/");
if (!dir)
return 0;
while (count < max) {
struct dirent *entry = readdir(dir);
size_t kb;
if (!entry)
break;
if (entry->d_type != DT_DIR)
continue;
if (sscanf(entry->d_name, "hugepages-%zukB", &kb) != 1)
continue;
sizes[count++] = kb * 1024;
ksft_print_msg("[INFO] detected hugetlb page size: %zu KiB\n",
kb);
}
closedir(dir);
return count;
}
unsigned long default_huge_page_size(void)
{
static struct hugetlb_settings *settings = &hugetlb_saved_settings;
unsigned long hps = 0;
char *line = NULL;
size_t linelen = 0;
FILE *f;
if (settings->default_size)
return settings->default_size;
f = fopen("/proc/meminfo", "r");
if (!f)
return 0;
while (getline(&line, &linelen, f) > 0) {
if (sscanf(line, "Hugepagesize: %lu kB", &hps) == 1) {
hps <<= 10;
break;
}
}
free(line);
fclose(f);
return hps;
}
static void hugetlb_sysfs_path(char *buf, size_t buflen,
unsigned long size, const char *attr)
{
snprintf(buf, buflen, "/sys/kernel/mm/hugepages/hugepages-%lukB/%s",
size / 1024, attr);
}
unsigned long hugetlb_nr_pages(unsigned long size)
{
char path[PATH_MAX];
hugetlb_sysfs_path(path, sizeof(path), size, "nr_hugepages");
return read_num(path);
}
void hugetlb_set_nr_pages(unsigned long size, unsigned long nr)
{
char path[PATH_MAX];
hugetlb_sysfs_path(path, sizeof(path), size, "nr_hugepages");
write_num(path, nr);
}
unsigned long hugetlb_free_pages(unsigned long size)
{
char path[PATH_MAX];
hugetlb_sysfs_path(path, sizeof(path), size, "free_hugepages");
return read_num(path);
}
static bool __hugetlb_setup(unsigned long size, unsigned long nr)
{
unsigned long free = hugetlb_free_pages(size);
unsigned long total = hugetlb_nr_pages(size);
if (free >= nr)
return true;
hugetlb_set_nr_pages(size, total + (nr - free));
return hugetlb_free_pages(size) >= nr;
}
bool hugetlb_setup_default(unsigned long nr)
{
unsigned long size;
hugetlb_save_settings();
size = default_huge_page_size();
if (!size)
return false;
return __hugetlb_setup(size, nr);
}
bool hugetlb_setup_default_exact(unsigned long nr)
{
unsigned long size;
hugetlb_save_settings();
size = default_huge_page_size();
if (!size)
return false;
hugetlb_set_nr_pages(size, nr);
return hugetlb_free_pages(size) == nr;
}
unsigned long hugetlb_setup(unsigned long nr, unsigned long sizes[],
int max)
{
unsigned long enabled[10];
int nr_sizes = 0;
int nr_enabled;
hugetlb_save_settings();
nr_enabled = detect_hugetlb_page_sizes(enabled, ARRAY_SIZE(enabled));
if (!nr_enabled)
return 0;
if (nr_enabled > max) {
ksft_print_msg("detected %d huge page sizes, will only test %d\n", nr_enabled, max);
nr_enabled = max;
}
/* request nr HugeTLB pages of every size. */
for (int i = 0; i < nr_enabled; i++) {
if (!__hugetlb_setup(enabled[i], nr))
continue;
sizes[nr_sizes++] = enabled[i];
}
return nr_sizes;
}
static void __hugetlb_save_settings(void)
{
struct hugetlb_settings *settings = &hugetlb_saved_settings;
int nr_sizes;
if (hugetlb_settings_saved)
return;
settings->default_size = default_huge_page_size();
if (!settings->default_size)
return;
nr_sizes = detect_hugetlb_page_sizes(settings->sizes,
HUGETLB_MAX_NR_PAGESIZES);
if (!nr_sizes) {
settings->default_size = 0;
return;
}
for (int i = 0; i < nr_sizes; i++) {
unsigned long sz = settings->sizes[i];
if (!sz)
continue;
settings->nr_hugepages[i] = hugetlb_nr_pages(sz);
}
settings->nr_sizes = nr_sizes;
hugetlb_settings_saved = true;
}
void hugetlb_restore_settings(void)
{
struct hugetlb_settings *settings = &hugetlb_saved_settings;
if (!hugetlb_settings_saved || !settings->default_size)
return;
for (int i = 0; i < HUGETLB_MAX_NR_PAGESIZES; i++) {
unsigned long sz = settings->sizes[i];
if (!sz)
continue;
hugetlb_set_nr_pages(sz, settings->nr_hugepages[i]);
}
}
static void hugepage_restore_settings_atexit(void)
{
if (thp_settings_saved)
thp_restore_settings();
if (hugetlb_settings_saved)
hugetlb_restore_settings();
}
static void hugepage_restore_settings_sighandler(int sig)
{
/* exit() will invoke the hugepage_restore_settings_atexit handler. */
exit(KSFT_FAIL);
}
void hugepage_save_settings(bool thp, bool hugetlb)
{
if (!thp && !hugetlb)
return;
if (thp)
__thp_save_settings();
if (hugetlb)
__hugetlb_save_settings();
/*
* setup exit hooks to make sure THP and HugeTLB settings are
* restored on graceful and error exits and signals
*/
atexit(hugepage_restore_settings_atexit);
signal(SIGTERM, hugepage_restore_settings_sighandler);
signal(SIGINT, hugepage_restore_settings_sighandler);
signal(SIGHUP, hugepage_restore_settings_sighandler);
signal(SIGQUIT, hugepage_restore_settings_sighandler);
}

82
tools/testing/selftests/mm/thp_settings.h → tools/testing/selftests/mm/hugepage_settings.h
View File

@ -1,11 +1,15 @@
/* SPDX-License-Identifier: GPL-2.0 */
#ifndef __THP_SETTINGS_H__
#define __THP_SETTINGS_H__
#ifndef __HUGEPAGE_SETTINGS_H__
#define __HUGEPAGE_SETTINGS_H__
#include <stdbool.h>
#include <stddef.h>
#include <stdint.h>
void hugepage_save_settings(bool thp, bool hugetlb);
/* Transparent Huge Pages (THP) */
enum thp_enabled {
THP_NEVER,
THP_ALWAYS,
@ -62,10 +66,6 @@ struct thp_settings {
struct shmem_hugepages_settings shmem_hugepages[NR_ORDERS];
};
int read_file(const char *path, char *buf, size_t buflen);
unsigned long read_num(const char *path);
void write_num(const char *path, unsigned long num);
int thp_read_string(const char *name, const char * const strings[]);
void thp_write_string(const char *name, const char *val);
unsigned long thp_read_num(const char *name);
@ -77,7 +77,11 @@ struct thp_settings *thp_current_settings(void);
void thp_push_settings(struct thp_settings *settings);
void thp_pop_settings(void);
void thp_restore_settings(void);
void thp_save_settings(void);
static inline void thp_save_settings(void)
{
hugepage_save_settings(/* thp = */ true, /* hugetlb = */ false);
}
void thp_set_read_ahead_path(char *path);
unsigned long thp_supported_orders(void);
@ -86,4 +90,66 @@ unsigned long thp_shmem_supported_orders(void);
bool thp_available(void);
bool thp_is_enabled(void);
#endif /* __THP_SETTINGS_H__ */
/* HugeTLB */
int detect_hugetlb_page_sizes(unsigned long sizes[], int max);
unsigned long default_huge_page_size(void);
unsigned long hugetlb_nr_pages(unsigned long size);
void hugetlb_set_nr_pages(unsigned long size, unsigned long nr);
unsigned long hugetlb_free_pages(unsigned long size);
static inline void hugetlb_save_settings(void)
{
hugepage_save_settings(/* thp = */ false, /* hugetlb = */ true);
}
void hugetlb_restore_settings(void);
static inline unsigned long hugetlb_nr_default_pages(void)
{
unsigned long size = default_huge_page_size();
if (!size)
return 0;
return hugetlb_nr_pages(size);
}
static inline void hugetlb_set_nr_default_pages(unsigned long nr)
{
unsigned long size = default_huge_page_size();
if (!size)
return;
hugetlb_set_nr_pages(size, nr);
}
static inline unsigned long hugetlb_free_default_pages(void)
{
unsigned long size = default_huge_page_size();
if (!size)
return 0;
return hugetlb_free_pages(size);
}
static inline bool hugetlb_available(void)
{
return default_huge_page_size() != 0;
}
bool hugetlb_setup_default(unsigned long nr);
bool hugetlb_setup_default_exact(unsigned long nr);
unsigned long hugetlb_setup(unsigned long nr, unsigned long sizes[],
int max);
#define HUGETLB_SETUP_DEFAULT_PAGES(nr_pages) \
static void __attribute__((constructor)) __hugetlb_setup_default(void) \
{ \
hugetlb_setup_default((nr_pages)); \
}
#endif /* __HUGEPAGE_SETTINGS_H__ */

219
tools/testing/selftests/mm/hugetlb-madvise.c
View File

@ -1,15 +1,9 @@
// SPDX-License-Identifier: GPL-2.0
/*
* hugepage-madvise:
* hugetlb-madvise:
*
* Basic functional testing of madvise MADV_DONTNEED and MADV_REMOVE
* on hugetlb mappings.
*
* Before running this test, make sure the administrator has pre-allocated
* at least MIN_FREE_PAGES hugetlb pages and they are free. In addition,
* the test takes an argument that is the path to a file in a hugetlbfs
* filesystem. Therefore, a hugetlbfs filesystem must be mounted on some
* directory.
*/
#define _GNU_SOURCE
@ -20,18 +14,18 @@
#include <fcntl.h>
#include "vm_util.h"
#include "kselftest.h"
#include "hugepage_settings.h"
#define MIN_FREE_PAGES 20
#define NR_HUGE_PAGES 10 /* common number of pages to map/allocate */
#define validate_free_pages(exp_free) \
do { \
int fhp = get_free_hugepages(); \
if (fhp != (exp_free)) { \
printf("Unexpected number of free huge " \
"pages line %d\n", __LINE__); \
exit(1); \
} \
unsigned long fhp = hugetlb_free_default_pages(); \
if (fhp != (exp_free)) \
ksft_exit_fail_msg("Unexpected number of free " \
"huge pages %lu, expected %lu line %d\n", \
fhp, (exp_free), __LINE__); \
} while (0)
unsigned long huge_page_size;
@ -57,28 +51,24 @@ int main(int argc, char **argv)
int fd;
int ret;
huge_page_size = default_huge_page_size();
if (!huge_page_size) {
printf("Unable to determine huge page size, exiting!\n");
exit(1);
}
base_page_size = sysconf(_SC_PAGE_SIZE);
if (!huge_page_size) {
printf("Unable to determine base page size, exiting!\n");
exit(1);
}
ksft_print_header();
ksft_set_plan(1);
free_hugepages = get_free_hugepages();
if (free_hugepages < MIN_FREE_PAGES) {
printf("Not enough free huge pages to test, exiting!\n");
exit(KSFT_SKIP);
}
huge_page_size = default_huge_page_size();
if (!huge_page_size)
ksft_exit_skip("Unable to determine huge page size\n");
base_page_size = sysconf(_SC_PAGE_SIZE);
if (!base_page_size)
ksft_exit_fail_msg("Unable to determine base page size\n");
if (!hugetlb_setup_default(MIN_FREE_PAGES))
ksft_exit_skip("Not enough free huge pages (have %lu, need %d)\n", hugetlb_free_default_pages(), MIN_FREE_PAGES);
free_hugepages = hugetlb_free_default_pages();
fd = memfd_create(argv[0], MFD_HUGETLB);
if (fd < 0) {
perror("memfd_create() failed");
exit(1);
}
if (fd < 0)
ksft_exit_fail_perror("memfd_create");
/*
* Test validity of MADV_DONTNEED addr and length arguments. mmap
@ -90,16 +80,13 @@ int main(int argc, char **argv)
PROT_READ | PROT_WRITE,
MAP_PRIVATE | MAP_ANONYMOUS | MAP_HUGETLB,
-1, 0);
if (addr == MAP_FAILED) {
perror("mmap");
exit(1);
}
if (addr == MAP_FAILED)
ksft_exit_fail_perror("mmap");
if (munmap(addr, huge_page_size) ||
munmap(addr + (NR_HUGE_PAGES + 1) * huge_page_size,
huge_page_size)) {
perror("munmap");
exit(1);
}
huge_page_size))
ksft_exit_fail_perror("munmap");
addr = addr + huge_page_size;
write_fault_pages(addr, NR_HUGE_PAGES);
@ -108,20 +95,14 @@ int main(int argc, char **argv)
/* addr before mapping should fail */
ret = madvise(addr - base_page_size, NR_HUGE_PAGES * huge_page_size,
MADV_DONTNEED);
if (!ret) {
printf("Unexpected success of madvise call with invalid addr line %d\n",
__LINE__);
exit(1);
}
if (!ret)
ksft_exit_fail_msg("madvise with invalid addr unexpectedly succeeded line %d\n", __LINE__);
/* addr + length after mapping should fail */
ret = madvise(addr, (NR_HUGE_PAGES * huge_page_size) + base_page_size,
MADV_DONTNEED);
if (!ret) {
printf("Unexpected success of madvise call with invalid length line %d\n",
__LINE__);
exit(1);
}
if (!ret)
ksft_exit_fail_msg("madvise with invalid length unexpectedly succeeded line %d\n", __LINE__);
(void)munmap(addr, NR_HUGE_PAGES * huge_page_size);
@ -132,10 +113,9 @@ int main(int argc, char **argv)
PROT_READ | PROT_WRITE,
MAP_PRIVATE | MAP_ANONYMOUS | MAP_HUGETLB,
-1, 0);
if (addr == MAP_FAILED) {
perror("mmap");
exit(1);
}
if (addr == MAP_FAILED)
ksft_exit_fail_perror("mmap");
write_fault_pages(addr, NR_HUGE_PAGES);
validate_free_pages(free_hugepages - NR_HUGE_PAGES);
@ -143,19 +123,14 @@ int main(int argc, char **argv)
ret = madvise(addr + base_page_size,
NR_HUGE_PAGES * huge_page_size - base_page_size,
MADV_DONTNEED);
if (!ret) {
printf("Unexpected success of madvise call with unaligned start address %d\n",
__LINE__);
exit(1);
}
if (!ret)
ksft_exit_fail_msg("madvise with unaligned start unexpectedly succeeded line %d\n", __LINE__);
/* addr + length should be aligned down to huge page size */
if (madvise(addr,
((NR_HUGE_PAGES - 1) * huge_page_size) + base_page_size,
MADV_DONTNEED)) {
perror("madvise");
exit(1);
}
MADV_DONTNEED))
ksft_exit_fail_perror("madvise");
/* should free all but last page in mapping */
validate_free_pages(free_hugepages - 1);
@ -170,17 +145,14 @@ int main(int argc, char **argv)
PROT_READ | PROT_WRITE,
MAP_PRIVATE | MAP_ANONYMOUS | MAP_HUGETLB,
-1, 0);
if (addr == MAP_FAILED) {
perror("mmap");
exit(1);
}
if (addr == MAP_FAILED)
ksft_exit_fail_perror("mmap");
write_fault_pages(addr, NR_HUGE_PAGES);
validate_free_pages(free_hugepages - NR_HUGE_PAGES);
if (madvise(addr, NR_HUGE_PAGES * huge_page_size, MADV_DONTNEED)) {
perror("madvise");
exit(1);
}
if (madvise(addr, NR_HUGE_PAGES * huge_page_size, MADV_DONTNEED))
ksft_exit_fail_perror("madvise");
/* should free all pages in mapping */
validate_free_pages(free_hugepages);
@ -190,29 +162,25 @@ int main(int argc, char **argv)
/*
* Test MADV_DONTNEED on private mapping of hugetlb file
*/
if (fallocate(fd, 0, 0, NR_HUGE_PAGES * huge_page_size)) {
perror("fallocate");
exit(1);
}
if (fallocate(fd, 0, 0, NR_HUGE_PAGES * huge_page_size))
ksft_exit_fail_perror("fallocate");
validate_free_pages(free_hugepages - NR_HUGE_PAGES);
addr = mmap(NULL, NR_HUGE_PAGES * huge_page_size,
PROT_READ | PROT_WRITE,
MAP_PRIVATE, fd, 0);
if (addr == MAP_FAILED) {
perror("mmap");
exit(1);
}
if (addr == MAP_FAILED)
ksft_exit_fail_perror("mmap");
/* read should not consume any pages */
read_fault_pages(addr, NR_HUGE_PAGES);
validate_free_pages(free_hugepages - NR_HUGE_PAGES);
/* madvise should not free any pages */
if (madvise(addr, NR_HUGE_PAGES * huge_page_size, MADV_DONTNEED)) {
perror("madvise");
exit(1);
}
if (madvise(addr, NR_HUGE_PAGES * huge_page_size, MADV_DONTNEED))
ksft_exit_fail_perror("madvise");
validate_free_pages(free_hugepages - NR_HUGE_PAGES);
/* writes should allocate private pages */
@ -220,10 +188,9 @@ int main(int argc, char **argv)
validate_free_pages(free_hugepages - (2 * NR_HUGE_PAGES));
/* madvise should free private pages */
if (madvise(addr, NR_HUGE_PAGES * huge_page_size, MADV_DONTNEED)) {
perror("madvise");
exit(1);
}
if (madvise(addr, NR_HUGE_PAGES * huge_page_size, MADV_DONTNEED))
ksft_exit_fail_perror("madvise");
validate_free_pages(free_hugepages - NR_HUGE_PAGES);
/* writes should allocate private pages */
@ -238,10 +205,9 @@ int main(int argc, char **argv)
* implementation.
*/
if (fallocate(fd, FALLOC_FL_PUNCH_HOLE | FALLOC_FL_KEEP_SIZE,
0, NR_HUGE_PAGES * huge_page_size)) {
perror("fallocate");
exit(1);
}
0, NR_HUGE_PAGES * huge_page_size))
ksft_exit_fail_perror("fallocate");
validate_free_pages(free_hugepages);
(void)munmap(addr, NR_HUGE_PAGES * huge_page_size);
@ -249,29 +215,25 @@ int main(int argc, char **argv)
/*
* Test MADV_DONTNEED on shared mapping of hugetlb file
*/
if (fallocate(fd, 0, 0, NR_HUGE_PAGES * huge_page_size)) {
perror("fallocate");
exit(1);
}
if (fallocate(fd, 0, 0, NR_HUGE_PAGES * huge_page_size))
ksft_exit_fail_perror("fallocate");
validate_free_pages(free_hugepages - NR_HUGE_PAGES);
addr = mmap(NULL, NR_HUGE_PAGES * huge_page_size,
PROT_READ | PROT_WRITE,
MAP_SHARED, fd, 0);
if (addr == MAP_FAILED) {
perror("mmap");
exit(1);
}
if (addr == MAP_FAILED)
ksft_exit_fail_perror("mmap");
/* write should not consume any pages */
write_fault_pages(addr, NR_HUGE_PAGES);
validate_free_pages(free_hugepages - NR_HUGE_PAGES);
/* madvise should not free any pages */
if (madvise(addr, NR_HUGE_PAGES * huge_page_size, MADV_DONTNEED)) {
perror("madvise");
exit(1);
}
if (madvise(addr, NR_HUGE_PAGES * huge_page_size, MADV_DONTNEED))
ksft_exit_fail_perror("madvise");
validate_free_pages(free_hugepages - NR_HUGE_PAGES);
/*
@ -279,29 +241,25 @@ int main(int argc, char **argv)
*
* madvise is same as hole punch and should free all pages.
*/
if (madvise(addr, NR_HUGE_PAGES * huge_page_size, MADV_REMOVE)) {
perror("madvise");
exit(1);
}
if (madvise(addr, NR_HUGE_PAGES * huge_page_size, MADV_REMOVE))
ksft_exit_fail_perror("madvise");
validate_free_pages(free_hugepages);
(void)munmap(addr, NR_HUGE_PAGES * huge_page_size);
/*
* Test MADV_REMOVE on shared and private mapping of hugetlb file
*/
if (fallocate(fd, 0, 0, NR_HUGE_PAGES * huge_page_size)) {
perror("fallocate");
exit(1);
}
if (fallocate(fd, 0, 0, NR_HUGE_PAGES * huge_page_size))
ksft_exit_fail_perror("fallocate");
validate_free_pages(free_hugepages - NR_HUGE_PAGES);
addr = mmap(NULL, NR_HUGE_PAGES * huge_page_size,
PROT_READ | PROT_WRITE,
MAP_SHARED, fd, 0);
if (addr == MAP_FAILED) {
perror("mmap");
exit(1);
}
if (addr == MAP_FAILED)
ksft_exit_fail_perror("mmap");
/* shared write should not consume any additional pages */
write_fault_pages(addr, NR_HUGE_PAGES);
@ -310,10 +268,8 @@ int main(int argc, char **argv)
addr2 = mmap(NULL, NR_HUGE_PAGES * huge_page_size,
PROT_READ | PROT_WRITE,
MAP_PRIVATE, fd, 0);
if (addr2 == MAP_FAILED) {
perror("mmap");
exit(1);
}
if (addr2 == MAP_FAILED)
ksft_exit_fail_perror("mmap");
/* private read should not consume any pages */
read_fault_pages(addr2, NR_HUGE_PAGES);
@ -324,17 +280,15 @@ int main(int argc, char **argv)
validate_free_pages(free_hugepages - (2 * NR_HUGE_PAGES));
/* madvise of shared mapping should not free any pages */
if (madvise(addr, NR_HUGE_PAGES * huge_page_size, MADV_DONTNEED)) {
perror("madvise");
exit(1);
}
if (madvise(addr, NR_HUGE_PAGES * huge_page_size, MADV_DONTNEED))
ksft_exit_fail_perror("madvise");
validate_free_pages(free_hugepages - (2 * NR_HUGE_PAGES));
/* madvise of private mapping should free private pages */
if (madvise(addr2, NR_HUGE_PAGES * huge_page_size, MADV_DONTNEED)) {
perror("madvise");
exit(1);
}
if (madvise(addr2, NR_HUGE_PAGES * huge_page_size, MADV_DONTNEED))
ksft_exit_fail_perror("madvise");
validate_free_pages(free_hugepages - NR_HUGE_PAGES);
/* private write should consume additional pages again */
@ -346,15 +300,16 @@ int main(int argc, char **argv)
* not correct. private pages should not be freed, but this is
* expected. See comment associated with FALLOC_FL_PUNCH_HOLE call.
*/
if (madvise(addr, NR_HUGE_PAGES * huge_page_size, MADV_REMOVE)) {
perror("madvise");
exit(1);
}
if (madvise(addr, NR_HUGE_PAGES * huge_page_size, MADV_REMOVE))
ksft_exit_fail_perror("madvise");
validate_free_pages(free_hugepages);
(void)munmap(addr, NR_HUGE_PAGES * huge_page_size);
(void)munmap(addr2, NR_HUGE_PAGES * huge_page_size);
close(fd);
return 0;
ksft_test_result_pass("MADV_DONTNEED and MADV_REMOVE on hugetlb\n");
ksft_finished();
}

143
tools/testing/selftests/mm/hugetlb-mmap.c Normal file
View File

@ -0,0 +1,143 @@
// SPDX-License-Identifier: GPL-2.0
/*
* hugetlb-mmap:
*
* Example of using huge page memory in a user application using the mmap
* system call. Before running this application, make sure that the
* administrator has mounted the hugetlbfs filesystem (on some directory
* like /mnt) using the command mount -t hugetlbfs nodev /mnt. In this
* example, the app is requesting memory of size 256MB that is backed by
* huge pages.
*/
#define _GNU_SOURCE
#include <stdlib.h>
#include <stdio.h>
#include <unistd.h>
#include <sys/mman.h>
#include <fcntl.h>
#include <linux/memfd.h>
#include "vm_util.h"
#include "kselftest.h"
#include "hugepage_settings.h"
#define LENGTH (256UL*1024*1024)
#define PROTECTION (PROT_READ | PROT_WRITE)
static void check_bytes(char *addr)
{
ksft_print_msg("First hex is %x\n", *((unsigned int *)addr));
}
static void write_bytes(char *addr, size_t length)
{
unsigned long i;
for (i = 0; i < length; i++)
*(addr + i) = (char)i;
}
static bool verify_bytes(char *addr, size_t length)
{
unsigned long i;
check_bytes(addr);
for (i = 0; i < length; i++)
if (*(addr + i) != (char)i) {
ksft_print_msg("Error: Mismatch at %lu(%p)\n", i, addr);
return false;
}
return true;
}
static void test_mmap(size_t length, int mmap_flags, int fd,
const char *test_name)
{
bool passed = true;
void *addr;
addr = mmap(NULL, length, PROTECTION, mmap_flags, fd, 0);
if (addr == MAP_FAILED)
ksft_exit_fail_perror("mmap");
ksft_print_msg("Returned address is %p\n", addr);
check_bytes(addr);
write_bytes(addr, length);
if (!verify_bytes(addr, length))
passed = false;
/* munmap() length of MAP_HUGETLB memory must be hugepage aligned */
if (munmap(addr, length))
ksft_exit_fail_perror("munmap");
ksft_test_result(passed, "%s\n", test_name);
}
static void test_anon_mmap(size_t length, int shift)
{
const char *test_name = "hugetlb anonymous mmap";
int mmap_flags = MAP_PRIVATE | MAP_ANONYMOUS | MAP_HUGETLB;
if (shift)
mmap_flags |= (shift & MAP_HUGE_MASK) << MAP_HUGE_SHIFT;
test_mmap(length, mmap_flags, -1, test_name);
}
static void test_file_mmap(size_t length, int shift)
{
const char *test_name = "hugetlb file mmap";
int mfd_flags = MFD_HUGETLB;
int fd;
if (shift)
mfd_flags |= (shift & MFD_HUGE_MASK) << MFD_HUGE_SHIFT;
fd = memfd_create("hugetlb-mmap", mfd_flags);
if (fd < 0)
ksft_exit_fail_perror("memfd_create");
test_mmap(length, MAP_SHARED, fd, test_name);
close(fd);
}
int main(int argc, char **argv)
{
size_t hugepage_size;
size_t length = LENGTH;
int shift = 0, nr;
ksft_print_header();
if (argc > 1)
length = atol(argv[1]) << 20;
if (argc > 2)
shift = atoi(argv[2]);
hugetlb_save_settings();
if (shift) {
hugepage_size = (1UL << shift);
ksft_print_msg("%lu kB hugepages\n", 1UL << (shift - 10));
} else {
hugepage_size = default_huge_page_size();
if (!hugepage_size)
ksft_exit_skip("Could not detect default hugetlb page size.");
ksft_print_msg("Default size hugepages (%lu kB)\n", hugepage_size >> 10);
}
/* munmap will fail if the length is not page aligned */
length = (length + hugepage_size - 1) & ~(hugepage_size - 1);
nr = length / hugepage_size;
hugetlb_set_nr_pages(hugepage_size, nr);
if (hugetlb_free_pages(hugepage_size) < nr)
ksft_exit_skip("Not enough %lu Kb pages\n", hugepage_size >> 10);
ksft_set_plan(2);
ksft_print_msg("Mapping %lu Mbytes\n", (unsigned long)length >> 20);
test_anon_mmap(length, shift);
test_file_mmap(length, shift);
ksft_finished();
}

47
tools/testing/selftests/mm/hugepage-mremap.c → tools/testing/selftests/mm/hugetlb-mremap.c
View File

@ -1,6 +1,6 @@
// SPDX-License-Identifier: GPL-2.0
/*
* hugepage-mremap:
* hugetlb-mremap:
*
* Example of remapping huge page memory in a user application using the
* mremap system call. The path to a file in a hugetlbfs filesystem must
@ -26,12 +26,13 @@
#include <stdbool.h>
#include "kselftest.h"
#include "vm_util.h"
#include "hugepage_settings.h"
#define DEFAULT_LENGTH_MB 10UL
#define MB_TO_BYTES(x) (x * 1024 * 1024)
#define PROTECTION (PROT_READ | PROT_WRITE | PROT_EXEC)
#define FLAGS (MAP_SHARED | MAP_ANONYMOUS)
#define FLAGS (MAP_HUGETLB | MAP_SHARED)
static void check_bytes(char *addr)
{
@ -85,31 +86,21 @@ static void register_region_with_uffd(char *addr, size_t len)
if (ioctl(uffd, UFFDIO_API, &uffdio_api) == -1)
ksft_exit_fail_msg("ioctl-UFFDIO_API: %s\n", strerror(errno));
/* Create a private anonymous mapping. The memory will be
* demand-zero paged--that is, not yet allocated. When we
* actually touch the memory, it will be allocated via
* the userfaultfd.
*/
addr = mmap(NULL, len, PROT_READ | PROT_WRITE,
MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
if (addr == MAP_FAILED)
ksft_exit_fail_msg("mmap: %s\n", strerror(errno));
ksft_print_msg("Address returned by mmap() = %p\n", addr);
/* Register the memory range of the mapping we just created for
* handling by the userfaultfd object. In mode, we request to track
* missing pages (i.e., pages that have not yet been faulted in).
/* Register the passed memory range for handling by the userfaultfd object.
* In mode, we request to track missing pages
* (i.e., pages that have not yet been faulted in).
*/
if (uffd_register(uffd, addr, len, true, false, false))
ksft_exit_fail_msg("ioctl-UFFDIO_REGISTER: %s\n", strerror(errno));
ksft_print_msg("Registered memory at address %p with userfaultfd\n", addr);
}
int main(int argc, char *argv[])
{
unsigned long hugepage_size;
int ret = 0, fd, nr;
size_t length = 0;
int ret = 0, fd;
ksft_print_header();
ksft_set_plan(1);
@ -125,30 +116,36 @@ int main(int argc, char *argv[])
else
length = DEFAULT_LENGTH_MB;
hugepage_size = default_huge_page_size();
if (!hugepage_size)
ksft_exit_skip("Could not detect default hugetlb page size\n");
length = MB_TO_BYTES(length);
length = (length + hugepage_size - 1) & ~(hugepage_size - 1);
nr = length / hugepage_size;
if (!hugetlb_setup_default(nr))
ksft_exit_skip("Not enough huge pages\n");
fd = memfd_create(argv[0], MFD_HUGETLB);
if (fd < 0)
ksft_exit_fail_msg("Open failed: %s\n", strerror(errno));
/* mmap to a PUD aligned address to hopefully trigger pmd sharing. */
unsigned long suggested_addr = 0x7eaa40000000;
void *haddr = mmap((void *)suggested_addr, length, PROTECTION,
MAP_HUGETLB | MAP_SHARED | MAP_POPULATE, fd, 0);
void *haddr = mmap((void *)suggested_addr, length, PROTECTION, FLAGS, fd, 0);
ksft_print_msg("Map haddr: Returned address is %p\n", haddr);
if (haddr == MAP_FAILED)
ksft_exit_fail_msg("mmap1: %s\n", strerror(errno));
/* mmap again to a dummy address to hopefully trigger pmd sharing. */
suggested_addr = 0x7daa40000000;
void *daddr = mmap((void *)suggested_addr, length, PROTECTION,
MAP_HUGETLB | MAP_SHARED | MAP_POPULATE, fd, 0);
void *daddr = mmap((void *)suggested_addr, length, PROTECTION, FLAGS, fd, 0);
ksft_print_msg("Map daddr: Returned address is %p\n", daddr);
if (daddr == MAP_FAILED)
ksft_exit_fail_msg("mmap3: %s\n", strerror(errno));
suggested_addr = 0x7faa40000000;
void *vaddr =
mmap((void *)suggested_addr, length, PROTECTION, FLAGS, -1, 0);
void *vaddr = mmap((void *)suggested_addr, length, PROTECTION, FLAGS, fd, 0);
ksft_print_msg("Map vaddr: Returned address is %p\n", vaddr);
if (vaddr == MAP_FAILED)
ksft_exit_fail_msg("mmap2: %s\n", strerror(errno));

121
tools/testing/selftests/mm/hugetlb-read-hwpoison.c
View File

@ -10,12 +10,10 @@
#include <sys/statfs.h>
#include <errno.h>
#include <stdbool.h>
#include <signal.h>
#include "kselftest.h"
#define PREFIX " ... "
#define ERROR_PREFIX " !!! "
#define MAX_WRITE_READ_CHUNK_SIZE (getpagesize() * 16)
#define MAX(a, b) (((a) > (b)) ? (a) : (b))
@ -25,17 +23,22 @@ enum test_status {
TEST_SKIPPED = 2,
};
static char *status_to_str(enum test_status status)
static void report_status(enum test_status status, const char *test_name,
size_t chunk_size)
{
switch (status) {
case TEST_PASSED:
return "TEST_PASSED";
ksft_test_result_pass("%s chunk_size=0x%lx\n",
test_name, chunk_size);
break;
case TEST_FAILED:
return "TEST_FAILED";
ksft_test_result_fail("%s chunk_size=0x%lx\n",
test_name, chunk_size);
break;
case TEST_SKIPPED:
return "TEST_SKIPPED";
default:
return "TEST_???";
ksft_test_result_skip("%s chunk_size=0x%lx\n",
test_name, chunk_size);
break;
}
}
@ -58,8 +61,8 @@ static bool verify_chunk(char *buf, size_t len, char val)
for (i = 0; i < len; ++i) {
if (buf[i] != val) {
printf(PREFIX ERROR_PREFIX "check fail: buf[%lu] = %u != %u\n",
i, buf[i], val);
ksft_print_msg("check fail: buf[%lu] = %u != %u\n",
i, buf[i], val);
return false;
}
}
@ -75,21 +78,21 @@ static bool seek_read_hugepage_filemap(int fd, size_t len, size_t wr_chunk_size,
ssize_t total_ret_count = 0;
char val = offset / wr_chunk_size + offset % wr_chunk_size;
printf(PREFIX PREFIX "init val=%u with offset=0x%lx\n", val, offset);
printf(PREFIX PREFIX "expect to read 0x%lx bytes of data in total\n",
expected);
ksft_print_msg("init val=%u with offset=0x%lx\n", val, offset);
ksft_print_msg("expect to read 0x%lx bytes of data in total\n",
expected);
if (lseek(fd, offset, SEEK_SET) < 0) {
perror(PREFIX ERROR_PREFIX "seek failed");
ksft_perror("seek failed");
return false;
}
while (offset + total_ret_count < len) {
ret_count = read(fd, buf, wr_chunk_size);
if (ret_count == 0) {
printf(PREFIX PREFIX "read reach end of the file\n");
ksft_print_msg("read reach end of the file\n");
break;
} else if (ret_count < 0) {
perror(PREFIX ERROR_PREFIX "read failed");
ksft_perror("read failed");
break;
}
++val;
@ -98,8 +101,8 @@ static bool seek_read_hugepage_filemap(int fd, size_t len, size_t wr_chunk_size,
total_ret_count += ret_count;
}
printf(PREFIX PREFIX "actually read 0x%lx bytes of data in total\n",
total_ret_count);
ksft_print_msg("actually read 0x%lx bytes of data in total\n",
total_ret_count);
return total_ret_count == expected;
}
@ -112,15 +115,15 @@ static bool read_hugepage_filemap(int fd, size_t len,
ssize_t total_ret_count = 0;
char val = 0;
printf(PREFIX PREFIX "expect to read 0x%lx bytes of data in total\n",
expected);
ksft_print_msg("expect to read 0x%lx bytes of data in total\n",
expected);
while (total_ret_count < len) {
ret_count = read(fd, buf, wr_chunk_size);
if (ret_count == 0) {
printf(PREFIX PREFIX "read reach end of the file\n");
ksft_print_msg("read reach end of the file\n");
break;
} else if (ret_count < 0) {
perror(PREFIX ERROR_PREFIX "read failed");
ksft_perror("read failed");
break;
}
++val;
@ -129,8 +132,8 @@ static bool read_hugepage_filemap(int fd, size_t len,
total_ret_count += ret_count;
}
printf(PREFIX PREFIX "actually read 0x%lx bytes of data in total\n",
total_ret_count);
ksft_print_msg("actually read 0x%lx bytes of data in total\n",
total_ret_count);
return total_ret_count == expected;
}
@ -142,14 +145,14 @@ test_hugetlb_read(int fd, size_t len, size_t wr_chunk_size)
char *filemap = NULL;
if (ftruncate(fd, len) < 0) {
perror(PREFIX ERROR_PREFIX "ftruncate failed");
ksft_perror("ftruncate failed");
return status;
}
filemap = mmap(NULL, len, PROT_READ | PROT_WRITE,
MAP_SHARED | MAP_POPULATE, fd, 0);
if (filemap == MAP_FAILED) {
perror(PREFIX ERROR_PREFIX "mmap for primary mapping failed");
ksft_perror("mmap for primary mapping failed");
goto done;
}
@ -162,7 +165,7 @@ test_hugetlb_read(int fd, size_t len, size_t wr_chunk_size)
munmap(filemap, len);
done:
if (ftruncate(fd, 0) < 0) {
perror(PREFIX ERROR_PREFIX "ftruncate back to 0 failed");
ksft_perror("ftruncate back to 0 failed");
status = TEST_FAILED;
}
@ -179,14 +182,14 @@ test_hugetlb_read_hwpoison(int fd, size_t len, size_t wr_chunk_size,
const unsigned long pagesize = getpagesize();
if (ftruncate(fd, len) < 0) {
perror(PREFIX ERROR_PREFIX "ftruncate failed");
ksft_perror("ftruncate failed");
return status;
}
filemap = mmap(NULL, len, PROT_READ | PROT_WRITE,
MAP_SHARED | MAP_POPULATE, fd, 0);
if (filemap == MAP_FAILED) {
perror(PREFIX ERROR_PREFIX "mmap for primary mapping failed");
ksft_perror("mmap for primary mapping failed");
goto done;
}
@ -201,7 +204,7 @@ test_hugetlb_read_hwpoison(int fd, size_t len, size_t wr_chunk_size,
*/
hwp_addr = filemap + len / 2 + pagesize;
if (madvise(hwp_addr, pagesize, MADV_HWPOISON) < 0) {
perror(PREFIX ERROR_PREFIX "MADV_HWPOISON failed");
ksft_perror("MADV_HWPOISON failed");
goto unmap;
}
@ -228,7 +231,7 @@ test_hugetlb_read_hwpoison(int fd, size_t len, size_t wr_chunk_size,
munmap(filemap, len);
done:
if (ftruncate(fd, 0) < 0) {
perror(PREFIX ERROR_PREFIX "ftruncate back to 0 failed");
ksft_perror("ftruncate back to 0 failed");
status = TEST_FAILED;
}
@ -241,17 +244,17 @@ static int create_hugetlbfs_file(struct statfs *file_stat)
fd = memfd_create("hugetlb_tmp", MFD_HUGETLB);
if (fd < 0) {
perror(PREFIX ERROR_PREFIX "could not open hugetlbfs file");
ksft_perror("could not open hugetlbfs file");
return -1;
}
memset(file_stat, 0, sizeof(*file_stat));
if (fstatfs(fd, file_stat)) {
perror(PREFIX ERROR_PREFIX "fstatfs failed");
ksft_perror("fstatfs failed");
goto close;
}
if (file_stat->f_type != HUGETLBFS_MAGIC) {
printf(PREFIX ERROR_PREFIX "not hugetlbfs file\n");
ksft_print_msg("not hugetlbfs file\n");
goto close;
}
@ -261,6 +264,10 @@ static int create_hugetlbfs_file(struct statfs *file_stat)
return -1;
}
static void sigbus_handler(int sig)
{
}
int main(void)
{
int fd;
@ -273,50 +280,44 @@ int main(void)
};
size_t i;
ksft_print_header();
ksft_set_plan(ARRAY_SIZE(wr_chunk_sizes) * 3);
signal(SIGBUS, sigbus_handler);
for (i = 0; i < ARRAY_SIZE(wr_chunk_sizes); ++i) {
printf("Write/read chunk size=0x%lx\n",
wr_chunk_sizes[i]);
ksft_print_msg("Write/read chunk size=0x%lx\n",
wr_chunk_sizes[i]);
fd = create_hugetlbfs_file(&file_stat);
if (fd < 0)
goto create_failure;
printf(PREFIX "HugeTLB read regression test...\n");
ksft_exit_fail_msg("Failed to create hugetlbfs file\n");
status = test_hugetlb_read(fd, file_stat.f_bsize,
wr_chunk_sizes[i]);
printf(PREFIX "HugeTLB read regression test...%s\n",
status_to_str(status));
close(fd);
if (status == TEST_FAILED)
return -1;
report_status(status, "HugeTLB read regression",
wr_chunk_sizes[i]);
fd = create_hugetlbfs_file(&file_stat);
if (fd < 0)
goto create_failure;
printf(PREFIX "HugeTLB read HWPOISON test...\n");
ksft_exit_fail_msg("Failed to create hugetlbfs file\n");
status = test_hugetlb_read_hwpoison(fd, file_stat.f_bsize,
wr_chunk_sizes[i], false);
printf(PREFIX "HugeTLB read HWPOISON test...%s\n",
status_to_str(status));
close(fd);
if (status == TEST_FAILED)
return -1;
report_status(status, "HugeTLB read HWPOISON",
wr_chunk_sizes[i]);
fd = create_hugetlbfs_file(&file_stat);
if (fd < 0)
goto create_failure;
printf(PREFIX "HugeTLB seek then read HWPOISON test...\n");
ksft_exit_fail_msg("Failed to create hugetlbfs file\n");
status = test_hugetlb_read_hwpoison(fd, file_stat.f_bsize,
wr_chunk_sizes[i], true);
printf(PREFIX "HugeTLB seek then read HWPOISON test...%s\n",
status_to_str(status));
close(fd);
if (status == TEST_FAILED)
return -1;
report_status(status, "HugeTLB seek then read HWPOISON",
wr_chunk_sizes[i]);
}
return 0;
create_failure:
printf(ERROR_PREFIX "Abort test: failed to create hugetlbfs file\n");
return -1;
ksft_finished();
}

73
tools/testing/selftests/mm/hugepage-shm.c → tools/testing/selftests/mm/hugetlb-shm.c
View File

@ -1,6 +1,6 @@
// SPDX-License-Identifier: GPL-2.0
/*
* hugepage-shm:
* hugetlb-shm:
*
* Example of using huge page memory in a user application using Sys V shared
* memory system calls. In this example the app is requesting 256MB of
@ -28,9 +28,27 @@
#include <sys/shm.h>
#include <sys/mman.h>
#include "vm_util.h"
#include "hugepage_settings.h"
#define LENGTH (256UL*1024*1024)
#define dprintf(x) printf(x)
static void prepare(void)
{
unsigned long length, hugepage_size, nr;
hugepage_size = default_huge_page_size();
if (!hugepage_size)
ksft_exit_skip("Unable to determine huge page size\n");
length = (LENGTH + hugepage_size - 1) & ~(hugepage_size - 1);
nr = length / hugepage_size;
if (!hugetlb_setup_default(nr))
ksft_exit_skip("Not enough free huge pages\n");
shm_limits_prepare(length);
}
int main(void)
{
@ -38,44 +56,45 @@ int main(void)
unsigned long i;
char *shmaddr;
ksft_print_header();
ksft_set_plan(1);
prepare();
shmid = shmget(2, LENGTH, SHM_HUGETLB | IPC_CREAT | SHM_R | SHM_W);
if (shmid < 0) {
perror("shmget");
exit(1);
}
printf("shmid: 0x%x\n", shmid);
if (shmid < 0)
ksft_exit_fail_perror("shmget");
ksft_print_msg("shmid: 0x%x\n", shmid);
shmaddr = shmat(shmid, NULL, 0);
if (shmaddr == (char *)-1) {
perror("Shared memory attach failure");
ksft_perror("Shared memory attach failure");
shmctl(shmid, IPC_RMID, NULL);
exit(2);
ksft_exit_fail();
}
printf("shmaddr: %p\n", shmaddr);
ksft_print_msg("shmaddr: %p\n", shmaddr);
dprintf("Starting the writes:\n");
for (i = 0; i < LENGTH; i++) {
shmaddr[i] = (char)(i);
if (!(i % (1024 * 1024)))
dprintf(".");
}
dprintf("\n");
dprintf("Starting the Check...");
ksft_print_msg("Starting the writes:\n");
for (i = 0; i < LENGTH; i++)
if (shmaddr[i] != (char)i) {
printf("\nIndex %lu mismatched\n", i);
exit(3);
}
dprintf("Done.\n");
shmaddr[i] = (char)(i);
ksft_print_msg("Starting the Check...");
for (i = 0; i < LENGTH; i++)
if (shmaddr[i] != (char)i)
ksft_exit_fail_msg("Data mismatch at index %lu\n", i);
ksft_print_msg("Done.\n");
if (shmdt((const void *)shmaddr) != 0) {
perror("Detach failure");
ksft_perror("Detach failure");
shmctl(shmid, IPC_RMID, NULL);
exit(4);
ksft_exit_fail();
}
shmctl(shmid, IPC_RMID, NULL);
return 0;
ksft_test_result_pass("hugepage using SysV shmget/shmat\n");
ksft_finished();
}
SHM_LIMITS_RESTORE()

45
tools/testing/selftests/mm/hugetlb-soft-offline.c
View File

@ -6,9 +6,7 @@
* - if enable_soft_offline = 1, a hugepage should be dissolved and
* nr_hugepages/free_hugepages should be reduced by 1.
*
* Before running, make sure more than 2 hugepages of default_hugepagesz
* are allocated. For example, if /proc/meminfo/Hugepagesize is 2048kB:
* echo 8 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
* The test allocates 8 default hugepages
*/
#define _GNU_SOURCE
@ -25,6 +23,7 @@
#include <sys/types.h>
#include "kselftest.h"
#include "hugepage_settings.h"
#ifndef MADV_SOFT_OFFLINE
#define MADV_SOFT_OFFLINE 101
@ -100,32 +99,6 @@ static int set_enable_soft_offline(int value)
return 0;
}
static int read_nr_hugepages(unsigned long hugepage_size,
unsigned long *nr_hugepages)
{
char buffer[256] = {0};
char cmd[256] = {0};
sprintf(cmd, "cat /sys/kernel/mm/hugepages/hugepages-%ldkB/nr_hugepages",
hugepage_size);
FILE *cmdfile = popen(cmd, "r");
if (cmdfile == NULL) {
ksft_perror(EPREFIX "failed to popen nr_hugepages");
return -1;
}
if (!fgets(buffer, sizeof(buffer), cmdfile)) {
ksft_perror(EPREFIX "failed to read nr_hugepages");
pclose(cmdfile);
return -1;
}
*nr_hugepages = atoll(buffer);
pclose(cmdfile);
return 0;
}
static int create_hugetlbfs_file(struct statfs *file_stat)
{
int fd;
@ -177,20 +150,14 @@ static void test_soft_offline_common(int enable_soft_offline)
ksft_exit_fail_msg("Failed to set enable_soft_offline\n");
}
if (read_nr_hugepages(hugepagesize_kb, &nr_hugepages_before) != 0) {
close(fd);
ksft_exit_fail_msg("Failed to read nr_hugepages\n");
}
nr_hugepages_before = hugetlb_nr_default_pages();
ksft_print_msg("Before MADV_SOFT_OFFLINE nr_hugepages=%ld\n",
nr_hugepages_before);
ret = do_soft_offline(fd, 2 * file_stat.f_bsize, expect_errno);
if (read_nr_hugepages(hugepagesize_kb, &nr_hugepages_after) != 0) {
close(fd);
ksft_exit_fail_msg("Failed to read nr_hugepages\n");
}
nr_hugepages_after = hugetlb_nr_default_pages();
ksft_print_msg("After MADV_SOFT_OFFLINE nr_hugepages=%ld\n",
nr_hugepages_after);
@ -219,6 +186,10 @@ static void test_soft_offline_common(int enable_soft_offline)
int main(int argc, char **argv)
{
ksft_print_header();
if (!hugetlb_setup_default(8))
ksft_exit_skip("not enough hugetlb pages\n");
ksft_set_plan(2);
test_soft_offline_common(1);

46
tools/testing/selftests/mm/hugepage-vmemmap.c → tools/testing/selftests/mm/hugetlb-vmemmap.c
View File

@ -11,6 +11,7 @@
#include <sys/mman.h>
#include <fcntl.h>
#include "vm_util.h"
#include "hugepage_settings.h"
#define PAGE_COMPOUND_HEAD (1UL << 15)
#define PAGE_COMPOUND_TAIL (1UL << 16)
@ -63,7 +64,7 @@ static int check_page_flags(unsigned long pfn)
read(fd, &pageflags, sizeof(pageflags));
if ((pageflags & HEAD_PAGE_FLAGS) != HEAD_PAGE_FLAGS) {
close(fd);
printf("Head page flags (%lx) is invalid\n", pageflags);
ksft_print_msg("Head page flags (%lx) is invalid\n", pageflags);
return -1;
}
@ -77,7 +78,7 @@ static int check_page_flags(unsigned long pfn)
if ((pageflags & TAIL_PAGE_FLAGS) != TAIL_PAGE_FLAGS ||
(pageflags & HEAD_PAGE_FLAGS) == HEAD_PAGE_FLAGS) {
close(fd);
printf("Tail page flags (%lx) is invalid\n", pageflags);
ksft_print_msg("Tail page flags (%lx) is invalid\n", pageflags);
return -1;
}
}
@ -91,44 +92,41 @@ int main(int argc, char **argv)
{
void *addr;
unsigned long pfn;
int ret;
ksft_print_header();
ksft_set_plan(1);
if (!hugetlb_setup_default(1))
ksft_exit_skip("Not enough free huge pages\n");
pagesize = psize();
maplength = default_huge_page_size();
if (!maplength) {
printf("Unable to determine huge page size\n");
exit(1);
}
if (!maplength)
ksft_exit_skip("Unable to determine huge page size\n");
addr = mmap(NULL, maplength, PROT_READ | PROT_WRITE,
MAP_PRIVATE | MAP_ANONYMOUS | MAP_HUGETLB, -1, 0);
if (addr == MAP_FAILED) {
perror("mmap");
exit(1);
}
if (addr == MAP_FAILED)
ksft_exit_fail_perror("mmap");
/* Trigger allocation of HugeTLB page. */
write_bytes(addr, maplength);
pfn = virt_to_pfn(addr);
if (pfn == -1UL) {
ksft_perror("virt_to_pfn");
munmap(addr, maplength);
perror("virt_to_pfn");
exit(1);
ksft_exit_fail();
}
printf("Returned address is %p whose pfn is %lx\n", addr, pfn);
ksft_print_msg("Returned address is %p whose pfn is %lx\n", addr, pfn);
if (check_page_flags(pfn) < 0) {
munmap(addr, maplength);
perror("check_page_flags");
exit(1);
}
ret = check_page_flags(pfn);
/* munmap() length of MAP_HUGETLB memory must be hugepage aligned */
if (munmap(addr, maplength)) {
perror("munmap");
exit(1);
}
if (munmap(addr, maplength))
ksft_exit_fail_perror("munmap");
return 0;
ksft_test_result(!ret, "HugeTLB vmemmap page flags\n");
ksft_finished();
}

15
tools/testing/selftests/mm/hugetlb_dio.c
View File

@ -20,6 +20,7 @@
#include <sys/syscall.h>
#include "vm_util.h"
#include "kselftest.h"
#include "hugepage_settings.h"
#ifndef STATX_DIOALIGN
#define STATX_DIOALIGN 0x00002000U
@ -84,19 +85,13 @@ static void run_dio_using_hugetlb(int fd, unsigned int start_off,
/* Get the default huge page size */
h_pagesize = default_huge_page_size();
if (!h_pagesize)
ksft_exit_fail_msg("Unable to determine huge page size\n");
/* Reset file position since fd is shared across tests */
if (lseek(fd, 0, SEEK_SET) < 0)
ksft_exit_fail_perror("lseek failed\n");
/* Get the free huge pages before allocation */
free_hpage_b = get_free_hugepages();
if (free_hpage_b == 0) {
close(fd);
ksft_exit_skip("No free hugepage, exiting!\n");
}
free_hpage_b = hugetlb_free_default_pages();
/* Allocate a hugetlb page */
orig_buffer = mmap(NULL, h_pagesize, mmap_prot, mmap_flags, -1, 0);
@ -120,7 +115,7 @@ static void run_dio_using_hugetlb(int fd, unsigned int start_off,
munmap(orig_buffer, h_pagesize);
/* Get the free huge pages after unmap*/
free_hpage_a = get_free_hugepages();
free_hpage_a = hugetlb_free_default_pages();
ksft_print_msg("No. Free pages before allocation : %d\n", free_hpage_b);
ksft_print_msg("No. Free pages after munmap : %d\n", free_hpage_a);
@ -140,8 +135,8 @@ int main(void)
ksft_print_header();
/* Check if huge pages are free */
if (!get_free_hugepages())
/* request a huge page */
if (!hugetlb_setup_default(1))
ksft_exit_skip("No free hugepage, exiting\n");
fd = open("/tmp", O_TMPFILE | O_RDWR | O_DIRECT, 0664);

9
tools/testing/selftests/mm/hugetlb_fault_after_madv.c
View File

@ -10,6 +10,7 @@
#include "vm_util.h"
#include "kselftest.h"
#include "hugepage_settings.h"
#define INLOOP_ITER 100
@ -53,7 +54,6 @@ void *madv(void *unused)
int main(void)
{
unsigned long free_hugepages;
pthread_t thread1, thread2;
/*
* On kernel 6.4, we are able to reproduce the problem with ~1000
@ -77,11 +77,8 @@ int main(void)
ksft_print_msg("[INFO] detected default hugetlb page size: %zu KiB\n",
huge_page_size / 1024);
free_hugepages = get_free_hugepages();
if (free_hugepages != 1) {
ksft_exit_skip("This test needs one and only one page to execute. Got %lu\n",
free_hugepages);
}
if (!hugetlb_setup_default(1))
ksft_exit_skip("Not enough HugeTLB pages\n");
while (max--) {
huge_ptr = mmap(NULL, huge_page_size, PROT_READ | PROT_WRITE,

22
tools/testing/selftests/mm/hugetlb_madv_vs_map.c
View File

@ -25,7 +25,7 @@
#include <unistd.h>
#include "vm_util.h"
#include "kselftest.h"
#include "hugepage_settings.h"
#define INLOOP_ITER 100
@ -77,7 +77,6 @@ void *map_extra(void *unused)
int main(void)
{
pthread_t thread1, thread2, thread3;
unsigned long free_hugepages;
void *ret;
/*
@ -86,12 +85,12 @@ int main(void)
*/
int max = 10;
free_hugepages = get_free_hugepages();
ksft_print_header();
ksft_set_plan(1);
if (free_hugepages != 1) {
if (!hugetlb_setup_default_exact(1))
ksft_exit_skip("This test needs one and only one page to execute. Got %lu\n",
free_hugepages);
}
hugetlb_free_default_pages());
mmap_size = default_huge_page_size();
@ -100,10 +99,8 @@ int main(void)
MAP_PRIVATE | MAP_ANONYMOUS | MAP_HUGETLB,
-1, 0);
if ((unsigned long)huge_ptr == -1) {
ksft_test_result_fail("Failed to allocate huge page\n");
return KSFT_FAIL;
}
if ((unsigned long)huge_ptr == -1)
ksft_exit_fail_msg("Failed to allocate huge page\n");
pthread_create(&thread1, NULL, madv, NULL);
pthread_create(&thread2, NULL, touch, NULL);
@ -115,12 +112,13 @@ int main(void)
if (ret) {
ksft_test_result_fail("Unexpected huge page allocation\n");
return KSFT_FAIL;
ksft_finished();
}
/* Unmap and restart */
munmap(huge_ptr, mmap_size);
}
return KSFT_PASS;
ksft_test_result_pass("No unexpected huge page allocations\n");
ksft_finished();
}

60
tools/testing/selftests/mm/hugetlb_reparenting_test.sh
View File

@ -12,6 +12,8 @@ if [[ $(id -u) -ne 0 ]]; then
fi
nr_hugepgs=$(cat /proc/sys/vm/nr_hugepages)
trap 'echo "$nr_hugepgs" > /proc/sys/vm/nr_hugepages' EXIT INT TERM
usage_file=usage_in_bytes
if [[ "$1" == "-cgroup-v2" ]]; then
@ -46,6 +48,13 @@ function get_machine_hugepage_size() {
}
MB=$(get_machine_hugepage_size)
if (( MB >= 1024 )); then
UNIT="GB"
MB_DISPLAY=$((MB / 1024))
else
UNIT="MB"
MB_DISPLAY=$MB
fi
function cleanup() {
echo cleanup
@ -56,7 +65,6 @@ function cleanup() {
rmdir "$CGROUP_ROOT"/a/b 2>/dev/null
rmdir "$CGROUP_ROOT"/a 2>/dev/null
rmdir "$CGROUP_ROOT"/test1 2>/dev/null
echo $nr_hugepgs >/proc/sys/vm/nr_hugepages
set -e
}
@ -87,6 +95,7 @@ function assert_with_retry() {
if [[ $elapsed -ge $timeout ]]; then
echo "actual = $((${actual%% *} / 1024 / 1024)) MB"
echo "expected = $((${expected%% *} / 1024 / 1024)) MB"
echo FAIL
cleanup
exit 1
fi
@ -96,22 +105,19 @@ function assert_with_retry() {
}
function assert_state() {
local expected_a="$1"
local expected_a_hugetlb="$2"
local expected_b=""
local expected_a_hugetlb="$1"
local expected_b_hugetlb=""
if [ ! -z ${3:-} ] && [ ! -z ${4:-} ]; then
expected_b="$3"
expected_b_hugetlb="$4"
if [ ! -z ${2:-} ]; then
expected_b_hugetlb="$2"
fi
assert_with_retry "$CGROUP_ROOT/a/memory.$usage_file" "$expected_a"
assert_with_retry "$CGROUP_ROOT/a/hugetlb.${MB}MB.$usage_file" "$expected_a_hugetlb"
assert_with_retry \
"$CGROUP_ROOT/a/hugetlb.${MB_DISPLAY}${UNIT}.$usage_file" "$expected_a_hugetlb"
if [[ -n "$expected_b" && -n "$expected_b_hugetlb" ]]; then
assert_with_retry "$CGROUP_ROOT/a/b/memory.$usage_file" "$expected_b"
assert_with_retry "$CGROUP_ROOT/a/b/hugetlb.${MB}MB.$usage_file" "$expected_b_hugetlb"
if [[ -n "$expected_b_hugetlb" ]]; then
assert_with_retry \
"$CGROUP_ROOT/a/b/hugetlb.${MB_DISPLAY}${UNIT}.$usage_file" "$expected_b_hugetlb"
fi
}
@ -143,18 +149,17 @@ write_hugetlbfs() {
local size="$3"
if [[ $cgroup2 ]]; then
echo $$ >$CGROUP_ROOT/$cgroup/cgroup.procs
cg_file="$CGROUP_ROOT/$cgroup/cgroup.procs"
else
echo 0 >$CGROUP_ROOT/$cgroup/cpuset.mems
echo 0 >$CGROUP_ROOT/$cgroup/cpuset.cpus
echo $$ >"$CGROUP_ROOT/$cgroup/tasks"
fi
./write_to_hugetlbfs -p "$path" -s "$size" -m 0 -o
if [[ $cgroup2 ]]; then
echo $$ >$CGROUP_ROOT/cgroup.procs
else
echo $$ >"$CGROUP_ROOT/tasks"
cg_file="$CGROUP_ROOT/$cgroup/tasks"
fi
# Spawn helper to join cgroup before exec to ensure correct cgroup accounting
bash -c 'echo $$ > "$1"; exec ./write_to_hugetlbfs -p "$2" -s "$3" -m 0 -o' _ \
"$cg_file" "$path" "$size" & pid=$!
wait "$pid"
echo
}
@ -192,21 +197,21 @@ if [[ ! $cgroup2 ]]; then
write_hugetlbfs a "$MNT"/test $size
echo Assert memory charged correctly for parent use.
assert_state 0 $size 0 0
assert_state $size 0
write_hugetlbfs a/b "$MNT"/test2 $size
echo Assert memory charged correctly for child use.
assert_state 0 $(($size * 2)) 0 $size
assert_state $(($size * 2)) $size
rmdir "$CGROUP_ROOT"/a/b
echo Assert memory reparent correctly.
assert_state 0 $(($size * 2))
assert_state $(($size * 2))
rm -rf "$MNT"/*
umount "$MNT"
echo Assert memory uncharged correctly.
assert_state 0 0
assert_state 0
cleanup
fi
@ -220,16 +225,16 @@ echo write
write_hugetlbfs a/b "$MNT"/test2 $size
echo Assert memory charged correctly for child only use.
assert_state 0 $(($size)) 0 $size
assert_state $(($size)) $size
rmdir "$CGROUP_ROOT"/a/b
echo Assert memory reparent correctly.
assert_state 0 $size
assert_state $size
rm -rf "$MNT"/*
umount "$MNT"
echo Assert memory uncharged correctly.
assert_state 0 0
assert_state 0
cleanup
@ -240,4 +245,3 @@ if [[ $do_umount ]]; then
rm -rf $CGROUP_ROOT
fi
echo "$nr_hugepgs" > /proc/sys/vm/nr_hugepages

559
tools/testing/selftests/mm/khugepaged.c
View File

File diff suppressed because it is too large Load Diff

184
tools/testing/selftests/mm/ksm_tests.c
View File

@ -15,7 +15,7 @@
#include "kselftest.h"
#include <include/vdso/time64.h>
#include "vm_util.h"
#include "thp_settings.h"
#include "hugepage_settings.h"
#define KSM_SYSFS_PATH "/sys/kernel/mm/ksm/"
#define KSM_FP(s) (KSM_SYSFS_PATH s)
@ -174,13 +174,13 @@ static void *allocate_memory(void *ptr, int prot, int mapping, char data, size_
{
void *map_ptr = mmap(ptr, map_size, PROT_WRITE, mapping, -1, 0);
if (!map_ptr) {
perror("mmap");
if (map_ptr == MAP_FAILED) {
ksft_perror("mmap");
return NULL;
}
memset(map_ptr, data, map_size);
if (mprotect(map_ptr, map_size, prot)) {
perror("mprotect");
ksft_perror("mprotect");
munmap(map_ptr, map_size);
return NULL;
}
@ -201,11 +201,11 @@ static int ksm_do_scan(int scan_count, struct timespec start_time, int timeout)
if (ksm_read_sysfs(KSM_FP("full_scans"), &cur_scan))
return 1;
if (clock_gettime(CLOCK_MONOTONIC_RAW, &cur_time)) {
perror("clock_gettime");
ksft_perror("clock_gettime");
return 1;
}
if ((cur_time.tv_sec - start_time.tv_sec) > timeout) {
printf("Scan time limit exceeded\n");
ksft_print_msg("Scan time limit exceeded\n");
return 1;
}
}
@ -218,12 +218,12 @@ static int ksm_merge_pages(int merge_type, void *addr, size_t size,
{
if (merge_type == KSM_MERGE_MADVISE) {
if (madvise(addr, size, MADV_MERGEABLE)) {
perror("madvise");
ksft_perror("madvise");
return 1;
}
} else if (merge_type == KSM_MERGE_PRCTL) {
if (prctl(PR_SET_MEMORY_MERGE, 1, 0, 0, 0)) {
perror("prctl");
ksft_perror("prctl");
return 1;
}
}
@ -242,7 +242,7 @@ static int ksm_unmerge_pages(void *addr, size_t size,
struct timespec start_time, int timeout)
{
if (madvise(addr, size, MADV_UNMERGEABLE)) {
perror("madvise");
ksft_perror("madvise");
return 1;
}
return 0;
@ -324,7 +324,7 @@ static int check_ksm_merge(int merge_type, int mapping, int prot,
struct timespec start_time;
if (clock_gettime(CLOCK_MONOTONIC_RAW, &start_time)) {
perror("clock_gettime");
ksft_perror("clock_gettime");
return KSFT_FAIL;
}
@ -338,7 +338,6 @@ static int check_ksm_merge(int merge_type, int mapping, int prot,
/* verify that the right number of pages are merged */
if (assert_ksm_pages_count(page_count)) {
printf("OK\n");
munmap(map_ptr, page_size * page_count);
if (merge_type == KSM_MERGE_PRCTL)
prctl(PR_SET_MEMORY_MERGE, 0, 0, 0, 0);
@ -346,7 +345,6 @@ static int check_ksm_merge(int merge_type, int mapping, int prot,
}
err_out:
printf("Not OK\n");
munmap(map_ptr, page_size * page_count);
return KSFT_FAIL;
}
@ -358,7 +356,7 @@ static int check_ksm_unmerge(int merge_type, int mapping, int prot, int timeout,
int page_count = 2;
if (clock_gettime(CLOCK_MONOTONIC_RAW, &start_time)) {
perror("clock_gettime");
ksft_perror("clock_gettime");
return KSFT_FAIL;
}
@ -380,13 +378,11 @@ static int check_ksm_unmerge(int merge_type, int mapping, int prot, int timeout,
/* check that unmerging was successful and 0 pages are currently merged */
if (assert_ksm_pages_count(0)) {
printf("OK\n");
munmap(map_ptr, page_size * page_count);
return KSFT_PASS;
}
err_out:
printf("Not OK\n");
munmap(map_ptr, page_size * page_count);
return KSFT_FAIL;
}
@ -398,7 +394,7 @@ static int check_ksm_zero_page_merge(int merge_type, int mapping, int prot, long
struct timespec start_time;
if (clock_gettime(CLOCK_MONOTONIC_RAW, &start_time)) {
perror("clock_gettime");
ksft_perror("clock_gettime");
return KSFT_FAIL;
}
@ -425,12 +421,10 @@ static int check_ksm_zero_page_merge(int merge_type, int mapping, int prot, long
else if (!use_zero_pages && !assert_ksm_pages_count(page_count))
goto err_out;
printf("OK\n");
munmap(map_ptr, page_size * page_count);
return KSFT_PASS;
err_out:
printf("Not OK\n");
munmap(map_ptr, page_size * page_count);
return KSFT_FAIL;
}
@ -465,16 +459,16 @@ static int check_ksm_numa_merge(int merge_type, int mapping, int prot, int timeo
int first_node;
if (clock_gettime(CLOCK_MONOTONIC_RAW, &start_time)) {
perror("clock_gettime");
ksft_perror("clock_gettime");
return KSFT_FAIL;
}
if (numa_available() < 0) {
perror("NUMA support not enabled");
ksft_print_msg("NUMA support not enabled\n");
return KSFT_SKIP;
}
if (numa_num_configured_nodes() <= 1) {
printf("At least 2 NUMA nodes must be available\n");
ksft_print_msg("At least 2 NUMA nodes must be available\n");
return KSFT_SKIP;
}
if (ksm_write_sysfs(KSM_FP("merge_across_nodes"), merge_across_nodes))
@ -485,7 +479,7 @@ static int check_ksm_numa_merge(int merge_type, int mapping, int prot, int timeo
numa1_map_ptr = numa_alloc_onnode(page_size, first_node);
numa2_map_ptr = numa_alloc_onnode(page_size, get_next_mem_node(first_node));
if (!numa1_map_ptr || !numa2_map_ptr) {
perror("numa_alloc_onnode");
ksft_perror("numa_alloc_onnode");
return KSFT_FAIL;
}
@ -510,13 +504,11 @@ static int check_ksm_numa_merge(int merge_type, int mapping, int prot, int timeo
numa_free(numa1_map_ptr, page_size);
numa_free(numa2_map_ptr, page_size);
printf("OK\n");
return KSFT_PASS;
err_out:
numa_free(numa1_map_ptr, page_size);
numa_free(numa2_map_ptr, page_size);
printf("Not OK\n");
return KSFT_FAIL;
}
@ -529,7 +521,7 @@ static int ksm_merge_hugepages_time(int merge_type, int mapping, int prot,
int pagemap_fd, n_normal_pages, n_huge_pages;
if (!thp_is_enabled()) {
printf("Transparent Hugepages not available\n");
ksft_print_msg("Transparent Hugepages not available\n");
return KSFT_SKIP;
}
@ -559,36 +551,35 @@ static int ksm_merge_hugepages_time(int merge_type, int mapping, int prot,
else
n_huge_pages++;
}
printf("Number of normal pages: %d\n", n_normal_pages);
printf("Number of huge pages: %d\n", n_huge_pages);
ksft_print_msg("Number of normal pages: %d\n", n_normal_pages);
ksft_print_msg("Number of huge pages: %d\n", n_huge_pages);
memset(map_ptr, '*', len);
if (clock_gettime(CLOCK_MONOTONIC_RAW, &start_time)) {
perror("clock_gettime");
ksft_perror("clock_gettime");
goto err_out;
}
if (ksm_merge_pages(merge_type, map_ptr, map_size, start_time, timeout))
goto err_out;
if (clock_gettime(CLOCK_MONOTONIC_RAW, &end_time)) {
perror("clock_gettime");
ksft_perror("clock_gettime");
goto err_out;
}
scan_time_ns = (end_time.tv_sec - start_time.tv_sec) * NSEC_PER_SEC +
(end_time.tv_nsec - start_time.tv_nsec);
printf("Total size: %lu MiB\n", map_size / MB);
printf("Total time: %ld.%09ld s\n", scan_time_ns / NSEC_PER_SEC,
ksft_print_msg("Total size: %lu MiB\n", map_size / MB);
ksft_print_msg("Total time: %ld.%09ld s\n", scan_time_ns / NSEC_PER_SEC,
scan_time_ns % NSEC_PER_SEC);
printf("Average speed: %.3f MiB/s\n", (map_size / MB) /
ksft_print_msg("Average speed: %.3f MiB/s\n", (map_size / MB) /
((double)scan_time_ns / NSEC_PER_SEC));
munmap(map_ptr_orig, len + HPAGE_SIZE);
return KSFT_PASS;
err_out:
printf("Not OK\n");
munmap(map_ptr_orig, len + HPAGE_SIZE);
return KSFT_FAIL;
}
@ -606,30 +597,29 @@ static int ksm_merge_time(int merge_type, int mapping, int prot, int timeout, si
return KSFT_FAIL;
if (clock_gettime(CLOCK_MONOTONIC_RAW, &start_time)) {
perror("clock_gettime");
ksft_perror("clock_gettime");
goto err_out;
}
if (ksm_merge_pages(merge_type, map_ptr, map_size, start_time, timeout))
goto err_out;
if (clock_gettime(CLOCK_MONOTONIC_RAW, &end_time)) {
perror("clock_gettime");
ksft_perror("clock_gettime");
goto err_out;
}
scan_time_ns = (end_time.tv_sec - start_time.tv_sec) * NSEC_PER_SEC +
(end_time.tv_nsec - start_time.tv_nsec);
printf("Total size: %lu MiB\n", map_size / MB);
printf("Total time: %ld.%09ld s\n", scan_time_ns / NSEC_PER_SEC,
ksft_print_msg("Total size: %lu MiB\n", map_size / MB);
ksft_print_msg("Total time: %ld.%09ld s\n", scan_time_ns / NSEC_PER_SEC,
scan_time_ns % NSEC_PER_SEC);
printf("Average speed: %.3f MiB/s\n", (map_size / MB) /
ksft_print_msg("Average speed: %.3f MiB/s\n", (map_size / MB) /
((double)scan_time_ns / NSEC_PER_SEC));
munmap(map_ptr, map_size);
return KSFT_PASS;
err_out:
printf("Not OK\n");
munmap(map_ptr, map_size);
return KSFT_FAIL;
}
@ -646,37 +636,36 @@ static int ksm_unmerge_time(int merge_type, int mapping, int prot, int timeout,
if (!map_ptr)
return KSFT_FAIL;
if (clock_gettime(CLOCK_MONOTONIC_RAW, &start_time)) {
perror("clock_gettime");
ksft_perror("clock_gettime");
goto err_out;
}
if (ksm_merge_pages(merge_type, map_ptr, map_size, start_time, timeout))
goto err_out;
if (clock_gettime(CLOCK_MONOTONIC_RAW, &start_time)) {
perror("clock_gettime");
ksft_perror("clock_gettime");
goto err_out;
}
if (ksm_unmerge_pages(map_ptr, map_size, start_time, timeout))
goto err_out;
if (clock_gettime(CLOCK_MONOTONIC_RAW, &end_time)) {
perror("clock_gettime");
ksft_perror("clock_gettime");
goto err_out;
}
scan_time_ns = (end_time.tv_sec - start_time.tv_sec) * NSEC_PER_SEC +
(end_time.tv_nsec - start_time.tv_nsec);
printf("Total size: %lu MiB\n", map_size / MB);
printf("Total time: %ld.%09ld s\n", scan_time_ns / NSEC_PER_SEC,
ksft_print_msg("Total size: %lu MiB\n", map_size / MB);
ksft_print_msg("Total time: %ld.%09ld s\n", scan_time_ns / NSEC_PER_SEC,
scan_time_ns % NSEC_PER_SEC);
printf("Average speed: %.3f MiB/s\n", (map_size / MB) /
ksft_print_msg("Average speed: %.3f MiB/s\n", (map_size / MB) /
((double)scan_time_ns / NSEC_PER_SEC));
munmap(map_ptr, map_size);
return KSFT_PASS;
err_out:
printf("Not OK\n");
munmap(map_ptr, map_size);
return KSFT_FAIL;
}
@ -695,24 +684,24 @@ static int ksm_cow_time(int merge_type, int mapping, int prot, int timeout, size
return KSFT_FAIL;
if (clock_gettime(CLOCK_MONOTONIC_RAW, &start_time)) {
perror("clock_gettime");
ksft_perror("clock_gettime");
return KSFT_FAIL;
}
for (size_t i = 0; i < page_count - 1; i = i + 2)
memset(map_ptr + page_size * i, '-', 1);
if (clock_gettime(CLOCK_MONOTONIC_RAW, &end_time)) {
perror("clock_gettime");
ksft_perror("clock_gettime");
return KSFT_FAIL;
}
cow_time_ns = (end_time.tv_sec - start_time.tv_sec) * NSEC_PER_SEC +
(end_time.tv_nsec - start_time.tv_nsec);
printf("Total size: %lu MiB\n\n", (page_size * page_count) / MB);
printf("Not merged pages:\n");
printf("Total time: %ld.%09ld s\n", cow_time_ns / NSEC_PER_SEC,
ksft_print_msg("Total size: %lu MiB\n\n", (page_size * page_count) / MB);
ksft_print_msg("Not merged pages:\n");
ksft_print_msg("Total time: %ld.%09ld s\n", cow_time_ns / NSEC_PER_SEC,
cow_time_ns % NSEC_PER_SEC);
printf("Average speed: %.3f MiB/s\n\n", ((page_size * (page_count / 2)) / MB) /
ksft_print_msg("Average speed: %.3f MiB/s\n\n", ((page_size * (page_count / 2)) / MB) /
((double)cow_time_ns / NSEC_PER_SEC));
/* Create 2000 pairs of duplicate pages */
@ -724,30 +713,29 @@ static int ksm_cow_time(int merge_type, int mapping, int prot, int timeout, size
goto err_out;
if (clock_gettime(CLOCK_MONOTONIC_RAW, &start_time)) {
perror("clock_gettime");
ksft_perror("clock_gettime");
goto err_out;
}
for (size_t i = 0; i < page_count - 1; i = i + 2)
memset(map_ptr + page_size * i, '-', 1);
if (clock_gettime(CLOCK_MONOTONIC_RAW, &end_time)) {
perror("clock_gettime");
ksft_perror("clock_gettime");
goto err_out;
}
cow_time_ns = (end_time.tv_sec - start_time.tv_sec) * NSEC_PER_SEC +
(end_time.tv_nsec - start_time.tv_nsec);
printf("Merged pages:\n");
printf("Total time: %ld.%09ld s\n", cow_time_ns / NSEC_PER_SEC,
ksft_print_msg("Merged pages:\n");
ksft_print_msg("Total time: %ld.%09ld s\n", cow_time_ns / NSEC_PER_SEC,
cow_time_ns % NSEC_PER_SEC);
printf("Average speed: %.3f MiB/s\n", ((page_size * (page_count / 2)) / MB) /
ksft_print_msg("Average speed: %.3f MiB/s\n", ((page_size * (page_count / 2)) / MB) /
((double)cow_time_ns / NSEC_PER_SEC));
munmap(map_ptr, page_size * page_count);
return KSFT_PASS;
err_out:
printf("Not OK\n");
munmap(map_ptr, page_size * page_count);
return KSFT_FAIL;
}
@ -765,6 +753,10 @@ int main(int argc, char *argv[])
bool use_zero_pages = KSM_USE_ZERO_PAGES_DEFAULT;
bool merge_across_nodes = KSM_MERGE_ACROSS_NODES_DEFAULT;
long size_MB = 0;
const char *test_descr = "KSM merging";
ksft_print_header();
ksft_set_plan(1);
while ((opt = getopt(argc, argv, "dha:p:l:z:m:s:t:MUZNPCHD")) != -1) {
switch (opt) {
@ -773,17 +765,13 @@ int main(int argc, char *argv[])
break;
case 'p':
page_count = atol(optarg);
if (page_count <= 0) {
printf("The number of pages must be greater than 0\n");
return KSFT_FAIL;
}
if (page_count <= 0)
ksft_exit_fail_msg("The number of pages must be greater than 0\n");
break;
case 'l':
ksm_scan_limit_sec = atoi(optarg);
if (ksm_scan_limit_sec <= 0) {
printf("Timeout value must be greater than 0\n");
return KSFT_FAIL;
}
if (ksm_scan_limit_sec <= 0)
ksft_exit_fail_msg("Timeout value must be greater than 0\n");
break;
case 'h':
print_help();
@ -805,19 +793,15 @@ int main(int argc, char *argv[])
break;
case 's':
size_MB = atoi(optarg);
if (size_MB <= 0) {
printf("Size must be greater than 0\n");
return KSFT_FAIL;
}
if (size_MB <= 0)
ksft_exit_fail_msg("Size must be greater than 0\n");
break;
case 't':
{
int tmp = atoi(optarg);
if (tmp < 0 || tmp > KSM_MERGE_LAST) {
printf("Invalid merge type\n");
return KSFT_FAIL;
}
if (tmp < 0 || tmp > KSM_MERGE_LAST)
ksft_exit_fail_msg("Invalid merge type\n");
merge_type = tmp;
}
break;
@ -845,82 +829,80 @@ int main(int argc, char *argv[])
test_name = KSM_COW_TIME;
break;
default:
return KSFT_FAIL;
ksft_exit_fail_msg("Unknown option\n");
}
}
if (prot == 0)
prot = str_to_prot(KSM_PROT_STR_DEFAULT);
if (access(KSM_SYSFS_PATH, F_OK)) {
printf("Config KSM not enabled\n");
return KSFT_SKIP;
}
if (access(KSM_SYSFS_PATH, F_OK))
ksft_exit_skip("Config KSM not enabled\n");
if (ksm_save_def(&ksm_sysfs_old)) {
printf("Cannot save default tunables\n");
return KSFT_FAIL;
}
if (ksm_save_def(&ksm_sysfs_old))
ksft_exit_fail_msg("Cannot save default tunables\n");
if (ksm_write_sysfs(KSM_FP("run"), 2) ||
ksm_write_sysfs(KSM_FP("sleep_millisecs"), 0) ||
numa_available() ? 0 :
ksm_write_sysfs(KSM_FP("merge_across_nodes"), 1) ||
ksm_write_sysfs(KSM_FP("pages_to_scan"), page_count))
return KSFT_FAIL;
ksft_exit_fail_msg("Cannot set up KSM tunables\n");
switch (test_name) {
case CHECK_KSM_MERGE:
test_descr = "KSM merging";
ret = check_ksm_merge(merge_type, MAP_PRIVATE | MAP_ANONYMOUS, prot, page_count,
ksm_scan_limit_sec, page_size);
break;
case CHECK_KSM_UNMERGE:
test_descr = "KSM unmerging";
ret = check_ksm_unmerge(merge_type, MAP_PRIVATE | MAP_ANONYMOUS, prot,
ksm_scan_limit_sec, page_size);
break;
case CHECK_KSM_ZERO_PAGE_MERGE:
test_descr = "KSM zero page merging";
ret = check_ksm_zero_page_merge(merge_type, MAP_PRIVATE | MAP_ANONYMOUS, prot,
page_count, ksm_scan_limit_sec, use_zero_pages,
page_size);
break;
case CHECK_KSM_NUMA_MERGE:
test_descr = "KSM NUMA merging";
ret = check_ksm_numa_merge(merge_type, MAP_PRIVATE | MAP_ANONYMOUS, prot,
ksm_scan_limit_sec, merge_across_nodes, page_size);
break;
case KSM_MERGE_TIME:
if (size_MB == 0) {
printf("Option '-s' is required.\n");
return KSFT_FAIL;
}
if (size_MB == 0)
ksft_exit_fail_msg("Option '-s' is required\n");
test_descr = "KSM merge time";
ret = ksm_merge_time(merge_type, MAP_PRIVATE | MAP_ANONYMOUS, prot,
ksm_scan_limit_sec, size_MB);
break;
case KSM_MERGE_TIME_HUGE_PAGES:
if (size_MB == 0) {
printf("Option '-s' is required.\n");
return KSFT_FAIL;
}
if (size_MB == 0)
ksft_exit_fail_msg("Option '-s' is required\n");
test_descr = "KSM merge time with huge pages";
ret = ksm_merge_hugepages_time(merge_type, MAP_PRIVATE | MAP_ANONYMOUS, prot,
ksm_scan_limit_sec, size_MB);
break;
case KSM_UNMERGE_TIME:
if (size_MB == 0) {
printf("Option '-s' is required.\n");
return KSFT_FAIL;
}
if (size_MB == 0)
ksft_exit_fail_msg("Option '-s' is required\n");
test_descr = "KSM unmerge time";
ret = ksm_unmerge_time(merge_type, MAP_PRIVATE | MAP_ANONYMOUS, prot,
ksm_scan_limit_sec, size_MB);
break;
case KSM_COW_TIME:
test_descr = "KSM COW time";
ret = ksm_cow_time(merge_type, MAP_PRIVATE | MAP_ANONYMOUS, prot,
ksm_scan_limit_sec, page_size);
break;
}
if (ksm_restore(&ksm_sysfs_old)) {
printf("Cannot restore default tunables\n");
return KSFT_FAIL;
}
if (ksm_restore(&ksm_sysfs_old))
ksft_print_msg("Cannot restore default tunables\n");
return ret;
ksft_test_result_report(ret, "%s\n", test_descr);
ksft_finished();
}

2
tools/testing/selftests/mm/madv_populate.c
View File

@ -34,7 +34,7 @@ static void sense_support(void)
addr = mmap(0, pagesize, PROT_READ | PROT_WRITE,
MAP_ANONYMOUS | MAP_PRIVATE, 0, 0);
if (!addr)
if (addr == MAP_FAILED)
ksft_exit_fail_msg("mmap failed\n");
ret = madvise(addr, pagesize, MADV_POPULATE_READ);

88
tools/testing/selftests/mm/map_hugetlb.c
View File

@ -1,88 +0,0 @@
// SPDX-License-Identifier: GPL-2.0
/*
* Example of using hugepage memory in a user application using the mmap
* system call with MAP_HUGETLB flag. Before running this program make
* sure the administrator has allocated enough default sized huge pages
* to cover the 256 MB allocation.
*/
#include <stdlib.h>
#include <stdio.h>
#include <unistd.h>
#include <sys/mman.h>
#include <fcntl.h>
#include "vm_util.h"
#include "kselftest.h"
#define LENGTH (256UL*1024*1024)
#define PROTECTION (PROT_READ | PROT_WRITE)
static void check_bytes(char *addr)
{
ksft_print_msg("First hex is %x\n", *((unsigned int *)addr));
}
static void write_bytes(char *addr, size_t length)
{
unsigned long i;
for (i = 0; i < length; i++)
*(addr + i) = (char)i;
}
static void read_bytes(char *addr, size_t length)
{
unsigned long i;
check_bytes(addr);
for (i = 0; i < length; i++)
if (*(addr + i) != (char)i)
ksft_exit_fail_msg("Mismatch at %lu\n", i);
ksft_test_result_pass("Read correct data\n");
}
int main(int argc, char **argv)
{
void *addr;
size_t hugepage_size;
size_t length = LENGTH;
int flags = MAP_PRIVATE | MAP_ANONYMOUS | MAP_HUGETLB;
int shift = 0;
hugepage_size = default_huge_page_size();
/* munmap with fail if the length is not page aligned */
if (hugepage_size > length)
length = hugepage_size;
ksft_print_header();
ksft_set_plan(1);
if (argc > 1)
length = atol(argv[1]) << 20;
if (argc > 2) {
shift = atoi(argv[2]);
if (shift)
flags |= (shift & MAP_HUGE_MASK) << MAP_HUGE_SHIFT;
}
if (shift)
ksft_print_msg("%u kB hugepages\n", 1 << (shift - 10));
else
ksft_print_msg("Default size hugepages\n");
ksft_print_msg("Mapping %lu Mbytes\n", (unsigned long)length >> 20);
addr = mmap(NULL, length, PROTECTION, flags, -1, 0);
if (addr == MAP_FAILED)
ksft_exit_fail_msg("mmap: %s\n", strerror(errno));
ksft_print_msg("Returned address is %p\n", addr);
check_bytes(addr);
write_bytes(addr, length);
read_bytes(addr, length);
/* munmap() length of MAP_HUGETLB memory must be hugepage aligned */
if (munmap(addr, length))
ksft_exit_fail_msg("munmap: %s\n", strerror(errno));
ksft_finished();
}

147
tools/testing/selftests/mm/migration.c
View File

@ -5,7 +5,7 @@
*/
#include "kselftest_harness.h"
#include "thp_settings.h"
#include "hugepage_settings.h"
#include <strings.h>
#include <pthread.h>
@ -23,6 +23,8 @@
#define MAX_RETRIES 100
#define ALIGN(x, a) (((x) + (a - 1)) & (~((a) - 1)))
HUGETLB_SETUP_DEFAULT_PAGES(1)
FIXTURE(migration)
{
pthread_t *threads;
@ -32,13 +34,26 @@ FIXTURE(migration)
int n2;
};
static void reset_signals(void)
{
struct sigaction sa = { .sa_handler = SIG_DFL };
sigemptyset(&sa.sa_mask);
sigaction(SIGTERM, &sa, NULL);
sigaction(SIGHUP, &sa, NULL);
sigaction(SIGINT, &sa, NULL);
sigaction(SIGQUIT, &sa, NULL);
}
FIXTURE_SETUP(migration)
{
int n;
reset_signals();
if (numa_available() < 0)
SKIP(return, "NUMA not available");
self->nthreads = numa_num_task_cpus() - 1;
self->nthreads = numa_num_task_cpus() - 2;
self->n1 = -1;
self->n2 = -1;
@ -52,6 +67,9 @@ FIXTURE_SETUP(migration)
}
}
if (self->nthreads < 1 || self->n1 < 0 || self->n2 < 0)
SKIP(return, "Not enough threads or NUMA nodes available");
self->threads = malloc(self->nthreads * sizeof(*self->threads));
ASSERT_NE(self->threads, NULL);
self->pids = malloc(self->nthreads * sizeof(*self->pids));
@ -64,6 +82,29 @@ FIXTURE_TEARDOWN(migration)
free(self->pids);
}
static bool kill_children(FIXTURE_DATA(migration) * self)
{
bool err = false;
pid_t pid;
int i;
for (i = 0; i < self->nthreads; i++) {
int status = 0;
pid = self->pids[i];
if (pid < 0)
continue;
if (kill(pid, SIGTERM))
err = true;
if (pid != waitpid(pid, &status, 0))
err = true;
if (!WIFSIGNALED(status) || WTERMSIG(status) != SIGTERM)
err = true;
}
return !err;
}
int migrate(uint64_t *ptr, int n1, int n2)
{
int ret, tmp;
@ -127,20 +168,17 @@ TEST_F_TIMEOUT(migration, private_anon, 2*RUNTIME)
uint64_t *ptr;
int i;
if (self->nthreads < 2 || self->n1 < 0 || self->n2 < 0)
SKIP(return, "Not enough threads or NUMA nodes available");
ptr = mmap(NULL, TWOMEG, PROT_READ | PROT_WRITE,
MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
ASSERT_NE(ptr, MAP_FAILED);
memset(ptr, 0xde, TWOMEG);
for (i = 0; i < self->nthreads - 1; i++)
for (i = 0; i < self->nthreads; i++)
if (pthread_create(&self->threads[i], NULL, access_mem, ptr))
perror("Couldn't create thread");
ASSERT_EQ(migrate(ptr, self->n1, self->n2), 0);
for (i = 0; i < self->nthreads - 1; i++)
for (i = 0; i < self->nthreads; i++)
ASSERT_EQ(pthread_cancel(self->threads[i]), 0);
}
@ -151,17 +189,14 @@ TEST_F_TIMEOUT(migration, shared_anon, 2*RUNTIME)
{
pid_t pid;
uint64_t *ptr;
int i;
if (self->nthreads < 2 || self->n1 < 0 || self->n2 < 0)
SKIP(return, "Not enough threads or NUMA nodes available");
int i, err;
ptr = mmap(NULL, TWOMEG, PROT_READ | PROT_WRITE,
MAP_SHARED | MAP_ANONYMOUS, -1, 0);
ASSERT_NE(ptr, MAP_FAILED);
memset(ptr, 0xde, TWOMEG);
for (i = 0; i < self->nthreads - 1; i++) {
for (i = 0; i < self->nthreads; i++) {
pid = fork();
if (!pid) {
prctl(PR_SET_PDEATHSIG, SIGHUP);
@ -174,9 +209,9 @@ TEST_F_TIMEOUT(migration, shared_anon, 2*RUNTIME)
}
}
ASSERT_EQ(migrate(ptr, self->n1, self->n2), 0);
for (i = 0; i < self->nthreads - 1; i++)
ASSERT_EQ(kill(self->pids[i], SIGTERM), 0);
err = migrate(ptr, self->n1, self->n2);
ASSERT_EQ(kill_children(self), true);
ASSERT_EQ(err, 0);
}
/*
@ -184,28 +219,30 @@ TEST_F_TIMEOUT(migration, shared_anon, 2*RUNTIME)
*/
TEST_F_TIMEOUT(migration, private_anon_thp, 2*RUNTIME)
{
uint64_t pmdsize;
uint64_t *ptr;
int i;
if (!thp_is_enabled())
SKIP(return, "Transparent Hugepages not available");
if (self->nthreads < 2 || self->n1 < 0 || self->n2 < 0)
SKIP(return, "Not enough threads or NUMA nodes available");
pmdsize = read_pmd_pagesize();
if (!pmdsize)
SKIP(return, "Reading PMD pagesize failed");
ptr = mmap(NULL, 2*TWOMEG, PROT_READ | PROT_WRITE,
ptr = mmap(NULL, 2 * pmdsize, PROT_READ | PROT_WRITE,
MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
ASSERT_NE(ptr, MAP_FAILED);
ptr = (uint64_t *) ALIGN((uintptr_t) ptr, TWOMEG);
ASSERT_EQ(madvise(ptr, TWOMEG, MADV_HUGEPAGE), 0);
memset(ptr, 0xde, TWOMEG);
for (i = 0; i < self->nthreads - 1; i++)
ptr = (uint64_t *) ALIGN((uintptr_t) ptr, pmdsize);
ASSERT_EQ(madvise(ptr, pmdsize, MADV_HUGEPAGE), 0);
memset(ptr, 0xde, pmdsize);
for (i = 0; i < self->nthreads; i++)
if (pthread_create(&self->threads[i], NULL, access_mem, ptr))
perror("Couldn't create thread");
ASSERT_EQ(migrate(ptr, self->n1, self->n2), 0);
for (i = 0; i < self->nthreads - 1; i++)
for (i = 0; i < self->nthreads; i++)
ASSERT_EQ(pthread_cancel(self->threads[i]), 0);
}
@ -215,25 +252,27 @@ TEST_F_TIMEOUT(migration, private_anon_thp, 2*RUNTIME)
TEST_F_TIMEOUT(migration, shared_anon_thp, 2*RUNTIME)
{
uint64_t pmdsize;
pid_t pid;
uint64_t *ptr;
int i;
int i, err;
if (!thp_is_enabled())
SKIP(return, "Transparent Hugepages not available");
if (self->nthreads < 2 || self->n1 < 0 || self->n2 < 0)
SKIP(return, "Not enough threads or NUMA nodes available");
pmdsize = read_pmd_pagesize();
if (!pmdsize)
SKIP(return, "Reading PMD pagesize failed");
ptr = mmap(NULL, 2 * TWOMEG, PROT_READ | PROT_WRITE,
ptr = mmap(NULL, 2 * pmdsize, PROT_READ | PROT_WRITE,
MAP_SHARED | MAP_ANONYMOUS, -1, 0);
ASSERT_NE(ptr, MAP_FAILED);
ptr = (uint64_t *) ALIGN((uintptr_t) ptr, TWOMEG);
ASSERT_EQ(madvise(ptr, TWOMEG, MADV_HUGEPAGE), 0);
ptr = (uint64_t *) ALIGN((uintptr_t) ptr, pmdsize);
ASSERT_EQ(madvise(ptr, pmdsize, MADV_HUGEPAGE), 0);
memset(ptr, 0xde, TWOMEG);
for (i = 0; i < self->nthreads - 1; i++) {
memset(ptr, 0xde, pmdsize);
for (i = 0; i < self->nthreads; i++) {
pid = fork();
if (!pid) {
prctl(PR_SET_PDEATHSIG, SIGHUP);
@ -246,9 +285,9 @@ TEST_F_TIMEOUT(migration, shared_anon_thp, 2*RUNTIME)
}
}
ASSERT_EQ(migrate(ptr, self->n1, self->n2), 0);
for (i = 0; i < self->nthreads - 1; i++)
ASSERT_EQ(kill(self->pids[i], SIGTERM), 0);
err = migrate(ptr, self->n1, self->n2);
ASSERT_EQ(kill_children(self), true);
ASSERT_EQ(err, 0);
}
/*
@ -256,23 +295,28 @@ TEST_F_TIMEOUT(migration, shared_anon_thp, 2*RUNTIME)
*/
TEST_F_TIMEOUT(migration, private_anon_htlb, 2*RUNTIME)
{
unsigned long hugepage_size;
uint64_t *ptr;
int i;
if (self->nthreads < 2 || self->n1 < 0 || self->n2 < 0)
SKIP(return, "Not enough threads or NUMA nodes available");
hugepage_size = default_huge_page_size();
if (!hugepage_size)
SKIP(return, "Reading HugeTLB pagesize failed");
ptr = mmap(NULL, TWOMEG, PROT_READ | PROT_WRITE,
if (hugetlb_free_default_pages() < 1)
SKIP(return, "Not enough huge pages");
ptr = mmap(NULL, hugepage_size, PROT_READ | PROT_WRITE,
MAP_PRIVATE | MAP_ANONYMOUS | MAP_HUGETLB, -1, 0);
ASSERT_NE(ptr, MAP_FAILED);
memset(ptr, 0xde, TWOMEG);
for (i = 0; i < self->nthreads - 1; i++)
memset(ptr, 0xde, hugepage_size);
for (i = 0; i < self->nthreads; i++)
if (pthread_create(&self->threads[i], NULL, access_mem, ptr))
perror("Couldn't create thread");
ASSERT_EQ(migrate(ptr, self->n1, self->n2), 0);
for (i = 0; i < self->nthreads - 1; i++)
for (i = 0; i < self->nthreads; i++)
ASSERT_EQ(pthread_cancel(self->threads[i]), 0);
}
@ -281,19 +325,24 @@ TEST_F_TIMEOUT(migration, private_anon_htlb, 2*RUNTIME)
*/
TEST_F_TIMEOUT(migration, shared_anon_htlb, 2*RUNTIME)
{
unsigned long hugepage_size;
pid_t pid;
uint64_t *ptr;
int i;
int i, err;
if (self->nthreads < 2 || self->n1 < 0 || self->n2 < 0)
SKIP(return, "Not enough threads or NUMA nodes available");
hugepage_size = default_huge_page_size();
if (!hugepage_size)
SKIP(return, "Reading HugeTLB pagesize failed");
ptr = mmap(NULL, TWOMEG, PROT_READ | PROT_WRITE,
if (hugetlb_free_default_pages() < 1)
SKIP(return, "Not enough huge pages");
ptr = mmap(NULL, hugepage_size, PROT_READ | PROT_WRITE,
MAP_SHARED | MAP_ANONYMOUS | MAP_HUGETLB, -1, 0);
ASSERT_NE(ptr, MAP_FAILED);
memset(ptr, 0xde, TWOMEG);
for (i = 0; i < self->nthreads - 1; i++) {
memset(ptr, 0xde, hugepage_size);
for (i = 0; i < self->nthreads; i++) {
pid = fork();
if (!pid) {
prctl(PR_SET_PDEATHSIG, SIGHUP);
@ -306,9 +355,9 @@ TEST_F_TIMEOUT(migration, shared_anon_htlb, 2*RUNTIME)
}
}
ASSERT_EQ(migrate(ptr, self->n1, self->n2), 0);
for (i = 0; i < self->nthreads - 1; i++)
ASSERT_EQ(kill(self->pids[i], SIGTERM), 0);
err = migrate(ptr, self->n1, self->n2);
ASSERT_EQ(kill_children(self), true);
ASSERT_EQ(err, 0);
}
TEST_HARNESS_MAIN

2
tools/testing/selftests/mm/page_frag/page_frag_test.c
View File

@ -131,6 +131,8 @@ static int __init page_frag_test_init(void)
init_completion(&wait);
if (test_alloc_len > PAGE_SIZE || test_alloc_len <= 0 ||
test_push_cpu < 0 || test_push_cpu >= nr_cpu_ids ||
test_pop_cpu < 0 || test_pop_cpu >= nr_cpu_ids ||
!cpu_active(test_push_cpu) || !cpu_active(test_pop_cpu))
return -EINVAL;

13
tools/testing/selftests/mm/pagemap_ioctl.c
View File

@ -7,8 +7,6 @@
#include <sys/mman.h>
#include <errno.h>
#include <malloc.h>
#include "vm_util.h"
#include "kselftest.h"
#include <linux/types.h>
#include <linux/memfd.h>
#include <linux/userfaultfd.h>
@ -23,6 +21,10 @@
#include <sys/ipc.h>
#include <sys/shm.h>
#include "vm_util.h"
#include "kselftest.h"
#include "hugepage_settings.h"
#define PAGEMAP_BITS_ALL (PAGE_IS_WPALLOWED | PAGE_IS_WRITTEN | \
PAGE_IS_FILE | PAGE_IS_PRESENT | \
PAGE_IS_SWAPPED | PAGE_IS_PFNZERO | \
@ -1554,6 +1556,9 @@ int main(int __attribute__((unused)) argc, char *argv[])
if (init_uffd())
ksft_exit_skip("Failed to initialize userfaultfd\n");
if (!hugetlb_setup_default(4))
ksft_print_msg("HugeTLB test will be skipped\n");
ksft_set_plan(117);
page_size = getpagesize();
@ -1605,7 +1610,7 @@ int main(int __attribute__((unused)) argc, char *argv[])
}
/* 5. SHM Hugetlb page testing */
mem_size = 2*1024*1024;
mem_size = default_huge_page_size();
mem = gethugetlb_mem(mem_size, &shmid);
if (mem) {
wp_init(mem, mem_size);
@ -1633,7 +1638,7 @@ int main(int __attribute__((unused)) argc, char *argv[])
}
/* 7. File Hugetlb testing */
mem_size = 2*1024*1024;
mem_size = default_huge_page_size();
fd = memfd_create("uffd-test", MFD_HUGETLB | MFD_NOEXEC_SEAL);
if (fd < 0)
ksft_exit_fail_msg("uffd-test creation failed %d %s\n", errno, strerror(errno));

15
tools/testing/selftests/mm/pkey-helpers.h
View File

@ -71,13 +71,14 @@ static inline void sigsafe_printf(const char *format, ...)
extern void abort_hooks(void);
#define pkey_assert(condition) do { \
if (!(condition)) { \
dprintf0("assert() at %s::%d test_nr: %d iteration: %d\n", \
__FILE__, __LINE__, \
test_nr, iteration_nr); \
dprintf0("errno at assert: %d", errno); \
abort_hooks(); \
exit(__LINE__); \
} \
dprintf0("# assert() at %s::%d test_nr: %d iteration: %d\n", \
__FILE__, __LINE__, \
test_nr, iteration_nr); \
dprintf0("# errno at assert: %d\n", errno); \
abort_hooks(); \
ksft_exit_fail_msg("test %d (iteration %d)\n", \
test_nr, iteration_nr); \
} \
} while (0)
#define barrier() __asm__ __volatile__("": : :"memory")

2
tools/testing/selftests/mm/prctl_thp_disable.c
View File

@ -14,7 +14,7 @@
#include <sys/wait.h>
#include "kselftest_harness.h"
#include "thp_settings.h"
#include "hugepage_settings.h"
#include "vm_util.h"
#ifndef PR_THP_DISABLE_EXCEPT_ADVISED

130
tools/testing/selftests/mm/protection_keys.c
View File

@ -46,6 +46,7 @@
#include <sys/ptrace.h>
#include <setjmp.h>
#include "hugepage_settings.h"
#include "pkey-helpers.h"
int iteration_nr = 1;
@ -61,6 +62,7 @@ noinline int read_ptr(int *ptr)
return *ptr;
}
#if CONTROL_TRACING > 0
static void cat_into_file(char *str, char *file)
{
int fd = open(file, O_RDWR);
@ -86,7 +88,6 @@ static void cat_into_file(char *str, char *file)
close(fd);
}
#if CONTROL_TRACING > 0
static int warned_tracing;
static int tracing_root_ok(void)
{
@ -136,6 +137,7 @@ static void tracing_off(void)
void abort_hooks(void)
{
fflush(stdout);
fprintf(stderr, "running %s()...\n", __func__);
tracing_off();
#ifdef SLEEP_ON_ABORT
@ -370,8 +372,8 @@ static void signal_handler(int signum, siginfo_t *si, void *vucontext)
if ((si->si_code == SEGV_MAPERR) ||
(si->si_code == SEGV_ACCERR) ||
(si->si_code == SEGV_BNDERR)) {
printf("non-PK si_code, exiting...\n");
exit(4);
dprintf0("# non-PK si_code: %d, exiting...\n", si->si_code);
exit(1);
}
si_pkey_ptr = siginfo_get_pkey_ptr(si);
@ -708,50 +710,28 @@ static void *malloc_pkey_anon_huge(long size, int prot, u16 pkey)
}
static int hugetlb_setup_ok;
#define SYSFS_FMT_NR_HUGE_PAGES "/sys/kernel/mm/hugepages/hugepages-%ldkB/nr_hugepages"
#define GET_NR_HUGE_PAGES 10
static void setup_hugetlbfs(void)
{
int err;
int fd;
char buf[256];
long hpagesz_kb;
long hpagesz_mb;
long hpagesz_mb = HPAGE_SIZE / 1024 / 1024;
unsigned long free_pages;
if (geteuid() != 0) {
fprintf(stderr, "WARNING: not run as root, can not do hugetlb test\n");
ksft_print_msg("WARNING: not run as root, can not do hugetlb test\n");
return;
}
cat_into_file(__stringify(GET_NR_HUGE_PAGES), "/proc/sys/vm/nr_hugepages");
/*
* Now go make sure that we got the pages and that they
* Make sure that we got the pages and that they
* are PMD-level pages. Someone might have made PUD-level
* pages the default.
*/
hpagesz_kb = HPAGE_SIZE / 1024;
hpagesz_mb = hpagesz_kb / 1024;
sprintf(buf, SYSFS_FMT_NR_HUGE_PAGES, hpagesz_kb);
fd = open(buf, O_RDONLY);
if (fd < 0) {
fprintf(stderr, "opening sysfs %ldM hugetlb config: %s\n",
hpagesz_mb, strerror(errno));
return;
}
/* -1 to guarantee leaving the trailing \0 */
err = read(fd, buf, sizeof(buf)-1);
close(fd);
if (err <= 0) {
fprintf(stderr, "reading sysfs %ldM hugetlb config: %s\n",
hpagesz_mb, strerror(errno));
return;
}
if (atoi(buf) != GET_NR_HUGE_PAGES) {
fprintf(stderr, "could not confirm %ldM pages, got: '%s' expected %d\n",
hpagesz_mb, buf, GET_NR_HUGE_PAGES);
hugetlb_save_settings();
hugetlb_set_nr_pages(HPAGE_SIZE, GET_NR_HUGE_PAGES);
free_pages = hugetlb_free_pages(HPAGE_SIZE);
if (free_pages < GET_NR_HUGE_PAGES) {
ksft_print_msg("could not confirm %ldM pages, got: '%lu' expected %d\n",
hpagesz_mb, free_pages, GET_NR_HUGE_PAGES);
return;
}
@ -855,7 +835,7 @@ void expected_pkey_fault(int pkey)
#define do_not_expect_pkey_fault(msg) do { \
if (last_pkey_faults != pkey_faults) \
dprintf0("unexpected PKey fault: %s\n", msg); \
dprintf0("# unexpected PKey fault: %s\n", msg); \
pkey_assert(last_pkey_faults == pkey_faults); \
} while (0)
@ -1128,7 +1108,7 @@ static void become_child(void)
/* in the child */
return;
}
exit(0);
_exit(0);
}
/* Assumes that all pkeys other than 'pkey' are unallocated */
@ -1507,18 +1487,18 @@ static void test_ptrace_modifies_pkru(int *ptr, u16 pkey)
* checking
*/
if (__read_pkey_reg() != new_pkru)
exit(1);
_exit(1);
/* Stop and allow the tracer to clear XSTATE_BV for PKRU */
raise(SIGSTOP);
if (__read_pkey_reg() != 0)
exit(1);
_exit(1);
/* Stop and allow the tracer to examine PKRU */
raise(SIGSTOP);
exit(0);
_exit(0);
}
pkey_assert(child == waitpid(child, &status, 0));
@ -1692,29 +1672,36 @@ static void test_mprotect_pkey_on_unsupported_cpu(int *ptr, u16 pkey)
pkey_assert(sret < 0);
}
static void (*pkey_tests[])(int *ptr, u16 pkey) = {
test_read_of_write_disabled_region,
test_read_of_access_disabled_region,
test_read_of_access_disabled_region_with_page_already_mapped,
test_write_of_write_disabled_region,
test_write_of_write_disabled_region_with_page_already_mapped,
test_write_of_access_disabled_region,
test_write_of_access_disabled_region_with_page_already_mapped,
test_kernel_write_of_access_disabled_region,
test_kernel_write_of_write_disabled_region,
test_kernel_gup_of_access_disabled_region,
test_kernel_gup_write_to_write_disabled_region,
test_executing_on_unreadable_memory,
test_implicit_mprotect_exec_only_memory,
test_mprotect_with_pkey_0,
test_ptrace_of_child,
test_pkey_init_state,
test_pkey_syscalls_on_non_allocated_pkey,
test_pkey_syscalls_bad_args,
test_pkey_alloc_exhaust,
test_pkey_alloc_free_attach_pkey0,
struct pkey_test {
void (*func)(int *ptr, u16 pkey);
const char *name;
};
#define PKEY_TEST(fn) { fn, #fn }
static struct pkey_test pkey_tests[] = {
PKEY_TEST(test_read_of_write_disabled_region),
PKEY_TEST(test_read_of_access_disabled_region),
PKEY_TEST(test_read_of_access_disabled_region_with_page_already_mapped),
PKEY_TEST(test_write_of_write_disabled_region),
PKEY_TEST(test_write_of_write_disabled_region_with_page_already_mapped),
PKEY_TEST(test_write_of_access_disabled_region),
PKEY_TEST(test_write_of_access_disabled_region_with_page_already_mapped),
PKEY_TEST(test_kernel_write_of_access_disabled_region),
PKEY_TEST(test_kernel_write_of_write_disabled_region),
PKEY_TEST(test_kernel_gup_of_access_disabled_region),
PKEY_TEST(test_kernel_gup_write_to_write_disabled_region),
PKEY_TEST(test_executing_on_unreadable_memory),
PKEY_TEST(test_implicit_mprotect_exec_only_memory),
PKEY_TEST(test_mprotect_with_pkey_0),
PKEY_TEST(test_ptrace_of_child),
PKEY_TEST(test_pkey_init_state),
PKEY_TEST(test_pkey_syscalls_on_non_allocated_pkey),
PKEY_TEST(test_pkey_syscalls_bad_args),
PKEY_TEST(test_pkey_alloc_exhaust),
PKEY_TEST(test_pkey_alloc_free_attach_pkey0),
#if defined(__i386__) || defined(__x86_64__) || defined(__aarch64__)
test_ptrace_modifies_pkru,
PKEY_TEST(test_ptrace_modifies_pkru),
#endif
};
@ -1735,7 +1722,7 @@ static void run_tests_once(void)
dprintf1("test %d starting with pkey: %d\n", test_nr, pkey);
ptr = malloc_pkey(PAGE_SIZE, prot, pkey);
dprintf1("test %d starting...\n", test_nr);
pkey_tests[test_nr](ptr, pkey);
pkey_tests[test_nr].func(ptr, pkey);
dprintf1("freeing test memory: %p\n", ptr);
free_pkey_malloc(ptr);
sys_pkey_free(pkey);
@ -1746,7 +1733,7 @@ static void run_tests_once(void)
tracing_off();
close_test_fds();
printf("test %2d PASSED (iteration %d)\n", test_nr, iteration_nr);
ksft_test_result_pass("test %s (iteration %d)\n", pkey_tests[test_nr].name, iteration_nr);
dprintf1("======================\n\n");
}
iteration_nr++;
@ -1766,27 +1753,30 @@ int main(void)
setup_handlers();
printf("has pkeys: %d\n", pkeys_supported);
ksft_print_header();
if (!pkeys_supported) {
int size = PAGE_SIZE;
int *ptr;
printf("running PKEY tests for unsupported CPU/OS\n");
ksft_set_plan(1);
ksft_print_msg("running PKEY tests for unsupported CPU/OS\n");
ptr = mmap(NULL, size, PROT_NONE, MAP_ANONYMOUS|MAP_PRIVATE, -1, 0);
assert(ptr != (void *)-1);
test_mprotect_pkey_on_unsupported_cpu(ptr, 1);
exit(0);
ksft_test_result_pass("pkey on unsupported CPU/OS\n");
ksft_finished();
}
ksft_set_plan(ARRAY_SIZE(pkey_tests) * nr_iterations);
pkey_setup_shadow();
printf("startup pkey_reg: %016llx\n", read_pkey_reg());
ksft_print_msg("startup pkey_reg: %016llx\n", read_pkey_reg());
setup_hugetlbfs();
while (nr_iterations-- > 0)
run_tests_once();
printf("done (all tests OK)\n");
return 0;
ksft_finished();
}

234
tools/testing/selftests/mm/run_vmtests.sh
View File

@ -132,7 +132,7 @@ test_selected() {
run_gup_matrix() {
# -t: thp=on, -T: thp=off, -H: hugetlb=on
local hugetlb_mb=$(( needmem_KB / 1024 ))
local hugetlb_mb=256
for huge in -t -T "-H -m $hugetlb_mb"; do
# -u: gup-fast, -U: gup-basic, -a: pin-fast, -b: pin-basic, -L: pin-longterm
@ -154,60 +154,6 @@ run_gup_matrix() {
done
}
# get huge pagesize and freepages from /proc/meminfo
while read -r name size unit; do
if [ "$name" = "HugePages_Free:" ]; then
freepgs="$size"
fi
if [ "$name" = "Hugepagesize:" ]; then
hpgsize_KB="$size"
fi
done < /proc/meminfo
# Simple hugetlbfs tests have a hardcoded minimum requirement of
# huge pages totaling 256MB (262144KB) in size. The userfaultfd
# hugetlb test requires a minimum of 2 * nr_cpus huge pages. Take
# both of these requirements into account and attempt to increase
# number of huge pages available.
nr_cpus=$(nproc)
uffd_min_KB=$((hpgsize_KB * nr_cpus * 2))
hugetlb_min_KB=$((256 * 1024))
if [[ $uffd_min_KB -gt $hugetlb_min_KB ]]; then
needmem_KB=$uffd_min_KB
else
needmem_KB=$hugetlb_min_KB
fi
# set proper nr_hugepages
if [ -n "$freepgs" ] && [ -n "$hpgsize_KB" ]; then
orig_nr_hugepgs=$(cat /proc/sys/vm/nr_hugepages)
needpgs=$((needmem_KB / hpgsize_KB))
tries=2
while [ "$tries" -gt 0 ] && [ "$freepgs" -lt "$needpgs" ]; do
lackpgs=$((needpgs - freepgs))
echo 3 > /proc/sys/vm/drop_caches
if ! echo $((lackpgs + orig_nr_hugepgs)) > /proc/sys/vm/nr_hugepages; then
echo "Please run this test as root"
exit $ksft_skip
fi
while read -r name size unit; do
if [ "$name" = "HugePages_Free:" ]; then
freepgs=$size
fi
done < /proc/meminfo
tries=$((tries - 1))
done
nr_hugepgs=$(cat /proc/sys/vm/nr_hugepages)
if [ "$freepgs" -lt "$needpgs" ]; then
printf "Not enough huge pages available (%d < %d)\n" \
"$freepgs" "$needpgs"
fi
HAVE_HUGEPAGES=1
else
echo "no hugetlbfs support in kernel?"
HAVE_HUGEPAGES=0
fi
# filter 64bit architectures
ARCH64STR="arm64 mips64 parisc64 ppc64 ppc64le riscv64 s390x sparc64 x86_64"
if [ -z "$ARCH" ]; then
@ -235,32 +181,61 @@ pretty_name() {
run_test() {
if test_selected ${CATEGORY}; then
local skip=0
local LOADED_HWPOISON_INJECT_MOD=0
# On memory constrainted systems some tests can fail to allocate hugepages.
# perform some cleanup before the test for a higher success rate.
if [ ${CATEGORY} == "thp" -o ${CATEGORY} == "hugetlb" ]; then
if [ "${HAVE_HUGEPAGES}" = "1" ]; then
mem_kb=$(awk '/MemAvailable/ {print $2}' /proc/meminfo)
mem_Mb=$((mem_kb / 1024))
if (( $mem_Mb < 256 )); then
echo 3 > /proc/sys/vm/drop_caches
sleep 2
echo 1 > /proc/sys/vm/compact_memory
sleep 2
else
echo "hugepages not supported" | tap_prefix
skip=1
fi
fi
# Ensure hwpoison_inject is available for memory-failure tests
if [ "${CATEGORY}" = "memory-failure" ]; then
# Try to load hwpoison_inject if not present.
HWPOISON_DIR=/sys/kernel/debug/hwpoison/
if [ ! -d "$HWPOISON_DIR" ]; then
if ! modprobe -n hwpoison_inject > /dev/null 2>&1; then
echo "Module hwpoison_inject not found, skipping..." \
| tap_prefix
skip=1
else
modprobe hwpoison_inject > /dev/null 2>&1
LOADED_HWPOISON_INJECT_MOD=1
if [ ! -d "$HWPOISON_DIR" ]; then
echo "hwpoison debugfs interface not present" \
| tap_prefix
skip=1
fi
fi
fi
fi
local test=$(pretty_name "$*")
local title="running $*"
local sep=$(echo -n "$title" | tr "[:graph:][:space:]" -)
printf "%s\n%s\n%s\n" "$sep" "$title" "$sep" | tap_prefix
if [ "${skip}" != "1" ]; then
if [ $skip -eq 1 ]; then
local ret=$ksft_skip
else
("$@" 2>&1) | tap_prefix
local ret=${PIPESTATUS[0]}
else
local ret=$ksft_skip
fi
# Unload hwpoison_inject if we loaded it
if [ "${LOADED_HWPOISON_INJECT_MOD}" = "1" ]; then
modprobe -r hwpoison_inject > /dev/null 2>&1
fi
count_total=$(( count_total + 1 ))
if [ $ret -eq 0 ]; then
count_pass=$(( count_pass + 1 ))
@ -270,7 +245,9 @@ run_test() {
count_skip=$(( count_skip + 1 ))
echo "[SKIP]" | tap_prefix
echo "ok ${count_total} ${test} # SKIP" | tap_output
exitcode=$ksft_skip
if [ $exitcode -eq 0 ]; then
exitcode=$ksft_skip
fi
else
count_fail=$(( count_fail + 1 ))
echo "[FAIL]" | tap_prefix
@ -282,31 +259,14 @@ run_test() {
echo "TAP version 13" | tap_output
CATEGORY="hugetlb" run_test ./hugepage-mmap
shmmax=$(cat /proc/sys/kernel/shmmax)
shmall=$(cat /proc/sys/kernel/shmall)
echo 268435456 > /proc/sys/kernel/shmmax
echo 4194304 > /proc/sys/kernel/shmall
CATEGORY="hugetlb" run_test ./hugepage-shm
echo "$shmmax" > /proc/sys/kernel/shmmax
echo "$shmall" > /proc/sys/kernel/shmall
CATEGORY="hugetlb" run_test ./map_hugetlb
CATEGORY="hugetlb" run_test ./hugepage-mremap
CATEGORY="hugetlb" run_test ./hugepage-vmemmap
CATEGORY="hugetlb" run_test ./hugetlb-mmap
CATEGORY="hugetlb" run_test ./hugetlb-shm
CATEGORY="hugetlb" run_test ./hugetlb-mremap
CATEGORY="hugetlb" run_test ./hugetlb-vmemmap
CATEGORY="hugetlb" run_test ./hugetlb-madvise
CATEGORY="hugetlb" run_test ./hugetlb_dio
if [ "${HAVE_HUGEPAGES}" = "1" ]; then
nr_hugepages_tmp=$(cat /proc/sys/vm/nr_hugepages)
# For this test, we need one and just one huge page
echo 1 > /proc/sys/vm/nr_hugepages
CATEGORY="hugetlb" run_test ./hugetlb_fault_after_madv
CATEGORY="hugetlb" run_test ./hugetlb_madv_vs_map
# Restore the previous number of huge pages, since further tests rely on it
echo "$nr_hugepages_tmp" > /proc/sys/vm/nr_hugepages
fi
CATEGORY="hugetlb" run_test ./hugetlb_fault_after_madv
CATEGORY="hugetlb" run_test ./hugetlb_madv_vs_map
if test_selected "hugetlb"; then
echo "NOTE: These hugetlb tests provide minimal coverage. Use" | tap_prefix
@ -333,44 +293,11 @@ CATEGORY="gup_test" run_test ./gup_longterm
CATEGORY="userfaultfd" run_test ./uffd-unit-tests
uffd_stress_bin=./uffd-stress
CATEGORY="userfaultfd" run_test ${uffd_stress_bin} anon 20 16
# Hugetlb tests require source and destination huge pages. Pass in almost half
# the size of the free pages we have, which is used for *each*. An adjustment
# of (nr_parallel - 1) is done (see nr_parallel in uffd-stress.c) to have some
# extra hugepages - this is done to prevent the test from failing by racily
# reserving more hugepages than strictly required.
# uffd-stress expects a region expressed in MiB, so we adjust
# half_ufd_size_MB accordingly.
adjustment=$(( (31 < (nr_cpus - 1)) ? 31 : (nr_cpus - 1) ))
half_ufd_size_MB=$((((freepgs - adjustment) * hpgsize_KB) / 1024 / 2))
CATEGORY="userfaultfd" run_test ${uffd_stress_bin} hugetlb "$half_ufd_size_MB" 32
CATEGORY="userfaultfd" run_test ${uffd_stress_bin} hugetlb-private "$half_ufd_size_MB" 32
CATEGORY="userfaultfd" run_test ${uffd_stress_bin} hugetlb 128 32
CATEGORY="userfaultfd" run_test ${uffd_stress_bin} hugetlb-private 128 32
CATEGORY="userfaultfd" run_test ${uffd_stress_bin} shmem 20 16
CATEGORY="userfaultfd" run_test ${uffd_stress_bin} shmem-private 20 16
# uffd-wp-mremap requires at least one page of each size.
have_all_size_hugepgs=true
declare -A nr_size_hugepgs
for f in /sys/kernel/mm/hugepages/**/nr_hugepages; do
old=$(cat $f)
nr_size_hugepgs["$f"]="$old"
if [ "$old" == 0 ]; then
echo 1 > "$f"
fi
if [ $(cat "$f") == 0 ]; then
have_all_size_hugepgs=false
break
fi
done
if $have_all_size_hugepgs; then
CATEGORY="userfaultfd" run_test ./uffd-wp-mremap
else
echo "# SKIP ./uffd-wp-mremap"
fi
#cleanup
for f in "${!nr_size_hugepgs[@]}"; do
echo "${nr_size_hugepgs["$f"]}" > "$f"
done
echo "$nr_hugepgs" > /proc/sys/vm/nr_hugepages
CATEGORY="userfaultfd" run_test ./uffd-wp-mremap
CATEGORY="compaction" run_test ./compaction_test
@ -395,12 +322,10 @@ CATEGORY="mremap" run_test ./mremap_test
CATEGORY="hugetlb" run_test ./thuge-gen
CATEGORY="hugetlb" run_test ./charge_reserved_hugetlb.sh -cgroup-v2
CATEGORY="hugetlb" run_test ./hugetlb_reparenting_test.sh -cgroup-v2
if $RUN_DESTRUCTIVE; then
nr_hugepages_tmp=$(cat /proc/sys/vm/nr_hugepages)
enable_soft_offline=$(cat /proc/sys/vm/enable_soft_offline)
echo 8 > /proc/sys/vm/nr_hugepages
CATEGORY="hugetlb" run_test ./hugetlb-soft-offline
echo "$nr_hugepages_tmp" > /proc/sys/vm/nr_hugepages
echo "$enable_soft_offline" > /proc/sys/vm/enable_soft_offline
CATEGORY="hugetlb" run_test ./hugetlb-read-hwpoison
fi
@ -437,9 +362,7 @@ CATEGORY="memfd_secret" run_test ./memfd_secret
fi
# KSM KSM_MERGE_TIME_HUGE_PAGES test with size of 100
if [ "${HAVE_HUGEPAGES}" = "1" ]; then
CATEGORY="ksm" run_test ./ksm_tests -H -s 100
fi
CATEGORY="ksm" run_test ./ksm_tests -H -s 100
# KSM KSM_MERGE_TIME test with size of 100
CATEGORY="ksm" run_test ./ksm_tests -P -s 100
# KSM MADV_MERGEABLE test with 10 identical pages
@ -458,7 +381,6 @@ CATEGORY="ksm_numa" run_test ./ksm_tests -N -m 0
CATEGORY="ksm" run_test ./ksm_functional_tests
# protection_keys tests
nr_hugepgs=$(cat /proc/sys/vm/nr_hugepages)
if [ -x ./protection_keys_32 ]
then
CATEGORY="pkey" run_test ./protection_keys_32
@ -468,7 +390,6 @@ if [ -x ./protection_keys_64 ]
then
CATEGORY="pkey" run_test ./protection_keys_64
fi
echo "$nr_hugepgs" > /proc/sys/vm/nr_hugepages
if [ -x ./soft-dirty ]
then
@ -490,24 +411,28 @@ CATEGORY="thp" run_test ./khugepaged all:shmem
CATEGORY="thp" run_test ./khugepaged -s 4 all:shmem
CATEGORY="thp" run_test ./transhuge-stress -d 20
# Try to create XFS if not provided
if [ -z "${SPLIT_HUGE_PAGE_TEST_XFS_PATH}" ]; then
if [ "${HAVE_HUGEPAGES}" = "1" ]; then
if test_selected "thp"; then
if grep xfs /proc/filesystems &>/dev/null; then
XFS_IMG=$(mktemp /tmp/xfs_img_XXXXXX)
SPLIT_HUGE_PAGE_TEST_XFS_PATH=$(mktemp -d /tmp/xfs_dir_XXXXXX)
truncate -s 314572800 ${XFS_IMG}
mkfs.xfs -q ${XFS_IMG}
mount -o loop ${XFS_IMG} ${SPLIT_HUGE_PAGE_TEST_XFS_PATH}
MOUNTED_XFS=1
fi
if test_selected "thp"; then
if grep xfs /proc/filesystems &>/dev/null; then
XFS_IMG=$(mktemp /tmp/xfs_img_XXXXXX)
SPLIT_HUGE_PAGE_TEST_XFS_PATH=$(mktemp -d /tmp/xfs_dir_XXXXXX)
truncate -s 314572800 ${XFS_IMG}
mkfs.xfs -q ${XFS_IMG}
mount -o loop ${XFS_IMG} ${SPLIT_HUGE_PAGE_TEST_XFS_PATH}
MOUNTED_XFS=1
fi
fi
fi
if [ -n "${SPLIT_HUGE_PAGE_TEST_XFS_PATH}" ]; then
CATEGORY="thp" run_test ./khugepaged all:file ${SPLIT_HUGE_PAGE_TEST_XFS_PATH}
elif test_selected thp; then
count_total=$(( count_total + 1 ))
count_skip=$(( count_skip + 1 ))
echo "[SKIP] ./khugepaged all:file" | tap_prefix
fi
CATEGORY="thp" run_test ./split_huge_page_test ${SPLIT_HUGE_PAGE_TEST_XFS_PATH}
if [ -n "${MOUNTED_XFS}" ]; then
@ -516,6 +441,8 @@ if [ -n "${MOUNTED_XFS}" ]; then
rm -f ${XFS_IMG}
fi
CATEGORY="thp" run_test ./transhuge-stress -d 20
CATEGORY="thp" run_test ./folio_split_race_test
CATEGORY="migration" run_test ./migration
@ -532,28 +459,7 @@ CATEGORY="page_frag" run_test ./test_page_frag.sh nonaligned
CATEGORY="rmap" run_test ./rmap
# Try to load hwpoison_inject if not present.
HWPOISON_DIR=/sys/kernel/debug/hwpoison/
if [ ! -d "$HWPOISON_DIR" ]; then
if ! modprobe -q -R hwpoison_inject; then
echo "Module hwpoison_inject not found, skipping..."
else
modprobe hwpoison_inject > /dev/null 2>&1
LOADED_MOD=1
fi
fi
if [ -d "$HWPOISON_DIR" ]; then
CATEGORY="memory-failure" run_test ./memory-failure
fi
if [ -n "${LOADED_MOD}" ]; then
modprobe -r hwpoison_inject > /dev/null 2>&1
fi
if [ "${HAVE_HUGEPAGES}" = 1 ]; then
echo "$orig_nr_hugepgs" > /proc/sys/vm/nr_hugepages
fi
CATEGORY="memory-failure" run_test ./memory-failure
echo "SUMMARY: PASS=${count_pass} SKIP=${count_skip} FAIL=${count_fail}" | tap_prefix
echo "1..${count_total}" | tap_output

6
tools/testing/selftests/mm/soft-dirty.c
View File

@ -9,7 +9,7 @@
#include "kselftest.h"
#include "vm_util.h"
#include "thp_settings.h"
#include "hugepage_settings.h"
#define PAGEMAP_FILE_PATH "/proc/self/pagemap"
#define TEST_ITERATIONS 10000
@ -143,7 +143,7 @@ static void test_mprotect(int pagemap_fd, int pagesize, bool anon)
if (anon) {
map = mmap(NULL, pagesize, PROT_READ|PROT_WRITE,
MAP_ANONYMOUS|MAP_PRIVATE, -1, 0);
if (!map)
if (map == MAP_FAILED)
ksft_exit_fail_msg("anon mmap failed\n");
} else {
test_fd = open(fname, O_RDWR | O_CREAT, 0664);
@ -155,7 +155,7 @@ static void test_mprotect(int pagemap_fd, int pagesize, bool anon)
ftruncate(test_fd, pagesize);
map = mmap(NULL, pagesize, PROT_READ|PROT_WRITE,
MAP_SHARED, test_fd, 0);
if (!map)
if (map == MAP_FAILED)
ksft_exit_fail_msg("file mmap failed\n");
}

29
tools/testing/selftests/mm/split_huge_page_test.c
View File

@ -21,7 +21,7 @@
#include <time.h>
#include "vm_util.h"
#include "kselftest.h"
#include "thp_settings.h"
#include "hugepage_settings.h"
uint64_t pagesize;
unsigned int pageshift;
@ -470,13 +470,18 @@ static void split_file_backed_thp(int order)
char tmpfs_template[] = "/tmp/thp_split_XXXXXX";
const char *tmpfs_loc = mkdtemp(tmpfs_template);
char testfile[INPUT_MAX];
unsigned long size = 2 * pmd_pagesize;
char opts[64];
ssize_t num_written, num_read;
char *file_buf1, *file_buf2;
char *file_buf1 = NULL, *file_buf2 = NULL;
uint64_t pgoff_start = 0, pgoff_end = 1024;
int i;
ksft_print_msg("Please enable pr_debug in split_huge_pages_in_file() for more info.\n");
if (!tmpfs_loc)
ksft_exit_fail_msg("mkdtemp failed\n");
file_buf1 = (char *)malloc(pmd_pagesize);
file_buf2 = (char *)malloc(pmd_pagesize);
@ -489,10 +494,13 @@ static void split_file_backed_thp(int order)
file_buf1[i] = (char)i;
memset(file_buf2, 0, pmd_pagesize);
status = mount("tmpfs", tmpfs_loc, "tmpfs", 0, "huge=always,size=4m");
snprintf(opts, sizeof(opts), "huge=always,size=%lu", size);
status = mount("tmpfs", tmpfs_loc, "tmpfs", 0, opts);
if (status)
ksft_exit_fail_msg("Unable to create a tmpfs for testing\n");
if (status) {
ksft_print_msg("Unable to create a tmpfs for testing\n");
goto out;
}
status = snprintf(testfile, INPUT_MAX, "%s/thp_file", tmpfs_loc);
if (status >= INPUT_MAX) {
@ -544,10 +552,13 @@ static void split_file_backed_thp(int order)
status = umount(tmpfs_loc);
if (status) {
rmdir(tmpfs_loc);
ksft_exit_fail_msg("Unable to umount %s\n", tmpfs_loc);
ksft_print_msg("Unable to umount %s\n", tmpfs_loc);
goto out;
}
free(file_buf1);
free(file_buf2);
status = rmdir(tmpfs_loc);
if (status)
ksft_exit_fail_msg("cannot remove tmp dir: %s\n", strerror(errno));
@ -560,8 +571,10 @@ static void split_file_backed_thp(int order)
close(fd);
cleanup:
umount(tmpfs_loc);
rmdir(tmpfs_loc);
out:
free(file_buf1);
free(file_buf2);
rmdir(tmpfs_loc);
ksft_exit_fail_msg("Error occurred\n");
}

96
tools/testing/selftests/mm/thuge-gen.c
View File

@ -1,17 +1,5 @@
// SPDX-License-Identifier: GPL-2.0
/* Test selecting other page sizes for mmap/shmget.
Before running this huge pages for each huge page size must have been
reserved.
For large pages beyond MAX_PAGE_ORDER (like 1GB on x86) boot options must
be used. 1GB wouldn't be tested if it isn't available.
Also shmmax must be increased.
And you need to run as root to work around some weird permissions in shm.
And nothing using huge pages should run in parallel.
When the program aborts you may need to clean up the shm segments with
ipcrm -m by hand, like this
sudo ipcs | awk '$1 == "0x00000000" {print $2}' | xargs -n1 sudo ipcrm -m
(warning this will remove all if someone else uses them) */
/* Test selecting other page sizes for mmap/shmget. */
#define _GNU_SOURCE
#include <sys/mman.h>
@ -21,13 +9,12 @@
#include <sys/ipc.h>
#include <sys/shm.h>
#include <sys/stat.h>
#include <glob.h>
#include <assert.h>
#include <unistd.h>
#include <stdarg.h>
#include <string.h>
#include "vm_util.h"
#include "kselftest.h"
#include "hugepage_settings.h"
#if !defined(MAP_HUGETLB)
#define MAP_HUGETLB 0x40000
@ -37,15 +24,6 @@
#ifndef SHM_HUGE_SHIFT
#define SHM_HUGE_SHIFT 26
#endif
#ifndef SHM_HUGE_MASK
#define SHM_HUGE_MASK 0x3f
#endif
#ifndef SHM_HUGE_2MB
#define SHM_HUGE_2MB (21 << SHM_HUGE_SHIFT)
#endif
#ifndef SHM_HUGE_1GB
#define SHM_HUGE_1GB (30 << SHM_HUGE_SHIFT)
#endif
#define NUM_PAGESIZES 5
#define NUM_PAGES 4
@ -63,32 +41,10 @@ int ilog2(unsigned long v)
void show(unsigned long ps)
{
char buf[100];
if (ps == getpagesize())
return;
ksft_print_msg("%luMB: ", ps >> 20);
fflush(stdout);
snprintf(buf, sizeof buf,
"cat /sys/kernel/mm/hugepages/hugepages-%lukB/free_hugepages",
ps >> 10);
system(buf);
}
unsigned long read_free(unsigned long ps)
{
unsigned long val = 0;
char buf[100];
snprintf(buf, sizeof(buf),
"/sys/kernel/mm/hugepages/hugepages-%lukB/free_hugepages",
ps >> 10);
if (read_sysfs(buf, &val) && ps != getpagesize())
ksft_print_msg("missing %s\n", buf);
return val;
ksft_print_msg("%luMB: %lu\n", ps >> 20, hugetlb_free_pages(ps));
}
void test_mmap(unsigned long size, unsigned flags)
@ -96,14 +52,14 @@ void test_mmap(unsigned long size, unsigned flags)
char *map;
unsigned long before, after;
before = read_free(size);
before = hugetlb_free_pages(size);
map = mmap(NULL, size*NUM_PAGES, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_ANONYMOUS|MAP_HUGETLB|flags, -1, 0);
if (map == MAP_FAILED)
ksft_exit_fail_msg("mmap: %s\n", strerror(errno));
memset(map, 0xff, size*NUM_PAGES);
after = read_free(size);
after = hugetlb_free_pages(size);
show(size);
ksft_test_result(size == getpagesize() || (before - after) == NUM_PAGES,
@ -120,7 +76,7 @@ void test_shmget(unsigned long size, unsigned flags)
struct shm_info i;
char *map;
before = read_free(size);
before = hugetlb_free_pages(size);
id = shmget(IPC_PRIVATE, size * NUM_PAGES, IPC_CREAT|0600|flags);
if (id < 0) {
if (errno == EPERM) {
@ -141,7 +97,7 @@ void test_shmget(unsigned long size, unsigned flags)
shmctl(id, IPC_RMID, NULL);
memset(map, 0xff, size*NUM_PAGES);
after = read_free(size);
after = hugetlb_free_pages(size);
show(size);
ksft_test_result(size == getpagesize() || (before - after) == NUM_PAGES,
@ -153,43 +109,15 @@ void test_shmget(unsigned long size, unsigned flags)
void find_pagesizes(void)
{
unsigned long largest = getpagesize();
unsigned long shmmax_val = 0;
int i;
glob_t g;
glob("/sys/kernel/mm/hugepages/hugepages-*kB", 0, NULL, &g);
assert(g.gl_pathc <= NUM_PAGESIZES);
for (i = 0; (i < g.gl_pathc) && (num_page_sizes < NUM_PAGESIZES); i++) {
sscanf(g.gl_pathv[i], "/sys/kernel/mm/hugepages/hugepages-%lukB",
&page_sizes[num_page_sizes]);
page_sizes[num_page_sizes] <<= 10;
ksft_print_msg("Found %luMB\n", page_sizes[i] >> 20);
num_page_sizes = hugetlb_setup(NUM_PAGES, page_sizes, ARRAY_SIZE(page_sizes));
if (page_sizes[num_page_sizes] > largest)
for (i = 0; i < num_page_sizes; i++)
if (page_sizes[i] > largest)
largest = page_sizes[i];
if (read_free(page_sizes[num_page_sizes]) >= NUM_PAGES)
num_page_sizes++;
else
ksft_print_msg("SKIP for size %lu MB as not enough huge pages, need %u\n",
page_sizes[num_page_sizes] >> 20, NUM_PAGES);
}
globfree(&g);
read_sysfs("/proc/sys/kernel/shmmax", &shmmax_val);
if (shmmax_val < NUM_PAGES * largest) {
ksft_print_msg("WARNING: shmmax is too small to run this test.\n");
ksft_print_msg("Please run the following command to increase shmmax:\n");
ksft_print_msg("echo %lu > /proc/sys/kernel/shmmax\n", largest * NUM_PAGES);
ksft_exit_skip("Test skipped due to insufficient shmmax value.\n");
}
#if defined(__x86_64__)
if (largest != 1U<<30) {
ksft_exit_skip("No GB pages available on x86-64\n"
"Please boot with hugepagesz=1G hugepages=%d\n", NUM_PAGES);
}
#endif
shm_limits_prepare(NUM_PAGES * largest);
}
int main(void)
@ -232,3 +160,5 @@ int main(void)
ksft_finished();
}
SHM_LIMITS_RESTORE()

2
tools/testing/selftests/mm/transhuge-stress.c
View File

@ -17,7 +17,7 @@
#include <sys/mman.h>
#include "vm_util.h"
#include "kselftest.h"
#include "thp_settings.h"
#include "hugepage_settings.h"
int backing_fd = -1;
int mmap_flags = MAP_ANONYMOUS | MAP_NORESERVE | MAP_PRIVATE;

18
tools/testing/selftests/mm/uffd-common.h
View File

@ -37,24 +37,24 @@
#include "kselftest.h"
#include "vm_util.h"
#include "hugepage_settings.h"
#define UFFD_FLAGS (O_CLOEXEC | O_NONBLOCK | UFFD_USER_MODE_ONLY)
#define _err(fmt, ...) \
do { \
int ret = errno; \
fprintf(stderr, "ERROR: " fmt, ##__VA_ARGS__); \
fprintf(stderr, " (errno=%d, @%s:%d)\n", \
ret, __FILE__, __LINE__); \
#define _err(fmt, ...) \
do { \
int ret = errno; \
ksft_print_msg("ERROR: " fmt " (errno=%d, @%s:%d)\n", \
##__VA_ARGS__, ret, __FILE__, __LINE__); \
} while (0)
#define errexit(exitcode, fmt, ...) \
#define errexit(fmt, ...) \
do { \
_err(fmt, ##__VA_ARGS__); \
exit(exitcode); \
ksft_exit_fail(); \
} while (0)
#define err(fmt, ...) errexit(1, fmt, ##__VA_ARGS__)
#define err(fmt, ...) errexit(fmt, ##__VA_ARGS__)
struct uffd_global_test_opts {
unsigned long nr_parallel, nr_pages, nr_pages_per_cpu, page_size;

48
tools/testing/selftests/mm/uffd-stress.c
View File

@ -286,18 +286,12 @@ static int userfaultfd_stress(uffd_global_test_opts_t *gopts)
pthread_attr_setstacksize(&attr, 16*1024*1024);
while (bounces--) {
printf("bounces: %d, mode:", bounces);
if (bounces & BOUNCE_RANDOM)
printf(" rnd");
if (bounces & BOUNCE_RACINGFAULTS)
printf(" racing");
if (bounces & BOUNCE_VERIFY)
printf(" ver");
if (bounces & BOUNCE_POLL)
printf(" poll");
else
printf(" read");
printf(", ");
ksft_print_msg("bounces: %d, mode:%s%s%s%s, ",
bounces,
bounces & BOUNCE_RANDOM ? " rnd" : "",
bounces & BOUNCE_RACINGFAULTS ? " racing" : "",
bounces & BOUNCE_VERIFY ? " ver" : "",
bounces & BOUNCE_POLL ? " poll" : " read");
fflush(stdout);
if (bounces & BOUNCE_POLL)
@ -461,6 +455,9 @@ int main(int argc, char **argv)
if (argc < 4)
usage();
ksft_print_header();
ksft_set_plan(1);
if (signal(SIGALRM, sigalrm) == SIG_ERR)
err("failed to arm SIGALRM");
alarm(ALARM_INTERVAL_SECS);
@ -483,17 +480,17 @@ int main(int argc, char **argv)
* Ensure nr_parallel - 1 hugepages on top of that to account
* for racy extra reservation of hugepages.
*/
if (gopts->test_type == TEST_HUGETLB &&
get_free_hugepages() < 2 * (bytes / gopts->page_size) + gopts->nr_parallel - 1) {
printf("skip: Skipping userfaultfd... not enough hugepages\n");
return KSFT_SKIP;
if (gopts->test_type == TEST_HUGETLB) {
unsigned long nr = 2 * (bytes / gopts->page_size) + gopts->nr_parallel - 1;
if (!hugetlb_setup_default(nr))
ksft_exit_skip("Skipping userfaultfd... not enough hugepages\n");
}
gopts->nr_pages_per_cpu = bytes / gopts->page_size / gopts->nr_parallel;
if (!gopts->nr_pages_per_cpu) {
_err("pages_per_cpu = 0, cannot test (%lu / %lu / %lu)",
bytes, gopts->page_size, gopts->nr_parallel);
usage();
ksft_exit_skip("pages_per_cpu = 0, cannot test (%zu / %lu / %lu)\n",
bytes, gopts->page_size, gopts->nr_parallel);
}
bounces = atoi(argv[3]);
@ -503,9 +500,12 @@ int main(int argc, char **argv)
}
gopts->nr_pages = gopts->nr_pages_per_cpu * gopts->nr_parallel;
printf("nr_pages: %lu, nr_pages_per_cpu: %lu\n",
gopts->nr_pages, gopts->nr_pages_per_cpu);
return userfaultfd_stress(gopts);
ksft_print_msg("nr_pages: %lu, nr_pages_per_cpu: %lu\n",
gopts->nr_pages, gopts->nr_pages_per_cpu);
ksft_test_result(!userfaultfd_stress(gopts),
"uffd-stress %s\n", argv[1]);
ksft_finished();
}
#else /* __NR_userfaultfd */
@ -514,8 +514,8 @@ int main(int argc, char **argv)
int main(void)
{
printf("skip: Skipping userfaultfd test (missing __NR_userfaultfd)\n");
return KSFT_SKIP;
ksft_print_header();
ksft_exit_skip("missing __NR_userfaultfd definition\n");
}
#endif /* __NR_userfaultfd */

138
tools/testing/selftests/mm/uffd-unit-tests.c
View File

@ -86,47 +86,28 @@ typedef struct {
uffd_test_case_ops_t *test_case_ops;
} uffd_test_case_t;
static void uffd_test_report(void)
{
printf("Userfaults unit tests: pass=%u, skip=%u, fail=%u (total=%u)\n",
ksft_get_pass_cnt(),
ksft_get_xskip_cnt(),
ksft_get_fail_cnt(),
ksft_test_num());
}
static char current_test[256];
static void uffd_test_pass(void)
{
printf("done\n");
ksft_inc_pass_cnt();
ksft_test_result_pass("%s\n", current_test);
}
#define uffd_test_start(...) do { \
printf("Testing "); \
printf(__VA_ARGS__); \
printf("... "); \
fflush(stdout); \
snprintf(current_test, sizeof(current_test), __VA_ARGS__); \
} while (0)
#define uffd_test_fail(...) do { \
printf("failed [reason: "); \
printf(__VA_ARGS__); \
printf("]\n"); \
ksft_inc_fail_cnt(); \
#define uffd_test_fail(fmt, ...) do { \
ksft_print_msg("failed reason: [" fmt "]\n", ##__VA_ARGS__); \
ksft_test_result_fail("%s\n", current_test); \
} while (0)
static void uffd_test_skip(const char *message)
{
printf("skipped [reason: %s]\n", message);
ksft_inc_xskip_cnt();
ksft_test_result_skip("%s (%s)\n", current_test, message);
}
/*
* Returns 1 if specific userfaultfd supported, 0 otherwise. Note, we'll
* return 1 even if some test failed as long as uffd supported, because in
* that case we still want to proceed with the rest uffd unit tests.
*/
static int test_uffd_api(bool use_dev)
static void test_uffd_api(bool use_dev)
{
struct uffdio_api uffdio_api;
int uffd;
@ -140,7 +121,7 @@ static int test_uffd_api(bool use_dev)
uffd = uffd_open_sys(UFFD_FLAGS);
if (uffd < 0) {
uffd_test_skip("cannot open userfaultfd handle");
return 0;
return;
}
/* Test wrong UFFD_API */
@ -177,8 +158,6 @@ static int test_uffd_api(bool use_dev)
uffd_test_pass();
out:
close(uffd);
/* We have a valid uffd handle */
return 1;
}
@ -320,7 +299,7 @@ static int pagemap_test_fork(uffd_global_test_opts_t *gopts, bool with_event, bo
if (test_pin)
unpin_pages(&args);
/* Succeed */
exit(0);
_exit(0);
}
waitpid(child, &result, 0);
@ -788,7 +767,7 @@ static void uffd_sigbus_test_common(uffd_global_test_opts_t *gopts, bool wp)
err("fork");
if (!pid)
exit(faulting_process(gopts, 2, wp));
_exit(faulting_process(gopts, 2, wp));
waitpid(pid, &err, 0);
if (err)
@ -842,7 +821,7 @@ static void uffd_events_test_common(uffd_global_test_opts_t *gopts, bool wp)
err("fork");
if (!pid)
exit(faulting_process(gopts, 0, wp));
_exit(faulting_process(gopts, 0, wp));
waitpid(pid, &err, 0);
if (err)
@ -1701,18 +1680,58 @@ static void usage(const char *prog)
exit(KSFT_FAIL);
}
static int uffd_count_tests(int n_tests, int n_mems, const char *test_filter)
{
uffd_test_case_t *test;
int i, j, count = 0;
if (!test_filter)
count += 2; /* test_uffd_api(false) + test_uffd_api(true) */
for (i = 0; i < n_tests; i++) {
test = &uffd_tests[i];
if (test_filter && !strstr(test->name, test_filter))
continue;
for (j = 0; j < n_mems; j++)
if (test->mem_targets & mem_types[j].mem_flag)
count++;
}
return count;
}
static unsigned long uffd_setup_hugetlb(void)
{
unsigned long nr_hugepages, hp_size;
hugetlb_save_settings();
hp_size = default_huge_page_size();
if (!hp_size)
return 0;
/* need twice UFFD_TEST_MEM_SIZE, one for src area and one for dst */
nr_hugepages = 2 * MAX(UFFD_TEST_MEM_SIZE, hp_size * 2) / hp_size;
hugetlb_set_nr_default_pages(nr_hugepages);
if (hugetlb_free_default_pages() < nr_hugepages)
return 0;
return hp_size;
}
int main(int argc, char *argv[])
{
int n_tests = sizeof(uffd_tests) / sizeof(uffd_test_case_t);
int n_mems = sizeof(mem_types) / sizeof(mem_type_t);
const char *test_filter = NULL;
unsigned long hugepage_size;
bool list_only = false;
uffd_test_case_t *test;
mem_type_t *mem_type;
uffd_test_args_t args;
const char *errmsg;
int has_uffd, opt;
int i, j;
int i, j, opt;
while ((opt = getopt(argc, argv, "f:hl")) != -1) {
switch (opt) {
@ -1730,24 +1749,30 @@ int main(int argc, char *argv[])
}
}
if (!test_filter && !list_only) {
has_uffd = test_uffd_api(false);
has_uffd |= test_uffd_api(true);
if (!has_uffd) {
printf("Userfaultfd not supported or unprivileged, skip all tests\n");
exit(KSFT_SKIP);
if (list_only) {
for (i = 0; i < n_tests; i++) {
test = &uffd_tests[i];
if (test_filter && !strstr(test->name, test_filter))
continue;
printf("%s\n", test->name);
}
return KSFT_PASS;
}
hugepage_size = uffd_setup_hugetlb();
ksft_print_header();
ksft_set_plan(uffd_count_tests(n_tests, n_mems, test_filter));
if (!test_filter) {
test_uffd_api(false);
test_uffd_api(true);
}
for (i = 0; i < n_tests; i++) {
test = &uffd_tests[i];
if (test_filter && !strstr(test->name, test_filter))
continue;
if (list_only) {
printf("%s\n", test->name);
continue;
}
for (j = 0; j < n_mems; j++) {
mem_type = &mem_types[j];
@ -1758,10 +1783,14 @@ int main(int argc, char *argv[])
uffd_test_ops = mem_type->mem_ops;
uffd_test_case_ops = test->test_case_ops;
if (!(test->mem_targets & mem_type->mem_flag))
continue;
uffd_test_start("%s on %s", test->name, mem_type->name);
if (mem_type->mem_flag & (MEM_HUGETLB_PRIVATE | MEM_HUGETLB)) {
gopts.page_size = default_huge_page_size();
gopts.page_size = hugepage_size;
if (gopts.page_size == 0) {
uffd_test_skip("huge page size is 0, feature missing?");
uffd_test_skip("not enough HugeTLB pages");
continue;
}
} else {
@ -1777,10 +1806,6 @@ int main(int argc, char *argv[])
/* Initialize test arguments */
args.mem_type = mem_type;
if (!(test->mem_targets & mem_type->mem_flag))
continue;
uffd_test_start("%s on %s", test->name, mem_type->name);
if (!uffd_feature_supported(test)) {
uffd_test_skip("feature missing");
continue;
@ -1794,10 +1819,7 @@ int main(int argc, char *argv[])
}
}
if (!list_only)
uffd_test_report();
return ksft_get_fail_cnt() ? KSFT_FAIL : KSFT_PASS;
ksft_finished();
}
#else /* __NR_userfaultfd */
@ -1806,8 +1828,8 @@ int main(int argc, char *argv[])
int main(void)
{
printf("Skipping %s (missing __NR_userfaultfd)\n", __file__);
return KSFT_SKIP;
ksft_print_header();
ksft_exit_skip("missing __NR_userfaultfd definition\n");
}
#endif /* __NR_userfaultfd */

33
tools/testing/selftests/mm/uffd-wp-mremap.c
View File

@ -8,16 +8,27 @@
#include <linux/mman.h>
#include <sys/mman.h>
#include "kselftest.h"
#include "thp_settings.h"
#include "hugepage_settings.h"
#include "uffd-common.h"
static int pagemap_fd;
static size_t pagesize;
static int nr_pagesizes = 1;
static unsigned long pagesize;
static int nr_thpsizes;
static size_t thpsizes[20];
static int nr_hugetlbsizes;
static size_t hugetlbsizes[10];
static unsigned long hugetlbsizes[10];
static void check_uffd_wp_feature_supported(void)
{
uint64_t features = 0;
if (uffd_get_features(&features))
ksft_exit_skip("failed to get available features (%d)\n", errno);
if (!(features & UFFD_FEATURE_PAGEFAULT_FLAG_WP))
ksft_exit_skip("uffd-wp feature not supported\n");
}
static int detect_thp_sizes(size_t sizes[], int max)
{
@ -245,7 +256,7 @@ static void test_one_folio(uffd_global_test_opts_t *gopts, size_t size, bool pri
}
struct testcase {
size_t *sizes;
unsigned long *sizes;
int *nr_sizes;
bool private;
bool swapout;
@ -336,14 +347,16 @@ int main(int argc, char **argv)
struct thp_settings settings;
int i, j, plan = 0;
hugepage_save_settings(true, true);
check_uffd_wp_feature_supported();
pagesize = getpagesize();
nr_thpsizes = detect_thp_sizes(thpsizes, ARRAY_SIZE(thpsizes));
nr_hugetlbsizes = detect_hugetlb_page_sizes(hugetlbsizes,
ARRAY_SIZE(hugetlbsizes));
nr_hugetlbsizes = hugetlb_setup(1, hugetlbsizes, ARRAY_SIZE(hugetlbsizes));
/* If THP is supported, save THP settings and initially disable THP. */
/* If THP is supported, initially disable THP. */
if (nr_thpsizes) {
thp_save_settings();
thp_read_settings(&settings);
for (i = 0; i < NR_ORDERS; i++) {
settings.hugepages[i].enabled = THP_NEVER;
@ -368,10 +381,6 @@ int main(int argc, char **argv)
tc->swapout, tc->hugetlb);
}
/* If THP is supported, restore original THP settings. */
if (nr_thpsizes)
thp_restore_settings();
i = ksft_get_fail_cnt();
if (i)
ksft_exit_fail_msg("%d out of %d tests failed\n",

42
tools/testing/selftests/mm/va_high_addr_switch.c
View File

@ -11,6 +11,7 @@
#include "vm_util.h"
#include "kselftest.h"
#include "hugepage_settings.h"
/*
* The hint addr value is used to allocate addresses
@ -257,40 +258,35 @@ void testcases_init(void)
switch_hint = addr_switch_hint;
}
static int run_test(struct testcase *test, int count)
static void run_test(struct testcase *test, int count)
{
void *p;
int i, ret = KSFT_PASS;
int i;
for (i = 0; i < count; i++) {
struct testcase *t = test + i;
p = mmap(t->addr, t->size, PROT_READ | PROT_WRITE, t->flags, -1, 0);
printf("%s: %p - ", t->msg, p);
if (p == MAP_FAILED) {
printf("FAILED\n");
ret = KSFT_FAIL;
ksft_perror("MAP_FAILED");
ksft_test_result_fail("%s\n", t->msg);
continue;
}
if (t->low_addr_required && p >= (void *)(switch_hint)) {
printf("FAILED\n");
ret = KSFT_FAIL;
ksft_print_msg("%p not below switch hint\n", p);
ksft_test_result_fail("%s\n", t->msg);
} else {
/*
* Do a dereference of the address returned so that we catch
* bugs in page fault handling
*/
memset(p, 0, t->size);
printf("OK\n");
ksft_test_result_pass("%s\n", t->msg);
}
if (!t->keep_mapped)
munmap(p, t->size);
}
return ret;
}
#ifdef __aarch64__
@ -322,19 +318,23 @@ static int supported_arch(void)
int main(int argc, char **argv)
{
int ret, hugetlb_ret = KSFT_PASS;
bool run_hugetlb = false;
ksft_print_header();
if (!supported_arch())
return KSFT_SKIP;
ksft_exit_skip("Architecture not supported\n");
if (hugetlb_setup_default(6))
run_hugetlb = true;
testcases_init();
ret = run_test(testcases, sz_testcases);
if (argc == 2 && !strcmp(argv[1], "--run-hugetlb"))
hugetlb_ret = run_test(hugetlb_testcases, sz_hugetlb_testcases);
ksft_set_plan(sz_testcases + (run_hugetlb ? sz_hugetlb_testcases : 0));
if (ret == KSFT_PASS && hugetlb_ret == KSFT_PASS)
return KSFT_PASS;
else
return KSFT_FAIL;
run_test(testcases, sz_testcases);
if (run_hugetlb)
run_test(hugetlb_testcases, sz_hugetlb_testcases);
ksft_finished();
}

41
tools/testing/selftests/mm/va_high_addr_switch.sh
View File

@ -9,7 +9,6 @@
# Kselftest framework requirement - SKIP code is 4.
ksft_skip=4
orig_nr_hugepages=0
skip()
{
@ -77,43 +76,5 @@ check_test_requirements()
esac
}
save_nr_hugepages()
{
orig_nr_hugepages=$(cat /proc/sys/vm/nr_hugepages)
}
restore_nr_hugepages()
{
echo "$orig_nr_hugepages" > /proc/sys/vm/nr_hugepages
}
setup_nr_hugepages()
{
local needpgs=$1
while read -r name size unit; do
if [ "$name" = "HugePages_Free:" ]; then
freepgs="$size"
break
fi
done < /proc/meminfo
if [ "$freepgs" -ge "$needpgs" ]; then
return
fi
local hpgs=$((orig_nr_hugepages + needpgs))
echo $hpgs > /proc/sys/vm/nr_hugepages
local nr_hugepgs=$(cat /proc/sys/vm/nr_hugepages)
if [ "$nr_hugepgs" != "$hpgs" ]; then
restore_nr_hugepages
skip "$0: no enough hugepages for testing"
fi
}
check_test_requirements
save_nr_hugepages
# The HugeTLB tests require 6 pages
setup_nr_hugepages 6
./va_high_addr_switch --run-hugetlb
retcode=$?
restore_nr_hugepages
exit $retcode
./va_high_addr_switch

136
tools/testing/selftests/mm/vm_util.c
View File

@ -2,7 +2,6 @@
#include <string.h>
#include <errno.h>
#include <fcntl.h>
#include <dirent.h>
#include <inttypes.h>
#include <sys/ioctl.h>
#include <linux/userfaultfd.h>
@ -291,53 +290,6 @@ int64_t allocate_transhuge(void *ptr, int pagemap_fd)
return -1;
}
unsigned long default_huge_page_size(void)
{
unsigned long hps = 0;
char *line = NULL;
size_t linelen = 0;
FILE *f = fopen("/proc/meminfo", "r");
if (!f)
return 0;
while (getline(&line, &linelen, f) > 0) {
if (sscanf(line, "Hugepagesize: %lu kB", &hps) == 1) {
hps <<= 10;
break;
}
}
free(line);
fclose(f);
return hps;
}
int detect_hugetlb_page_sizes(size_t sizes[], int max)
{
DIR *dir = opendir("/sys/kernel/mm/hugepages/");
int count = 0;
if (!dir)
return 0;
while (count < max) {
struct dirent *entry = readdir(dir);
size_t kb;
if (!entry)
break;
if (entry->d_type != DT_DIR)
continue;
if (sscanf(entry->d_name, "hugepages-%zukB", &kb) != 1)
continue;
sizes[count++] = kb * 1024;
ksft_print_msg("[INFO] detected hugetlb page size: %zu KiB\n",
kb);
}
closedir(dir);
return count;
}
int pageflags_get(unsigned long pfn, int kpageflags_fd, uint64_t *flags)
{
size_t count;
@ -396,25 +348,6 @@ int uffd_unregister(int uffd, void *addr, uint64_t len)
return ret;
}
unsigned long get_free_hugepages(void)
{
unsigned long fhp = 0;
char *line = NULL;
size_t linelen = 0;
FILE *f = fopen("/proc/meminfo", "r");
if (!f)
return fhp;
while (getline(&line, &linelen, f) > 0) {
if (sscanf(line, "HugePages_Free: %lu", &fhp) == 1)
break;
}
free(line);
fclose(f);
return fhp;
}
static bool check_vmflag(void *addr, const char *flag)
{
char buffer[MAX_LINE_LENGTH];
@ -463,7 +396,7 @@ bool softdirty_supported(void)
/* New mappings are expected to be marked with VM_SOFTDIRTY (sd). */
addr = mmap(0, pagesize, PROT_READ | PROT_WRITE,
MAP_ANONYMOUS | MAP_PRIVATE, 0, 0);
if (!addr)
if (addr == MAP_FAILED)
ksft_exit_fail_msg("mmap failed\n");
supported = check_vmflag(addr, "sd");
@ -765,6 +698,27 @@ int unpoison_memory(unsigned long pfn)
return ret > 0 ? 0 : -errno;
}
int read_file(const char *path, char *buf, size_t buflen)
{
int fd;
ssize_t numread;
fd = open(path, O_RDONLY);
if (fd == -1)
return 0;
numread = read(fd, buf, buflen - 1);
if (numread < 1) {
close(fd);
return 0;
}
buf[numread] = '\0';
close(fd);
return (unsigned int) numread;
}
void write_file(const char *path, const char *buf, size_t buflen)
{
int fd, saved_errno;
@ -788,3 +742,49 @@ void write_file(const char *path, const char *buf, size_t buflen)
ksft_exit_fail_msg("%s write(%.*s) is truncated, expected %zu bytes, got %zd bytes\n",
path, (int)(buflen - 1), buf, buflen - 1, numwritten);
}
unsigned long read_num(const char *path)
{
char buf[21];
if (read_file(path, buf, sizeof(buf)) < 0)
ksft_exit_fail_perror("read_file()");
return strtoul(buf, NULL, 10);
}
void write_num(const char *path, unsigned long num)
{
char buf[21];
sprintf(buf, "%lu", num);
write_file(path, buf, strlen(buf) + 1);
}
static unsigned long shmall, shmmax;
void __shm_limits_restore(void)
{
if (shmmax)
write_num("/proc/sys/kernel/shmmax", shmmax);
if (shmall)
write_num("/proc/sys/kernel/shmall", shmall);
}
void shm_limits_prepare(unsigned long length)
{
unsigned long nr = length / psize();
unsigned long val;
val = read_num("/proc/sys/kernel/shmmax");
if (val < length) {
write_num("/proc/sys/kernel/shmmax", length);
shmmax = val;
}
val = read_num("/proc/sys/kernel/shmall");
if (val < nr) {
write_num("/proc/sys/kernel/shmall", nr);
shmall = val;
}
}

15
tools/testing/selftests/mm/vm_util.h
View File

@ -94,8 +94,6 @@ bool check_huge_anon(void *addr, int nr_hpages, uint64_t hpage_size);
bool check_huge_file(void *addr, int nr_hpages, uint64_t hpage_size);
bool check_huge_shmem(void *addr, int nr_hpages, uint64_t hpage_size);
int64_t allocate_transhuge(void *ptr, int pagemap_fd);
unsigned long default_huge_page_size(void);
int detect_hugetlb_page_sizes(size_t sizes[], int max);
int pageflags_get(unsigned long pfn, int kpageflags_fd, uint64_t *flags);
int uffd_register(int uffd, void *addr, uint64_t len,
@ -103,7 +101,6 @@ int uffd_register(int uffd, void *addr, uint64_t len,
int uffd_unregister(int uffd, void *addr, uint64_t len);
int uffd_register_with_ioctls(int uffd, void *addr, uint64_t len,
bool miss, bool wp, bool minor, uint64_t *ioctls);
unsigned long get_free_hugepages(void);
bool check_vmflag_io(void *addr);
bool check_vmflag_pfnmap(void *addr);
bool check_vmflag_guard(void *addr);
@ -168,3 +165,15 @@ int unpoison_memory(unsigned long pfn);
#define PAGEMAP_PFN(ent) ((ent) & ((1ull << 55) - 1))
void write_file(const char *path, const char *buf, size_t buflen);
int read_file(const char *path, char *buf, size_t buflen);
unsigned long read_num(const char *path);
void write_num(const char *path, unsigned long num);
void shm_limits_prepare(unsigned long length);
void __shm_limits_restore(void);
#define SHM_LIMITS_RESTORE() \
static void __attribute__((destructor)) shm_limits_restore(void) \
{ \
__shm_limits_restore(); \
}